Support building docker image from M1 Mac

[NixBiks]

Is your feature request related to a problem? Please describe.

I'm trying to build a docker container with PyMuPDF but unfortunately I can't get it to work on my Mac M1.

Describe the solution you'd like

My Dockerfile looks like this

FROM python:3.10-bullseye
RUN apt-get update && apt-get install --no-install-recommends -y mupdf
RUN pip3 install --upgrade pip
RUN pip3 install pymupdf

When I run docker build . then I get the following error

#7 0.558 Collecting pymupdf                                                                                                                                            
#7 0.750   Downloading PyMuPDF-1.19.4.tar.gz (2.3 MB)                                                                                                                  
#7 0.955   Preparing metadata (setup.py): started                                                                                                                      
#7 1.134   Preparing metadata (setup.py): finished with status 'done'                                                                                                  
#7 1.135 Building wheels for collected packages: pymupdf
#7 1.136   Building wheel for pymupdf (setup.py): started
#7 1.294   Building wheel for pymupdf (setup.py): finished with status 'error'
#7 1.294   ERROR: Command errored out with exit status 1:
#7 1.294    command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-rdtyj0dj
#7 1.294        cwd: /tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/
#7 1.294   Complete output (20 lines):
#7 1.294   running bdist_wheel
#7 1.294   running build
#7 1.294   running build_py
#7 1.294   creating build
#7 1.294   creating build/lib.linux-aarch64-3.10
#7 1.294   creating build/lib.linux-aarch64-3.10/fitz
#7 1.294   copying fitz/__init__.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.294   copying fitz/fitz.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.294   copying fitz/utils.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.294   copying fitz/__main__.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.294   running build_ext
#7 1.294   building 'fitz._fitz' extension
#7 1.294   creating build/temp.linux-aarch64-3.10
#7 1.294   creating build/temp.linux-aarch64-3.10/fitz
#7 1.294   gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include/mupdf -I/usr/local/include/mupdf -Imupdf/thirdparty/freetype/include -I/usr/include/freetype2 -I/usr/local/include/python3.10 -c fitz/fitz_wrap.c -o build/temp.linux-aarch64-3.10/fitz/fitz_wrap.o
#7 1.294   fitz/fitz_wrap.c:2755:10: fatal error: fitz.h: No such file or directory
#7 1.294    2755 | #include <fitz.h>
#7 1.294         |          ^~~~~~~~
#7 1.294   compilation terminated.
#7 1.294   error: command '/usr/bin/gcc' failed with exit code 1
#7 1.294   ----------------------------------------
#7 1.294   ERROR: Failed building wheel for pymupdf
#7 1.294   Running setup.py clean for pymupdf
#7 1.375 Failed to build pymupdf
#7 1.386 Installing collected packages: pymupdf
#7 1.387     Running setup.py install for pymupdf: started
#7 1.517     ERROR: Command errored out with exit status 1:
#7 1.517      command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h77owsa2/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.10/pymupdf
#7 1.517          cwd: /tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/
#7 1.517     Complete output (20 lines):
#7 1.517     running install
#7 1.517     running build
#7 1.517     running build_py
#7 1.517     creating build
#7 1.517     creating build/lib.linux-aarch64-3.10
#7 1.517     creating build/lib.linux-aarch64-3.10/fitz
#7 1.517     copying fitz/__init__.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.517     copying fitz/fitz.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.517     copying fitz/utils.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.517     copying fitz/__main__.py -> build/lib.linux-aarch64-3.10/fitz
#7 1.517     running build_ext
#7 1.517     building 'fitz._fitz' extension
#7 1.517     creating build/temp.linux-aarch64-3.10
#7 1.517     creating build/temp.linux-aarch64-3.10/fitz
#7 1.517     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include/mupdf -I/usr/local/include/mupdf -Imupdf/thirdparty/freetype/include -I/usr/include/freetype2 -I/usr/local/include/python3.10 -c fitz/fitz_wrap.c -o build/temp.linux-aarch64-3.10/fitz/fitz_wrap.o
#7 1.517     fitz/fitz_wrap.c:2755:10: fatal error: fitz.h: No such file or directory
#7 1.517      2755 | #include <fitz.h>
#7 1.517           |          ^~~~~~~~
#7 1.517     compilation terminated.
#7 1.517     error: command '/usr/bin/gcc' failed with exit code 1
#7 1.517     ----------------------------------------
#7 1.517     Running setup.py install for pymupdf: finished with status 'error'
#7 1.517 ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7ttde7if/pymupdf_5e778e7ede114e88a541cb42ff556098/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h77owsa2/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.10/pymupdf Check the logs for full command output.

Additional context
I've also questioned this in this closed thread but I thought it was better to start a new issue instead.

[JorjMcKie]

I cannot help you here. I have no Mac machine and also no M1.
The CI service of Github only supports Mac on Intel servers, and does not support docker services in this case.
Your error is fairly typical for building from sources: it always occurs when setup.py cannot find the libraries of the installed MuPDF.
There also has been quite some discussion going on about M1 support here. I will turn this issue into a discussion and put it close to those other comments.

You are more than welcome to develop something that can help other M1 users to more easily install PyMuPDF.

[xiongchiamiov]

I did a little binary searching and found that 1.18.13 is the first version that works (for me, at least) - I get a prebuilt wheel, whereas earlier versions fall back to compilation.

[benthorner]

The latest version - 1.19.6 - also works for me. It was just version 1.19.1 where I encountered the error.

[Aea]

If you're running on macOS/ARM using homebrew you may need to revert to an older version (PyMuPDF 1.20.0 has a compilation error: fitz/fitz_wrap.c:15265:20: error: returning 'fz_location' from a function with incompatible result type 'int').

# Formula Commit History: https://github.com/Homebrew/homebrew-core/commits/39220aa34b4c1250fb2b11c9657376abed3de322/Formula/mupdf.rb

cd "$(brew --repo homebrew/core)" && git checkout 457cb59a52ff7666dc146c1c480d2ea98d63564f;

# If you already have mupdf installed use `brew reinstall` instead of `brew install`
cd "$(brew --repo homebrew/core)" && HOMEBREW_NO_AUTO_UPDATE=1 brew install mupdf;
cd "$(brew --repo homebrew/core)" && git checkout master && brew pin mupdf;

pip install https://github.com/pymupdf/PyMuPDF/archive/refs/tags/1.19.6.tar.gz;

[julian-smith-artifex-com]

If you're running on macOS/ARM using homebrew you may need to revert to an older version (PyMuPDF 1.20.0 has a compilation error: fitz/fitz_wrap.c:15265:20: error: returning 'fz_location' from a function with incompatible result type 'int').

That error looks like the pymupdf build is trying to use an incompatible version of mupdf - pymupdf-1.20.0 requires mupdf-1.20.x. pymupdf-1.20.0's setup.py defaults to downloading and building the required mupdf source release; and sdists already include it. So in theory such incompatibilities shouldn't happen in normal usage.

Could you say what command you used when you tried to build pymupdf-1.20.0?

Someone at Artifex with an M1 has succeeded in installing pymupdf-1.20.0 with pip install pymupdf; the only issues was that they had to install SWIG.

I'm not familiar with MacOS or homebrew, so it's likely that i'm missing something here.

[Aea]

I did not compile, just installed the mupdf library via homebrew:

# This previously worked out of the box for mupdf < 1.20.0
$ brew install swig freetype mupdf
$ pip install pymupdf

Here's the info on the version of mupdf that did not work (i.e. pip install pymupdf==1.20.0 would fail)

$ brew info mupdf
mupdf: stable 1.20.0 (bottled), HEAD
Lightweight PDF and XPS viewer
https://mupdf.com/
Conflicts with:
  mupdf-tools (because mupdf and mupdf-tools install the same binaries)
/opt/homebrew/Cellar/mupdf/1.20.0 (129 files, 49.4MB) *
  Poured from bottle on 2022-06-22 at 19:31:29
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/mupdf.rb
License: AGPL-3.0-or-later
==> Dependencies
Build: lzip ✘, pkg-config ✔
Required: freeglut ✔, mesa ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 1,847 (30 days), 4,346 (90 days), 9,278 (365 days)
install-on-request: 1,628 (30 days), 3,866 (90 days), 8,084 (365 days)
build-error: 0 (30 days)

I closed the shell that had full build log (it was 1 error and 80 warnings), but I can reproduce it later today if that would be helpful.

Edit: Added clarification.

[julian-smith-artifex-com]

If you could reproduce the full output from pip install pymupdf, that would be very helpful.

During development of the 1.20.0 release, there was a similar-looking problem on OpenBSD where any system-installed mupdf in /usr/local was being used in preference to the one that is downloaded and built by setup.py. This goes wrong if the system mupdf is older than 1.20.0 (which is very likely the case at the moment of course). So in setup.py there's some openbsd-specific code to force linking of pymupdf with the locally-build mupdf's .a files. Maybe we'll have to do something similar on other systems, or even make it the default.

One thing worth trying would be to uninstall mupdf from your system before pip install pymupdf. If that works, we'll have a good idea about what's going on.

[Aea]

Turns out it was user error, I didn't have the wheel package installed in my environment so mupdf couldn't build through pip install, and the fallback failed.

Maybe this error message can be expanded: Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed.?

Log is provided: mupdf.txt

[julian-smith-artifex-com]

Ah, thanks for providing mupdf.txt, i think i know what's going on now.

Your suggested workaround, running pip install wheel before pip install pymupdf, makes sense, and will actually apply to at least one other issue here.

[The error message Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed. is from pip, not PyMuPDF's setup.py so we can't improve it in PyMuPDF.]

Here are the details of what's happening:

Some background: building Python packages which need to build something like mupdf, which if far too big and complicated for setuptools to build itself, is quite tricky. So PyMuPDF's setup.py has a hack where it looks for specific terms build or bdist or bdist_wheel anywhere in sys.argv; if found, we preemptively download/build mupdf, before using setuptools.setup() to handle the args passed on the command line.

It looks like in the fallback case you describe where wheel is not installed, pip runs setup.py with args that do not include any of our specific terms, specifying install instead. So setup.py doesn't build mupdf locally, so setuptools' building of PyMuPDF fails.

I think a simple fix is to add install to the list of terms in setup.py; this will be in the next PyMuPDF release.

Longer term i'll be looking at removing the hack and instead make setup.py work in a more elegant way, including providing a PEP-517 interface for pip to use.

[Aea]

Glad I could help, and I appreciate the extra context as to how the python build / package system works.