This article follows my meandering path to building a simple Python script (and dependencies) as a binary that I could easily run on multiple machines. If you just want to see what I ended up using, scroll down to the last section.

PyInstaller

I started out using PyInstaller as it was really simple to get running. To build a binary, ready for distribution, from a single script your just need to:

pyinstaller --onefile my-python-script

It will work out what Python libraries need to be included and bundle everything up, including a Python interpreter.

To start with I thought that last bit sounded great, including the Python interpreter, the trouble is Python has dependencies too. One of which is GLIBC - meaning that building on a laptop with the latest things and running on an older server doesn't work so well:

[3081] Error loading Python lib '/tmp/_MEIdcU0to/libpython3.6m.so.1.0': dlopen: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/_MEIdcU0to/libpython3.6m.so.1.0)

Supposedly there's an answer - StaticX - but it didn't work for me. The statically linked binary produced a different error:

/tmp/staticx-CamJai/.staticx.prog: relocation error: /lib/x86_64-linux-gnu/libnss_dns.so.2: symbol __res_maybe_init version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

...but it was still an error and I switched tactic at this point.

XAR (or not, yet)

Next I found XAR which sounds great - smaller, faster and generally just better than other options.

Unfortunately it requires a newer squashfuse than I found in my current desktop distribution and I'm not so desperate for smaller and faster that I didn't try to find a simpler alternative first.

PEX

Next up was PEX - this looked like it should be almost as easy as PyInstaller. The docs seem to suggest you need to manually provide the list of dependencies plus the script to run and that should be about it.

My experience was certainly not that simple though. Initially I thought something as simple as this might work:

pex -o my-executable.pex requests -- my-python-script

This does build a PEX file (without errors) and you can run it, but it will always drop at a python shell. The built PEX file doesn't actually include the required script:

unzip -t my-executable.pex | grep my-python-script

... will show it's not been included at all.

Trying to use the -e parameter to specify the execution target provides a similar result and using -c produces a build error a bit like this:

pex.pex_builder.InvalidExecutableSpecification: Could not find script 'my-python-script' in any distribution certifi 2018.8.24, idna 2.7, chardet 3.0.4, requests 2.19.1, urllib3 1.23 within PEX!

My testing around this wasn't entirely useless though as I did discover along the way that by default PEX will pick up the full python interpreter name to use as a target. Unfortunately building on a newer machine (with Python 3.6) and running on an older server (with Python 3.5) this produced:

/usr/bin/env: ‘python3.6’: No such file or directory

The solution was easy with an extra parameter, --python-shebang='/usr/bin/env python3', to pex though; making it target a slightly more generic Python version:

pex -o my-executable.pex --python-shebang='/usr/bin/env python3' requests -- my-python-script

PEX with a package

Some searching online suggested that putting the script into a mini Python package could be a way to get PEX to actually include it. So armed with a new directory and a very simple setup.py:

from distutils.core import setup

setup(name='my-python-script',
    version='1.0',
    scripts=['my-python-script'],
)

... I tried again (note the extra "." before "requests" which asks pex to include the package in the current directory) ...

pex -o my-executable.pex --python-shebang='/usr/bin/env python3' . requests -- my-python-script

This did succeed in getting the script included in the resulting PEX file, but unfortunately running it still just provides a Python shell. I thought perhaps -- should be -c or -e instead but neither of those worked either. The first refused to build, declaring that the script could not be found (despite correctly listing my new package) and the second built but then failed to execute with an ImportError:

ImportError: No module named my-python-script

I decided it was time for a new approach.

PEX via Pants

Apparently the "main" way to build PEX files is via another tool, such as Pants. Supposedly Pants is easy to get going, here's the one-line install process:

curl -L -O https://pantsbuild.github.io/setup/pants && chmod +x pants && touch pants.ini

Quite why you need an empty pants.ini file I do not know! But anyway, having pants installed is a lot less than half of the battle because then you need to make a BUILD file, thankfully there's a Python specific guide to BUILD files. The only trouble is, there's no example or guidance on how to something as simple as including an external library dependency! After a lot of searching, reading of the Pants documentation and random experimentation I did end up with a working BUILD file:

python_requirement_library(
  name='requests',
  requirements=[
    python_requirement(name='requests', requirement='requests'),
  ]
)

python_binary(
  name='my-python-script',
  dependencies=[
    ':requests',
  ],
  source='my-python-script.py',
)

With this BUILD file in place and Pants installed alongside my script, I can build a working PEX file with:

./pants binary :my-python-script

It's worth pointing out that to get this working I had to rename my-python-script to my-python-script.py, otherwise it build but the PEX file didn't work, again complaining about not being able to find the module.

PEX, the simple way

My experience getting Pants to work hadn't been particularly satisfying and I felt the process was really more than should be needed for a simple script. I also have several small scripts that I want to be able to distribute and the idea of making complete BUILD files for each one, especially with tedious and long dependency definitions didn't fill me with joy. I could of course, have written a wrapper script that took a simple list of dependencies and built a BUILD file, to then actually build the PEX file. Again this didn't feel ideal though.

I felt sure there must be a better, simpler way to do such a simple task. I went back to the PEX documentation to re-read, I searched some more, I found the PEX source repository and started poking around. In the end I came across a bug report and the related commits that included automated tests that provided the vital clues I needed.

So it turns out you can build a (fairly) portable, executable PEX file for a single, simple Python script with the following command:

pex -o ../my-executable.pex --python-shebang='/usr/bin/env python3' -D . requests -e my-python-script

With only one important tweak to my original script - the file had to be renamed to end in .py. With that in place the -D . parameter tells pex to include the current directory (and hence the script) in the archive and the -e my-python-script (without .py ending) was suitable to pick it up as the PEX execution target.