Versioned Python commands on Mac

Sunday 15 December 2013

I was experimenting with pydoc yesterday, and was baffled by how it was running. Turns out Mac OS X does some tricky stuff to support multiple versions of Python.

If you type "pydoc" at a shell prompt, it works properly:

$ pydoc
pydoc - the Python documentation tool <name> ...
    Show text documentation on something.  <name> may be the name of a

If you ask which file is being run, it's /usr/bin/pydoc, and you can look at that file:

$ which pydoc
$ more /usr/bin/pydoc

import sys, os
import glob, re

partA = """\
python version %d.%d.%d can't run %s.  Try the alternative(s):

partB = """
Run "man python" for more information about multiple version support in
Mac OS X.

sys.stderr.write(partA % (sys.version_info[:3] + (sys.argv[0],)))

Notice that this file will always write "python version X.Y.Z can't run...," which is not the output we're getting. Weird!

(BTW, if you have activated a virtualenv, you may have an alias, so that "pydoc" is actually "python -m pydoc". Is there an equivalent to "which" that will include that fact it in its output?)

But even without the virtualenv alias, this file isn't being run. Why not? The answer is in the shebang line:


Unix uses the shebang line to find a program to run the file. So typing "pydoc" at the prompt will find /usr/bin/pydoc, then find the shebang line, and will actually run this:

/usr/bin/python /usr/bin/pydoc

Seems simple enough: invoke Python, and have it run the code in /usr/bin/pydoc. So why isn't Python running the Python code we saw? The answer is that /usr/bin/python is not a Python interpreter!

On OS X, /usr/bin/python examines various settings and then invokes a real Python interpreter of the correct version: python.1 man page. A quick look at the readable text inside the executable confirms that it is not a full interpreter, and that it is concerned with versions:

$ strings /usr/bin/python
%s environment variable error (ignored)
no user %d
%s: too long
%s: Can't append "%s"
prefer32bit=%d version=%s
Executable path too long
alloca: out of memory
realpath couldn't resolve "%s"
%s: Can't append "%s"
argv=%s path=%s
posix_spawnattr_setbinpref_np only copied %d of %d
posix_spawn: %s
%s -> %s
versionarg: Out of memory
read_plist: %s: open
read_plist: %s: stat
read_plist: %s: mmap
read_plist: %s: CFDataCreateWithBytesNoCopy failed
(cfstrdup also failed)
read_plist: %s: CFPropertyListCreateFromXMLData failed: %s
read_plist: %s: plist not a dictionary
read_plist: %s: Prefer-32-Bit not a boolean
read_plist: %s: %s: Version unknown
read_plist: %s: Version not a string

So this finder is given "/usr/bin/pydoc" as an argument, and it decides what to really run. It's special-casing "/usr/bin", and actually invoking /usr/bin/pydoc2.7. The real /usr/bin/pydoc file is there only to be executed when the version-selection mechanism fails, which is why it simply prints messages about not being able to find the right version.

It seems that the switcher doesn't care much what command you're trying to run. If it's in /usr/bin, and there's a file alongside it with the same name but the current Python version appended, then it will run the versioned one.

All this can be verified by an experiment. I created /usr/bin/foo with these contents:

import sys
print sys.argv, "Plain!"

and a /usr/bin/foo2.7 with this:

import sys
print sys.argv, "Versioned!"

Then I ran it a number of different ways (the current directory is in the prompt):

~ $ foo
['/usr/bin/foo2.7'] Versioned!
~ $ /usr/bin/foo
['/usr/bin/foo2.7'] Versioned!
~ $ foo2.7
['/usr/bin/foo2.7'] Versioned!
~ $ /usr/bin/foo2.7
['/usr/bin/foo2.7'] Versioned!
~ $ cd /usr/bin
/usr/bin $ foo
['/usr/bin/foo2.7'] Versioned!
/usr/bin $ foo2.7
['/usr/bin/foo2.7'] Versioned!
/usr/bin $ ./foo
['./foo'] Plain!

Notice that the Python switcher runs the versioned file whenever it's identified as being from /usr/bin. But when run so that the shell doesn't identify it that way, even when it's the exact same file, the Python switcher decides it shouldn't interfere, and it runs the exact file specified.



Nick Sloan 6:38 PM on 15 Dec 2013

Very weird. Yet another reason to forget about the bundled Python, install multiple versions with Homebrew, and run everything in a virtual environment. I usually keep a scrap virtualenv for each Python version, so I can quickly activate and play around when needed.

Wes Turner 6:52 PM on 15 Dec 2013

I can't help but wonder what '#!/usr/bin/env python' in combination with virtualenvwrapper's postactivate (for $PATH mangling) would do to ease Python complexity on that AppleScript platform you've got there. +1 for Homebrew: 'brew install pyenv'

Marius Gedminas 8:11 PM on 15 Dec 2013

> Is there an equivalent to "which" that will include that fact it in its output?

The bash builtin 'type' does exactly that.

Ned Batchelder 9:40 PM on 15 Dec 2013

@Marius. Yes! "type" does what I want. Thanks!

Add a comment:

Ignore this:
not displayed and no spam.
Leave this empty:
not searched.
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.