Running code at Python startup

Thursday 14 January 2010

I recently had need to run a piece of Python code before every invocation of the interpreter. In my case, it was to enable coverage testing of processes spawned by a test suite. Certain tests would run Python code in a subprocess, and I wanted to measure coverage within that process.

I added a function to coverage.py that is suitable for executing on startup. It examines the environment to determine whether to start measuring coverage. (This code isn't released yet, but if you want to try it, it's on the trunk at bitbucket.)

Perl has a command line switch (-M) for this, as does Ruby (-r), but Python doesn't. I came up with two ways to do it for Python, both depending on the site module which is imported automatically on startup.

The first way is to create a sitecustomize.py file on the path somewhere. This module will be imported automatically by site.py. Any lines you add to that file are executed when Python starts:

import coverage
coverage.process_startup()

This is simple, but editing an existing sitecustomize.py could be awkward, and you may not even have permissions to do it.

The second way is to create a .pth file with an executable line. Usually lines in .pth files are directory names, but if the line starts with "import ", it is executed (what a hack!), so you can create a .pth file like this:

import coverage; coverage.process_startup()

This has the advantage that you can create a new file without worrying about touching existing files, all .pth files will be examined and interpreted. Unfortunately, you might have to give the file a strange name, since they are executed alphabetically, and a package you need may only be available after its .pth file has been seen. In my case, I named my .pth file zzz_coverage_process_start.pth to be sure it ran after coverage was added to the path.

Neither of these feels clean or elegant. I'm wondering if there are better techniques I'm overlooking? Python has a clean way to register functions to be called at process exit (atexit), which has been designed so that multiple authors of exit functions don't need to cooperate to avoid interfering with each other. It'd be nice to have a similar facility for startup functions.

tagged: » 12 reactions

Comments

[gravatar]
Tobu 3:19 PM on 16 Jan 2010

Doesn't Python have `python -m module`?

[gravatar]
Ned Batchelder 4:11 PM on 16 Jan 2010

@Tobu, it does, but it merely runs the module. It isn't a way to run the module before running your own program.

[gravatar]
Jim Sizelove 4:29 PM on 16 Jan 2010

Would the PYTHONSTARTUP environment variable meet your need? See http://docs.python.org/using/cmdline.html#environment-variables

[gravatar]
Ned Batchelder 4:41 PM on 16 Jan 2010

PYTHONSTARTUP does precisely what I need, but *only* for interactive prompts. I need the same thing, but for non-interactive invocations of the interpreter also.

[gravatar]
Andy Jennings 10:18 AM on 18 Jan 2010

But `python -m cProfile myscript.py` runs myscript.py after it loads the profiler. Is that because there is special code in the cProfile module to parse that parameter, load it, and run it? Would that same code work for your scenario?

[gravatar]
Ned Batchelder 10:47 AM on 18 Jan 2010

@Andy: the cProfile module reads the myscript.py argument and runs the Python file. I could write a similar module, but it doesn't really solve my problem. I have a test suite with many invocations of the Python interpreter in subprocesses. I'd have to change the way Python is invoked in those cases to use my new module. I was looking for a way to transparently invoke a module before the real program starts.

[gravatar]
Paul Ollis 2:24 PM on 18 Jan 2010

Python (at least version 2.6) supports a usercustomization feature. You can put a usercustomize.py module in your USER_SITE directory. For my Linux python install this is ~/.local/lib/python2.6/site-packages.

You can run this to find out where USER_SITE is for your Python installation.

python -c "import site; site._script()"

The usercustomize module is imported late during; after sitecustomize.

[gravatar]
Mkch 2:49 AM on 20 Apr 2010

About the file names sorting problem ; there is a hint commonly used under Unix : prepend by a number instead of an alphabetic sequence.

01_root_startup.pth
...
50_user_startup.pth
...
99_coverage_process_start.pth

I find this cleaner, but is a matter of taste.

[gravatar]
Geoff Bache 4:10 AM on 23 Aug 2010

I just ran into this issue myself. The problem with the .pth files solution is that this works differently on different platforms (apparently). On Linux you can put them anywhere on PYTHONPATH, but on Windows you have to put them in certain pre-defined global locations. This last makes it very difficult to do this sort of thing without risking screwing up some other Python process.

My own solution was as follows: use your sitecustomize.py one (using a temporary directory to store it), but add the following code at the end so that the real sitecustomize.py is also loaded:

import os, sys
currDir = os.path.dirname(__file__)
sys.path.remove(currDir)
del sys.modules["sitecustomize"]
try:
    import sitecustomize
finally:
    sys.path.insert(0, currDir)
Yes it's a hack, but it only needs doing once for any sitecustomize setup.

[gravatar]
Karol M. Langner 6:25 AM on 19 Jan 2012

There is also the -i option, which executes the script given after this option before entering interactive mode. That script can contain any code you like, and that is what I use to customize certain automatic python calls. Also, you can make use of temporary files if the code is not too long, like so:

t=`mktemp`; echo "print 'ok'" > $t; python -i $t; rm -rvf $t

[gravatar]
Karol M. Langner 7:27 AM on 19 Jan 2012

Oops, haven't used this in a while, and forgot to pipe the actual script to run to the interpreter. So, looking at my scripts, it seems actually more sloppy and hackish than I remembered, but here it is:

t=`mktemp`; echo "print 'ok'" > $t; cat script.py | python -i $t; rm -rvf $t
and it seems this also print a lot of ">>" while executing script.py (the actual script you want to run). I remember getting rid of that somehow, but I can only find redirection to /dev/null, as I didn't care abotu output.

[gravatar]
Reader 8:58 PM on 2 Apr 2016

The standard module 'user.py' does something like what you want.

I suppose you would put

import user

at the top of 'myscript.py', using the convention of Andy Jenning's response, without the need to alter your invocations as he suggests. Any imported names in the ~/.pythonrc.py file are appended to the following help(user) text, so for example, if the rc file has

from math import *

you can access 'user.pi' in your sub-process, but 'pi' is not defined. Executable code in the file is, obviously, simply executed. If the rc file does not exist, it is as if your 'user' module import were omitted.

The doc string for 'user' is:

Help on module user:

NAME
user - Hook to allow user-specified customization code to run.

FILE
/usr/lib/python2.7/user.py

MODULE DOCS
http://docs.python.org/library/user

DESCRIPTION
As a policy, Python doesn't run user-specified code on startup of
Python programs (interactive sessions execute the script specified in
the PYTHONSTARTUP environment variable if it exists).

However, some programs or sites may find it convenient to allow users
to have a standard customization file, which gets run when a program
requests it. This module implements such a mechanism. A program
that wishes to use the mechanism must execute the statement

import user

The user module looks for a file .pythonrc.py in the user's home
directory and if it can be opened, execfile()s it in its own global
namespace. Errors during this phase are not caught; that's up to the
program that imports the user module, if it wishes.

The user's .pythonrc.py could conceivably test for sys.version if it
wishes to do different things depending on the Python version.

DATA
home = '/home/'
pythonrc = '/home//.pythonrc.py'

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.