Installing Python packages from Windows installers into virtualenv

Saturday 17 July 2010This is more than 14 years old. Be careful.

Note (Feb 3, 2016): the latest and best version of this is Alex Waters’ onetruepython. Use that.

I’m a recent convert to virtualenv, it’s a great way to maintain a number of different Python installations so that you can install packages for one project without it polluting the environment for all your projects.

I also work on Windows, which can be a pain. In particular, many interesting Python packages involve compiling extensions, which is not always easy, and especially not easy on Windows. So I’m glad when package authors provide pre-built binaries for Windows. These are typically delivered as .exe Windows installers.

Here’s the problem: these installers know to look in the registry to find the Python installation. There are many things developers dislike about Windows, and the registry is often at the top of the list. One of the bad things about it is that it encourages a mindset of their being one of everything. Starting with the concept of “one registry”, it seeps into the whole culture of Windows, invading even to Python, which cannot abide more than one installation of a major release.

So when running a Windows package installer, it will find the Python 2.6 installation in the registry, and that’s the only option you’ve got for where the code is going to go. Your nice isolated virtualenvs are completely out of the picture.

I asked on Stack Overflow if there’s a way to install Windows package installers into virtualenvs, and didn’t get the answers I wanted.

So I decided the best approach was to change the registry, install my package, then change the registry back. I adapted a classic script to register Python installations, to create what I’ve called the_python.py:

# script to register Python 2.0 or later for use with win32all
# and other extensions that require Python registry settings
#
# Adapted by Ned Batchelder and Alex Waters from a script
# written by Joakim Low for Secret Labs AB / PythonWare
#
# source:
# http://www.pythonware.com/products/works/articles/regpy20.htm

import sys

from _winreg import *
import ctypes
import ctypes.wintypes
import os.path

# tweak as necessary
version = sys.version[:3]
installpath = sys.prefix

regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version)
installkey = "InstallPath"
pythonkey = "PythonPath"
pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % (
    installpath, installpath, installpath
)

def RegisterPy():
    try:
        reg = OpenKey(HKEY_LOCAL_MACHINE, regpath)
    except EnvironmentError:
        try:
            reg = CreateKey(HKEY_LOCAL_MACHINE, regpath)
        except Exception, e:
            print "*** Unable to register: %s" % e
            return

    SetValue(reg, installkey, REG_SZ, installpath)
    SetValue(reg, pythonkey, REG_SZ, pythonpath)
    CloseKey(reg)
    print "--- Python %s at %s is now registered!" % (version, installpath)
    raw_input('Press any key...')


class SHELLEXECUTEINFO(ctypes.Structure):
    _fields_ = (
        ("cbSize",ctypes.wintypes.DWORD),
        ("fMask",ctypes.c_ulong),
        ("hwnd",ctypes.wintypes.HANDLE),
        ("lpVerb",ctypes.c_char_p),
        ("lpFile",ctypes.c_char_p),
        ("lpParameters",ctypes.c_char_p),
        ("lpDirectory",ctypes.c_char_p),
        ("nShow",ctypes.c_int),
        ("hInstApp",ctypes.wintypes.HINSTANCE),
        ("lpIDList",ctypes.c_void_p),
        ("lpClass",ctypes.c_char_p),
        ("hKeyClass",ctypes.wintypes.HKEY),
        ("dwHotKey",ctypes.wintypes.DWORD),
        ("hIconOrMonitor",ctypes.wintypes.HANDLE),
        ("hProcess",ctypes.wintypes.HANDLE),
    )

def need_admin():
    ShellExecuteEx = ctypes.windll.shell32.ShellExecuteEx
    ShellExecuteEx.restype = ctypes.wintypes.BOOL
    sei = SHELLEXECUTEINFO()
    sei.cbSize = ctypes.sizeof(sei)
    sei.lpVerb = "runas"
    sei.lpFile = sys.executable
    sei.lpParameters = os.path.abspath(__file__)
    sei.nShow = 1
    ShellExecuteEx(ctypes.byref(sei))


if __name__ == "__main__":
    try:
        RegisterPy()
    except WindowsError as e:
        if e.errno == 13:
            need_admin()
        else:
            raise

Use your desired Python to run this script, and it will be entered into the registry as the Python. When you run your Windows package installer, it will go into your virtualenv. Don’t forget to run it again at the end to put things back the way they were.

Updated Nov 12, 2013: Alex Waters ([Tritium] on #python) revised the program to detect the need for elevated privileges and automatically re-run itself with privileges. This makes it much more convenient. Thanks, Alex!

Updated Feb 3, 2016: Alex Waters continues dominance in this area: his latest version is at onetruepython.

Comments

[gravatar]
This could be a nice addition to the activate and deavtivate scripts of virtualenv when called with a cmd line switch. With deactivating reversing the changes.
[gravatar]
Sounds like an interesting addition to our pythonselect (http://bitbucket.org/activestate/pythonselect/src) that we use for select "the python" in /Library/Frameworks/Python.framework/Versions on Mac.
[gravatar]
For me, calling "python.exe pip install" instead of just "pip install" does the trick. The former picks up the currently visible interpreter (made visible by virtualenv) instead of resorting to the registry.
[gravatar]
@guillermo - Normal Python packages get installed in the virtualenv correctly once they are activated. Ned is talking about installing from a Windows installers (i.e. a *.exe) which typically looks up the installed version of Python in the registry and installs itself there instead of a virualenv that maybe activated.

Some installers are smart enough to pickup multiple versions of Python installed but I haven't come across any that actually pick up on an active virtualenv. This is the issue that Ned's script is dealing with.
[gravatar]
easy_install I think can install Windows .exe files, not just .eggs or source. pip can't, but hopefully that'll be added this summer.
[gravatar]
@John M. Camara: I see. I was talking about regular packages only. Also, the fact that Win seems to use different methods to find the python interpreter depending on whether you do $ ./script.py or $ python script.py on the command line is rather confusing.

I might add the register/unregister functionality to my port of virtualenvwrapper: http://bitbucket.org/guillermooo/virtualenvwrapper/src
[gravatar]
Thank you! Very helpful. Allowed me to quickly get the MySQLdb binaries into my virtualenv.
[gravatar]
Piotr Dobrogost 7:37 AM on 26 Mar 2011
Ian Bicking is right. You can install using .exe installer as a source with easy_install. Take a look at my answer to your question on SO ->http://stackoverflow.com/questions/3271590/can-i-install-python-windows-packages-into-virtualenvs/5442340#5442340
[gravatar]
Hi,
thank you very much Ned, you saved my evening. I never though that installing what I developed under Mac OS X would be so painfull to install under Windows.
[gravatar]
It should be noted that sourcing an exe or msi installer with easy_install only works if the packager used the distutils installer building features, not some other installer. Other installers WILL check the registry, so this is a more general solution.
[gravatar]
My contribution to this post not withstanding....

pip install wheel
wheel convert pyOpenSSL-0.13.1.win32-py2.7.exe
pip install pyOpenSSL-0.13.1-cp27-none-win32.whl
[gravatar]
There is much easier way nowadays: easy_install binary_installer_built_with_distutils.exe

Works flawless!
[gravatar]
Kevin Dahlhausen 3:47 PM on 15 Nov 2014
Thanks. I just came across this 3 days ago and said "That's good, might need it one day" and saved it. Well, today was the day and it worked great.

Thanks!

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.