Django is easily the most popular Python web framework these days. For all of its features, and ease of use, though, sometimes it just seems misleading on purpose. This morning I fixed a mysterious problem, and once again I was reminded of how Django can seem simple until things go wrong, and then it's weirdly complex.

In particular, how the settings work is just odd. There are two ways that Django does two things when it would be better to do only one.

For Ibis Reader, our settings machinery is elaborate: the settings file imports from product_settings.py, then from a host-specific settings file, then from a local_settings.py which isn't committed to source control:

# Settings.py
    
#.. lots of settings ..

from product_settings import *

# Settings particular to this host.
# For a host named xyz01.myapp.com, 
# create a file host_settings/xyz01_myapp_com.py
import platform
host_name = platform.node().replace('.', '_').replace('-', '_')
try:
    exec "from ibis.host_settings.%s import *" % host_name
except ImportError:
    pass

# Last resort (good for dev machines): 
# import settings that aren't in the repo.
try:
    from local_settings import *
except ImportError:
    pass

This scheme works great: you can put settings in the file that corresponds logically to why the setting needs the value.

But something odd was happening: if a setting was in both product_settings.py and the host settings file, then the value in product_settings won. How could this be? The host settings file is applied after product_settings!

Part of the answer is the first thing that Django does twice that should only happen once: the settings file is imported twice. This flies in the face of everything we know about Python modules, but it happens. So the actual order of imports for my settings files is:

  1. from product_settings import *
  2. from ibis.host_settings.my_host import *
  3. from local_settings import *
  4. from product_settings import *
  5. from ibis.host_settings.my_host import *
  6. from local_settings import *

I don't know why Django imports twice, but it's long been true, and I've had to rediscover it the hard way a few times.

But this still doesn't explain the mystery: every time product_settings is applied, host settings should then be applied over it, so why would a setting in product_settings take effect over one in host settings? The answer is in the second thing that Django does twice: adding directories to the Python path.

I don't know if this is really Django's fault, or something about the way people seem to always configure their Django projects, but it seems to very often be true: your source files are available through two different import paths, because your source tree has been added to the Python path twice at two different levels.

A Django project has a top level corresponding to the project ("ibis" in this case), and then apps beneath that. The Python path is constructed so that you can import a file as "my_project.my_app", or just as "my_app". Except that for some reason, this double-view of the source tree isn't always available, and it isn't during that second series of settings imports!? The path is being modified between the two import sequences!

So the import march actually looks like this:

  1. from product_settings import *
  2. from ibis.host_settings.my_host import *
  3. from local_settings import *
  4. from product_settings import *
  5. from ibis.host_settings.my_host import *: Import failed!
  6. from local_settings import *

The net result is that settings in both product_settings and host settings will keep the value from product_settings, even though host settings is imported second.

The fix is really easy: remove "ibis." from the host settings import line, taking advantage of the fact that either form will work, and in fact, the second form is more robust since it seems to always be available on the Python path. The settings files still get imported twice, but at least the same thing happens both times.

I still don't understand why all these things happen. I hope part of this is my fault, because then I can fix it for real.

tagged: » 13 reactions

Comments

[gravatar]
zsiciarz 6:58 AM on 6 Dec 2011

Perhaps the change in the default project layout, which will be introduced in 1.4, will prevent the double import, or at least establish a better convention.

[gravatar]
Ned Batchelder 7:35 AM on 6 Dec 2011

@zsiciarz: it certainly looks like that change will stop the double imports. That release note doesn't actually say that Django is doing the double-path stuff, though.

[gravatar]
Jacob Kaplan-Moss 9:41 AM on 6 Dec 2011

Yeah, this is a wart. It's very very slightly your fault (if you pay really close attention to PYTHONPATH this doesn't happen) but it's mostly Django's fault. manage.py puts both . and .. on PYTHONPATH, which is just an accident waiting to happen. Further, the from local_settings import * trick is a bit of an antipattern, but the docs don't do anything to encourage you to do it differently.

We're starting to resolve some of this:

* As zsiciarz points out, Django 1.4 changes the default project layout, in the process getting rid of the module-on-PYTHONPATH-twice nastiness.

* I'd like to kill the double-import thing, but it's a side effect of how Django discovers custom management commands and it's a bit tricky to get rid of.

* I'm starting to try to spread a new pattern of handling multiple settings files, and hopefully I can find some time to get it written up and in the docs. You can check out this slide deck, especially slides 47 - 51.

[gravatar]
Ricardo Kirkner 9:50 AM on 6 Dec 2011

Hi Ned,

while this is not a lightweight solution to your problem, you may want to look at django-configglue (http://django-configglue.readthedocs.org). It's a library to allow django to work with configglue (http://configglue.readthedocs.org) generated files.

configglue makes it easy to manage multiple layered configuration files, amongst other things, and the integration with django works fine. An example of the nice things you get is to be able to lookup in which file a setting was lastly defined.

Just wanted to share this with you, it might be useful

[gravatar]
Ned Batchelder 12:24 PM on 6 Dec 2011

@Jacob, Your slides suggest a simple change to the settings choreography, but I'm not sure it would have solved this problem. I certainly agree with you that using local.py to handle server differences is a bad idea, which is why I use a hostname-based scheme.

@Ricardo: I hadn't seen configglue before, thanks for the links.

[gravatar]
Sever Băneșiu 1:10 PM on 6 Dec 2011

While this won't solve your problem, I think an entry-point based solution for extending the settings is better than hardcoding the imports. django-extraconfig can serve as an example.

[gravatar]
Idan Gazit 1:46 PM on 6 Dec 2011

Hey Ned,

You can still use a hostname-based scheme with the kind of approach that Jacob lays out. I use settings/__init__.py as a traffic cop; it contains any logic about which settings file to import. I used this approach in epio_skel to good effect, there it toggles based on the presence of an environment variable.

The only key thing (as pointed out by Jacob) is to import specific->general, e.g. import your most specific settings file, and have that import more generic ones at the top of your file.

[gravatar]
Ned Batchelder 7:26 AM on 7 Dec 2011

@Sever: thanks for the pointer to django-extraconfig. Both it and configglue seeem to add extra steps: configglue wants me to make explicit a schema for my settings, and django-extraconfig requires a setup.py for each extra settings file? Solutions will need to be as simple as "one more settings.py file" to succeed, I think.

@Idan: You are right, I can combine the host-based scheme with the idea of importing specific to general. My point was just that specific-to-general seems to be independent of the issues I was having. I would still have had double importing, and I would still have had mysteriously changing python paths.

[gravatar]
Nick Coghlan 4:59 PM on 7 Dec 2011

While it's great for app-level configuration and calculation of default settings, I personally consider using Python for production settings a bit of an anti-pattern (since it makes life harder for sysadmins). Accordingly, I have my settings.py set up to read the real settings with ConfigParser:

http://git.fedorahosted.org/git/?p=pulpdist.git;a=blob;f=src/pulpdist/django_site/settings.py

(and yes, I know I need to move at least SECRET_KEY into the production config file - it's on the to-do list)

[gravatar]
Gareth Rees 5:03 PM on 8 Mar 2012

Thank you, Ned, this helped me track down a puzzling wrinkle with WSGI deployment.

The Django 1.3 tutorial basically tells you to make the mistake of referring to your modules under two different paths. Imagine you are a new Django programmer and are following the tutorial.

The first thing the tutorial suggests you do is run

django-admin.py startsite mysite
Suppose you do this in the directory ~. Then you get the file ~/mysite/settings.py which contains the line:
ROOT_URLCONF = 'mysite.urls'
So this will only work if ~ is on the path.

Then the tutorial suggests that you cd mysite and run
python manage.py startapp polls
And then it says, "Edit the settings.py file again, and change the INSTALLED_APPS setting to include the string 'polls'."

So this will only work if ~/mysite is on the path.

All goes well as long as you are running the development server via manage.py, because it puts both directories on the path. But when it comes to deploying the site, it won't work unless you put both directories on the path. (And the Apache/mod_wsgi documentation doesn't mention this.)

Is this a bug in the tutorial, the startsite command, or the Apache/mod_wsgi documentation?

[gravatar]
Gareth Rees 5:06 PM on 8 Mar 2012

Looks like it's all fixed in 1.4, though.

[gravatar]
Jonathan Hartley 11:29 AM on 27 Dec 2012

@Jacob

I know this is a very old issue, but I'm working on a Django 1.2 codebase, and am having trouble interpreting the slide 51 to which you refer. For ease of reference, it reads:


51. The one true way

    
    settings
        __init__.py
        base.py
            INSTALLED_APPS = [...]
        staging.py
        production.py
        local.py
            from settings.base import *
            INSTALLED_APPS += ['debug_toolbar']

    $ django-admin.py shell --settings=settings.local
    
    # deploy.wsgi
    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings.deploy'
What is the 'settings.deploy' referred to in the last line? How are settings.staging or .production intended to be used? How is this local.py different from the antipattern of using local.py you condone in the previous slides?

TBH, I have similar questions about many of the slides in this presentation. In the spirit of Christmas, would you reconsider posting an intact version of the speakers notes for that presentation, even if they were never intended for publication? For the children???

Cheers.

[gravatar]
Jonathan Hartley 11:45 AM on 27 Dec 2012

I said 'condone' in my last post when I meant the oppositish. Criticize, maybe.

Add a comment:

name
email
Ignore this:
not displayed and no spam.
Leave this empty:
www
not searched.
 
Name and either email or www are required.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.