Tuesday 6 December 2011 — This is nearly 13 years old. Be careful.
Django is easily the most popular Python web framework these days. For all of its features, and ease of use, though, sometimes it just seems misleading on purpose. This morning I fixed a mysterious problem, and once again I was reminded of how Django can seem simple until things go wrong, and then it’s weirdly complex.
In particular, how the settings work is just odd. There are two ways that Django does two things when it would be better to do only one.
For Ibis Reader, our settings machinery is elaborate: the settings file imports from product_settings.py, then from a host-specific settings file, then from a local_settings.py which isn’t committed to source control:
# Settings.py
#.. lots of settings ..
from product_settings import *
# Settings particular to this host.
# For a host named xyz01.myapp.com,
# create a file host_settings/xyz01_myapp_com.py
import platform
host_name = platform.node().replace('.', '_').replace('-', '_')
try:
exec "from ibis.host_settings.%s import *" % host_name
except ImportError:
pass
# Last resort (good for dev machines):
# import settings that aren't in the repo.
try:
from local_settings import *
except ImportError:
pass
This scheme works great: you can put settings in the file that corresponds logically to why the setting needs the value.
But something odd was happening: if a setting was in both product_settings.py and the host settings file, then the value in product_settings won. How could this be? The host settings file is applied after product_settings!
Part of the answer is the first thing that Django does twice that should only happen once: the settings file is imported twice. This flies in the face of everything we know about Python modules, but it happens. So the actual order of imports for my settings files is:
- from product_settings import *
- from ibis.host_settings.my_host import *
- from local_settings import *
- from product_settings import *
- from ibis.host_settings.my_host import *
- from local_settings import *
I don’t know why Django imports twice, but it’s long been true, and I’ve had to rediscover it the hard way a few times.
But this still doesn’t explain the mystery: every time product_settings is applied, host settings should then be applied over it, so why would a setting in product_settings take effect over one in host settings? The answer is in the second thing that Django does twice: adding directories to the Python path.
I don’t know if this is really Django’s fault, or something about the way people seem to always configure their Django projects, but it seems to very often be true: your source files are available through two different import paths, because your source tree has been added to the Python path twice at two different levels.
A Django project has a top level corresponding to the project (“ibis” in this case), and then apps beneath that. The Python path is constructed so that you can import a file as “my_project.my_app”, or just as “my_app”. Except that for some reason, this double-view of the source tree isn’t always available, and it isn’t during that second series of settings imports!? The path is being modified between the two import sequences!
So the import march actually looks like this:
- from product_settings import *
- from ibis.host_settings.my_host import *
- from local_settings import *
- from product_settings import *
- from ibis.host_settings.my_host import *: Import failed!
- from local_settings import *
The net result is that settings in both product_settings and host settings will keep the value from product_settings, even though host settings is imported second.
The fix is really easy: remove “ibis.” from the host settings import line, taking advantage of the fact that either form will work, and in fact, the second form is more robust since it seems to always be available on the Python path. The settings files still get imported twice, but at least the same thing happens both times.
I still don’t understand why all these things happen. I hope part of this is my fault, because then I can fix it for real.
Comments
We're starting to resolve some of this:
* As zsiciarz points out, Django 1.4 changes the default project layout, in the process getting rid of the module-on-PYTHONPATH-twice nastiness.
* I'd like to kill the double-import thing, but it's a side effect of how Django discovers custom management commands and it's a bit tricky to get rid of.
* I'm starting to try to spread a new pattern of handling multiple settings files, and hopefully I can find some time to get it written up and in the docs. You can check out this slide deck, especially slides 47 - 51.
while this is not a lightweight solution to your problem, you may want to look at django-configglue (http://django-configglue.readthedocs.org). It's a library to allow django to work with configglue (http://configglue.readthedocs.org) generated files.
configglue makes it easy to manage multiple layered configuration files, amongst other things, and the integration with django works fine. An example of the nice things you get is to be able to lookup in which file a setting was lastly defined.
Just wanted to share this with you, it might be useful
@Ricardo: I hadn't seen configglue before, thanks for the links.
You can still use a hostname-based scheme with the kind of approach that Jacob lays out. I use settings/__init__.py as a traffic cop; it contains any logic about which settings file to import. I used this approach in epio_skel to good effect, there it toggles based on the presence of an environment variable.
The only key thing (as pointed out by Jacob) is to import specific->general, e.g. import your most specific settings file, and have that import more generic ones at the top of your file.
@Idan: You are right, I can combine the host-based scheme with the idea of importing specific to general. My point was just that specific-to-general seems to be independent of the issues I was having. I would still have had double importing, and I would still have had mysteriously changing python paths.
http://git.fedorahosted.org/git/?p=pulpdist.git;a=blob;f=src/pulpdist/django_site/settings.py
(and yes, I know I need to move at least SECRET_KEY into the production config file - it's on the to-do list)
The Django 1.3 tutorial basically tells you to make the mistake of referring to your modules under two different paths. Imagine you are a new Django programmer and are following the tutorial.
The first thing the tutorial suggests you do is runSuppose you do this in the directory ~. Then you get the file ~/mysite/settings.py which contains the line:So this will only work if ~ is on the path.
Then the tutorial suggests that you cd mysite and runAnd then it says, "Edit the settings.py file again, and change the INSTALLED_APPS setting to include the string 'polls'."
So this will only work if ~/mysite is on the path.
All goes well as long as you are running the development server via manage.py, because it puts both directories on the path. But when it comes to deploying the site, it won't work unless you put both directories on the path. (And the Apache/mod_wsgi documentation doesn't mention this.)
Is this a bug in the tutorial, the startsite command, or the Apache/mod_wsgi documentation?
I know this is a very old issue, but I'm working on a Django 1.2 codebase, and am having trouble interpreting the slide 51 to which you refer. For ease of reference, it reads:
51. The one true way
settings __init__.py base.py INSTALLED_APPS = [...] staging.py production.py local.py from settings.base import * INSTALLED_APPS += ['debug_toolbar'] $ django-admin.py shell --settings=settings.local # deploy.wsgi os.environ['DJANGO_SETTINGS_MODULE'] = 'settings.deploy'What is the 'settings.deploy' referred to in the last line? How are settings.staging or .production intended to be used? How is this local.py different from the antipattern of using local.py you condone in the previous slides?TBH, I have similar questions about many of the slides in this presentation. In the spirit of Christmas, would you reconsider posting an intact version of the speakers notes for that presentation, even if they were never intended for publication? For the children???
Cheers.
settings.py is not exactly loaded twice, but rather in two separate proccesses (cf https://stackoverflow.com/questions/11149730/django-settings-py-seems-to-load-multiple-times ).
About the path, I don't know what's going on there but it might be related to referencing the project folder on a django level, while referencing the app folder on the app level....
Add a comment: