Tuesday 18 June 2013 — This is more than 11 years old. Be careful.
I’ve always found descriptors to be one of the more confusing corners of Python. Partly, I think that’s to do with the explanations in the docs, which I find opaque:
In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol.
What is binding behavior, and why is it in quotes? This lead sentence hints at overriding attribute access, but doesn’t tell me how it happens. It’s a tall wall to scale right at the start of the learning process.
The best explanation I’ve seen of what descriptors do and why you’d want to write them was in Chris Beaumont’s lightning talk at Boston Python, Demystifying Descriptors in 5 Minutes. The video quality was not great, but now Chris has written it up: Python Descriptors Demystified.
(Sorry about the quality, we’re getting much better... PS: subscribe to our YouTube channel!)
Chris’ insight was that instead of defining descriptors, and then showing how you could make properties with them, he flipped that explanation around: explain properties, then show how descriptors are like generalized properties. Read the whole thing: Python Descriptors Demystified.
When explaining things, you have to build from what people already know, a step at a time. I picture a student’s understanding being like geography. What they know is a land mass, and when you teach them, you are extending their land out into the unknown ocean. You want to make their island bigger, and there’s a particular point out in the ocean you want to encompass.
Some technical descriptions will explain that distant point, and either hope you can build the peninsula yourself, or expect to be able to build backwards toward the mainland. The classic descriptor explanation is like that: provide a definition of the distant concept, and hope students can make the leap.
Chris’ explanation is more incremental. Start with what we know, and extend a little bit at a time, with motivations as we go. I love it.
BTW: I made some edits to the Python documentation, but they haven’t been adopted: Edits to descriptor howto. Others have also suggested reorganizations of the docs about descriptors: Harmonizing descriptor protocol documentation.
Descriptors are still an advanced feature, and I don’t expect everyone to understand and use them. But they are not as complicated as they first seem, and the explanations can do a better job helping people up that learning curve.
Comments
So using descriptors is easy (it's so easy people don't realise they're doing it whenever they retrieve a method from an object). It's defining new ones that can be a little arcane (that's why it's often easier to use nested functions + property in a factory function than it is to use the descriptor protocol to define a new descriptor from scratch).
They're a lot like decorators that way (and, indeed, it's not an accident that most descriptors offer a decorator form).
Another problem I spot was that storing values in a dictionary inside a descriptor can cause a huge memory leak. Descriptors live in the class so everything stored into the dictionary lives "forever". Using weakref.WeakKeyDictionary, or storing values to instances, is safer.
Right, so I might have been sloppy with some of the terminology in the article. I can address that.
Still, I agree with Ned that many easy-to-google discussions of descriptors (http://docs.python.org/2/howto/descriptor.html#id1, http://stackoverflow.com/questions/101268/hidden-features-of-python#102062) explain things backwards. As a consumer of Python, I don't *care* if descriptors can implement properties, class methods, etc. Those things already exist, and don't communicate why writing custom descriptors would be useful. Furthermore, none of those examples illustrate how to deal with instance-specific state, which confused me for a long time.
@Pekka's comment about WeakKeyDictionaries is a really excellent point.
http://m.youtube.com/#/watch?v=D3-NZXHO5QI&desktop_uri=%2Fwatch%3Fv%3DD3-NZXHO5QI
And for what it's worth, I found this blog post http://python-history.blogspot.com/2010/06/inside-story-on-new-style-classes.html?m=1 by Guido Van Rossum to be the best description of descriptors that I've seen (that whole blog is great, actually).
Add a comment: