Explaining descriptors

Tuesday 18 June 2013This is more than 11 years old. Be careful.

I’ve always found descriptors to be one of the more confusing corners of Python. Partly, I think that’s to do with the explanations in the docs, which I find opaque:

In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol.

What is binding behavior, and why is it in quotes? This lead sentence hints at overriding attribute access, but doesn’t tell me how it happens. It’s a tall wall to scale right at the start of the learning process.

The best explanation I’ve seen of what descriptors do and why you’d want to write them was in Chris Beaumont’s lightning talk at Boston Python, Demystifying Descriptors in 5 Minutes. The video quality was not great, but now Chris has written it up: Python Descriptors Demystified.

(Sorry about the quality, we’re getting much better... PS: subscribe to our YouTube channel!)

Chris’ insight was that instead of defining descriptors, and then showing how you could make properties with them, he flipped that explanation around: explain properties, then show how descriptors are like generalized properties. Read the whole thing: Python Descriptors Demystified.

When explaining things, you have to build from what people already know, a step at a time. I picture a student’s understanding being like geography. What they know is a land mass, and when you teach them, you are extending their land out into the unknown ocean. You want to make their island bigger, and there’s a particular point out in the ocean you want to encompass.

Some technical descriptions will explain that distant point, and either hope you can build the peninsula yourself, or expect to be able to build backwards toward the mainland. The classic descriptor explanation is like that: provide a definition of the distant concept, and hope students can make the leap.

Chris’ explanation is more incremental. Start with what we know, and extend a little bit at a time, with motivations as we go. I love it.

BTW: I made some edits to the Python documentation, but they haven’t been adopted: Edits to descriptor howto. Others have also suggested reorganizations of the docs about descriptors: Harmonizing descriptor protocol documentation.

Descriptors are still an advanced feature, and I don’t expect everyone to understand and use them. But they are not as complicated as they first seem, and the explanations can do a better job helping people up that learning curve.

Comments

[gravatar]
As I noted on Twitter, my only objection to Chris's explanation is that he uses the general term "descriptor" when he specifically means "custom descriptor". People will walk away from that explanation without realising that things like functions, classmethod, staticmethod, property and abc.abstractmethod are all descriptors.

So using descriptors is easy (it's so easy people don't realise they're doing it whenever they retrieve a method from an object). It's defining new ones that can be a little arcane (that's why it's often easier to use nested functions + property in a factory function than it is to use the descriptor protocol to define a new descriptor from scratch).

They're a lot like decorators that way (and, indeed, it's not an accident that most descriptors offer a decorator form).
[gravatar]
@Nick: I'm not sure I see the benefit of people knowing that functions, classmethods, etc, are descriptors. There's no reason to know the term "descriptor," or know how they work, until you want to write one of your own.
[gravatar]
@Ned I found that when teaching people how to write custom descriptors it's helps to show how the known concepts methods, static methods, class methods etc. can be easily expressed using the descriptor API. These are examples people can relate to.
[gravatar]
I agree with @Nick that terminology in the otherwise really good article is confusing.

Another problem I spot was that storing values in a dictionary inside a descriptor can cause a huge memory leak. Descriptors live in the class so everything stored into the dictionary lives "forever". Using weakref.WeakKeyDictionary, or storing values to instances, is safer.
[gravatar]
Thanks for the article, Ned!

Right, so I might have been sloppy with some of the terminology in the article. I can address that.

Still, I agree with Ned that many easy-to-google discussions of descriptors (http://docs.python.org/2/howto/descriptor.html#id1, http://stackoverflow.com/questions/101268/hidden-features-of-python#102062) explain things backwards. As a consumer of Python, I don't *care* if descriptors can implement properties, class methods, etc. Those things already exist, and don't communicate why writing custom descriptors would be useful. Furthermore, none of those examples illustrate how to deal with instance-specific state, which confused me for a long time.

@Pekka's comment about WeakKeyDictionaries is a really excellent point.
[gravatar]
You've seen discovering descriptors from europython last year?

http://m.youtube.com/#/watch?v=D3-NZXHO5QI&desktop_uri=%2Fwatch%3Fv%3DD3-NZXHO5QI
[gravatar]
Here is another use-case for descriptors: http://asmeurersympy.wordpress.com/2013/04/06/how-to-make-attributes-un-inheritable-in-python-using-descriptors/ (making attributes un-inheritable).

And for what it's worth, I found this blog post http://python-history.blogspot.com/2010/06/inside-story-on-new-style-classes.html?m=1 by Guido Van Rossum to be the best description of descriptors that I've seen (that whole blog is great, actually).
[gravatar]
looks like a verbose way of doing properties. unless i am missing something C# and recently Dart do this a lot tersely (and elegantly) through "get" and "set" modifiers.
[gravatar]
@numan You're right, in a way: properties are implemented by descriptors. For example,
@property
def x(self):
    return True
creates an object whose __get__ method calls the function x.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.