« | » Main « | »

Python is well-known for its duck-typing: objects are examined for what they can do rather than for what type they are. But if you like being strict about the methods derived classes have to implement, you can use the abstract base classes in the abc module.

They let you define a class, with some methods defined as abstract, and if those methods aren't defined in a subclass, the subclass can't be instantiated:

# abstract.py
from abc import ABCMeta, abstractmethod

class Abstract(metaclass=ABCMeta):

    def concrete(self):
        print("I am concrete")

    @abstractmethod
    def not_defined_yet(self):
        raise NotImplementedError

a = Abstract()

produces:

Traceback (most recent call last):
  File "abstract.py", line 13, in <module>
    a = Abstract()
TypeError: Can't instantiate abstract class Abstract with abstract methods not_defined_yet

This is great when you want to be strict, and can remind you of your pleasant days writing Java! But like Java, you can find yourself in situations where you have an abstract base class with a handful of abstract methods, and know that you only need a few of them. The usual remedy at this point is to define all the missing methods knowing they'll never be called. This is the worst of "keeping the compiler happy": you know what you need, but the type checking insists that you go through the motions.

Here's another option: a class decorator that erases the list of abstract methods, so that the class can be instantiated:

def unabc(cls):
    cls.__abstractmethods__ = ()
    return cls

Now we can make a subclass of our abstract base class, not define any methods, and still instantiate the class:

@unabc
class ShutUpAbc(Abstract):
    pass

just_do_it = ShutUpAbc()    # yay!

If we want to get fancier, we can! The missing abstract methods aren't going to be called (we think!) but we can provide stub methods just in case. The stub methods will raise an error with a message naming the method. For extra bells and whistles, the message will be settable in the decorator, and the decorator will be usable with or without a customized message:

def unabc(arg):
    """
    Add stub methods to a class to satisfy abstract base classes.

    Usage::

        @unabc
        class NotAbstract(SomeAbstractClass):
            pass

        @unabc('Fake {}')
        class NotAbstract(SomeAbstractClass):
            pass
    """

    def _unabc(cls, msg=arg):
        def make_stub_method(ab_name):
            def stub_method(self, *args, **kwargs):
                meth_name = cls.__name__ + "." + ab_name
                raise NotImplementedError(msg.format(meth_name))
            return stub_method

        for ab_name in cls.__abstractmethods__:
            setattr(cls, ab_name, make_stub_method(ab_name))

        # No more abstract methods!
        cls.__abstractmethods__ = ()
        return cls

    # Handle the possibility that unabc is called without a custom message.
    if isinstance(arg, type):
        return _unabc(arg, "{} isn't implemented, and won't be!")
    else:
        return _unabc

Here the _unabc function is the actual decorator. It loops over all the abstract method names, and makes a new stub method for each one. The make_stub_method function is needed because we need to close over the ab_name variable so it will have the proper value when called.

Then stub_method is defined as the actual method that will be added to the class with setattr. Yes, this is four defs nested inside each other: one to define the decorator you use, one to be the actual decorator applied to the class, one to form a closure so we can define stub methods, and one to create the stub methods themselves!

The last part here is to deal with the two ways the unabc decorator can be used: if it's used without an argument, then the class in question will be the argument, and the isinstance check will be true. In that case, we'll use the argument as the class, and provide a default message. If the argument isn't a class, then we return _unabc, and the argument is already provided as a default msg for the _unabc function.

BTW: all the code above is Python 3. The only thing to change for Python 2 is how the ABCMeta metaclass is associated with your abstract class:

class Abstract(object):
    __metaclass__ = ABCMeta
    ...
tagged: » 4 reactions

I was experimenting with pydoc yesterday, and was baffled by how it was running. Turns out Mac OS X does some tricky stuff to support multiple versions of Python.

If you type "pydoc" at a shell prompt, it works properly:

$ pydoc
pydoc - the Python documentation tool

pydoc.py <name> ...
    Show text documentation on something.  <name> may be the name of a
    (etc...)

If you ask which file is being run, it's /usr/bin/pydoc, and you can look at that file:

$ which pydoc
/usr/bin/pydoc
$ more /usr/bin/pydoc
#!/usr/bin/python

import sys, os
import glob, re

partA = """\
python version %d.%d.%d can't run %s.  Try the alternative(s):

"""
partB = """
Run "man python" for more information about multiple version support in
Mac OS X.
"""

sys.stderr.write(partA % (sys.version_info[:3] + (sys.argv[0],)))
(etc...)

Notice that this file will always write "python version X.Y.Z can't run...," which is not the output we're getting. Weird!

(BTW, if you have activated a virtualenv, you may have an alias, so that "pydoc" is actually "python -m pydoc". Is there an equivalent to "which" that will include that fact it in its output?)

But even without the virtualenv alias, this file isn't being run. Why not? The answer is in the shebang line:

#!/usr/bin/python

Unix uses the shebang line to find a program to run the file. So typing "pydoc" at the prompt will find /usr/bin/pydoc, then find the shebang line, and will actually run this:

/usr/bin/python /usr/bin/pydoc

Seems simple enough: invoke Python, and have it run the code in /usr/bin/pydoc. So why isn't Python running the Python code we saw? The answer is that /usr/bin/python is not a Python interpreter!

On OS X, /usr/bin/python examines various settings and then invokes a real Python interpreter of the correct version: python.1 man page. A quick look at the readable text inside the executable confirms that it is not a full interpreter, and that it is concerned with versions:

$ strings /usr/bin/python
python
/System/Library/Frameworks/Python.framework/Versions/
/Resources/Python.app/Contents/MacOS/Python
pythonw
VERSIONER_DEBUG
VERSIONER_PYTHON_VERSION
%s environment variable error (ignored)
VERSIONER_PYTHON_PREFER_32_BIT
no user %d
%s: too long
%s: Can't append "%s"
prefer32bit=%d version=%s
Executable path too long
proc_pidpath
PATH
/usr/bin:/bin
alloca: out of memory
realpath couldn't resolve "%s"
%s: Can't append "%s"
argv=%s path=%s
posix_spawnattr_init
posix_spawnattr_setflags
posix_spawnattr_setbinpref_np
posix_spawnattr_setbinpref_np only copied %d of %d
posix_spawn: %s
%s -> %s
versionarg: Out of memory
read_plist: %s: open
read_plist: %s: stat
read_plist: %s: mmap
read_plist: %s: CFDataCreateWithBytesNoCopy failed
(cfstrdup also failed)
read_plist: %s: CFPropertyListCreateFromXMLData failed: %s
read_plist: %s: plist not a dictionary
Prefer-32-Bit
read_plist: %s: Prefer-32-Bit not a boolean
Version
read_plist: %s: %s: Version unknown
read_plist: %s: Version not a string
true
false
0123456789
/Preferences/com.apple.versioner.python.plist
/usr/bin/

So this finder is given "/usr/bin/pydoc" as an argument, and it decides what to really run. It's special-casing "/usr/bin", and actually invoking /usr/bin/pydoc2.7. The real /usr/bin/pydoc file is there only to be executed when the version-selection mechanism fails, which is why it simply prints messages about not being able to find the right version.

It seems that the switcher doesn't care much what command you're trying to run. If it's in /usr/bin, and there's a file alongside it with the same name but the current Python version appended, then it will run the versioned one.

All this can be verified by an experiment. I created /usr/bin/foo with these contents:

#!/usr/bin/python
import sys
print sys.argv, "Plain!"

and a /usr/bin/foo2.7 with this:

#!/usr/bin/python
import sys
print sys.argv, "Versioned!"

Then I ran it a number of different ways (the current directory is in the prompt):

~ $ foo
['/usr/bin/foo2.7'] Versioned!
~ $ /usr/bin/foo
['/usr/bin/foo2.7'] Versioned!
~ $ foo2.7
['/usr/bin/foo2.7'] Versioned!
~ $ /usr/bin/foo2.7
['/usr/bin/foo2.7'] Versioned!
~ $ cd /usr/bin
/usr/bin $ foo
['/usr/bin/foo2.7'] Versioned!
/usr/bin $ foo2.7
['/usr/bin/foo2.7'] Versioned!
/usr/bin $ ./foo
['./foo'] Plain!

Notice that the Python switcher runs the versioned file whenever it's identified as being from /usr/bin. But when run so that the shell doesn't identify it that way, even when it's the exact same file, the Python switcher decides it shouldn't interfere, and it runs the exact file specified.

Tricky.

tagged: , , » 4 reactions

I've just released the latest version of coverage.py: coverage.py v3.7.1. The only changes are performance improvements in the HTML report, and a little restructuring to make Debian packaging a little easier.

The next version will be 4.0, and I'm looking for feedback about major changes you'd like to see. Coverage.py has never had an API-centric mindset, for example. Breaking backward compatibility will be OK if there's a good reason, and I'm dropping support for older Pythons, so it should be easier to make changes.

Send me your ideas.

I help people with Python questions, and often those people are only casual acquaintances (at Boston Python) or complete strangers (in the #python IRC channel). There's a common misunderstanding that can crop up in these situations. It has to do with asking why.

While helping, I often ask the asker, "why?" For example, someone will need help with the two Python installations on their laptop. I'll ask, "Why did you install a second Python?" When I did this the other night, another guy chuckled, because he thought I meant, "You shouldn't have installed a second Python."

This is a common reaction. Asking why is perceived as a rebuke. "Why did you XYZ?" is taken to mean, "You dummy, you shouldn't have XYZed!" But when I ask it, I really do mean, "I want to understand what led you to XYZ."

English can be difficult that way, especially over a purely textual, cue-less medium like IRC. One way to soften the apparent bluntness is to add more words. For example, instead of:

Why did you install another Python?

this might go over better:

Do you mind if I ask you why you installed another Python? Understanding the reason might provide an important clue.

So if I ask you why, don't take it personally. I really do want to know the reason for something. If I want to chide you for making a mistake, I'll say something like, "You shouldn't have XYZed," or, "I don't think it was a good idea to XYZ."

Sometimes even when people understand I'm looking for reasons, they bristle at the question. They insist the reasons aren't important, and why can't I just answer a simple question?

I ask because 75% of the time, my answer changes once I know the bigger picture. It's common for people to start with a problem, and work towards a solution, and get stuck. Then they ask about the thing they are stuck on. That's natural. But it's also common that the reason they are stuck is there was a better path to start down in the first place. Understanding the reasons can help find that better path.

Asking why can help find those earlier choices, and lay out the entire problem. Having all the information means we can work together to find the best solution.

If I ask you to explain the larger problem, don't take it personally. Solving complex problems is hard, and it's easy to choose a first step that makes the second step more difficult than it needs to be. When a helper asks about the bigger picture, they aren't trying to make you look silly, they're trying to help you find the best solution.

This phenomenon has a name: the XY problem. You have problem X, you choose solution Y, and when it doesn't work, you ask about Y instead of asking about X. Despite some people's disdain for this dynamic, it's very common and people shouldn't be put down for it. Sometimes the asker can't see that there were alternatives earlier on. Sometimes the asker is trying to not take up too much of the helper's time by asking a focused question. Whatever the reason, it happens all the time, and helpers should get out of the helping business if XY problems make them mad.

Askers: if someone asks you why, or says you have an "XY problem," that means they are trying to help you find the best solution. Don't feel bad. We're all struggling together to learn and overcome complexity.

« | » Main « | »