Multiple inheritance is hard

Sunday 28 October 2012This is more than 12 years old. Be careful.

Multiple inheritance is hard, we all know it, but even after warning people about it myself, I found myself tripped up by it yesterday. All I wanted was a mixin for my unittest.TestCase class.

Unittest’s TestCases use setUp and tearDown to affect the test state, and I wanted a way to share setUp and tearDown implementations between two different test classes. A mixin seemed like a good solution. I already have a BaseTestCase of my own that inherits from unittest.TestCase, so my mixin looked like this:

class BaseTestCase(unittest.TestCase):
    def setUp(self):
        super(BaseTestCase, self).setUp()
        #.. set up that everyone needs ..

    def tearDown(self):
        #.. tear down that everyone needs ..
        super(BaseTestCase, self).tearDown()

class MyMixin(object):
    def setUp(self):
        super(MyMixin, self).setUp()
        #.. do something here ..

    def tearDown(self):
        #.. do something here ..
        super(MyMixin, self).tearDown()

class RealTestCase(BaseTestCase, MyMixin):
    # I didn't need setUp and tearDown here..

    def test_foo(self):
        #.. etc ..

The theory is that mixins avoid some complications of multiple inheritance: they provide methods, but no attributes, and don’t inherit in a way that produces diamonds in the inheritance hierarchy. And I did the right things here, using super to keep the MRO (method resolution order) intact.

But this code doesn’t work, MyMixin.setUp is never invoked. Why? It’s because TestCase in unittest doesn’t invoke super():

# unittest.py (some details omitted :)
class TestCase(object):
    def setUp(self):
        pass

The method resolution order of RealTestCase is:

  • RealTestCase
  • BaseTestCase
  • unittest.TestCase
  • MyMixin
  • object

Since TestCase.setUp doesn’t use super, the sequence stops there, and MyMixin is never consulted. At first I thought, “TestCase.setup should use super!” But, if it did, it would fail in simpler hierarchies that don’t use a mixin, because it would try to invoke object.setUp(), which doesn’t exist.

I suppose TestCase could be re-written like this:

# unittest.py (hypothetical)
class TestCase(object):
    def setUp(self):
        parent = super(Base, self)
        if hasattr(parent, "setUp"):
            parent.setUp()

This works, and now MyMixin.setUp() is invoked, but then it fails for the same reason: it tries to invoke object.setUp, which doesn’t exist, so MyMixin also needs to have the defensive check of its parent. Yuk.

The simple solution is to swap the order of the mixin and the base class:

class RealTestCase(MyMixin, BaseTestCase):
    #...

With this class declaration, the MRO is: RealTestCase, MyMixin, BaseTestCase, TestCase, object. All the setUp’s are invoked, and it ends cleanly with no complicated parentage check.

But I can’t help feeling this isn’t great. For one thing, conceptually, BaseTestCase is the real parent, and the mixin is just tossed in, so I would rather be able to write my base classes in the other order. But more worrying, this solution also means that I have to be very careful to know the lineage of all my classes all the way up to object.

Maybe this is the fundamental truth about Multiple Inheritance, and why it is so difficult: details about base classes that you thought were abstracted away from you can suddenly be critical to understand. Like all action at a distance, this is mysterious and confusing.

My take-aways from this:

  • Mixins come first even though it looks odd.
  • Multiple inheritance is hard.
  • There are yet more Python details to master.

The question remaining in my mind: would class hierarchies be better if the top-most classes (derived from object) used the defensive super style? Or is that overkill that defers rather than removes the pain? Would something else bite me later?

Comments

[gravatar]
Can you elaborate about why listing the mixin before the base class "looks odd"? (I don't have the same experience.) It seems like the entire problem you've described here hinges on that.
[gravatar]
@Michael, when I think of my RealTestCase class, it is mostly a TestCase, with MyMixin mixed in. To me, it's most natural to talk about TestCase first.
[gravatar]
Multiple implementation inheritance is indicative of poor design. Even more generally speaking, inheritance should never be the construct of choice for re-use. Inheritance imposes extremely tight coupling between parent and child, as you indicated by saying "details about base classes that you thought were abstracted away from you can suddenly be critical to understand".

Inheritance should model a true, intensional generalization relationship. In your case, the discriminator between various subclasses of TestCase is only extensional, i.e. a minor detail about coincidental setup/teardown implementation.

Remember the heuristic "favor association over inheritance". For example, you could pass a setup/teardown implementor into your various test case classes (something like a Strategy pattern).
[gravatar]
I guess I'm in the minority but I disagree that multiple inheritance is "so difficult" or "indicative of poor design."

I just keep in mind that python searches for methods and attributes left-to-right through the list of inherited classes (obviously a simplification of the MRO algorithm, but it's good enough,) so the most-base class goes on the right. Call super() when overriding methods, and everything tends to Just Work. It's a bit hand-wavey I know, but Python hasn't let me down yet.
[gravatar]
@Eric, thanks for reminding me of the "composition over inheritance" maxim, I need to keep that in mind more. In this case, I can't pass an implementor into the test classes, because I don't own the code that constructs them, but I could certainly reference an implementor instead.
[gravatar]
Russell Borogove 12:37 PM on 28 Oct 2012
What prevented you from making MyMixin a subclass of BaseTestCase and inheriting singly from MyMixin?
[gravatar]
@Russell: in this case I could have, but I've wanted a way to separate the setUp and tearDown details from the main line of inheritance. Probably I should go all the way to composition instead, as Eric suggests.
[gravatar]
@eric: I'm glad I'm not the only one that feels composition over inheritance is a better reu-use model.
[gravatar]
Just putting my vote in for "multiple inheritance is just fine". The "multiple inheritance is a design flaw" meme IMHO is one of those rules people created under the justification of "if practice X confuses anyone, is poorly implemented in language Q, or generally can ever be mis-used by anyone anywhere, then it is always bad for everyone, all the time, in all languages" - and it has its roots within C++ where it developed a bad reputation. That's not an architectural straightjacket I'm willing to accept.

I use multiple inheritance usually for mixins (where I find it quite natural to apply them first, as I typically want their methods to take precedence), and more often than not in test code also (funny how it finds itself there quite a bit, as in this blog post). In *extremely rare* cases I do have some cases (well just one I can think of) where a particular subclass is truly an amalgam of two distinct parent hierarchies. I certainly don't do that lightly.
[gravatar]
I had the exact same problem and did exactly this, swap the order in which the base classes were introduced.

I've never fully understood MRO and how metaclasses affect it, it's supposed to be useful in this cases.

I also wonder if this changes in some way in Python3 with the new super().
[gravatar]
To echo Michael Lamb, there's a fundamental misunderstanding here about the ordering of the superclass list: it's not accidental, or freely alterable.

In Python (and any C3 class system), every class declaration should be read as a single partial order: class A(B, C) should be understood to mean that A < B < C, without putting special weight on A. (It can help to mentally ignore the parentheses and actually read the syntax as "class A < B < C: ...", when in doubt.)

In other words, the implication that B becomes a subclass of C is just as important and meaningful as the implication that A becomes a subclass of B. In particular, the question of swapping B and C around is no different in general than the question of swapping A and B around, and can have equally large repercussions.

Looking back at the example:
class RealTestCase(BaseTestCase, MyMixin):
This declaration should immediately jump out as a red flag: why is the mixin being declared more general (BaseTestCase < MyMixin), when it actually depends on the setUp() method provided by BaseTestCase or its ancestors (implying MyMixin < BaseTestCase, or MyMixin < unittest.TestCase)?

This points to the more serious underlying bug:
class MyMixin(object):
    def setUp(self):
        super(MyMixin, self).setUp()
As it stands, this code is unconditionally buggy: it declares a dependency on object alone (MyMixin < object), but proceeds to super-call a method (setUp()) that object does not provide.

Fixing this bug requires either removing the super-call (making MyMixin the introducer of a new setUp() method), or inserting a dependency on an appropriate base class (to extend its setUp() method):
class MyMixin(unittest.TestCase, object)  # MyMixin < TestCase < object
class MyMixin(unittest.TestCase)          # equivalent shorthand
The bug can also be papered over by adding the missing dependency in a subclass:
class Foo(MyMixin, unittest.TestCase)  # MyMixin < TestCase
class Bar(MyMixin, BaseTestCase)       # MyMixin < BaseTestCase (< TestCase)
but this merely hides the underlying bug: any subclass that does not declare MyMixin's dependency for it will still hit the problem.
[gravatar]
Regarding the use of super() in TestCase.setUp(), it should never be sensical to attempt a super-call in a base method implementation.

super() only exists to allow subclasses to call and extend methods that already exist in their superclasses: there may be any number of super-calls, but by definition, the super-calls must eventually stop at a base class that first introduces the method, without using any super-calls.

In this example, TestCase is the class that actually introduces setUp() and tearDown(): all its subclasses (and only its subclasses) can use super() to extend those methods, but TestCase itself provides them from scratch.
[gravatar]
Samus: None of this changed in Python 3, except that super() no longer requires any arguments in the default case.
[gravatar]
Thanks all for the thoughtful comments. As usual, half of what I learned was before I wrote the post, and half after. The idea that Mixin must derive from TestCase is an interesting one. On the one hand, I see how that solves problems, ironically by introducing a dreaded diamond to the hierarchy. On the other hand a Mixin is not itself a TestCase, and isn't meant to be used as one. I invoke super.setUp because I know the class is only ever meant to be mixed into an actual TestCase.
[gravatar]
I wanted to thank Ned for introducing the topic and Piet for explaining it so greatfully.

Best wishes.
[gravatar]
Deriving Mixin from (TestCase,object) doesn't introduce a diamond, because new-style classes don't have diamonds: they have linearized MROs.

When Python creates a new style class, it "sorts" the diamonds into a linear ordering, such that the resulting list has the same ordering relationships found in every multiple inheritance declaration in the entire tree.

Thus, by making the mixin derive from (TestCase,object), you are telling Python that TestCase must always come before object and *after* the mixin in every subclass of Mixin. Thus, TestCase will never shadow the mixin's methods.
[gravatar]
It is basically a rule in Python (because of the MRO) that you list mixin classes *first*. So doing it the other way round always looks odd to me. Composition is fine, but it's generally *more work* than inheritance because you have to explicitly delegate rather than have Python do it for you.
[gravatar]
"""when I think of my RealTestCase class, it is mostly a TestCase, with MyMixin mixed in. To me, it's most natural to talk about TestCase first"""

It seems to make sense to me that the last parent is the "most fundamental," and that minor contributions to behavior should precede them, thus modifying the behaviors of the "principal" parent. But that's just me.
[gravatar]
Hey Ned,

If you decide to go down the "association instead of inheritance" route, could you make sure to post the final product? I'm JUST BARELY starting to scratch the surface of that paradigm and seeing a real world example would be tremendously helpful.

Also, thank you so much for all you do for the python community. I can't tell you how much I've appreciated coverage.py alone!
[gravatar]
When I write mixin classes I try to avoid make them change the original workflow of the class.

In this case, I would add the common code in a method, say "prepare_something()", on the mixin class, and call it from TestCase setUp(). I like to think of mixins as away to offer more features to the class, and not change the class original features. This is a different use case than Piet Delport explained above.
class MyMixin(object):
    def prepare_something(self):
        #.. do something here ..

    def restore_something(self):
        #.. do something here ..

class BaseTestCase(unittest.TestCase, MyMixin):
    def setUp(self):
        super(BaseTestCase, self).setUp()
        self.prepare_something()

    def tearDown(self):
	self.restore_something()
        super(BaseTestCase, self).tearDown()

class RealTestCase(BaseTestCase):

    def test_foo(self):
        #.. etc ..
For tests I usually have mixin classes with domain specific asserts, for example.
[gravatar]
Thanks Ned,

I ran into this *exact* issue earlier this year. Having the mixin first does look weird, but it works.

Thanks for publishing this and hopefully saving some others a bit of time.
[gravatar]
Pamela McA'Nulty 2:58 PM on 22 Jan 2016
Thanks Ned,

Perfectly on-point to an issue I had today adding unittest functionality that was used by two Django test cases - one a subclass of TestCase, the other a subclass of TransactionTestCase
[gravatar]
I'm afraid I have to agree with others and especially Piet.
If you call super(), the class by definition can't be a mixin. Calling super implies you already know what that parent function does so you don't replicate the code. This means you must tie it to the parent class to create a true child class. Mixin's shouldn't modify the parent code but should add to or replace it.
I like the "is a" argument though it can be a difficult one to get your head around. Thinking of an animal, is it a dog, or is it an animal with four legs? In class terms you could declare a Dog class two ways: you could inherit from Animal and add all the stuff to make it a Dog, or you could inherit from Animal and add the mixins for Species, Limbs, Surface, etc. Fundamentally your Dog class "is an" Animal, but it has additional attributes covered by the mixins, and those mixins could be used on other classes, like Tree (okay contrived example I know).
The "Mixins come first even though it looks odd" argument doesn't wash. If you commit to using a language, you have to learn how to use it properly and not use it the way you think it should work, or only use the bits you like the look of. Otherwise it's going to be an uphill struggle!
Great article though, I and many others have learned some more.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.