Sunday 28 October 2012 — This is 12 years old. Be careful.
Multiple inheritance is hard, we all know it, but even after warning people about it myself, I found myself tripped up by it yesterday. All I wanted was a mixin for my unittest.TestCase class.
Unittest’s TestCases use setUp and tearDown to affect the test state, and I wanted a way to share setUp and tearDown implementations between two different test classes. A mixin seemed like a good solution. I already have a BaseTestCase of my own that inherits from unittest.TestCase, so my mixin looked like this:
class BaseTestCase(unittest.TestCase):
def setUp(self):
super(BaseTestCase, self).setUp()
#.. set up that everyone needs ..
def tearDown(self):
#.. tear down that everyone needs ..
super(BaseTestCase, self).tearDown()
class MyMixin(object):
def setUp(self):
super(MyMixin, self).setUp()
#.. do something here ..
def tearDown(self):
#.. do something here ..
super(MyMixin, self).tearDown()
class RealTestCase(BaseTestCase, MyMixin):
# I didn't need setUp and tearDown here..
def test_foo(self):
#.. etc ..
The theory is that mixins avoid some complications of multiple inheritance: they provide methods, but no attributes, and don’t inherit in a way that produces diamonds in the inheritance hierarchy. And I did the right things here, using super to keep the MRO (method resolution order) intact.
But this code doesn’t work, MyMixin.setUp is never invoked. Why? It’s because TestCase in unittest doesn’t invoke super():
# unittest.py (some details omitted :)
class TestCase(object):
def setUp(self):
pass
The method resolution order of RealTestCase is:
- RealTestCase
- BaseTestCase
- unittest.TestCase
- MyMixin
- object
Since TestCase.setUp doesn’t use super, the sequence stops there, and MyMixin is never consulted. At first I thought, “TestCase.setup should use super!” But, if it did, it would fail in simpler hierarchies that don’t use a mixin, because it would try to invoke object.setUp(), which doesn’t exist.
I suppose TestCase could be re-written like this:
# unittest.py (hypothetical)
class TestCase(object):
def setUp(self):
parent = super(Base, self)
if hasattr(parent, "setUp"):
parent.setUp()
This works, and now MyMixin.setUp() is invoked, but then it fails for the same reason: it tries to invoke object.setUp, which doesn’t exist, so MyMixin also needs to have the defensive check of its parent. Yuk.
The simple solution is to swap the order of the mixin and the base class:
class RealTestCase(MyMixin, BaseTestCase):
#...
With this class declaration, the MRO is: RealTestCase, MyMixin, BaseTestCase, TestCase, object. All the setUp’s are invoked, and it ends cleanly with no complicated parentage check.
But I can’t help feeling this isn’t great. For one thing, conceptually, BaseTestCase is the real parent, and the mixin is just tossed in, so I would rather be able to write my base classes in the other order. But more worrying, this solution also means that I have to be very careful to know the lineage of all my classes all the way up to object.
Maybe this is the fundamental truth about Multiple Inheritance, and why it is so difficult: details about base classes that you thought were abstracted away from you can suddenly be critical to understand. Like all action at a distance, this is mysterious and confusing.
My take-aways from this:
- Mixins come first even though it looks odd.
- Multiple inheritance is hard.
- There are yet more Python details to master.
The question remaining in my mind: would class hierarchies be better if the top-most classes (derived from object) used the defensive super style? Or is that overkill that defers rather than removes the pain? Would something else bite me later?
Comments
Inheritance should model a true, intensional generalization relationship. In your case, the discriminator between various subclasses of TestCase is only extensional, i.e. a minor detail about coincidental setup/teardown implementation.
Remember the heuristic "favor association over inheritance". For example, you could pass a setup/teardown implementor into your various test case classes (something like a Strategy pattern).
I just keep in mind that python searches for methods and attributes left-to-right through the list of inherited classes (obviously a simplification of the MRO algorithm, but it's good enough,) so the most-base class goes on the right. Call super() when overriding methods, and everything tends to Just Work. It's a bit hand-wavey I know, but Python hasn't let me down yet.
I use multiple inheritance usually for mixins (where I find it quite natural to apply them first, as I typically want their methods to take precedence), and more often than not in test code also (funny how it finds itself there quite a bit, as in this blog post). In *extremely rare* cases I do have some cases (well just one I can think of) where a particular subclass is truly an amalgam of two distinct parent hierarchies. I certainly don't do that lightly.
I've never fully understood MRO and how metaclasses affect it, it's supposed to be useful in this cases.
I also wonder if this changes in some way in Python3 with the new super().
In Python (and any C3 class system), every class declaration should be read as a single partial order: class A(B, C) should be understood to mean that A < B < C, without putting special weight on A. (It can help to mentally ignore the parentheses and actually read the syntax as "class A < B < C: ...", when in doubt.)
In other words, the implication that B becomes a subclass of C is just as important and meaningful as the implication that A becomes a subclass of B. In particular, the question of swapping B and C around is no different in general than the question of swapping A and B around, and can have equally large repercussions.
Looking back at the example:This declaration should immediately jump out as a red flag: why is the mixin being declared more general (BaseTestCase < MyMixin), when it actually depends on the setUp() method provided by BaseTestCase or its ancestors (implying MyMixin < BaseTestCase, or MyMixin < unittest.TestCase)?
This points to the more serious underlying bug:
class MyMixin(object): def setUp(self): super(MyMixin, self).setUp()As it stands, this code is unconditionally buggy: it declares a dependency on object alone (MyMixin < object), but proceeds to super-call a method (setUp()) that object does not provide.Fixing this bug requires either removing the super-call (making MyMixin the introducer of a new setUp() method), or inserting a dependency on an appropriate base class (to extend its setUp() method):The bug can also be papered over by adding the missing dependency in a subclass:but this merely hides the underlying bug: any subclass that does not declare MyMixin's dependency for it will still hit the problem.
super() only exists to allow subclasses to call and extend methods that already exist in their superclasses: there may be any number of super-calls, but by definition, the super-calls must eventually stop at a base class that first introduces the method, without using any super-calls.
In this example, TestCase is the class that actually introduces setUp() and tearDown(): all its subclasses (and only its subclasses) can use super() to extend those methods, but TestCase itself provides them from scratch.
Best wishes.
When Python creates a new style class, it "sorts" the diamonds into a linear ordering, such that the resulting list has the same ordering relationships found in every multiple inheritance declaration in the entire tree.
Thus, by making the mixin derive from (TestCase,object), you are telling Python that TestCase must always come before object and *after* the mixin in every subclass of Mixin. Thus, TestCase will never shadow the mixin's methods.
It seems to make sense to me that the last parent is the "most fundamental," and that minor contributions to behavior should precede them, thus modifying the behaviors of the "principal" parent. But that's just me.
If you decide to go down the "association instead of inheritance" route, could you make sure to post the final product? I'm JUST BARELY starting to scratch the surface of that paradigm and seeing a real world example would be tremendously helpful.
Also, thank you so much for all you do for the python community. I can't tell you how much I've appreciated coverage.py alone!
In this case, I would add the common code in a method, say "prepare_something()", on the mixin class, and call it from TestCase setUp(). I like to think of mixins as away to offer more features to the class, and not change the class original features. This is a different use case than Piet Delport explained above.
class MyMixin(object): def prepare_something(self): #.. do something here .. def restore_something(self): #.. do something here .. class BaseTestCase(unittest.TestCase, MyMixin): def setUp(self): super(BaseTestCase, self).setUp() self.prepare_something() def tearDown(self): self.restore_something() super(BaseTestCase, self).tearDown() class RealTestCase(BaseTestCase): def test_foo(self): #.. etc ..For tests I usually have mixin classes with domain specific asserts, for example.I ran into this *exact* issue earlier this year. Having the mixin first does look weird, but it works.
Thanks for publishing this and hopefully saving some others a bit of time.
Perfectly on-point to an issue I had today adding unittest functionality that was used by two Django test cases - one a subclass of TestCase, the other a subclass of TransactionTestCase
If you call super(), the class by definition can't be a mixin. Calling super implies you already know what that parent function does so you don't replicate the code. This means you must tie it to the parent class to create a true child class. Mixin's shouldn't modify the parent code but should add to or replace it.
I like the "is a" argument though it can be a difficult one to get your head around. Thinking of an animal, is it a dog, or is it an animal with four legs? In class terms you could declare a Dog class two ways: you could inherit from Animal and add all the stuff to make it a Dog, or you could inherit from Animal and add the mixins for Species, Limbs, Surface, etc. Fundamentally your Dog class "is an" Animal, but it has additional attributes covered by the mixins, and those mixins could be used on other classes, like Tree (okay contrived example I know).
The "Mixins come first even though it looks odd" argument doesn't wash. If you commit to using a language, you have to learn how to use it properly and not use it the way you think it should work, or only use the bits you like the look of. Otherwise it's going to be an uphill struggle!
Great article though, I and many others have learned some more.
Add a comment: