Austin Bingham from Good With Computers
In the previous articles in this series we uncovered a small mystery
regarding how Python's super() works, and we looked at some of the
underlying mechanics of how super() really works. In this article we'll see
how those details work together to resolve the mystery.
The mystery revisited
As you'll recall, the mystery we uncovered has to do with how a single
use of super() seemed to invoke two separate implementations of a
method in our SortedIntList example. As a reminder, our class
diagram looks like this:
SimpleList defines a simplified list API, and IntList and SortedList
each enforce specific constraints on the contents of the list. The class
SortedIntList inherits from both IntList and SortedList and enforces
the constraints of both base classes. Interestingly, though, SortedIntList
has a trivial implementation:
class SortedIntList(IntList, SortedList):
pass
How does SortedIntList do what it does? From the code it seems that
SortedIntList does no coordination between its base classes, and
certainly the base classes don't know anything about each other. Yet
somehow SortedIntList manages to invoke both implementations of
add(), thereby enforcing all of the necessary constraints.
super() with no arguments
We've already looked at method resolution order, C3 linearization, and
the general behavior of super instances. The final bit of
information we need in order to resolve the mystery is how super()
behaves when called with no arguments. Both IntList and
SortedList use super() this way, in both their initializers and
in add().
For instance methods such as these, calling super() with no
arguments is the same as calling super() with the method's class as
the first argument and self as the second. In other words, it
constructs a super proxy that uses the MRO from self starting at
the class implementing the method. Knowing this, it's easy to see how,
in simple cases, using super() is equivalent to "calling the base
class implementation". In these cases, type(self) is the same as the
class implementing the method, so the method is resolved using
everything in that class's MRO except the class itself. The next entry
in the MRO will, of course, be the class's first base class, so simple
uses of super() are equivalent to invoking a method on the first
base class.
A key point to notice here is that type(self) will not always be the
same as the class implementing a specific method. If you invoke
super() in a method on a class with a subclass, then type(self)
may well be that subclass. You can see this in a trivial example:
>>> class Base:
... def f(self):
... print('Type of self in Base.f() =', type(self))
...
>>> class Sub(Base):
... pass
...
>>> Sub().f()
Type of self in Base.f() = <class '__main__.Sub'>
Understanding this point is the final key to seeing how
SortedIntList works. If type(self) in a base class is not
necessarily the type of the class implementing the method, then the MRO
that gets used by super() is not necessarily the MRO of the class
implementing the method...it may be that of the subclass. Since the
entries in type(self).mro() may include entries that are not in the
MRO for the implementing class, calls to super() in a base class may
resolve to implementations that are not in the base class's own MRO. In
other words, Python's method resolution is - as you might have guessed -
extremely dynamic and depends not just on a class's base classes but its
subclasses as well.
Method resolution order to the rescue
With that in mind, let's finally see how all of the elements - MRO,
no-argument super(), and multiple inheritance - coordinate to make
SortedIntList work. As we've just seen, when super() is used
with no arguments in an instance method, the proxy uses the MRO of
self which, of course, will be that of SortedIntList in our
example. So the first thing to look at is the MRO of SortedIntList
itself:
>>> SortedIntList.mro()
[<class 'SortedIntList'>,
<class 'IntList'>,
<class 'SortedList'>,
<class 'SimpleList'>,
<class 'object'>]
A critical point here is that the MRO contains IntList and
SortedList, meaning that both of them can contribute implementations
when super() is used.
Next, let's examine the behavior of SortedIntList when we call its
add() method. Because IntList is the first
class in the MRO which implements add(), a call to add() on a
SortedIntList resolves to IntList.add():
class IntList(SimpleList):
# ...
def add(self, item):
self._validate(item)
super().add(item)
And this is where things get interesting!
In IntList.add() there is a call to super().add(item), and because of
how no-argument super() calls work, this is equivalent to super(IntList,
self).add(item). Since type(self) == SortedIntList, this call to
super() uses the MRO for SortedIntList and not just IntList. As a
result, even though IntList doesn't really "know" anything about
SortedList, it can access SortedList methods via a subclass's MRO.
In the end, the call to super().add(item) in IntList.add() takes the MRO
of SortedIntList, find's IntList in that MRO, and uses everything after
IntList in the MRO to resolve the method invocation. Since that MRO-tail
looks like this:
[<class 'SortedList'>, <class 'SimpleList'>, <class 'object'>]
and since method resolution uses the first class in an MRO that implements a
method, the call resolves to SortedList.add() which, of course, enforces the
sorting constraint.
So by including both of its base classes in its MRO - and because IntList and SortedList use
super() in a cooperative way - SortedIntList ensures that both
the sorting and type constraint are enforced.
No more mystery
We've seen that a subclass can leverage MRO and super() to do some
pretty interesting things. It can create entirely new method resolutions
for its base classes, resolutions that aren't apparent in the base class
definitions and are entirely dependent on runtime context.
Used properly, this can lead to some really powerful designs. Our
SortedIntList example is just one instance of what can be done. At
the same time, if used naively, super() can have some surprising and
unexpected effects, so it pays to think deeply about the consequences of
super() when you use it. For example, if you really do want to
just call a specific base class implementation, you might be better off
calling it directly rather than leaving the resolution open to the whims
of subclass developers. It may be cliche, but it's true: with great
power comes great responsibility.
For more information on this topic, you can always see the Inheritance
and Subtype Polymorphism module of our Python: Beyond the Basics
course on PluralSight.
