A More Full-Featured Emacs company-mode Backend

Austin Bingham from Good With Computers

In the first article in this series we looked at how to define the simplest company-mode backend. [1] This backend drew completion candidates from a predefined list of options, and allowed you to do completion in buffers in fundamental mode. The main purpose of that article was to introduce the essential plumbing of a company-mode backend.

In this article we'll expand upon the work of the first, adding some useful UI elements like annotations and metadata. We'll also implement a rough form of fuzzy matching, wherein candidates will be presented to the user when they mostly match the prefix. After this article you'll know almost everything you need to know about writing company-mode backends, and you'll be in a great position to learn the rest on your own.

Most of what we'll be doing in the article revolves around handling completion candidate "metadata", data associated in some way with our completion candidates. In practice this kind of data covers things like documentation strings, function signatures, symbols types, and so forth, but for our purposes we'll simply associate some biographical data with the names in our completion set sample-completions.

company-mode provides affordances for displaying metadata as part of the completion process. For example, if your backend is showing completions for function names, you could display the currently-selected function's signature in the echo area. We'll develop a backend that displays a sentence about the selected candidate in the echo area, and we'll also display their initials as an annotation in the candidate selection popup menu.

Adding more data to our completion candidates

First we need to add some metadata to our existing completion candidates. To do this we'll use Emacs text properties. ((Text properties allow you to associate arbitrary data with strings. You can read about them here. Specifically, we use the special read syntax for text properties.)) For each completion candidate we define an :initials property containing their initials and a :summary property containing a one-sentence summary of the candidate. [2] To add these properties, update sample-completions to look like this:

(defconst sample-completions
  '(#("alan" 0 1
      (:initials
      "AMT"
      :summary
      (concat "Alan Mathison Turing, OBE, FRS (/ˈtjʊərɪŋ/ "
              "tewr-ing; 23 June 1912 – 7 June 1954) was a "
              "British mathematician, logician, cryptanalyst, "
              "philosopher, pioneering computer scientist, "
              "mathematical biologist, and marathon and ultra "
              "distance runner.")))
    #("john" 0 1
      (:initials
      "JVN"
      :summary
      (concat "John von Neumann (/vɒn ˈnɔɪmən/; December 28, "
              "1903 – February 8, 1957) was a Hungarian and "
              "American pure and applied mathematician, physicist, "
              "inventor and polymath.")))
    #("ada" 0 1
      (:initials
      "AAK"
      :summary
      (concat "Augusta Ada King, Countess of Lovelace (10 December "
              "1815 – 27 November 1852), born Augusta Ada Byron "
              "and now commonly known as Ada Lovelace, was an "
              "English mathematician and writer chiefly known for "
              "her work on Charles Babbage's early mechanical "
              "general-purpose computer, the Analytical Engine.")))
    #("don" 0 1
      (:initials
      "DEK"
      :summary
      (concat "Donald Ervin Knuth (/kəˈnuːθ/[1] kə-nooth; born "
              "January 10, 1938) is an American computer "
              "scientist, mathematician, and Professor Emeritus "
              "at Stanford University.")))))

Attaching properties like this is a very convenient way to store metadata for completion candidates. Of course in a real backend you probably wouldn't have a hard-coded list of candidates, and you'd be fetching them dynamically from a server, database, or external process. In that case, you'd need to also dynamically fetch the metadata you want and attach it to the candidate strings you serve through your backend. In the end, text properties work well in this context because they transparently transport the metadata - which company-mode doesn't know about - with the completion strings that company-mode definitely knows about.

Adding completion menu annotations

This change by itself doesn't really do anything, of course. All we've done is add properties to some strings, and we need to instruct company-mode on how to actually use them for display. The first way we'll use this metadata, then, is to add a small annotation to each entry in the popup menu used for candidate selection. To add this annotation, we need to update company-sample-backend to respond to the annotation command. This command should resolve to the annotation you want to use for the given candidate. Typically this means calling a function taking the completion candidate string arg and returning the annotation string.

First let's define a function that takes a completion candidate string and returns an annotation. Remember that our candidate strings store their metadata as text properties, so fundamentally this function simply needs to extract a property. For the annotation, we'll extract the :initials property and return it (prefixed with a blank.) That function looks like this:

(defun sample-annotation (s)
  (format " [%s]" (get-text-property 0 :initials s)))

Next we need to update our backend to respond to the annotation command like this:

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))``

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (string-prefix-p arg c))
      sample-completions))
    (annotation (sample-annotation arg))))

In the last line we tell the backend to call sample-annotation with the candidate string to produce an annotation.

Now when we do completion we see the candidates' initials in the popup menu:

candidates-initials

Displaying metadata in the echo area

Where the annotation command adds a small annotation to the completion popup menu, the meta backend command produces text to display in the echo area. [3] The process for producing the metadata string is almost exactly like that of producing the annotation string. First we write a function that extracts the string from the candidate text properties. Then we wire that function into the backend through the meta command.

As you've probably guessed, the function for extracting the metadata string will simply read the :summary property from a candidate string. It looks like this:

(defun sample-meta (s)
  (get-text-property 0 :summary s))

The changes to the backend look like this:

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (string-prefix-p arg c))
      sample-completions))
    (annotation (sample-annotation arg))
    (meta (sample-meta arg))))

As before, in the last line we associate the meta command with our sample-meta function.

Here's how the metadata looks when displayed in the echo area:

Screen Shot 2014-11-03 at 12.02.10 PM

Fuzzy matching

As a final improvement to our backend, let's add support for fuzzy matching. This will let us do completion on prefixes which don't exactly match a candidate, but which are close enough. [4] For our purposes we'll implement a very crude form of fuzzy matching wherein a prefix matches a candidate if the set of letters in the prefix is a subset of the set of letters in the candidate. The function for performing fuzzy matching looks like this:

(defun sample-fuzzy-match (prefix candidate )
  (cl-subsetp (string-to-list prefix)
              (string-to-list candidate)))

Now we just need to modify our backend a bit. First we need to modify our response to the candidates command to use our new fuzzy matcher. Then we need to respond to the no-cache command by returning true. [5] Here's how that looks:

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (sample-fuzzy-match arg c))
      sample-completions))
    (annotation (sample-annotation arg))
    (meta (sample-meta arg))
    (no-cache 't)))

As you can see, we've replaced string-prefix-p in the candidates response with sample-fuzzy-match, and we've added (no-cache 't).

Here's how our fuzzy matching looks in action:

Screen Shot 2014-11-03 at 12.23.45 PM

That's all, folks

We've seen how to use Emacs' text properties to attach metadata to candidate strings. This is a really useful technique to use when developing company-mode backends, and one that you'll see used in real-world backends. With that metadata in place, we've also seen that it's very straightforward to tell your backend to display annotations in popup menus and metadata in the echo area. Once you've got the basic techniques under your belt, you can display anything you want as part of completion.

There are still more aspects to developing company-mode backends, but with what we've covered in this series you can get very far. More importantly, you know the main concepts and infrastructure for the backends, so you can learn the rest on your own. If you want to delve into all of the gory details, you'll need to read the company-mode source code, and specifically the documentation for company-backends. [6]

For an example of a fairly full-featured backend implementation that's currently in use (and under active development), you can see the emacs-ycmd project. [7] Happy hacking!

[1]The first article in this series..
[2]The summaries are simply the first sentences of the respective Wikipedia articles.
[3]The Emacs manual entry on the echo area.
[4]Fuzzy matching is commonly used for completion tools because it addresses common cases where users transpose characters, accidentally leave characters out, or consciously leverage fuzzy matching for increased speed.
[5]The details of why this is the case are murky, but the company-mode source code specifically states this. The same source also says that we technically should be implementing a response to match, but that doesn't seem to affect this implementation.
[6]The company-mode project page.
[7]The emacs-ycmd project is on github. In particular, see company-ycmd.el.

Writing the Simplest Emacs company-mode Backend

Austin Bingham from Good With Computers

In Emacs, company-mode (short for "complete anything") is a framework for performing completion in buffers. [1] It's an alternative to the popular auto-complete-mode. company-mode supports extension via backends which provide the framework with lists of possible completions in various contexts. So, for example, there's a backend th(at provides completion support for Emacs lisp and one that does the same for Python. Backends can use very different technologies as long as they conform to the backend interface specified by the mode.

I recently decided to write a company-mode backend for ycmd, a completion server for languages including C/C++/Objective-C and Python. [2] All in all it was a relatively pain-free experience, but the process isn't as well documented as I would have liked. So I want to use this series to describe how it's done with the hope of making it easier for others and of helping me remember how to do it in the future.

I won't be covering all of the details of company-mode backends (partially because I don't know them all), but this series should tell you what you need to know to create your own fully-armed and operational backend. [3] In this article we'll define the simplest possible backend in order to familiarize you with the concepts and infrastructure involved. In the next article we'll add some sophistication to that backend to improve the user experience.

The simplest possible backend

For our example we need to define a source of completion candidates. Ultimately, any completion source is just a sequence of strings that meet some criteria. Examples might include:

  • A list of English words starting with some prefix
  • Methods for a particular object in Java
  • Modules available for import in Python program

company-mode doesn't care about the nature of these strings. It just takes them and makes it easy for the user to select from the available options.

In this case, we'll just define a fixed list of strings:

(defconst sample-completions
  '("alan" "john" "ada" "don"))

That's it. [4] Completion sources don't need to (though they generally will) be more complex than that.

Defining the backend

Backends take the form of a function which takes a command as its first argument. This command can take any of a number of values, and backends are required to respond to a handful of them. Before we get into those details, let's look at our very basic backend implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 (require 'cl-lib)
 (require 'company)

 (defun company-sample-backend (command &optional arg &rest ignored)
   (interactive (list 'interactive))

   (cl-case command
     (interactive (company-begin-backend 'company-sample-backend))
     (prefix (and (eq major-mode 'fundamental-mode)
                 (company-grab-symbol)))
     (candidates
     (cl-remove-if-not
       (lambda (c) (string-prefix-p arg c))
       sample-completions))))

The signature of this function is mandated by company-mode. Line 5 makes the function interactive so that you can easily drive your backend without invoking company-mode, something we'll do in a bit. The cl-case statement on line 7 is where we decide what to do based on command. In this case, we respond to interactive, prefix, and candidates.

The interactive command is passed once, before the other commands are used, and it is used to initialize the company-mode infrastructure. All you need to do as a backend developer is pass your backend to company-begin-backend as in this example.

The prefix command

The prefix command is probably the most complex command to handle. This command should return the text that is to be completed. Determining this text can be complex depending on what you're trying to complete, but company-grab-symbol often does "the right thing" if your completion context is space-delimited.

If the prefix command returns nil, this tells company-mode that the backend is not suitable for doing completion on this context. On line 9 of our example we check to see if we're in fundamental-mode and, if not, return nil. In other words, we're saying here that our backend only applies to fundamental-mode. Programming language-oriented backends can make a similar check for their specific modes. When a backend responds to prefix with nil, other backends are given a chance to do the completion.

On the other hand, if a backend is appropriate for the current completion but it can't provide any completions for some reason, the backend should return 'stop. This tells company-mode that no other backends should be used for this completion.

So our backend is effectively saying that it can do completion for anything in fundamental mode. There are more details to prefix, but that's covers the important parts.

The candidates commands

The response to the candidates command is where you actually generate a list of possible completions at a point in a buffer. When this command is passed in, the arg argument holds the value returned by prefix. In other words, you construct your candidates based on the text that you previously indicated was to be completed.

In our case, the prefix we indicated was whatever came before point in the buffer. To calculate our possible completions, we filter the sample-completions values with that prefix using remove-if-not, returning only those candidates which begin with the prefix.

As with prefix calculations, real candidate calculations can be much more complex. But if you understand how the data is piped around, then constructing these complex candidate lists should be fairly straightforward.

Test-driving the backend

To test out our backend, first enter all of the code into a buffer and evaluate it (e.g. with M-x eval-buffer.) Then create a new buffer and run M-x fundamental-mode and M-x company-mode. [5]

In this new buffer enter the single character "a" and then, with the cursor immediately after the "a", run M-x company-sample-backend. This should give you completion options something like this:

Screen Shot 2014-08-28 at 7.17.11 PM

If that works correctly, then you've done almost everything you need to for a fully working backend.

Plugging the backend into company-mode

The final thing you need to do to make your backend available to company-mode is to add it the list company-backends. One simple way to do that is with add-to-list list this:

(add-to-list 'company-backends 'company-sample-backend)

Once you've done this, you can use the command company-complete to do completions, and your new backend will be used in concert with all of the other backends in that list. Generally speaking, company-complete is the command you'll use for completion with company-mode, and it'll often be bound to a simple keystroke.

A complete company-mode backend

That's all there is to writing a basic company-mode backend. In the next article in this series we'll look at adding a few more details to what we have already.

Here's a complete listing of the code used in this article:

(require 'company)

(defconst sample-completions
  '("alan" "john" "ada" "don"))

(defun company-sample-backend (command &optional arg &rest ignored)
  (interactive (list 'interactive))

  (case command
    (interactive (company-begin-backend 'company-sample-backend))
    (prefix (and (eq major-mode 'fundamental-mode)
                (company-grab-symbol)))
    (candidates
    (remove-if-not
      (lambda (c) (string-prefix-p arg c))
      sample-completions))))

(add-to-list 'company-backends 'company-sample-backend)
[1]company-mode project site
[2]The *ycmd* github repository and my Emacs client.
[3]Sorry, I couldn't resist the Star Wars reference.
[4]We'll filter the strings later based on context.
[5]This puts your buffer in major mode "fundamental" and minor mode "company".

How to write a company-mode backends

Austin Bingham from Good With Computers

In Emacs, company-mode (short for "complete anything") is a framework for performing completion in buffers. It's an alternative to the popular auto-complete-mode. company-mode supports extension via backends which provide the framework with lists of possible completions in various contexts. So, for example, there's a backend that provides completion support for Emacs lisp and one that does the same for Python. Backends can use very different technologies as long as they conform to the backend interface specified by the mode.

The super() Mystery Resolved

Austin Bingham from Good With Computers

In the previous articles in this series [1] we uncovered a small mystery regarding how Python's super() works, and we looked at some of the underlying mechanics of how super() really works. In this article we'll see how those details work together to resolve the mystery.

The mystery revisited

As you'll recall, the mystery we uncovered has to do with how a single use of super() seemed to invoke two separate implementations of a method in our SortedIntList example. As a reminder, our class diagram looks like this:

Inheritance graph

SimpleList defines a simplified list API, and IntList and SortedList each enforce specific constraints on the contents of the list. The class SortedIntList inherits from both IntList and SortedList and enforces the constraints of both base classes. Interestingly, though, SortedIntList has a trivial implementation:

class SortedIntList(IntList, SortedList):
    pass

How does SortedIntList do what it does? From the code it seems that SortedIntList does no coordination between its base classes, and certainly the base classes don't know anything about each other. Yet somehow SortedIntList manages to invoke both implementations of add(), thereby enforcing all of the necessary constraints.

super() with no arguments

We've already looked at method resolution order, C3 linearization, and the general behavior of super instances. The final bit of information we need in order to resolve the mystery is how super() behaves when called with no arguments. [2] Both IntList and SortedList use super() this way, in both their initializers and in add().

For instance methods such as these, calling super() with no arguments is the same as calling super() with the method's class as the first argument and self as the second. In other words, it constructs a super proxy that uses the MRO from self starting at the class implementing the method. Knowing this, it's easy to see how, in simple cases, using super() is equivalent to "calling the base class implementation". In these cases, type(self) is the same as the class implementing the method, so the method is resolved using everything in that class's MRO except the class itself. The next entry in the MRO will, of course, be the class's first base class, so simple uses of super() are equivalent to invoking a method on the first base class.

A key point to notice here is that type(self) will not always be the same as the class implementing a specific method. If you invoke super() in a method on a class with a subclass, then type(self) may well be that subclass. You can see this in a trivial example:

>>> class Base:
...     def f(self):
...         print('Type of self in Base.f() =', type(self))
...
>>> class Sub(Base):
...     pass
...
>>> Sub().f()
Type of self in Base.f() = <class '__main__.Sub'>

Understanding this point is the final key to seeing how SortedIntList works. If type(self) in a base class is not necessarily the type of the class implementing the method, then the MRO that gets used by super() is not necessarily the MRO of the class implementing the method...it may be that of the subclass. Since the entries in type(self).mro() may include entries that are not in the MRO for the implementing class, calls to super() in a base class may resolve to implementations that are not in the base class's own MRO. In other words, Python's method resolution is - as you might have guessed - extremely dynamic and depends not just on a class's base classes but its subclasses as well.

Method resolution order to the rescue

With that in mind, let's finally see how all of the elements - MRO, no-argument super(), and multiple inheritance - coordinate to make SortedIntList work. As we've just seen, when super() is used with no arguments in an instance method, the proxy uses the MRO of self which, of course, will be that of SortedIntList in our example. So the first thing to look at is the MRO of SortedIntList itself:

>>> SortedIntList.mro()
[<class 'SortedIntList'>,
<class 'IntList'>,
<class 'SortedList'>,
<class 'SimpleList'>,
<class 'object'>]

A critical point here is that the MRO contains IntList and SortedList, meaning that both of them can contribute implementations when super() is used.

Next, let's examine the behavior of SortedIntList when we call its add() method. [3] Because IntList is the first class in the MRO which implements add(), a call to add() on a SortedIntList resolves to IntList.add():

class IntList(SimpleList):
    # ...

    def add(self, item):
        self._validate(item)
        super().add(item)

And this is where things get interesting!

In IntList.add() there is a call to super().add(item), and because of how no-argument super() calls work, this is equivalent to super(IntList, self).add(item). Since type(self) == SortedIntList, this call to super() uses the MRO for SortedIntList and not just IntList. As a result, even though IntList doesn't really "know" anything about SortedList, it can access SortedList methods via a subclass's MRO.

In the end, the call to super().add(item) in IntList.add() takes the MRO of SortedIntList, find's IntList in that MRO, and uses everything after IntList in the MRO to resolve the method invocation. Since that MRO-tail looks like this:

[<class 'SortedList'>, <class 'SimpleList'>, <class 'object'>]

and since method resolution uses the first class in an MRO that implements a method, the call resolves to SortedList.add() which, of course, enforces the sorting constraint.

So by including both of its base classes in its MRO [4] - and because IntList and SortedList use super() in a cooperative way - SortedIntList ensures that both the sorting and type constraint are enforced.

No more mystery

We've seen that a subclass can leverage MRO and super() to do some pretty interesting things. It can create entirely new method resolutions for its base classes, resolutions that aren't apparent in the base class definitions and are entirely dependent on runtime context.

Used properly, this can lead to some really powerful designs. Our SortedIntList example is just one instance of what can be done. At the same time, if used naively, super() can have some surprising and unexpected effects, so it pays to think deeply about the consequences of super() when you use it. For example, if you really do want to just call a specific base class implementation, you might be better off calling it directly rather than leaving the resolution open to the whims of subclass developers. It may be cliche, but it's true: with great power comes great responsibility.

For more information on this topic, you can always see the Inheritance and Subtype Polymorphism module of our Python: Beyond the Basics course on PluralSight. [5]

[1]Part 1 and Part 2
[2]Note that calling super() with no arguments is only supported in Python 3. Python 2 users will need to use the longer explicit form.
[3]The same logic applies to it's __init__() which also involves calls to super().
[4]Thanks, of course, to how C3 works.
[5]Python: Beyond the Basics on PluralSight

Method Resolution Order, C3, and Super Proxies

Austin Bingham from Good With Computers

In the previous article in this series we looked at a seemingly simple class graph with some surprising behavior. The central mystery was how a class with two bases can seem to invoke two different method implementations with just a single invocation of super(). In order to understand how that works, we need to delve into the details of how super() works, and this involves understanding some design details of the Python language itself.

Method Resolution Order

The first detail we need to understand is the notion of method resolution order or simply MRO. Put simply a method resolution order is the ordering of an inheritance graph for the purposes of deciding which implementation to use when a method is invoked on an object. Let's look at that definition a bit more closely.

First, we said that an MRO is an "ordering of an inheritance graph". Consider a simple diamond class structure like this:

>>> class A: pass
...
>>> class B(A): pass
...
>>> class C(A): pass
...
>>> class D(B, C): pass
...

The MRO for these classes could be, in principle, any ordering of the classes A, B, C, and D (and object, the ultimate base class of all classes in Python.) Python, of course, doesn't just pick the order randomly, and we'll cover how it picks the order in a later section. For now, let's examine the MROs for our classes using the mro() class method:

>>> A.mro()
[<class '__main__.A'>,
<class 'object'>]
>>> B.mro()
[<class '__main__.B'>,
<class '__main__.A'>,
<class 'object'>]
>>> C.mro()
[<class '__main__.C'>,
<class '__main__.A'>,
<class 'object'>]
>>> D.mro()
[<class '__main__.D'>,
<class '__main__.B'>,
<class '__main__.C'>,
<class '__main__.A'>,
<class 'object'>]

We can see that all of our classes have an MRO. But what is it used for? The second half of our definition said "for the purposes of deciding which implementation to use when a method is invoked on an object". What this means is that Python looks at a class's MRO when a method is invoked on an instance of that class. Starting at the head of the MRO, Python examines each class in order looking for the first one which implements the invoked method. That implementation is the one that gets used.

For example, let's augment our simple example with a method implemented in multiple locations:

>>> class A:
...     def foo(self):
...         print('A.foo')
...
>>> class B(A):
...     def foo(self):
...         print('B.foo')
...
>>> class C(A):
...     def foo(self):
...         print('C.foo')
...
>>> class D(B, C):
...     pass
...

What will happen if we invoke foo() on an instance of D? Remember that the MRO of D was [D, B, C, A, object]. Since the first class in that sequence to support foo() is B, we would expect to see "B.foo" printed, and indeed that is exactly what happens:

>>> D().foo()
B.foo

What if remove the implementation in B? We would expect to see "C.foo", which again is what happens:

>>> class A:
...     def foo(self):
...         print('A.foo')
...
>>> class B(A):
...     pass
...
>>> class C(A):
...     def foo(self):
...         print('C.foo')
...
>>> class D(B, C):
...     pass
...
>>> D().foo()
C.foo

To reiterate, method resolution order is nothing more than some ordering of the inheritance graph that Python uses to find method implementations. It's a relatively simple concept, but it's one that many developers understand only intuitively and partially. But how does Python calculate an MRO? We hinted earlier – and you probably suspected – that it's not just any random ordering, and in the next section we'll look at precisely how Python does this.

C3 superclass linearization

The short answer to the question of how Python determines MRO is "C3 superclass linearization", or simply C3. C3 is an algorithm initially developed for the Dylan programming language [1], and it has since been adopted by several prominent programming languages including Perl, Parrot, and of course Python. [2] We won't go into great detail on how C3 works, though there is plenty of information on the web that can tell you everything you need to know. [3]

What's important to know about C3 is that it guarantees three important features:

  1. Subclasses appear before base classes
  2. Base class declaration order is preserved
  3. For all classes in an inheritance graph, the relative orderings guaranteed by 1 and 2 are preserved at all points in the graph.

In other words, by rule 1, you will never see an MRO where a class is preceded by one of its base classes. If you have this:

>>> class Foo(Fred, Jim, Shiela):
...     pass
...

you will never see an MRO where Foo comes after Fred, Jim, or Shiela. This, again, is because Fred, Jim, and Shiela are all base classes of Foo, and C3 puts base classes after subclasses.

Likewise, by rule 2, you will never see an MRO where the base classes specified to the class keyword are in a different relative order than that definition. Given the same code above, this means that you will never see and MRO with Fred after either Jim or Shiela. Nor will you see an MRO with Jim after Shiela. This is because the base class declaration order is preserved by C3.

The third constraint guaranteed by C3 simply means that the relative orderings determined by one class in an inheritance graph – i.e. the ordering constraints based on one class's base class declarations – will not be violated in any MRO for any class in that graph.

C3 limits your inheritance options

One interesting side-effect of the use of C3 is that not all inheritance graphs are legal. It's possible to construct inheritance graphs which make it impossible to meet all of the constraints of C3. When this happens, Python raises an exception and prevents the creation of the invalid class:

>>> class A:
...     pass
...
>>> class B(A):
...     pass
...
>>> class C(A):
...     pass
...
>>> class D(B, A, C):
...     pass
...
Traceback (most recent call last):
  File "<input>", line 1, in <module>
TypeError: Cannot create a consistent method resolution
order (MRO) for bases A, C

In this case, we've asked for D to inherit from B, A, and C, in that order. Unfortunately, C3 wants to enforce two incompatible constraints in this case:

  1. It wants to put C before A because A is a base class of C
  2. It wants to put A before C because of D's base class ordering

Since these are obviously mutually exclusive states, C3 rejects the inheritance graph and Python raises a TypeError.

That's about it, really. These rules provide a consistent, predictable basis for calculating MROs. Understanding C3, or even just knowing that it exists, is perhaps not important for day-to-day Python development, but it's an interesting tidbit for those interested in the details of language design.

Super proxies

The third detail we need to understand in order to resolve our mystery is the notion of a "super proxy". When you invoke super() in Python [4], what actually happens is that you construct an object of type super. In other words, super is a class, not a keyword or some other construct. You can see this in a REPL:

>>> s = super(C)
>>> type(s)
<class 'super'>
>>> dir(s)
['__class__', '__delattr__', '__dir__',
'__doc__', '__eq__', '__format__', '__ge__',
'__get__', '__getattribute__', '__gt__',
'__hash__', '__init__', '__le__', '__lt__',
'__ne__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__self__',
'__self_class__', '__setattr__', '__sizeof__',
'__str__', '__subclasshook__', '__thisclass__']

In most cases, super objects have two important pieces of information: an MRO and a class in that MRO. [5] When you use an invocation like this:

super(a_type, obj)

then the MRO is that of the type of obj, and the class within that MRO is a_type. [6]

Likewise, when you use an invocation like this:

super(type1, type2)

the MRO is that of type2 and the class within that MRO is type1. [7]

Given that, what exactly does super() do? It's hard to put it in a succinct, pithy, or memorable form, but here's my best attempt so far. Given a method resolution order and a class C in that MRO, super() gives you an object which resolves methods using only the part of the MRO which comes after C.

In other words, rather than resolving method invocation using the full MRO like normal, super uses only the tail of an MRO. In all other ways, though, method resolution is occurring exactly as it normally would.

For example, suppose I have an MRO like this:

[A, B, C, D, E, object]

and further suppose that I have a super object using this MRO and the class C in this MRO. In that case, the super instance would only resolve to methods implemented in D, E, or object (in that order.) In other words, a call like this:

super(C, A).foo()

would only resolve to an implementation of foo() in D or E. [8]

Why the name super-proxy

You might wonder why we've been using the name super-proxy when discussing super instances. The reason is that instances of super are designed to respond to any method name, resolving the actual method implementation based on their MRO and class configuration. In this way, super instances act as proxies for all objects. They simply pass arguments through to an underlying implementation.

The mystery is almost resolved!

We now know everything we need to know to resolve the mystery described in the first article in this series. You can (and probably should) see if you can figure it out for yourself at this point. By applying the concepts we discussed in this article - method resolution order, C3, and super proxies - you should be able to see how SortedIntList is able to enforce the constraints of IntList and SortedList even though it only makes a single call to super().

If you'd rather wait, though, the third article in this series will lay it all out for you. Stay tuned!

[1]The Dylan programming language
[2]Presumably starting with the letter "P" is not actually a requirement for using C3 in a language.
[3]Python's introduction of C3 in version 2.3 includes a great description. You can also track down the original research describing C3.
[4]With zero or more arguments. I'm using zero here to keep things simple.
[5]In the case of an unbound super object you don't have the MRO, but that's a detail we can ignore for this article.
[6]Remember that this form of super() requires that isinstance(obj, a_type) be true.
[7]Remember that this form of super() requires that issubclass(type2, type1) be true.
[8]Since object does not (yet...though I'm working on a PEP) implement foo().

Python’s super(): Not as Simple as You Thought

Austin Bingham from Good With Computers

Python's super() is one of those aspects of the language that many developers use without really understanding what it does or how it works. [1] To many people, super() is simply how you access your base-class's implementation of a method. And while this is true, it's far from the full story.

In this series I want to look at the mechanics and some of the theory behind super(). I want to show that, far from just letting you access your base-class, Python's super() is the key to some interesting and elegant design options that promote composability, separation of concerns, and long-term maintainability. In this first article I'll introduce a small set of classes, and in these classes we'll find a bit of a mystery. In subsequent articles we'll investigate this mystery by seeing how Python's super() really works, looking at topics like method resolution order, the C3 algorithm, and proxy objects.

In the end, you'll find that super() is both more complex than you probably expected, yet also surprisingly elegant and easy-to-understand. This improved understanding of super() will help you understand and appreciate Python on at a deeper level, and it will give you new tools for developing your own Python code.

A note on Python versions

This series is written using Python 3, and some of the examples and concepts don't apply completely to Python 2. In particular, this series assumes that classes are "new-style" classes. ((For a discussion of the difference between "old-style" and "new-style" classes, see the Python wiki.)) In Python 3 all classes are new-style, while in Python 2 you have to explicitly inherit from object to be a new-style class.

For example, where we might use the following Python 3 code in this series:

class IntList:
    . . .

the equivalent Python 2 code would be:

class IntList(object):
    . . .

Also, throughout this series we'll call super() with no arguments. This is only supported in Python 3, but it has an equivalent form in Python 2. In general, when you see a method like this:

class IntList:
    def add(self, x):
        super().add(x)

the equivalent Python 2 code has to pass the class name and self to super(), like this:

class IntList:
    def add(self, x):
        super(IntList, self).add(x)

If any other Python2/3 differences occur in the series, I'll be sure to point them out. [2]

The Mystery of the SortedIntList

To begin, we're going to define a small family of classes that implement some constrained list-like functionality. These classes form a diamond inheritance graph like this:

Inheritance graph

At the root of these classes is SimpleList:

class SimpleList:
    def __init__(self, items):
        self._items = list(items)

    def add(self, item):
        self._items.append(item)

    def __getitem__(self, index):
        return self._items[index]

    def sort(self):
        self._items.sort()

    def __len__(self):
        return len(self._items)

    def __repr__(self):
        return "{}({!r})".format(
            self.__class__.__name__,
            self._items)

SimpleList uses a standard list internally, and it provides a smaller, more limited API for interacting with the list data. This may not be a very realistic class from a practical point of view, but, as you'll see, it let's us explore some interesting aspects of inheritance relationships in Python.

Next let's create a subclass of SimpleList which keeps the list contents sorted. We'll call this class SortedList:

class SortedList(SimpleList):
    def __init__(self, items=()):
        super().__init__(items)
        self.sort()

    def add(self, item):
        super().add(item)
        self.sort()

The initializer calls SimpleList's initializer and then immediately uses SimpleList.sort() to sort the contents. SortedList also overrides the add method on SimpleList to ensure that the list always remains sorted.

In SortedList we already see a call to super(), and the intention of this code is pretty clear. In SortedList.add(), for example, super() is used to call SimpleList.add() - that is, deferring to the base-class - before sorting the list contents. There's nothing mysterious going on...yet.

Next let's define IntList, a SimpleList subclass which only allows integer elements. This list subclass prevents the insertion of non-integer elements, and it does so by using the isinstance() function:

class IntList(SimpleList):
    def __init__(self, items=()):
        for item in items: self._validate(item)
        super().__init__(items)

    @classmethod
    def _validate(cls, item):
        if not isinstance(item, int):
            raise TypeError(
                '{} only supports integer values.'.format(
                    cls.__name__))

    def add(self, item):
        self._validate(item)
        super().add(item)

You'll immediately notice that IntList is structurally similar to SortedList. It provides its own initializer and, like SortedList, overrides the add() method to perform extra checks. In this case, IntList calls its _validate() method on every item that goes into the list. _validate() uses isinstance() to check the type of the candidates, and if a candidate is not an instance of int, _validate() raises a TypeError.

Chances are, neither SortedList nor IntList are particularly surprising. They both use super() for the job that most people find most natural: calling base-class implementations. With that in mind, then, let's introduce one more class, SortedIntList, which inherits from both SortedList and IntList, and which enforces both constraints at once:

class SortedIntList(IntList, SortedList):
    pass

It doesn't look like much, does it? We've simply defined a new class and given it two base classes. In fact, we haven't added any new implementation code at all. But if we go to the REPL we can see that it works as we expect. The initializer sorts the input sequence:

>>> sil = SortedIntList([42, 23, 2])
>>> sil
SortedIntList([2, 23, 42])

but rejects non-integer values:

>>> SortedIntList([3, 2, '1'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sorted_int_list.py", line 43, in __init__
    for x in items: self._validate(x)
  File "sorted_int_list.py", line 49, in _validate
    raise TypeError(
                '{} only supports integer values.'.format(
                    cls.__name__))
TypeError: SortedIntList only supports integer values.

Likewise, add() maintains both the sorting and type constraints defined by the base classes:

>>> sil.add(-1234)
>>> sil
SortedIntList([-1234, 2, 23, 42])
>>> sil.add('the smallest uninteresting number')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sorted_int_list.py", line 45, in add
    self._validate(item)
  File "sorted_int_list.py", line 42, in _validate
    raise TypeError(
                '{} only supports integer values.'.format(
                    cls.__name__))
TypeError: SortedIntList only supports integer values.

You should spend some time playing with SortedIntList to convince yourself that it works as expected. You can get the code here.

It may not be immediately apparent how all of this works, though. After all, both IntList and SortedList define add(). How does Python know which add() to call? More importantly, since both the sorting and type constraints are being enforced by SortedIntList, how does Python seem to know to call both of them? This is the mystery that this series will unravel, and the answers to these questions have to do with the method resolution order we mentioned earlier, along with the details of how super() really works. So stay tuned!

[1]If you do know how it works, then congratulations! This series probably isn't for you. But believe me, lots of people don't.
[2]For a detailed look at the differences between the language versions, see Guido's list of differences.

Python’s super() explained

Austin Bingham from Good With Computers

You probably already know how to use Python's super() to call base-class implementations of methods. But do you really know what it's doing? The details of super() are elegant, interesting, and powerful, and while super() is probably more complex than you expect, it's also surprisingly easy to understand. In this series we'll explore super() by first uncovering a bit of a mystery. To resolve the mystery, we'll look a bit under Python's hood to see how super() really works.

The Primacy of Testability: Modularity

Austin Bingham from Good With Computers

In the first post in this series I set the stage for a discussion of how testability can serve as a proxy or enabler for other, more directly desirable qualities in a software system. In this post I'd like to look at the first such quality, modularity.

Modularity is perhaps the most obvious quality to correlate with testability. Modularity is a bit like motherhood and apple pie, and we generally recognize it as a "good thing." Most good developers instinctively foster modularity and reject non-modular designs, both because they've been taught that modularity is good and because they've learned through experience that it's important for many, many reasons.

What do we mean by modularity? A simple definition of modularity is "breaking designs into pieces", but that obviously misses important points. We don't just pull a system into arbitrary pieces, but instead we create modules with specific qualities in order to support and foster the goals of whatever it is we're building. A typical (and important) goal with modules is to have high internal cohesion and low external coupling. A highly cohesive module is one in which all of the module elements are related, work together, and are necessary for performing the work that the module does. A module with low coupling is one that doesn't rely unnecessarily or too extensively on other modules.

There are plenty of ways to ways to discuss and even measure modularity in software designs [1], but for our purposes the basic concepts of cohesion and coupling suffice. So how does modular code - code structured as modules with high internal cohesion and low external coupling - correlate with testability? The short answer is that modular code is generally very testable, so let's see how and why.

One way to see the relationship between testability and modularity is to recognize that modular code, by virtue of its low coupling, is easier to instantiate and execute in isolation. The fewer things (classes, services, whatever) that a module needs, the less work it is to write tests for it. That is, you simply have to type less for each test. Likewise, tests for modules with fewer external dependencies often execute more quickly - at least marginally so - because there's less stuff to do for each test. The result is that developers can run the tests more often and will be more inclined to do so.

If you use mocks in your testing system, low coupling means that you generally have to design fewer mock objects (because your modules have fewer dependencies that need mocking). Moreover, your mocks will often be simpler because they won't have to simulate interactions between far-flung dependencies.

The relationship between modularity and testability goes even deeper, though. Beyond the somewhat mechanical questions of instantiation times, test length, and so forth, there are serious cognitive problems associated with testing poorly-modularized code. The first of these problems is that your tests can end up testing the wrong thing. If a test for some bit of functionality has to invoke code in other modules, the success or failure of the test depends not just on the code under test but on all of those other modules. On some level this is unavoidable [2], but unnecessary dependencies unnecessarily increase the risk that the results you're seeing for your tests are caused by faults in other modules. These kinds of external failures take time to debug, can distort tests as developers try to work around them, or - in the case of incorrectly passing tests - can mask actual defects. [3] Ideally, the output from your tests should give you precise and unambiguous information about just the element under test, and reducing coupling helps developers approach that ideal.

A second way that poorly modularized code increases cognitive load for test developers is that it increases the state space with which a test developer needs to contend. For every external element that plays a role in a module's functionality, the state space that needs to be considered for testing grows to include the potential states of that element. For example, if a test relies on the some global state managed by an external module - perhaps something like an application object managed by the GUI library - then test developers need to ensure that this external object is in the correct state when the test is executed. This means that no matter what other tests have been run, what order they've been run in, or their results, the test developer needs to ensure that this external state is acceptable. As external test dependencies grow, this management of external state can become a significant or even dominant aspect of test development. In other words, the more unnecessary dependencies a module has, the more time a developer will need to spend thinking about those dependencies rather than focusing on testing the functionality at hand. Babysitting unnecessary dependencies doesn't generate information or value. [4]

So we can see that poorly modularized code is generally harder to test, and that's good to know. But the goal of this series is to explore how testability reflects other qualities, not the other way around. How, then, can the testability of our code inform us about its modularity?

The important point is this: if you find that your code is easy to test, then that's a good indication that your code is modular! If you sit down to write some tests and you find that your fixtures are easy to write, the extent of the tests is easy to define, and you don't find that you're constantly needing to consider distant side-effects, then you're probably testing modular code. To carry on with the barometer metaphor, you've got high-pressure, and you can see blue skies and little white puffy clouds. It's time for a picnic.

On the other hand, if your testing efforts involve lots of ceremony - instantiating objects, wiring them together, and bootstrapping subsystems - or if you feel like your really just shotgunning tests at the code rather than pinpointing the "obvious" testing points, then chances are that you've got some modularity issues.

Like I said, modularity is perhaps the most obvious quality that we can associate with testability. And the way to leverage testability in this case is to pay attention to what your testing efforts tell you. There's a lot of information in the work involved in developing tests, and drawing insight from this work is a valuable doubling-up of that effort.

A corollary to this insight is that, when you find yourself having difficulty writing some tests, step back and ask yourself if these problems stem from poor modularity. Testing can be a bit repetitive, though often with minor tweaks, and this is a wonderful environment in which to find repeated bad patterns. And sometime consideration of these patterns will show where you might have an opportunity for better modularization of your code.

[1]See for example some of Martin Fowler's thoughts on the topic, discussions about the interaction between type-systems and modularity, and even some information-theoretic approaches.
[2]Operating systems and standard libraries are "other modules" by most measures.
[3]Consider a test for a specific exception. If some dependency is throwing that exception and the function under test is, in fact, not throwing the exception, then the test is passing erroneously.
[4]And as Gerald Weinberg says, tests are about producing information.

Series: The Primacy of Testability

Austin Bingham from Good With Computers

In this series we look at how software architects - or really anyone involved in creating software - can use testability to help manage other quality attributes. From modularity to performance to the SOLID principles, testability can act as a proxy and an enabler for many of the cross-cutting, interacting concerns that architects need to shepherd. Over several articles we'll explore testability's relationship to these qualities, and we'll see how paying attention to testability can help simplify the job of managing them.

The Primacy of Testability

Austin Bingham from Good With Computers

The job of a software architect [1] is difficult, just like almost every role in software development. They have to keep track of many subtly interacting quality attributes, often on multiple projects, any one of which may be too big or evolving too quickly to meaningfully keep in mental cache. To make matters worse, architects don't have near the level of tool support - compilers, static analysis tools, auto-completion - available to developers. They are much more reliant on experience, awareness, intuition, and heuristics.

In light of this, it's interesting and useful to consider what tools are available to help architects. In particular, I want to look at the role of testability in the architect's job, and to try to show how it can serve as a meaningful proxy for other, perhaps more important qualities in a software system. Testability is a quality that can promote the health of other desirable qualities, and it can serve as an indicator of whether these requirements are being met. The metaphor I like to use is that testability is a kind of barometer for software architects. A barometer only really tells you the air-pressure, but you can often use this to determine if there's going to be rain. Testability only really tells you how amenable your code is to useful testing, but you can often use this to help determine if your system is modular, organizationally scalable, and so forth.

What is testability anyway?

To meaningfully discuss testability as a tool, we need to establish some definition of what it means. Like "software architect", there is no perfect answer. On some level all software is testable in that you can test it. By hook or by crook you can write some code that verifies the behavior of pretty much anything with a specification. So clearly just "being testable" isn't a sufficient definition.

At the same time, it's also pretty clear that it's simpler to test some software than other. It may be easy to test for a number of reasons. Perhaps it's easy to understand, so that you have a clear understanding of how to test it thoroughly and properly. Perhaps the chunk of code is easy to instantiate without requiring a whole bunch of scaffolding and support objects. This not only saves on keystrokes but it also has other big benefits: it isolates behavior, it may mean your tests are faster, and it generally means that your tests are easier to understand and thus maintain.

If you do a little poking around you'll find that people have hit upon certain code qualities that generally influence the testability of a piece of code. [2] But in the end we don't really have a "testability-o-meter" that we can point at a piece of code. There's no accepted way to assign a "testability rating" to software that tells you if code is more or less testable than other code, or even if it's "easy to test" or "hard to test". We can sometimes get these kinds of numbers for other qualities like modularity or complexity, and things like "scalability" also lend themselves to being measured, but testability isn't (yet) in that realm.

Instead, determining if something is testable is a decision that people need to make, and it's a decision that you can only make in an informed way if you understand code. And this is why my definition of "software architect" - from a practical standpoint - includes being able to understand code well at many levels. You have to be able to recognize when, say, dependency injection could replace local object construction to reduce coupling in a system. You need to be able to spot - or at least know to be on the lookout for - circular dependencies between modules. And in general you're going to need to be able to do this not only with code that you're writing but with code that you only see in reviews or maybe only see described in documents.

Why testability?

So I've just told you that testability is hard to measure or even to define. In fact, I've told you that to make heads or tails of it you need to be an experienced programmer. On its face, then, it sounds like the cure is worse than the disease: yes, you've got complexity in your projects to deal with, but now I want to you do something even harder to make those problems go away.

On some level that's true! Gauging testability isn't simple and it's not perfect, but by targetting testability we get a couple of important benefits because testability is special.

Testability represents your first customer, your first users: your tests! Tests are very often the first place your code is used outside of your head. This means that this is where you'll first spot difficult APIs or awkward relationships that slipped through your design.

Tests force us to use code, and they force us to consider it at many different zoom levels - from unit tests to functional tests to integration tests, we get to see it all. And tests can - and should - happen early and often in the development process. This is how you get maximum benefit from them.

If you're paying attention you'll notice that I just made a significant shift in terminology. I went from talking about "testability" to "tests", from "code that can be tested" to "code with tests". I guess it's arguable that you can have testable code without actually having tests, but that seems a bit academic to me. I've gone on and on about how difficult it is to measure testability, but one of most effective and practical ways to asses testability is to simply test your code!

So for my purposes, testable code is also tested code. I won't quibble able precisely how much testing is enough, or at what level it should be done; there are plenty of other people who are happy to tell you that. [3]

But if your tests add value to your software system, then I'd wager that they exercise your code enough to highlight a lot of the software qualities for which testability is a barometer.

Qualities correlated with testability

This article lays the foundation for the rest of this series in which we'll look at various software qualities that correlate with testability. Some of these qualities, such as modularity, are directly reflected in the testability of a system. Other qualities, like performance, are supported or enabled by testability but aren't directly related to it.

[1]This is an ill-defined term, to be sure, but I'm essentially talking about the person tasked with shepherding the so-called non-functional requirements...whether their job title is "software architect" or not.
[2]Wikipedia's got a nice, non-controversial list of things like "observability", "heterogeneity", and "understandability", and all of these things certainly would influence how easy or hard it is to test a piece of code.
[3]See for example TDD, BDD, or James Coplien's thoughts.