As posted in a few places, Emacs 26.1-RC1 has been released. Following up my previous experiments with running Emacs on the Windows Subsystem for Linux, I naturally had to see how the latest version would work out. For that, I built the RC1 on an up-to-date Ubuntu WSL. I actually built it twice – once […]
A quick follow-up to my last post where I was experimenting with running emacsclient from an ansi-term running in the main Emacs. Interestingly, you can run Emacs in text mode within an ansi-term, just not emacsclient: Yes, the whole thing got a little recursive. Yes, it’s a little silly, and yes, I’m one of those […]
I’m experimenting with screen recordings at the moment and just out of curiosity decided to see if I can load and edit a text file inside the main Emacs process from inside an ansi-term using emacsclient. Spoiler alert – yes, you can. At least the way it is set up on my system, emacsclient doesn’t […]
I’ve had the Linux Subsystem for Windows enabled for quite a while during the time it was in Beta. With the release of the Fall Creators Update, I ended up redoing my setup from scratch. As usual I grabbed Emacs and a bunch of other packages and was initially disappointed that I was looking at […]
Emacs 25.3 was released on Monday. Given that it’s a security fix I’m downloading the source as I write this. If you’re using the latest Emacs I’d recommend you update your Emacs. The vulnerability as been around since Emacs 19.29, you probably want to upgrade anyway. Build instructions for Ubuntu and friends are the same […]
I haven’t done much with Ubuntu recently, but had to set up a laptop with XUbuntu 17.04. That came with Emacs 24.5 as the default emacs package, and as skeeto pointed out in the comments, with a separate emacs25 package for Emacs 25.1. I tend to run the latest release Emacs everywhere out of habit, […]
In the first article in this series we looked at how to define the simplest company-mode backend.  This backend drew completion candidates from a predefined list of options, and allowed you to do completion in buffers in fundamental mode. The main purpose of that article was to introduce the essential plumbing of a company-mode backend.
In this article we'll expand upon the work of the first, adding some useful UI elements like annotations and metadata. We'll also implement a rough form of fuzzy matching, wherein candidates will be presented to the user when they mostly match the prefix. After this article you'll know almost everything you need to know about writing company-mode backends, and you'll be in a great position to learn the rest on your own.
Most of what we'll be doing in the article revolves around handling completion candidate "metadata", data associated in some way with our completion candidates. In practice this kind of data covers things like documentation strings, function signatures, symbols types, and so forth, but for our purposes we'll simply associate some biographical data with the names in our completion set sample-completions.
company-mode provides affordances for displaying metadata as part of the completion process. For example, if your backend is showing completions for function names, you could display the currently-selected function's signature in the echo area. We'll develop a backend that displays a sentence about the selected candidate in the echo area, and we'll also display their initials as an annotation in the candidate selection popup menu.
Adding more data to our completion candidates
First we need to add some metadata to our existing completion candidates. To do this we'll use Emacs text properties. ((Text properties allow you to associate arbitrary data with strings. You can read about them here. Specifically, we use the special read syntax for text properties.)) For each completion candidate we define an :initials property containing their initials and a :summary property containing a one-sentence summary of the candidate.  To add these properties, update sample-completions to look like this:
(defconst sample-completions '(#("alan" 0 1 (:initials "AMT" :summary (concat "Alan Mathison Turing, OBE, FRS (/ˈtjʊərɪŋ/ " "tewr-ing; 23 June 1912 – 7 June 1954) was a " "British mathematician, logician, cryptanalyst, " "philosopher, pioneering computer scientist, " "mathematical biologist, and marathon and ultra " "distance runner."))) #("john" 0 1 (:initials "JVN" :summary (concat "John von Neumann (/vɒn ˈnɔɪmən/; December 28, " "1903 – February 8, 1957) was a Hungarian and " "American pure and applied mathematician, physicist, " "inventor and polymath."))) #("ada" 0 1 (:initials "AAK" :summary (concat "Augusta Ada King, Countess of Lovelace (10 December " "1815 – 27 November 1852), born Augusta Ada Byron " "and now commonly known as Ada Lovelace, was an " "English mathematician and writer chiefly known for " "her work on Charles Babbage's early mechanical " "general-purpose computer, the Analytical Engine."))) #("don" 0 1 (:initials "DEK" :summary (concat "Donald Ervin Knuth (/kəˈnuːθ/ kə-nooth; born " "January 10, 1938) is an American computer " "scientist, mathematician, and Professor Emeritus " "at Stanford University.")))))
Attaching properties like this is a very convenient way to store metadata for completion candidates. Of course in a real backend you probably wouldn't have a hard-coded list of candidates, and you'd be fetching them dynamically from a server, database, or external process. In that case, you'd need to also dynamically fetch the metadata you want and attach it to the candidate strings you serve through your backend. In the end, text properties work well in this context because they transparently transport the metadata - which company-mode doesn't know about - with the completion strings that company-mode definitely knows about.
Displaying metadata in the echo area
Where the annotation command adds a small annotation to the completion popup menu, the meta backend command produces text to display in the echo area.  The process for producing the metadata string is almost exactly like that of producing the annotation string. First we write a function that extracts the string from the candidate text properties. Then we wire that function into the backend through the meta command.
As you've probably guessed, the function for extracting the metadata string will simply read the :summary property from a candidate string. It looks like this:
(defun sample-meta (s) (get-text-property 0 :summary s))
The changes to the backend look like this:
(defun company-sample-backend (command &optional arg &rest ignored) (interactive (list 'interactive)) (case command (interactive (company-begin-backend 'company-sample-backend)) (prefix (and (eq major-mode 'fundamental-mode) (company-grab-symbol))) (candidates (remove-if-not (lambda (c) (string-prefix-p arg c)) sample-completions)) (annotation (sample-annotation arg)) (meta (sample-meta arg))))
As before, in the last line we associate the meta command with our sample-meta function.
Here's how the metadata looks when displayed in the echo area:
As a final improvement to our backend, let's add support for fuzzy matching. This will let us do completion on prefixes which don't exactly match a candidate, but which are close enough.  For our purposes we'll implement a very crude form of fuzzy matching wherein a prefix matches a candidate if the set of letters in the prefix is a subset of the set of letters in the candidate. The function for performing fuzzy matching looks like this:
(defun sample-fuzzy-match (prefix candidate ) (cl-subsetp (string-to-list prefix) (string-to-list candidate)))
Now we just need to modify our backend a bit. First we need to modify our response to the candidates command to use our new fuzzy matcher. Then we need to respond to the no-cache command by returning true.  Here's how that looks:
(defun company-sample-backend (command &optional arg &rest ignored) (interactive (list 'interactive)) (case command (interactive (company-begin-backend 'company-sample-backend)) (prefix (and (eq major-mode 'fundamental-mode) (company-grab-symbol))) (candidates (remove-if-not (lambda (c) (sample-fuzzy-match arg c)) sample-completions)) (annotation (sample-annotation arg)) (meta (sample-meta arg)) (no-cache 't)))
As you can see, we've replaced string-prefix-p in the candidates response with sample-fuzzy-match, and we've added (no-cache 't).
Here's how our fuzzy matching looks in action:
That's all, folks
We've seen how to use Emacs' text properties to attach metadata to candidate strings. This is a really useful technique to use when developing company-mode backends, and one that you'll see used in real-world backends. With that metadata in place, we've also seen that it's very straightforward to tell your backend to display annotations in popup menus and metadata in the echo area. Once you've got the basic techniques under your belt, you can display anything you want as part of completion.
There are still more aspects to developing company-mode backends, but with what we've covered in this series you can get very far. More importantly, you know the main concepts and infrastructure for the backends, so you can learn the rest on your own. If you want to delve into all of the gory details, you'll need to read the company-mode source code, and specifically the documentation for company-backends. 
For an example of a fairly full-featured backend implementation that's currently in use (and under active development), you can see the emacs-ycmd project.  Happy hacking!
|||The first article in this series..|
|||The summaries are simply the first sentences of the respective Wikipedia articles.|
|||The Emacs manual entry on the echo area.|
|||Fuzzy matching is commonly used for completion tools because it addresses common cases where users transpose characters, accidentally leave characters out, or consciously leverage fuzzy matching for increased speed.|
|||The details of why this is the case are murky, but the company-mode source code specifically states this. The same source also says that we technically should be implementing a response to match, but that doesn't seem to affect this implementation.|
|||The company-mode project page.|
|||The emacs-ycmd project is on github. In particular, see company-ycmd.el.|
In Emacs, company-mode (short for "complete anything") is a framework for performing completion in buffers.  It's an alternative to the popular auto-complete-mode. company-mode supports extension via backends which provide the framework with lists of possible completions in various contexts. So, for example, there's a backend th(at provides completion support for Emacs lisp and one that does the same for Python. Backends can use very different technologies as long as they conform to the backend interface specified by the mode.
I recently decided to write a company-mode backend for ycmd, a completion server for languages including C/C++/Objective-C and Python.  All in all it was a relatively pain-free experience, but the process isn't as well documented as I would have liked. So I want to use this series to describe how it's done with the hope of making it easier for others and of helping me remember how to do it in the future.
I won't be covering all of the details of company-mode backends (partially because I don't know them all), but this series should tell you what you need to know to create your own fully-armed and operational backend.  In this article we'll define the simplest possible backend in order to familiarize you with the concepts and infrastructure involved. In the next article we'll add some sophistication to that backend to improve the user experience.
The simplest possible backend
For our example we need to define a source of completion candidates. Ultimately, any completion source is just a sequence of strings that meet some criteria. Examples might include:
- A list of English words starting with some prefix
- Methods for a particular object in Java
- Modules available for import in Python program
company-mode doesn't care about the nature of these strings. It just takes them and makes it easy for the user to select from the available options.
In this case, we'll just define a fixed list of strings:
(defconst sample-completions '("alan" "john" "ada" "don"))
That's it.  Completion sources don't need to (though they generally will) be more complex than that.
Defining the backend
Backends take the form of a function which takes a command as its first argument. This command can take any of a number of values, and backends are required to respond to a handful of them. Before we get into those details, let's look at our very basic backend implementation:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
(require 'cl-lib) (require 'company) (defun company-sample-backend (command &optional arg &rest ignored) (interactive (list 'interactive)) (cl-case command (interactive (company-begin-backend 'company-sample-backend)) (prefix (and (eq major-mode 'fundamental-mode) (company-grab-symbol))) (candidates (cl-remove-if-not (lambda (c) (string-prefix-p arg c)) sample-completions))))
The signature of this function is mandated by company-mode. Line 5 makes the function interactive so that you can easily drive your backend without invoking company-mode, something we'll do in a bit. The cl-case statement on line 7 is where we decide what to do based on command. In this case, we respond to interactive, prefix, and candidates.
The interactive command is passed once, before the other commands are used, and it is used to initialize the company-mode infrastructure. All you need to do as a backend developer is pass your backend to company-begin-backend as in this example.
The prefix command
The prefix command is probably the most complex command to handle. This command should return the text that is to be completed. Determining this text can be complex depending on what you're trying to complete, but company-grab-symbol often does "the right thing" if your completion context is space-delimited.
If the prefix command returns nil, this tells company-mode that the backend is not suitable for doing completion on this context. On line 9 of our example we check to see if we're in fundamental-mode and, if not, return nil. In other words, we're saying here that our backend only applies to fundamental-mode. Programming language-oriented backends can make a similar check for their specific modes. When a backend responds to prefix with nil, other backends are given a chance to do the completion.
On the other hand, if a backend is appropriate for the current completion but it can't provide any completions for some reason, the backend should return 'stop. This tells company-mode that no other backends should be used for this completion.
So our backend is effectively saying that it can do completion for anything in fundamental mode. There are more details to prefix, but that's covers the important parts.
The candidates commands
The response to the candidates command is where you actually generate a list of possible completions at a point in a buffer. When this command is passed in, the arg argument holds the value returned by prefix. In other words, you construct your candidates based on the text that you previously indicated was to be completed.
In our case, the prefix we indicated was whatever came before point in the buffer. To calculate our possible completions, we filter the sample-completions values with that prefix using remove-if-not, returning only those candidates which begin with the prefix.
As with prefix calculations, real candidate calculations can be much more complex. But if you understand how the data is piped around, then constructing these complex candidate lists should be fairly straightforward.
Test-driving the backend
To test out our backend, first enter all of the code into a buffer and evaluate it (e.g. with M-x eval-buffer.) Then create a new buffer and run M-x fundamental-mode and M-x company-mode. 
In this new buffer enter the single character "a" and then, with the cursor immediately after the "a", run M-x company-sample-backend. This should give you completion options something like this:
If that works correctly, then you've done almost everything you need to for a fully working backend.
Plugging the backend into company-mode
The final thing you need to do to make your backend available to company-mode is to add it the list company-backends. One simple way to do that is with add-to-list list this:
(add-to-list 'company-backends 'company-sample-backend)
Once you've done this, you can use the command company-complete to do completions, and your new backend will be used in concert with all of the other backends in that list. Generally speaking, company-complete is the command you'll use for completion with company-mode, and it'll often be bound to a simple keystroke.
A complete company-mode backend
That's all there is to writing a basic company-mode backend. In the next article in this series we'll look at adding a few more details to what we have already.
Here's a complete listing of the code used in this article:
(require 'company) (defconst sample-completions '("alan" "john" "ada" "don")) (defun company-sample-backend (command &optional arg &rest ignored) (interactive (list 'interactive)) (case command (interactive (company-begin-backend 'company-sample-backend)) (prefix (and (eq major-mode 'fundamental-mode) (company-grab-symbol))) (candidates (remove-if-not (lambda (c) (string-prefix-p arg c)) sample-completions)))) (add-to-list 'company-backends 'company-sample-backend)
|||company-mode project site|
|||The *ycmd* github repository and my Emacs client.|
|||Sorry, I couldn't resist the Star Wars reference.|
|||We'll filter the strings later based on context.|
|||This puts your buffer in major mode "fundamental" and minor mode "company".|