Automate Only What You Need To

Chris Oldwood from The OldWood Thing

The meme tells us to “automate all the things” and it’s a noble cause which has sprung up as a backlash against the ridiculous amount of manual work we’ve often had to do in the past. However in our endeavour to embrace the meme we should not go overboard and lose sight of what we’re automating and why.

The Value

The main reason we tend to automate things is to save ourselves time (and by extension, money) by leveraging tools that can perform tasks quicker than we can, but also with more determinism and reliability, thereby saving even more time. For example, pasting a complex set of steps off a wiki page into a command prompt to perform a task is slower than an interpreter running a script and is fraught with danger as we might screw up at various points along the way and so end up not doing exactly what we’d intended. Ultimately computers are good at boring repetitive tasks whilst we humans are not.

However if we only do this operation once every six months and there are too many potential points of failure we might spend far longer trying to automate it than it actually takes to do carefully, manually. It’s a classic trade-off and like most things in IT there are some XKCD’s for that – “Automation” and “Is It Worth the Time”. They make sobering reading when you’re trying to work out how much time you might save automating something and therefore also gives a good indication of the maximum amount of time you should spend on achieving that.

Orchestration First, Actor Later

Where I think the meme starts to break down is when we get this balance wrong and begin to lose sight of where the real value is, thereby wasting time trying to automate not only all the steps but also wire it into some job scheduling system (e.g. CI server) so that once in a blue moon we can push a button and the whole task from start to finish is executed for us without further intervention.

The dream suggests at that point we can go off and do something else more valuable instead. Whilst this notion of autonomy is idyllic it can also come with a considerable extra up-front cost and any shortcuts are likely to buy us false security (i.e. it silently fails and we lose time investigating downstream failures instead).

For example there are many crude command prompt one-liners I’ve written in the past to pick up common mistakes that are trivial for me to run because they’ve been written to automate the expensive bit, not the entire problem. I often rely on my own visual system to filter out the noise and compensate for the impurities within the process. Removing these wrinkles is often where the proverbial “last 10% that takes 90% of the time” goes [1].

It’s all too easy to get seduced by the meme and believe that no automation task is truly complete until it’s fully automated.

An Example

In .Net when you publish shared libraries as NuGet packages you have a .nuspec file which lists the package dependencies. The library .csproj build file also has project dependencies for use with compilation. However these two sets of dependencies should be kept in sync [2].

Initially with only a couple of NuGet packages it was easy to do manually as I knew it was unlikely to change. However once the monolithic library got split it up the dependencies started to grow and manually comparing the relevant sections got harder and more laborious.

Given the text based nature of the two files (XML) it was pretty easy to write a simple shell one-liner to grep the values from the two sets of relevant XML tags, dump them in a file, and then use diff to show a side-by-side comparison. Then it just needed wrapping in a for loop to traverse the solution workspace.

Because the one-liner was mine I got to take various shortcuts like hardcoding the input path and temporary files along with “knowing” that a certain project was always misreported. At this point a previously manual process has largely been automated and as long as I run it regularly will catch any mistakes.

Of course it’s nice to share things like this so that others can take advantage after I’m gone, and it might be even better if the process can be added as a build step so that it’s caught the moment the problem surfaces rather than later in response to a more obscure issue. Now things begin to get tricky and we start to see diminishing returns.

First, the Gnu on Windows (GoW) toolset I used isn’t standard on Windows so now I need to make the one-liner portable or make everyone else match my tooling choice [3]. I also need to fix the hard coded paths and start adding a bit of error handling. I also need to find a way to remove the noise caused by the one “awkward” project.

None of this is onerous, but this all takes time and whilst I’m doing it I’m not doing something (probably) more valuable. The majority of the value was in being able to scale out this safety check, there is (probably) far less value in making it portable and making it run reliably as part of an automated build. This is because essentially it only needs to be run whenever the project dependencies change and that was incredibly rare once the initial split was done. Additionally the risk of not finding an impedance mismatch was small and should be caught by other automated aspects of the development process, i.e. the deployment and test suite.

Knowing When to Automate More

This scenario of cobbling something together and then finding you need to do it more often is the bread and butter of build & deployment pipelines. You often start out with a bunch of hacked together scripts which do just enough to allow the team to bootstrap itself in to an initial fluid state of delivery. This is commonly referred to as a walking skeleton because it forms the basis for the entire development process.

The point of starting with the walking skeleton rather than just diving headlong into features is to try and tackle some of the problems that historically got left until it was too late, such as packaging and deployment. In the modern era of continuous delivery we look to deliver a thin slice of functionality quickly and then build upon it piecemeal.

However it’s all too easy to get bogged down early on in a project and spend lots of time just getting the build pipeline up and running and have nothing functional to show for it. This has always made me feel a little uncomfortable as it feels as though we should be able to get away with far less than perhaps we think we need to.

In “Building the Pipeline - Process Led or Automation Led” and my even earlier post “Layered Builds” I’ve tried to promote a more organic approach that focuses on what I think really matters most which is a consistent and extensible approach. In essence we focus first on producing a simple, repeatable process that can be used locally to enable the application skeleton to safely evolve and then balance the need for automating this further along with the other features. If quality or speed of delivery drops and more automation looks to be the answer then it can be added with the knowledge that it’s being done for deliberate reasons, rather than because we’ve got carried away gold plating the build system based on what other people think it should do (i.e. a cargo cult mentality).

Technical Risk

The one caveat to being leaner about your automation is that you may (accidentally) put off addressing one or more technical risks because you don’t perceive them as risks. This leads us back to why the meme exists in the first place – failing to address certain aspects of software delivery until it’s too late. If there is a technical concern, address it, but only to the extent that the risk is understood, you may not need to do anything about it now.

With a team of juniors there is likely to be far more unknowns [4] than with a team of experienced programmers, therefore the set of perceived risks will be higher. Whilst you might not know the most elegant approach to solving a problem, knowing an approach already reduces the risk because you know that you can trade technical debt in the short term for something else more valuable if necessary.

Everything is Negotiable

The thing I like most about an agile development process is that every trade-off gets put front-and-centre, everything is now negotiable [5]. Every task now comes with an implicit question: is this the most valuable thing we could be doing?

Whilst manually building a private cloud for your production system using a UI is almost certainly not the most scalable approach, neither is starting day one of a project by diving into, say, Terraform when you don’t even know what you’re supposed to be building. There is nothing wrong with starting off manually, you just need to be diligent and ensure that your decision to only automate “enough of the things” is always working in your favour.

 

[1] See “The Curse of NTLM Based HTTP Proxies”.

[2] I’m not aware of Visual Studio doing this yet although there may now be extensions and tools written by others I’m not aware of.

[3] Yes, the Unix command line tools should be ubiquitous and maybe finally they will be with Bash on Windows.

[4] See “Turning Unconscious Incompetence to Conscious Incompetence”.

[5] See “Estimating is Liberating”.

LINQ: Did You Mean First(), or Really Single()?

Chris Oldwood from The OldWood Thing

TL;DR: if you see someone using the LINQ method First() without a comparator it’s probably a bug and they should have used Single().

I often see code where the author “knows” that a sequence (i.e. an Enumerable<T>) will result in just one element and so they use the LINQ method First() to retrieve the value, e.g.

var value = sequence.First();

However there is also the Single() method which could be used to achieve a similar outcome:

var value = sequence.Single();

So what’s the difference and why do I think it’s probably a bug if you used First?

Both First and Single have the same semantics for the case where the the sequence is empty (they throw) and similarly when the sequence contains only a single element (they return it). The difference however is when the sequence contains more than one element – First discards the extra values and Single will throw an exception.

If you’re used to SQL it’s the difference between using “top” to filter and trying extract a single scalar value from a subquery:

select top 1 x as [value] from . . .

and

select a, (select x from . . .) as [value] from . . .

(The latter tends to complain loudly if the result set from the subquery is not just a single scalar value or null.)

While you might argue that in the face of a single-value sequence both methods could be interchangeable, to me they say different things with Single begin the only “correct” choice.

Seeing First says to me that the author knows the sequence might contain multiple values and they have expressed an ordering which ensures the right value will remain after the others have been consciously discarded.

Whereas Single suggests to me that the author knows this sequence contains one (and only one) element and that any other number of elements is wrong.

Hence another big clue that the use of First is probably incorrect is the absence of a comparator function used to order the sequence. Obviously it’s no guarantee as the sequence might be being returned from a remote service or function which will do the sorting instead but I’d generally expect to see the two used together or some other clue (method or variable name, or parameter) nearby which defines the order.

The consequence of getting this wrong is that you don’t detect a break in your expectations (a multi-element sequence). If you’re lucky it will just be a test that starts failing for a strange reason, which is where I mostly see this problem showing up. If you’re unlucky then it will silently fail and you’ll be using the wrong data which will only manifest itself somewhere further down the road where it’s harder to trace back.

Generating Sequences

Anthony Williams from Just Software Solutions Blog

I was having a discussion with my son over breakfast about C++ and Python, and he asked me if C++ had anything equivalent to Python's range() function for generating a sequence of integers. I had to tell him that no, the C++ standard library didn't supply such a function, but there were algorithms for generating sequences (std::generate and std::generate_n) into an existing container, and you could write something that would provide a "virtual" container that would supply a sequence as you iterated over it with range-for.

He was happy with that, but I felt the urge to write such a container, and blog about it. There are existing implementations available (e.g. boost::counting_range), but hopefully people will find this instructive, if not useful.

Iterating over a "virtual" container

Our initial goal is to be able write

for(int x: range(10))
  std::cout<<x<<std::endl;

and have it print out the numbers 0 to 9. This requires that our new range function returns something we can use with range-for — something for which we can call begin and end to get iterators.

    class numeric_range{
    public:
        class iterator;
        iterator begin();
        iterator end();
    };

    numeric_range range(int count);

Our container is virtual, so we don't have real values we can refer to. This means our iterators must be input iterators — forward iterators need real objects to refer to. That's OK though; we're designing this for use with range-for, which only does one pass anyway.

Input Iterator requirements

All iterators have 5 associated types. They can either be provided as member types/typedefs, or by specializing std::iterator_traits<>. If you provide them as part of your iterator class, then the std::iterator_traits<> primary template works fine, so we'll do that. The types are:

iterator_category

The tag type for the iterator category: for input iterators it's std::input_iterator_tag.

value_type

This is the type of the element in the sequence. For an integer sequence, it would be int, but we'll want to allow sequences of other types, such as unsigned, double, or even a custom class.

reference

The result of operator* on an iterator. For forward iterators it has to be an lvalue reference, but for our input iterator it can be anything convertible to value_type. For simplicity, we'll make it the same as value_type.

pointer

The return type of operator-> on an iterator. value_type* works for us, and is the simplest option.

difference_type

The type used to represent the difference between two iterators. For input iterators, this is irrelevant, as they provide a one-pass abstraction with no exposed size. We can therefore use void.

The types aren't the only requirements on input iterators: we also have to meet the requirements on valid expressions.

Input Iterator expression requirements

Input iterators can be valid or invalid. If an iterator is invalid (e.g. because it was invalidated by some operation), then doing anything to it except assigning to it or destroying it is undefined behaviour, so we don't have to worry about that. All requirements below apply only to valid iterators.

Valid iterators can either refer to an element of a sequence, in which case they are dereferencable, or they can be past-the-end iterators, which are not dereferencable.

Comparisons

For iterators a and b, a==b is only a valid expression if a and b refer to the same sequence. If they do refer to the same sequence, a==b is true if and only if both a and b are past-the-end iterators, or both a and b are dereferencable iterators that refer to the same element in the sequence.

a!=b returns !(a==b), so is again only applicable if a and b are from the same sequence.

Incrementing

Only dereferencable iterators can be incremented.

If a is dereferencable, ++a must move a to the next value in the sequence, or make a a past-the-end iterator. Incrementing an iterator a may invalidate all copies of a: input iteration is single-pass, so you cannot use iterators to an element after you've incremented any iterator that referenced that element. ++a returns a reference to a.

(void)a++ is equivalent to (void)++a. The return type is unspecified, and not relevant to the increment.

Dereferencing

For a dereferencable iterator a, *a must return the reference type for that iterator, which must be convertible to the value_type. Dereferencing the same iterator multiple times without modifying the iterator must return an equivalent value each time. If a and b are iterators such that a==b is true, *a and *b must be return equivalent values.

a->m must be equivalent to (*a).m.

*a++ must return a value convertible to the value_type of our iterator, and is equivalent to calling following the function:

iterator::value_type increment_and_dereference(iterator& a) {
  iterator::value_type temp(*a);
  ++a;
  return temp;
}

This is why the return type for a++ is unspecified: for some iterators it will need to be a special type that can generate the required value on dereferencing; for others it can be a reference to a real value_type object.

Basic implementation

Our initial implementation can be really simple. We essentially just need a current value, and a final value. Our dereferencable iterators can then just hold a pointer to the range object, and past-the-end iterators can hold a null pointer. Iterators are thus equal if they are from the same range. Comparing invalidated iterators, or iterators from the wrong range is undefined behaviour, so that's OK.

class numeric_range{
    int current;
    int final;
public:
    numeric_range(int final_):current(0),final(final_){}

    iterator begin(){ return iterator(this); }
    iterator end() { return iterator(nullptr); }

    class iterator{
      numeric_range* range;
    public:
      using value_type = T;
      using reference = T;
      using iterator_category=std::input_iterator_tag;
      using pointer = T*;
      using difference_type = void;

      iterator(numeric_range* range_):range(range_){}

      int operator*() const{ return range->current; }
      int* operator->() const{ return &range->current;}

      iterator& operator++(){ // preincrement
          ++range->current;
          if(range->current==range->final)
            range=nullptr;
            return *this;
      }

      ??? operator++(int); // postincrement

      friend bool operator==(iterator const& lhs,iterator const& rhs){
        return lhs.range==rhs.range;
      }
      friend bool operator!=(iterator const& lhs,iterator const& rhs){
        return !(lhs==rhs);
      }
    };
};

numeric_range range(int max){
  return numeric_range(max);
}

Note that I've left out the definition of the iterator postincrement operator. That's because it warrants a bit more discussion.

Remember the spec for *a++: equivalent to calling

iterator::value_type increment_and_dereference(iterator& a) {
  iterator::value_type temp(*a);
  ++a;
  return temp;
}

Since this is a combination of two operators, we can't do it directly: instead, our postincrement operator has to return a special type to hold the temporary value. Our postincrement operator thus looks like this:

class numeric_range::iterator{
  class postinc_return{
    int value;
  public:
    postinc_return(int value_): value(value_){}
    int operator*(){ return value; }
  };
public:
  postinc_return operator++(int){
    postinc_return temp(range->current);
    ++*this;
    return temp;
  }
};

That's enough for our initial requirement:

int main(){
    for(int x: range(10)){
        std::cout<<x<<",";
    }
    std::cout<<std::endl;
}

will now print 0 to 9.

More complete implementation

The Python range function provides more options than just this, though: you can also specify start and end values, or start, end and step values, and the step can be negative. We might also like to handle alternative types, such as unsigned or double, and we need to ensure we handle things like empty ranges.

Alternative types is mostly easy: just make numeric_range a class template, and replace int with our new template parameter T everywhere.

Setting the initial and final values is also easy: just provide a new constructor that takes both the current and final values, rather than initializing the current value to 0.

Steps are a bit more tricky: if the final value is not initial+n*step for an integer n, then the direct comparison of current==final to check for the end of the sequence won't work, as it will always be false. We therefore need to check for current>=final, to account for overshoot. This then doesn't work for negative steps, so we need to allow for that too: if the step is negative, we check for current<=final instead.

Final code

The final code is available for download with simple examples, under the BSD license.

Posted by Anthony Williams
[/ cplusplus /] permanent link
Tags: , ,

| Stumble It! stumbleupon logo | Submit to Reddit reddit logo | Submit to DZone dzone logo

Comment on this post

Follow me on Twitter

Fixing package download performance problems in Manjaro Linux

The Lone C++ Coder's Blog from The Lone C++ Coder&#039;s Blog

My adventures with Manjaro Linux continue and I’ve even moved my “craptop” - a somewhat ancient Lenovo X240 that I use as a semi-disposable travel laptop - from XUbuntu to Manjaro Linux. But that’s a subject for another blog post. Today, I wanted to write about package download performance issues I started encountering on my desktop recently and how I managed to fix them. I was trying to install terminator this morning and kept getting errors from Pamac that the downloads timed out.

Visual Lint 6.0 has been released

Products, the Universe and Everything from Products, the Universe and Everything

The first public build of Visual Lint 6.0 has just been uploaded to our website. As of today, Visual Lint 6.0 replaces Visual Lint 5.5 as the current supported Visual Lint version. Customers with active Visual Lint 5.x priority support, floating and site licence subscriptions should shortly be receiving updated licence keys for the new version, and upgrades for Visual Lint 1.x, 2.x, 3.x, 4.x and 5.x per user licences should become available in our online store shortly. If you have purchased any full per-user Visual Lint licences since the start of October, your licence keys are already v6.x compatible so can start using the new version of the software right now without any need to upgrade. If you have purchased any upgrade licences during the same period, you will shortly receive new Visual Lint 6.x compatible licence keys. The new version adds more complete support for the forthcoming PC-lint Plus, as well as support for Visual Studio 2017 RC and many other improvements. Full details of the changes in this version are as follows: Host Environments:
  • Added support for analysing standalone makefile projects.
  • Added support for makefiles within custom project (.vlproj) files. This allows makefiles requiring additional parameters to be analysed more easily.
  • Added support for Visual Studio 2017 RC. Note that the IDE will currently issue a warning that the extension is not compatible during installation.
  • Speeded up the loading of solutions and workspaces by loading the projects within them in parallel.
Analysis Tools:
  • PC-lint Plus crashes ("Serious Errors") are now parsed and reported as Fatal Errors with ID 399.
  • PC-lint Plus messages 890-897 are now correctly recognised as referencing ("supplemental", in PC-lint Plus parlance) issues.
  • Added PC-lint Plus specific compiler indirect files co-rb-vs2012.lnt, co-rb-vs2013.lnt, co-rb-vs2015.lnt and co-rb-vs2017.lnt, as well as a PC-lint Plus version of lib-rb-win32.lnt. All such indirect files are installed into the Visual Lint installation folder and referenced via -i directives on the generated PC-lint Plus command line.
  • When a PC-lint Plus per-project analysis task which uses more than one thread is running, Visual Lint can now dynamically reduce the number of active analysis tasks to take account of the number of analysis threads launched by the analysis task. [Visual Lint Professional and Enterprise Editions]
  • PC-lint and PC-lint Plus project indirect (project.lnt) files are now written when analysis tasks requiring them are created rather than when a solution/workspace is loaded.
Installation:
  • If the Visual Studio plug-in is selected for installation, a VSIX extension (VisualLintPackage.vsix) supporting Visual Studio 2017 RC is now installed into the Visual Lint installation folder. If Visual Studio 2017 is installed on the target system, the installer will automatically attempt to run the appropriate instance of VsixInstaller.exe to install the extension into Visual Studio 2017 RC.
Configuration:
  • When invoked, the Configuration Wizard now opens on the Analysis Tools page. As a result, the "Introductory" page has been removed.
  • The indirect files listed in the Configuration Wizard PC-lint Compiler Configuration page are now sorted primarily by description rather than filename. This works better than sorting by filename as the naming conventions for MSVC compiler indirect files between PC-lint and PC-lint Plus are slightly different.
  • Added an "analysis threads" option to the Command Line Options page to allow the value of the PC-lint Plus -max_threads option to be set on the command line in per-project analysis. [Visual Lint Professional and Enterprise Editions]
  • Renamed the "Maximum background analysis thread count" option on the Analysis Options page to "Maximum background analysis tasks".
User Interface:
  • Added a new Projects Display to VisualLintGui to provide a way of viewing the structure and properties of projects.
  • Added a new Project Properties Dialog to provide a way of viewing (and in the case of custom projects, editing) project properties. The dialog is available via context menu commands in the Projects Display and Analysis Status Display.
  • Added a new Project Variables Dialog to display the build variables defined in the associated project configuration.
  • Added an "Issue IDs" control (similar to the one in the Analysis Statistics Display) to the Display Filter Dialog as an alternative to editing the raw issue filter directly. Note that this control will be hidden if the dialog does not have access to a list of the issues which can be detected by the current analysis tool.
  • Added a new Makefile Properties Dialog. Where appropriate, the makefile command line can be edited here and the makefile output seen immediately.
  • Replaced the discrete "Clear Text Filter" buttons in the primary displays with integrated ones built into the corresponding text filter edit controls.
  • Added infotips to the Analysis Statistics Display.
  • When the current host environment is changed (e.g. when projects for different versions of Visual Studio are loaded) the details are now reflected in the Status View.
  • The "Analysis Tool:" message displayed in the Status View when Visual Lint starts now shows which host environment/project type the active analysis tool relates to.
Commands:
  • Added an "Export Project" command to the Analysis Status Display context menu to export the current project as a custom project (.vlproj) file.
  • Added an "Export Analysis Results" context menu command to the Analysis Status Display context menu.
  • Added a "File | New Project" command to VisualLintGui to allow new custom projects to be created directly within the environment.
  • Rearranged the VisualLintGui "File" menu and added accelerators to the "File | Open | Solution/Workspace/Project..." and "File | Open | Eclipse Workspace..." commands.
  • Added a "Send Feedback" command to the main menus in VisualLintGui and the Visual Studio and Eclipse plug-ins.
Reports:
  • HTML analysis reports are now displayed in the internal browser of the host environment (if available) by default.
Bug Fixes:
  • The platform name is now inferred from the targetPlatform element while reading Eclipse C/C++ (.cproject) project files.
  • Variables within file paths are now expanded correctly while reading IAR Embedded Workbench project files.
  • The Analysis Status Display "Troubleshooting | View Project Indirect File" context menu command is now disabled if Visual Lint has not been configured for PC-lint analysis of the associated project type, or if the project is still loading.
  • Fixed a bug relating to ProjectVersion elements within Atmel AVR Studio 5.0 project files.
  • Fixed a potential bug when interfacing to CppCheck, which could have caused cppcheck.exe to be omitted from the analysis tool pathname under some conditions - potentially causing analysis errors.
Download Visual Lint 6.0.0.273

Dumbing Down Code

Chris Oldwood from The OldWood Thing

A conversation with a colleague, which was originally sparked off by “C# BAD PRACTICES: Learn how to make a good code by bad example” (an article about writing clear code), reminded me about a recent change I had just made. When I reflected back I began to consider whether I had just “dumbed down” the code instead of keeping my original possibly simpler version. I also couldn’t decide whether the reason I had changed it was because I was using an old idiom which might now be unfamiliar or because I just didn’t think it was worth putting the burden on the reader.

Going Loopy

The change I made was to add a simple retry loop to some integration test clean-up code that would occasionally fail on a laggy VM. The loop just needed to retry a small piece of code a few times with a brief pause in between each iteration.

My gut instinct was to write the loop like this:

int attempt = 5;

while (attempt-- > 0)
{
  . . .
}

Usually when I need to write a “for” loop these days, I don’t. By far the most common use case is to iterate over a sequence and so I’d use LINQ in C# or, say, a pipeline (or foreach) in PowerShell. If I did need to manually index an array (which is rare) I’d probably go straight for a classic for loop.

After I wrote and tested the code I took a step back just to review what I’d written and realised I felt uncomfortable with the while loop. Yes this was only test code but I don’t treat it as a second class citizen, it still gets the same TLC as production code. So what did I not like?

  • While loops are much rarer than for loops these days.
  • The use of the pre and post-decrement operators in a conditional expression is also uncommon.
  • The loop counts down instead of up.

Of these the middle one – the use of the post-decrement operator in a conditional expression – was the most concerning. Used by themselves (in pre or post form) to just mutate a variable, such as in the final clause of a for loop [1], seems trivial to comprehend, but once you include it in a conditional statement the complexity goes up.

Hence there is no difference between --attempt and attempt-- when used in a simple arithmetic adjustment, but when used in a conditional expression it matters whether the value is decremented before or after the comparison is made. If you get it wrong the loop executes one less time than you expect (or vice-versa) [2].

Which leads me to my real concern – how easy is it to reason about how many times the loop executes? Depending on the placement of the decrement operator it might be one less than the number “attempt” is initialised with at the beginning.

Of course you can also place the “while” comparison at the end of the block which means it would execute at least once, so subconsciously this might also cause you to question the number of iterations, at least momentarily.

Ultimately I know I’ve written the loop so that the magic number “5” which is used to initialise the attempt counter represents the maximum number of iterations, but will a reader trust me to have done the same? I think they’ll err on the side of caution and work it out for themselves.

Alternatives

The solution I went with in the end was this:

const int maxAttempts = 5;

for (int attempt = 0; attempt != maxAttempts;
                                            ++attempt)
{
  . . .
}

Now clearly this is more verbose, but not massively more verbose than my original “while” loop. However the question is whether (to quote Sir Tony Hoare [3]) it obviously contains less bugs that the original. Being au fait with both forms it’s hard for me to decide so I’m trying to guess that the reader would prefer the latter.

Given that we don’t care what the absolute value of “attempt” is, only that we execute the loop N times, I did consider some other approaches, at least momentarily. Both of these examples use methods from the Enumerable class.

The first generates a sequence of 5 numbers starting from the “arbitrary” number 1:

const int maxAttempts = 5;

foreach (var _ in Enumerable.Range(1, maxAttempts))
{
  . . .
}

The second also generates a sequence of 5 numbers but this time by repeating the “arbitrary” number 0:

const int maxAttempts = 5;

foreach (var _ in Enumerable.Repeat(0, maxAttempts))
{
  . . .
}

In both cases I have dealt with the superfluous naming of the loop variable by simply calling it “_”.

I discounted both these ideas purely on the grounds that they’re overkill. It just feels wrong to be bringing so much machinery into play just to execute a loop a fixed number of times. Maybe my brain is addled from too much assembly language programming in my early years but seemingly unnecessary waste is still a hard habit to shake.

As an aside there are plenty of C# extension methods out there which people have written to try and reduce this further so you only need write, say, “5.Times()” or “0.To(5)” but they still feel like syntactic sugar just for the sake of it.

Past Attempts

This is not the first time I’ve questioned whether it’s possible to write code that’s perhaps considered too clever. Way back in 2012 I wrote “Can Code Be Too Simple?” which looked at some C++ code I had encountered 10 years ago and which first got me thinking seriously about the subject.

What separates the audience back then and the one now is the experience level of the programmers who will likely tackle this codebase. A couple of years later in “Will Your Successor Be a Superstar Programmer?” I questioned whether you write code for people of your own calibre or the (inevitable) application support team who have the unenviable task of having to be experts at two disciplines – support and software development. As organisations move towards development teams owning their services this issue is diminishing.

My previous musings were also driven by my perception of the code other people wrote, whereas this time I’m reflecting solely on my own actions. In particular I’m now beginning to wonder if my approach is in fact patronising rather than aiding? Have I gone too far this time and should I give my successors far more credit? Or doesn’t it matter as long as we just don’t write “really weird” code?

 

[1] In C++ iterators can be implemented as simple pointers or complex objects (e.g. STL container iterators in debug builds) and so you tend to be aware of the difference because of the performance impacts it can have.

[2] I was originally going to write while (attempt-- != 0), again because in C++ you normally iterate from “begin” to “!= end”, but many devs seem to be overly defensive and favour using the > and < comparison operators instead.

[3] “There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.” – Sir Tony Hoare.

Journey Code

Chris Oldwood from The OldWood Thing

In the move from a Waterfall approach to a more Agile way of working we need to learn to be more comfortable with our software being in a less well defined state. It’s not any less correct per-se, it’s just that we may choose to prioritise our work differently now so that we tackle the higher value features first.

Destination Unknown

A consequence of this approach is that conversations between, say, traditional architects (who are talking about the system as they expect it to end up) and those developing it (who know it’s only somewhere between started and one possible future), need interpretation. Our job as programmers is to break these big features down into much smaller units that can be (ideally) independently scoped and prioritised. Being agile is about adapting to changing requirements and that’s impossible if every feature needs to designed, delivered and tested in it’s entirely.

In an attempt to bridge these two perspectives I’ve somewhat turned into a broken record by frequently saying “but that’s the destination, we only need to start the journey” [1]. This is very much a statement to remind our (often old and habitual) selves that we don’t build software like that anymore.

Build the Right Thing

The knock on effect is that any given feature is likely only partially implemented, especially in the early days when what we are still trying to explore what it is we’re actually trying to build in the first place. For example, data validation is feature we want in the product before going live, but we can probably make do with only touching on the subject lightly whilst we explore what data even needs validating in the first place.

This partially implemented feature has started to go by the name “journey code”. While you can argue that all code is essentially malleable as the system grows from its birth to its eventual demise, what we are really trying to convey here is code which cannot really be described as having been “designed”. As such it does not indicate in any meaningful way the thoughts and intended direction of the original author – they literally did the simplest possible thing to make the acceptance test pass and that’s all.

Destination in Sight

When we finally come to play out the more detailed aspects of the feature that becomes the time at which we intend to replace what we did along the journey with where we think the destination really is. Journey code does not even have to have been written in response to the feature itself, it may just be some infrastructure code required to bootstrap exploring a different one, such as logging or persistence.

What it does mean is that collaboration is even more essential when the story is eventually picked up for fleshing out to ensure that we are not wasting time trying to fit in with some design that never existed in the first place. Unless you are someone who happily ignores whatever code anyone else ever writes anyway, and they do exist, you will no doubt spend at least some time trying to understand what’s gone before you. Hence being able to just say “journey code” has become a nice shorthand for “it wasn’t designed so feel free to chuck it away and do the right thing now we know what really needs doing”.

 

[1] I covered a different side-effect of this waterfall/agile impedance mismatch in “Confusion Over Waste”.

Weak and Strong Idempotency

Chris Oldwood from The OldWood Thing

In my recent post “PUT vs POST and Idempotency” I touched on the different degrees of idempotency. In retrospect I think it’s better to describe our idempotency guarantee as weak or strong. Either way this post looks at why you might choose one over the other.

Weak Idempotency

At one end of the spectrum we can choose to classify our unique operations with a single scalar value such as a large number, UUID or complex string value (with other embedded values like the date & time). In this scenario we assume that a simple value represents the identity of the operation in much the same way that a reference (or pointer) is the unique address of a value in languages like C# and Java (or C++).

In SQL terms your write with an idempotency check might look something like this:

insert into Operations values(...)
where not exists
(
  select 1
  from Operations
  where OperationId <> @operationId
)

For a document oriented database like MongoDB where you might store everything in a single object to achieve atomic updates you could do an equivalent find-and-modify-where style query. If the data model requires multiple writes you might still be able to order them carefully to achieve the desired effect (See “Observable State versus Persisted State”).

Strong Idempotency

In contrast, at the other end, we define an idempotent operation based not only on some unique operation identifier but also the arguments to the operation itself, i.e. the request body in HTTP. In this instance we are not willing to accept that a replay can be distinguished solely by it’s handle but also wish to ensure the actual details match too, just like a “deep” value equality comparison.

In SQL terms we potentially now have two steps, one to attempt the write and, if a conflict occurs, a subsequent read to validate it:

insert into Operations values(...)
where not exists
(
  select 1
  from Operations
  where OperationId <> @operationId
)

select *
from Operations
where OperationId = @operationId

If replays are rare (and I’d expect them to be) you should find this adds only a little extra overhead but also allows you to report broken requests properly.

Identity and Equality

Although it might seem like these weak and strong guarantees are directly analogous to the concept of identity and equality in normal programming there is a subtle difference.

Whilst in the weak case we are happy for the unique operation identifier to act as our sole definition of equality, in the strong guarantee both the identifier and the input values [1] take part in the comparison. In a programming language we tend to choose either the reference based comparison or the value based approach but not both [2].

Failure Modes

In theory it should be adequate for us to take the simpler approach, and as the provider of a service it’s almost certainly slightly cheaper for us to do that. However the downside is that it makes it harder to diagnose missed updates caused by the service erroneously treating them as replays. But that’s just an edge case right?

During development when your clients are coding against your API for the first time they may make a simple mistake and not correctly generate unique requests. This is not quite as uncommon as you may think.

Another potential source of mistaken uniqueness can come from transforming requests, e.g. from a message queue. If you’re integrating with a legacy system what you may think is unique might in fact turn out to be an artefact of the small data set you sampled and it could turn out only to be mostly unique [3].

Answering these kinds of support queries (“what happened to my request?“) can easily chip away at a team’s time. The more effort you put into making it harder for consumers to incorrectly use your API the less time you’ll need to support them.

Debug Switch

Although the need to verify a replay should be pretty infrequent you could easily wrap the validation logic in a feature switch so that development and test environments provide better diagnostics whilst production has less work to do. But you should measure before considering dropping such a low-cost feature.

Monitoring

What will help both the service and it’s consumers is if you can afford to capture when replays do occur so that the monitoring can alert if there appears to be a sudden rise in them. For example a client may go into an failure state that keeps replaying the same message over and over again. Alternatively the rise might suggest some configuration problem where upstream traffic is being duplicated. From a functional point of view the entire system could be behaving correctly but wasting resources in the process that could lead to a more catastrophic failure state.

Although a client should never need to behave differently from a functional perspective when a replay occurs, the flip side to the service tracking the event is for the client to report it as well. They could do this if, say, the return code changes in the replay case. In HTTP for example there is 200 OK and 201 Created which could be used to tell the client what happened.

Middle Ground

As I described in my previous post there can also be points between the two extremes. One of the problems with modern document oriented databases is that they have limits on the size of a single document. Depending on the anticipated depth of history this may exceed the practical document limit and therefore require some compromises to be made.

If no false replays can be tolerated than perhaps the model needs splitting across documents which removes the ability to do simple atomic updates. If the history grows slowly and data retention is negotiable then maybe the fine details of much older operations can be discarded to make room for the new.

Brave New World

The new era of databases continues to bring interesting challenges and trade offs due to their differing goals from a traditional big iron RDBMS. Pushing back on some requirements (e.g. “Don’t Be Afraid to Throw Away Data”) is essential if we are to balance their constraints with correctness.

 

[1] You may store server derived properties as well, like a “created at” timestamp. These are of course ignored in any comparison.

[2] Of course breaking an invariant like the relationship between Equals() and GetHashCode() is a popular mistake which can make objects appear to go missing in maps & sets.

[3] I know of one organisation where many people mistakenly believed their customer identifiers were unique, but we discovered they were only truly unique if you took their legacy back-office sources into account too.

the DevOps Handbook

Jon Jagger from less code, more software

is an excellent book by Gene Kim, Jez Humble, Patrick Debois, and John Willis (isbn 978-1-942788-00-3). As usual I'm going to quote from a few pages.

Make infrastructure easier to rebuild than to repair.
The average age of a Netflix AWS instance is twenty-four days.
Interrupting technology workers is easy, because the consequences are invisible to almost everyone.
In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures.
Over the following year, they eliminated testing as a separate phase of work, instead integrating it into everyone's daily work. They doubled the features being delivered per month and halved the number of defects.
Bureaucracies are incredibly resilient and are designed to survive adverse conditions - one can remove half the bureaucrats, and the process will still survive.
When we have a tightly coupled architecture, small changes can result in large scale failure.
Our deployment pipeline infrastructure becomes as foundational for our development processes as our version control infrastructure.
If we find that unit or acceptance tests are too difficult and expensive to write and maintain, it's likely that we have an architecture that is too tightly coupled.
Any successful product or organization will necessarily evolve over its life cycle... eBay and Google are each on their fifth entire rewrite of their architecture from top to bottom.
... which can lead to the unfortunate metric of mean time until declared innocent.
The principle of small batch sizes also applies to code reviews.
80% of MTTR (mean time to recovery) is spent trying to determine what changed.
High performing DevOps organizations will fail and make mistakes more often... If high performers are performing thirty times more frequently but with only half the change failure rate, they're obviously having more failures. [Roy Rapoport, Netflix]
Spiders repair rips and tears in the web as they occur, not waiting for the failures to accumulate. [Dr Steven Spear]


Catch Up

Phil Nash from level of indirection

Trolley

Stock image from Shutterstock

It's been just over six years since I first announced Catch to the world as a brand new C++ test framework!

In that time it has matured to the point that it can take on the heavyweights - while still staying true to its original goals of being lightweight, easy to get started with and low-friction to work with.

In the last couple of years or so it has also increased dramatically in popularity! That sounds like a good thing - and it is - but with that comes a greater diversity of environments and usage, and more people raising issues and submitting pull requests.

Again, it's great to have so much input from the community - especially in the form of pull requests - where other developers have gone to some effort to implement a change, or a fix, and present it back for inclusion in the main project. So it's been heart-breaking for me that, between this increase in volume and finding my meagre free-time stretched even further, so many issues and PRs have been left unacknowledged - many not even seen by me in the first place.

But two things have happened, recently, that completely change this state of affairs. We're moving firmly in the right direction again.

Firstly, as mentioned in On Joining JetBrains, I've recently changed jobs to one that should give me much more time and opportunity to work on Catch - as well as the opportunity to do so in my home office - with stable internet (as opposed to on the train while commuting to and from work). The first few months were a bit of a wash for the reasons discussed in that post, but, as I also suggested there, this year has seen that change and I've been able to put in quite a lot of work on Catch already.

But that's not really enough. There's a huge back-log - and I'm still only doing this part time - and I want to spend time working on Catch2 as well (more on that soon). I don't want to end up back in the situation where everything is backing up and there's no hope of recovery.

I've been hoping to find someone else to be a key maintainer of Catch for a couple of years now. I've not been very active in this search - for all the same reasons - but it's been on my mind.

But, just last month, after I appeared on CppCast talking about JetBrains and Catch, there was a thread on Reddit about it - with many expressing concern over the Catch situation. I brought the subject up on there again and got the attention of one of the commenters.

I didn't know it at the time, but Martin Hořeňovský has been responsible for a good number of those PRs and issues that had been left unaddressed - as well as an active community member in helping address other people's issues. So it's with great pleasure (and relief!) that I can announce that Martin now has full commit rights to Catch on GitHub and has been prolific in working through the currently outstanding tickets.

Martin seems to really "get" Catch, and the design goals around it - so working with him on this the last couple of weeks has been very rewarding. From some queries I just ran on GitHub it looks like 39 issues have been closed and 38 PRs merged or closed in that time! That's compared to 9 new issues and 7 PRs - about half of which were created by Martin and I in the process. And that's not to mention all the labels we've been using to categorise the other tickets - with many marked as "Resolved - pending review" - which usually means we think it's resolved but we're just waiting for feedback (or a chance for more testing).

With 219 open issues and 41 PRs still outstanding, at time of writing, there's a lot more work to do yet - but I hope this reassures you that we're going in the right direction - and fast!

And we're not stopping with Martin. We have at least one other volunteer that I'll be bringing up to speed soon.

Catch2

I've referred to Catch2 a number of times now, and talked a little about what it will be. The biggest reason for making it a major release, according to Semantic Versioning, is that it will drop support for pre-C++11. For that reason Catch Classic (1.x) will continue to receive at least bug fix updates - but no more new features once Catch2 is fully released. A few major features in the pipeline have been explicitly deferred to Catch2: concurrency support and generators/ property-based testing in particular.

Moving to C++11 provides a very large scope for cleaning up the code-base - which has a significant volume of code dedicated to platform-specific workarounds for compiler shortcomings, missing library features such as smart pointers, and boilerplate that will no longer be necessary with things like range-based-for, auto and others. Lambdas will be useful too, but are not quite so important.

Because taking advantage of C++11 has the potential to touch almost every line of code, I'm taking the opportunity to rewrite the core of Catch - primarily the assertion macros and the infrastructure to support that. This is code that is #included in every test file, and expanded (in the case of macros) in every test case or even every assertion. Keeping this code lightweight is essential to avoiding a compile time hit. There's a number of ways this foot-print can be reduced and the rewrite will strive for this as much as possible.

The rest of the code, concerned with maintaining the registry of tests, parsing and interpreting the command line, running tests and reporting results, will be updated more incrementally.

I already have a (not-yet-public) proof-of-concept version of the re-written code. It's not yet complete but, so far, has only one standard library dependency and minimal templates. The compile-time overhead is imperceptible.

In addition to compile-time, runtime performance is also a goal of Catch2. It's not an overriding goal - I won't be obfuscating the code in the name of wringing out the last few milliseconds of performance - but this is a definite change from Catch Classic where runtime performance was a non-goal. This is in recognition of the fact that Catch is used for more than just isolated unit tests - and will also become more important with property based testing.

I don't have a timeline, yet, for when I expect Catch2 to be ready - and in the immediate term getting Catch Classic back under control is the priority. Despite the partial re-write, and the major version increment, I expect tests written against Catch Classic to mostly "just work" with Catch2 - or require very minimal changes in a some rare cases.

You

As already mentioned many developers have also spent time and effort contributing issues, fixes and even feature PRs over the years. So Catch has really been a community project for years now and I'm very grateful for all the help and support. I think Catch has shown that having a low-friction approach to testing C++ code is very important to a lot of people and I'm hoping we'll continue to build on that. Thank you all.