Posted on

Money talks: A tale of two change programs

Allan Kelly from Allan Kelly Associates

iStock_000005509580Small-2018-11-21-13-12.jpg

Like many in the agile community – or what sometimes gets called the agile industrial complex – I am all for piecemeal adoption, small scale before large scale, get good at doing stuff then expand it out… – what else would you expect from Mr Diseconomies of Scale?

The “start small” and grow might even be regarded as the canonical approach to agile implementation. But from time to time I run across something that makes me wonder…

Four or five years back I got involved with an agile transformation programme at a large financial institution, not a bank, more of a mass market asset manager. I was attached to a small team trying to make the whole company agile.

The coaches viewed themselves as a gorilla movement, changing the organization from within. They had some success, there was a bunch of stuff the agile teams and the coaches were doing wrong but that is another story. This was a licensed insurrection.

As is often the case, this team found it lacked the ability to ask the big questions and get people outside the team (often the higher ups) to engage or change themselves. The organization wanted agile down in the engine room – at the code face – but they didn’t want to rethink how they set requirements and approached projects. The whole organization was chronically project driven, obsessed with long term planning and offshore development. Economies of scale thinking ran riot.

Because the agile change was at the team level the product owners lacked authority to make real decisions – like not delivering functionality. Yes the organization wanted to “be agile” but the management cadre didn’t see any need to change their own behaviour.

One day I met two men who ran the company’s “Software Process Group.” They were guardians of the formal process and “working practices”. My immediate reaction was that they wanted to kill agile and stick with ISO-9000, PRINCE2, CMMI and certified approaches. They scared me. But actually they were very clued up. They got agile. They saw it was better than the current process.

These two had no role in the agile transformation, their role was to ensure the company kept its CMMI level two certification. This was really important to the company because this allowed the company to do business. They told me a story…

During the previous 20 years the company had grown large, very large, by buying up competitors and companies in related markets. These companies had been thrown together and costs stripped out. One day the financial regulator came to the company and said:

“We have been examining your IT functions. They are not fit for purpose. If you don’t fix it within 12 months we are going to withdraw your license to do business.”

Shit hitting the fan doesn’t come much bigger than this.

The company went to IBM and said “Help! – Fix it – we’ll will pay anything.”

IBM flooded the company with people. IBM imposed a process – a traditional CMMI compliant process. IBM changed the company, not just the programmers but everything. The company did as IBM told them.

And don’t imagine it was cheap. I bet that the change and IBM fees were on the agenda at every board meeting during that period. The men I had met were the remnants of that programme, they worked for the company not IBM, their job was to ensure the company maintained accreditation and the financial regulator wouldn’t have cause to come back.

Now contrast this with the piecemeal, small scale, bottom-up change that us agile folk are so fond of. Time and time again we get stuck: “the business won’t change”, “we can’t get access to the senior people”, “existing processes and expectations are unassailable”, “projects are killing us” and so on.

I’m sure IBM faced many of those same problems but they had one big advantage: They were expensive.

OK, they had a second: the loss of license would destroy the client company. But when threatened people often respond by sticking with that they know so maybe it was a double edged sword.

Because IBM were expensive they had access to anyone they wanted access too. Because they were expensive they had authority. And if someone didn’t want to make the changes IBM suggested then IBM could simply ask the next person up.

Once again money is information: by spending lots of money with IBM the company was signalling it really wanted these changes to happen.

Agile changers may not like big change, they may point to the inherent risks, they may point that use of authority conflicts with self-organization, they may understand that diseconomies of scale rule and they may point to a bunch of other risks.

But they should also note one clear advantage: a big expensive change programme brings authority all of its own.

The post Money talks: A tale of two change programs appeared first on Allan Kelly Associates.

Posted on

Some pair programming benefits may be mathematical artefacts

Derek Jones from The Shape of Code

Many claims are made about the advantages of pair programming. The claim that the performance of pairs is better than the performance of individuals may actually be the result of the mathematical consequences of two people working together, rather than working independently (at least for some tasks).

Let’s say that individuals have to find a fault in code, and then fix it. Some people will find the fault and then its fix much more quickly than others. The data for the following analysis comes from the report Experimental results on software debugging (late Rome period), via Lutz Prechelt and shows the density of the time taken by each developer to find and fix a fault in a short Fortran program.

Fixing faults is different from many other development tasks in that if often requires a specific insight to spot the mistake; once found, the fixing task tends to be trivial.

Density plot of time taken to find a fault by developers.

The mean time taken, for task t1, is 22.2 minutes (standard deviation 13).

How long might pairs of developers have taken to solve the same problem. We can take the existing data, create pairs, and estimate (based on individual developer time) how long the pair might take (code+data).

Averaging over every pair of 17 individuals would take too much compute time, so I used bootstrapping. Assuming the time taken by a pair was the shortest time taken by the two of them, when working individually, sampling without replacement produces a mean of 14.9 minutes (sd 1.4) (sampling with replacement is complicated…).

By switching to pairs we appear to have reduced the average time taken by 30%. However, the apparent saving is nothing more than the mathematical consequence of removing larger values from the sample.

The larger the variability of individuals, the larger the apparent saving from working in pairs.

When working as a pair, there will be some communication overhead (unless one is much faster and ignores the other developer), so the saving will be slightly less.

If the performance of a pair was the mean of their individual times, then pairing would not change the mean performance, compared to working alone. The performance of a pair has to be less than the mean of the performance of the two individuals, for pairs to show an improved performance.

There is an analytic solution for the distribution of the minimum of two values drawn from the same distribution. If f(x) is a probability density function and F(x) the corresponding cumulative distribution function, then the corresponding functions for the minimum of a pair of values drawn from this distribution is given by: F_p(x)=1-(1-F(x))^2 and f_p(x)=2f(x)(1-F(x)).

The presence of two peaks in the above plot means the data is not going to be described by a single distribution. So, the above formula look interesting but are not useful (in this case).

Posted on

Christmas books for 2018

Derek Jones from The Shape of Code

The following are the really interesting books I read this year (only one of which was actually published in 2018, everything has to work its way through several piles). The list is short because I did not read many books and/or there is lots of nonsense out there.

The English and their history by Robert Tombs. A hefty paperback, at nearly 1,000 pages, it has been the book I read on train journeys, for most of this year. Full of insights, along with dull sections, a narrative that explains lots of goings-on in a straight-forward manner. I still have a few hundred pages left to go.

The mind is flat by Nick Chater. We experience the world through a few low bandwidth serial links and the brain stitches things together to make it appear that our cognitive hardware/software is a lot more sophisticated. Chater’s background is in cognitive psychology (these days he’s an academic more connected with the business world) and describes the experimental evidence to back up his “mind is flat” model. I found that some of the analogues dragged on too long.

In the readable social learning and evolution category there is: Darwin’s unfinished symphony by Leland and The secret of our success by Henrich. Flipping through them now, I cannot decide which is best. Read the reviews and pick one.

Group problem solving by Laughin. Eye opening. A slim volume, packed with data and analysis.

I have already written about Experimental Psychology by Woodworth.

The Digital Flood: The Diffusion of Information Technology Across the U.S., Europe, and Asia by Cortada. Something of a specialist topic, but if you are into the diffusion of technology, this is surely the definitive book on the diffusion of software systems (covers mostly hardware).

Posted on

Onwards And Downwards – baron m.

baron m. from thus spake a.k.

Greetings Sir R-----! Might I suggest that you take one of these spiced beef pies and a mug of mulled cider to stave off this winter chill? And perhaps a wager to fire the blood?

Good man! Good man!

I propose a game that ever puts me in mind of my ill-fated expedition to recover for the glory of the Empress of Russia the priceless Amulet of Yendor from the very depths of Hell.
Posted on

Is it worth attending an academic conference or workshop?

Derek Jones from The Shape of Code

If you work in industry, is it worth attending an academic conference or workshop?

The following observations are based on my attending around 50 software engineering and compiler related conferences/workshops, plus discussion with a few other people from industry who have attended such events.

Short answer: No.

Slightly longer answer: Perhaps, if you are looking to hire somebody knowledgeable in a particular domain.

Much longer answer: Academics go to conferences to network. They are looking for future collaborators, funding, jobs, and general gossip. What is the point of talking to somebody from industry? Academics will make small talk and be generally friendly, but they don’t know how to interact, at the professional level, with people from industry.

Why are academics generally hopeless at interacting, at the professional level, with people from industry?

Part of the problem is lack of practice, many academic researchers live in a world that rarely intersects with people from industry.

Impostor syndrome is another. I have noticed that academics often think that people in industry have a much better understanding of the realities of their field. Those who have had more contact with people from industry might have noticed that impostor syndrome is not limited to academia.

Talking of impostor syndrome, and feeling of being a fraud, academics don’t seem to know how to handle direct criticism. Again I think it is a matter of practice. Industry does not operate according to: I won’t laugh at your idea, if you don’t laugh at mine, which means people within industry are practiced at ‘robust’ discussion (this does not mean they like it, and being good at handling such discussions smooths the path into management).

At the other end of the impostor spectrum, some academics really do regard people working in industry as simpletons. I regularly have academics express surprise that somebody in industry, i.e., me, knows about this-that-or-the-other. My standard reply is to say that its because I paid more for my degree and did not have the usual labotomy before graduating. Not a reply guaranteed to improve industry/academic relations, but I enjoy the look on their faces (and I don’t expect they express that opinion again to anyone else from industry).

The other reason why I don’t recommend attending academic conferences/workshops, is that lots of background knowledge is needed to understand what is being said. There is no point attending ‘cold’, you will not understand what is being presented (academic presentations tend to be much better organized than those given by people in industry, so don’t blame the speaker). Lots of reading is required. The point of attending is to talk to people, which means knowing something about the current state of research in their area of interest. Attending simply to learn something about a new topic is a very poor use of time (unless the purpose is to burnish your c.v.).

Why do I continue to attend conferences/workshops?

If a conference/workshop looks like it will be attended by people who I will find interesting, and it’s not too much hassle to attend, then I’m willing to go in search of gold nuggets. One gold nugget per day is a good return on investment.

Posted on

Feeling Isolated

Chris Oldwood from The OldWood Thing

By and large I think I’ve been fairly lucky with my time as a contract programmer. Virtually all the teams I’ve worked in and systems I’ve worked on have been pretty decent. None of them are going to change the world but they’ve been enjoyable, which is probably why I’ve ended up working on them for a decent length of time [1].

I can only say “virtually all” because one contract sadly fell way short of the mark. Although I was technically part of a team it only really felt that way from a managerial perspective, even though we shared a codebase. I felt somewhat isolated both physically and mentally. Aside from the morning stand-up I could easily have gone the rest of the day without speaking to my teammates if I had chosen to do so.

Physical Isolation

I started the contract on a separate floor from the rest of my team with a couple of other recent joiners [2]. We were the only people on that floor with the air conditioning on full blast so we had to wear our coats in the afternoon to stay warm. None of the rest of my team had an office pass that could access the floor either, should they want to talk face-to-face while getting us up to speed.

Even when they moved us onto the same floor a month later we were still on the opposite side of the room. In the next desk shuffle I got to swap colleagues although they were working on an entirely separate area of the system with a totally different bunch of people so we had little need to collaborate per-se, only to make small talk. Also the two desks next to me only seemed to be used for a game of Tower of Hanoi by the office movers given how the occupants came and went.

Even my “customer”, at least, the one I knew about, because they were paying for the project, was situated in a different country and spoke a different language. Although their English was way better than any knowledge I have of a second language I quickly discovered why most communication was via email or IM instead of vocally.

Project Isolation

Being an enterprise scale organisation the work was all about projects, and who was sponsoring how many “resources”. Nowhere was this more apparent than the Scrum Board with its project-oriented swim-lanes. Each swim-lane had the names of the team members assigned to that project, and as the stand-up proceeded it walked down the board a project at a time with each member of the sub-team providing an update.

It was fairly apparent right from the moment I started, just by reading the body language of the team members, that there was often little real interest in what the rest of the team was doing. Those that did, cut across projects to some degree because they tended to nurse the build system, deployments and monitoring. A couple of team members never attended our stand-up because they already attended a different one that encompassed their project.

To be fair some of the apathy at the stand-up was almost certainly down to its excessive length. And with little reason for attending except to provide a status update for the managers it’s no surprise those mostly on the periphery zoned out. Sometimes the only common goal of the team seemed to be to not break the system.

Code Isolation

During my short stint I effectively had one feature to work on. There were a couple of other minor tweaks to begin with but ultimately my project was one feature (nay, user story) and it took 5 months to deliver. That one feature involved making a change in an area of the codebase that nobody else knew except one of the tech leads who I soon discovered was leaving. In fact, taking away his days off after the announcement of his departure, I effectively had 3 days for any handover.

Not only were there no docs to work from there were no tests either. The only real knowledge about how any of the service was expected to behave had left firmly inside the head of the author. This pretty much just left doing a spot of software archaeology with the VCS in the hope that the commit messages might contain some extra clues. Many features had been tracked in a feature tracking tool but there were not enough licenses to go round so I had to hassle a teammate to look things up. Even then it often wasn’t worth it as there were no useful details; it felt like the ticket was just there to “tick a box”.

The code relied heavily on the caller “doing the right thing” so any understanding only made sense if you already knew what the caller was supposed to do, and that relied heavily on knowledge of the problem domain and the organisation’s other systems. (At the interview I made it perfectly clear that I still knew little about the problem domain, despite the many years I have worked in it [3].)

Methodology Isolation

Ever since I had my epiphany [4] around testing all those years ago I have become a firm believer in TDD and automated testing as the preferred approach to the sustainable delivery of quality software. Being told early in the project that “you won’t have time to write tests”, despite being asked in the interview about what your approach is, did not bode well.

It soon became apparent that the previous approach had been to rush something out and rely on manual, end-to-end testing and the customer doing things “right”. Validation was almost entirely left to the underlying maths library and so bizarre errors manifested and needed investigating by the developers due to a lack of basic error handling and reporting [5].

With no way of knowing if I had broken anything, because I didn’t know for sure what anything was supposed to do, my only recourse was to write new code with tests and then refactor later when someone (potentially me) could be sure that it was safe to do so. For existing code that I had to change or understand I would write a barrage of tests first to try and ensure I didn’t accidentally break anything. In some cases it was hard to know what was “by design” and what was “by accident”.

Clearly not everyone took this approach, as you can see in “It Compiles, Ship It!”. My pessimism paid off though once the edge cases and little extras started appearing as I could turn around a fix or improvement (safely) in minutes due to my suite of automated unit and regression tests.

Environment Isolation

Sadly, despite my ability to push through changes quickly into the integration test environment, it still took weeks for them to actually appear in the production environment. When my first task, a handful of lines of boilerplate code, took 6 weeks to make it into production I assumed continuous delivery was not something they cared about.

On the contrary, for one aspect of the business, releases were very frequent. It was just that I was on the other side and due to some (IMHO) poor architecture and deployment decisions my part of the distributed system was tightly-coupled to another (major) system’s release cycle.

While it might seem great having my own integration test environment to play with, I ran into issues no one else knew about and I had no idea who was really using it and for what. Once again that information pretty much departed with the author.

Parting Thoughts

On reflection I have to look at my own behaviour first and ask myself whether I was at least partly responsible for feeling left out. Once we moved onto the same floor it was definitely easier to wander over and ask people questions, which I did. However when the response is “well I worked all this out by myself originally” and “that’s more than anyone ever gave me” I think it’s not entirely unfair to assume that knowledge sharing isn’t high on some people’s agenda.

I believe I was as welcoming as I normally am and was happy to help out where possible, given the limited knowledge I had acquired. I guess that culturally there was such a large drive for autonomy that the idea of just chatting about stuff to see what improvements in the system or process would be beneficial just wasn’t on the cards. A couple of times what should have been a constructive comment or question definitely came out of me more as a snide remark which is never a good sign. I’ve been trying hard to be more aware of any sarcasm, which unfortunately comes all too easily to me, and so not add to any unnecessary negativity but I know I failed a few times.

Ultimately I think it says a lot about an organisation that rejects your approach because “they are not a start-up” when your application of that approach has only ever been in large enterprises and none of them has ever had an issue with it before. On the contrary they have often been grateful for the insights and improvements that I’ve brought.

Maybe if I was a lot younger I’d not have known any better and stuck it out a bit more but these days I know it’s just not worth the effort. I feel comfortable that I left the place in a better state than I joined it by documenting various things and writing tests for the code I wrote. After a slightly rocky start my customer seemed pretty pleased with everything I delivered, which I guess is largely what matters most.

As ever, my main regret is leaving behind some people that I wish I could have gotten to know better. Maybe I will, in another life, one where the benefits of collaboration are more positively encouraged.

 

[1] Mostly my tenure has been measured in years, not months.

[2] Only one of which was left when I called it a day – the other two barely lasted a month or so.

[3] See “Problem Domain Expert or Technical Expert or Even Both” for more on this recurring theme.

[4] See “My [Unit] Testing Epiphany” and my more recent ACCU / Agile on the Beach talk “A Test of Strength” for what lead to my enlightenment.

[5] Poor error messages is a popular topic of mine, see “Terse Exception Messages”. Also “The Perils of DateTime.Parse()” covers one specific example.

Posted on

Practical ecosystem books for software engineers

Derek Jones from The Shape of Code

So you have read my (draft) book on evidence-based software engineering and want to learn more about ecosystems. What books do I suggest?

Biologists have been studying ecosystems for a long time, and more recently social scientists have been investigating cultural ecosystems. Many of the books written in these fields are oriented towards solving differential equations and are rather subject specific.

The study of software ecosystems has been something of a niche topic for a long time. Problems for researchers have included gaining access to ecosystems and the seeming proliferation of distinct ecosystems. The state of ecosystem research in software engineering is rudimentary; historians are starting to piece together what has happened.

Most software ecosystems are not even close to being in what might be considered a steady state. Eventually most software will be really old, and this will be considered normal (“Shock Of The Old: Technology and Global History since 1900″ by Edgerton; newness is a marketing ploy to get people to buy stuff). In the meantime, I have concentrated on the study of ecosystems in a state of change.

Understanding ecosystems is about understanding how the interaction of participant’s motivation, evolves the environment in which they operate.

“Modern Principles of Economics” by Cowen and Tabarrok, is a very readable introduction to economics. Economics might be thought of as a study of the consequences of optimizing the motivation of maximizing return on investment. “Principles of Corporate Finance” by Brealey and Myers, focuses on the topic in its title.

“The Control Revolution: Technological and Economic Origins of the Information Society” by Beniger: the ecosystems in which software ecosystems coexist and their motivations.

“Evolutionary dynamics: exploring the equations of life” by Nowak, is a readable mathematical introduction to the subject given in the title.

“Mathematical Models of Social Evolution: A Guide for the Perplexed” by McElreath and Boyd, is another readable mathematical introduction, but focusing on social evolution.

“Social Learning: An Introduction to Mechanisms, Methods, and Models” by Hoppitt and Laland: developers learn from each other and from their own experience. What are the trade-offs for the viability of an ecosystem that preferentially contains people with specific ways of learning?

“Robustness and evolvability in living systems” by Wagner, survival analysis of systems built from components (DNA in this case). Rather specialised.

Books with a connection to technology ecosystems.

“Increasing returns and path dependence in the economy” by Arthur, is now a classic, containing all the basic ideas.

“The red queen among organizations” by Barnett, includes a chapter on computer manufacturers (has promised me data, but busy right now).

“Information Foraging Theory: Adaptive Interaction with Information” by Pirolli, is an application of ecosystem know-how, i.e., how best to find information within a given environment. Rather specialised.

“How Buildings Learn: What Happens After They’re Built” by Brand, yes building are changed just like software and the changes are just as messy and expensive.

Several good books have probably been omitted, because I failed to spot them sitting on the shelf. Suggestions for books covering topics I have missed welcome, or your own preferences.

Posted on

Crack the motivation code! at the nor(DEV):con 2019 Thursday Workshops!

Paul Grenyer from Paul Grenyer


Join one of our interactive workshops at nor(DEV):con 2019 on Thursday 21st of February from 9am, Get your tickets here: http://nordevcon-2019.eventbrite.com

Crack the motivation code! - Cassandra Andrews

Imagine if you knew precisely what motivated each member of your team, how motivated they were and what you could do to improve their motivation!
Cassandra Andrews interactive workshop focuses on motivation in the workplace and introduces ‘motivational maps’, an incredibly accurate and user-friendly tool which enables us to unlock and measure employee motivation.

Understanding exactly what motivates individuals in an organisation can be used with significant impact to support business growth and profitability by maximising employee and team motivation, retain employees and recruit the right people to complement existing teams.

At the workshop you will discover how to create high performing teams by learning:

  • What motivation is, how it can be measured and how it impacts the workforce.
  • About the nine motivators identified in motivational maps.
  • How to increase team performance by identifying conflicts in motivation.

To enable all delegates to get their own ‘wow’ moment from the session, when you register for this event you will receive a complimentary motivational map (worth £150) for completion prior to the event.


Day of Deliberate Practice - Jon Jagger.

During the session we will discuss and practice software craftsmanship, communication and creativity, focusing on TDD, pair programming, trying out new things and revisiting old ones. Deliberate practice involves careful repetition and reflection to master, explore and understand a technique, to know its boundaries, to make it second nature.

  • Hands on workshops & lectures
  • Shared Learning Environment
  • Effective coding habits
  • Collaborative Development

If you code every day, keep in mind that a change is as good as a rest — yes, there will be coding, but it’s not your workaday coding. We will be using cyber-dojo, an innovative, collaborative, open source environment. We are going to have fun — join us!

  • Test Driven Development – TDD
  • Pair & Mob Programming
  • Cyber-dojo

With time to look at different code, and code differently, this session will open your mind to different ways of doing things and new ways of approaching old problems with great rewards. Using Jon Jagger’s innovative cyber-dojo environment, you will get to re-examine coding habits and debate modern practices with experts and peers.

This session is open to individuals and teams of all abilities, the only pre-requisite is some experience of software development and a desire to learn:

  • All programmers who want to step back and examine how they can be better
  • Junior programmers who are looking to master their craft
  • Experienced developers who want to reflect on their own practices and hone their skills

Bring a laptop – wifi and access to http://cyber-dojo.org/ required.


Real time analysis of streaming data with Hadoop and Druid - Tom Barber

In this workshop we will build and deploy a Hadoop cluster along with Apache Druid which is an analytical database designed for high volume time series data. We will then work with the attendees to stream data into Druid, process that data and return it for real time analysis. This type of workflow is useful for log file analysis, intrusion detection, sensor analysis or similar and very relevant to businesses of today. All of the software used is open source and free to deploy, attendees will leave with a high level understanding of the various technologies used and a framework that can be deployed quickly and easily in the real world.

Level: Intermediate