Object oriented hierarchy? – a postscript

Allan Kelly from Allan Kelly Associates

1143px-CPT-Structured_Chart_Example.svg-2020-01-31-12-05.png

My last post on hierarchy generated a fair bit of interest – both online and offline. Once it is pointed out people recognise there is a positive side to hierarchy, it is not so much “hierarchy bad, no hierarchy good” as it is “how much hierarchy?” and the culture around that hierarchy.

But there is something else I have to say, something that has been bugging me for years. Look at the picture above, it is a structure chart, this one from Wikipedia (public domain). I suspect few people below the age of 40 have seen one, this is the way I was taught to design software at University. A structure chart is like a flowchart but horizontal, not vertical.

Look familiar?

A structure chart looks a lot like a hierarchy chart. Compare the picture above with the (fantasy) organization chart on my previous post.

Structure charts are used when doing structured programming, a technique used in procedural programming. So Pascal, Modula-2, Algol, even C. In this world (experienced) programmers spend a lot of time thinking (and worrying) about layering. They aim for layered systems, these are drawn as dependency diagrams which – can you guess? – look a lot like hierarchies too.

While all of these techniques still have relevance, in our modern world things have changed. Primarily, procedural programming has given way to object-oriented programming. In OOP the object is the thing: the object is an idea, code and data reside in the object. We build our systems with self contained objects (well ideally).

Sound familiar?

I like to talk about Amoeba teams, similar ideas emerge under names such as Feature teams, or stand-alone, or even Spotify Squad. The repeated idea is that teams have a purpose and contain the people and skills they need to pursue that purpose.

In the 1970s and 80s we had procedural programming, structure charts and hierarchies. The organizational form of hierarchy matched our programming model. In the 1990s we switched to object-oriented programming and now our organizations are playing catch up in switching to “object oriented teams” (if I can coin a new expression, which naturally abbreviates to OOT).

Conway’s Law strikes again!

OOP changes how you design software, it also changes how organizations structure themselves. While OOP changes the way programmes are structured and reduces the way programs are layered (and dependencies managed) it doesn’t do away with a structure. There is still a framework in place. Similarly, OOTs don’t exist in isolation, they need to exist in a framework – a hierarchy of sorts.

(And of course modern UI models and micro-services means main() isn’t always the top of the program any more!)

Conway’s Law implies that a hierarchical organization will adopt procedural programming, which was true in the 60s and 70s. New companies, start-ups, born in the OOP age natural structure as OOTs. Existing companies first see tension because the two models rub against one another.

Then reverse Conway’s Law (Yawnoc as it is sometimes called) would suggest companies move towards OOT – which of course we see with all the big companies adopting “Spotify.”

Which raises the question:

If stand-alone/self-contained teams, and reduced hierarchy, are the organizational structure which parallels object-oriented design and programming… what is the organizational form that parallels functional programming? How do you structure your teams when you are working in Lisp, Haskell or F# ?

What about concurrent (parallel) languages like Occam?
What organizational structure parallels message passing systems?
Or Data flow architecture? And quantum computing?


Like this post? – Like to receive these posts by e-mail? XanpanNewlite-2020-01-31-12-05.jpgXanpan

Subscribe to my newsletter & receive a free eBook “Xanpan: Team Centric Agile Software Development”

The post Object oriented hierarchy? – a postscript appeared first on Allan Kelly Associates.

How are C functions different from Java methods?

Derek Jones from The Shape of Code

According to the right plot below, most of the code in a C program resides in functions containing between 5-25 lines, while most of the code in Java programs resides in methods containing one line (code+data; data kindly supplied by Davy Landman):

Number of C/Java functions of a given length and percentage of code in these functions.

The left plot shows the number of functions/methods containing a given number of lines, the right plot shows the total number of lines (as a percentage of all lines measured) contained in functions/methods of a given length (6.3 million functions and 17.6 million methods).

Perhaps all those 1-line Java methods are really complicated. In C, most lines contain a few tokens, as seen below (code+data):

Number of lines containing a given number of C tokens.

I don’t have any characters/tokens per line data for Java.

Is Java code mostly getters and setters?

I wonder what pattern C++ will follow, i.e., C-like, Java-like, or something else? If you have data for other languages, please send me a copy.

How useful are automatically generated compiler tests?

Derek Jones from The Shape of Code

Over the last decade, testing compilers using automatically generated source code has been a popular research topic (for those working in the compiler field; Csmith kicked off this interest). Compilers are large complicated programs, and they will always contain mistakes that lead to faults being experienced. Previous posts of mine have raised two issues on the use of automatically generated tests: a financial issue (i.e., fixing reported faults costs money {most of the work on gcc and llvm is done by people working for large companies}, and is intended to benefit users not researchers seeking bragging rights for their latest paper), and applicability issue (i.e., human written code has particular characteristics and unless automatically generated code has very similar characteristics the mistakes it finds are unlikely to commonly occur in practice).

My claim that mistakes in compilers found by automatically generated code are unlikely to be the kind of mistakes that often lead to a fault in the compilation of human written code is based on the observations (I don’t have any experimental evidence): the characteristics of automatically generated source is very different from human written code (I know this from measurements of lots of code), and this difference results in parts of the compiler that are infrequently executed by human written code being more frequently executed (increasing the likelihood of a mistake being uncovered; an observation based on my years working on compilers).

An interesting new paper, Compiler Fuzzing: How Much Does It Matter?, investigated the extent to which fault experiences produced by automatically generated source are representative of fault experiences produced by human written code. The first author of the paper, Michaël Marcozzi, gave a talk about this work at the Papers We Love workshop last Sunday (videos available).

The question was attacked head on. The researchers instrumented the code in the LLVM compiler that was modified to fix 45 reported faults (27 from four fuzzing tools, 10 from human written code, and 8 from a formal verifier); the following is an example of instrumented code:

warn ("Fixing patch reached");
if (Not.isPowerOf2()) {
   if (!(C-> getValue().isPowerOf2()  // Check needed to fix fault
         && Not != C->getValue())) {
      warn("Fault possibly triggered");
   } else { /* CODE TRANSFORMATION */ } } // Original, unfixed code

The instrumented compiler was used to build 309 Debian packages (around 10 million lines of C/C++). The output from the builds were (possibly miscompiled) built versions of the packages, and log files (from which information could be extracted on the number of times the fixing patches were reached, and the number of cases where the check needed to fix the fault was triggered).

Each built package was then checked using its respective test suite; a package built from miscompiled code may successfully pass its test suite.

A bitwise compare was run on the program executables generated by the unfixed and fixed compilers.

The following (taken from Marcozzi’s slides) shows the percentage of packages where the fixing patch was reached during the build, the percentages of packages where code added to fix a fault was triggered, the percentage where a different binary was generated, and the percentages of packages where a failure was detected when running each package’s tests (0.01% is one failure):

Percentage of packages where patched code was reached during builds, and packages with failing tests.

The takeaway from the above figure is that many packages are affected by the coding mistakes that have been fixed, but that most package test suites are not affected by the miscompilations.

To find out whether there is a difference, in terms of impact on Debian packages, between faults reported in human and automatically generated code, we need to compare number of occurrences of “Fault possibly triggered”. The table below shows the break-down by the detector of the coding mistake (i.e., Human and each of the automated tools used), and the number of fixed faults they contributed to the analysis.

Human, Csmith and EMI each contributed 10-faults to the analysis. The fixes for the 10-fault reports in human generated code were triggered 593 times when building the 309 Debian packages, while each of the 10 Csmith and EMI fixes were triggered 1,043 and 948 times respectively; a lot more than the Human triggers :-O. There are also a lot more bitwise compare differences for the non-Human fault-fixes.

Detector  Faults   Reached    Triggered   Bitwise-diff   Tests failed
Human       10      1,990         593         56              1
Csmith      10      2,482       1,043        318              0
EMI         10      2,424         948        151              1
Orange       5        293          35          8              0
yarpgen      2        608         257          0              0
Alive        8      1,059         327        172              0

Is the difference due to a few packages being very different from the rest?

The table below breaks things down by each of the 10-reported faults from the three Detectors.

Ok, two Human fault-fix locations are never reached when compiling the Debian packages (which is a bit odd), but when the locations are reached they are just not triggering the fault conditions as often as the automatic cases.

Detector   Reached    Triggered
Human
              300       278
              301         0
              305         0
                0         0
                0         0
              133        44
              286       231
              229         0
              259        40
               77         0
Csmith
              306         2
              301       118
              297       291
              284         1
              143         6
              291       286
              125       125
              245         3
              285        16
              205       205
EMI      
              130         0
              307       221
              302       195
              281        32
              175         5
              122         0
              300       295
              297       215
              306       191
              287        10

It looks like I am not only wrong, but that fault experiences from automatically generated source are more (not less) likely to occur in human written code (than fault experiences produced by human written code).

This is odd. At best I would expect fault experiences from human and automatically generated code to have the same characteristics.

Ideas and suggestions welcome.

Converting HTML slides to a PDF with Firefox

Andy Balaam from Andy Balaam's Blog

I have not found an automated way to generate a nice PDF from some slides written in HTML – if you know of one please add a comment!

In the meantime, if you create slides using my HTML Slides template, then you can make a decent-ish-looking PDF like this:

  1. View your slides in Firefox and open the Print dialog (press Ctrl-p).
  2. Select Print to File and choose a filename to save to.
  3. Under Options deselect “Ignore Scaling…” and select “–blank–” for all the headers and footers. Ensure “Print Background Colours” and “Print Background Images” are selected.
  4. Under Page Setup set Scale to 70.0 and Orientation to Landscape.
  5. Select the dropdown next to Paper size and choose Manage Custom Sizes…
  6. Create a new custom size called “Screen 16:9” with Width 157.5 mm and Height 280.0 mm. Set all the Paper Margins to 5.0 mm. You can modify the name of the custom size by clicking on it in the list on the left.
    Click Close.
  7. Back in Page Setup, make sure your new custom size is selected next to Paper Size.
  8. Click Print.

If that does not work well for you, try experimenting with different Scale settings.

Support the Software Freedom Conservancy

Andy Balaam from Andy Balaam's Blog

The Software Freedom Conservancy helps Free/Open Source software projects by providing infrastructure, financial structures, and legal help. It is a not-for-profit organisation that is dedicated to software freedom, something that I think is an important prerequisite for a decent world in the future.

Conservancy looks after lots of projects, including these ones that I personally use: Boost, BusyBox, Etherpad, Git, Godot Engine, Inkscape, phpMyAdmin, QEMU, Racket, Samba, Selenium, Squeak and Wine.

It provides an easy way for projects to accept donations, hold assets and negotiate contracts, as well as mentoring and legal advice. It also leads the difficult and thankless work to persuade (and force) companies to comply with the GPL (a Free Software license). Without this work a great deal more Free Software would be distributed without the required access to source code, meaning it is not free at all.

Further, it is a key supporter of Outreachy, which does important work to address the under-representation, systemic bias, and discrimination in the technology industry.

I encourage you to join me and support the Software Freedom Conservancy.

You might also like to listen to the Free As In Freedom podcast.

Cargo Culting GitFlow

Chris Oldwood from The OldWood Thing

A few years back I got to spend a couple of weeks consulting at a small company involved in the production of smart cards. My team had been brought in by the company’s management to cast our critical eye over their software development process and provide a report on what we found along with any recommendations on how it could be improved.

The company only had a few developers and while the hardware side of the business seemed to be running pretty smoothly the software side was seriously lacking. To give you some indication of how bad things used to be, they weren’t even using version control for their source code. Effectively when a new customer came on board they would find the most recent and relevant existing customer’s version (stored in a .zip file), copy their version of the system, and then start hacking out a new one just for the new customer.

As you can imagine in a set-up like this, if a bug is found it would need to be fixed in every version and therefore it only gets fixed if a customer noticed and reported it. This led to more divergence. Also as the software usually went in a kiosk the hardware and OS out in the wild was often ancient (Windows 2000 in some cases) [1].

When I say “how bad things used to be” this was some months before we started our investigation. The company had already brought in a previous consultant to do an “Agile Transformation” and they had recognised these issues and made a number of very sensible recommendations, like introducing version control, automated builds, unit testing, more collaboration with the business, etc.

However, we didn’t think they looked too hard at the way the team were actually working and only addressed the low hanging fruit by using whatever they found in their copy of The Agile Transformation Playbook™, e.g. Scrum. Naturally we weren’t there at the time but through the course of our conversations with the team it became apparent that a cookie-cutter approach had been prescribed despite it being (in our opinion) far too heavyweight for the handful of people in the team.

As the title of this post suggests, and the one choice I found particularly amusing, was the introduction of VSTS (Visual Studio Team Services; rebranded Azure DevOps) and a GitFlow style workflow for the development team. While I applaud the introduction of version control and isolated, repeatable builds to the company, this feels like another heavyweight choice. The fact that they were already using Visual Studio and writing their web service in C# probably means it’s not that surprising if you wanted to pick a Big Iron product.

The real kicker though was the choice of a GitFlow style workflow for the new product team where there were only two developers – one for the front-end and another for the back-end. They were using feature branches and pull requests despite the fact that they were the only people working in their codebase. While the company might have hired another developer at some point in the future they had no immediate plans to to grow the team to any significant size [2] so there would never be any merge conflicts to resolve in the short to medium term! Their project was a greenfield one to create a configurable product instead of the many one-offs to date, so they had no regressions to worry about at this point either – it was all about learning and building a prototype.

It’s entirely possible the previous consultant was working on different information to us but there was nothing in our conversations with the team or management that suggested they previously had different goals to what they were asking from us now. Sadly this is all too common an occurrence – a company hires an agile coach or consultant who may know how to handle the transformation from the business end [3] but they don’t really know the technical side. Adopting an agile mindset requires the XP technical practices too to be successful and so, unless the transformation team really knows its development onions, the practices are going to be rolled out and applied with a cargo cult mentality instead of being taught in a way that the team understands which practices are most pertinent to their situation and why.

In contrast, the plan we put forward was to strip out much of the fat and focus on making it easy to develop something which could be easily demo’d to the stakeholders for rapid feedback. We also proposed putting someone who was “more developer than scrum master” into the team for a short period so they could really grok the XP practices and see why they matter. (This was something I personally pushed quite hard for because I’ve seen how this has played out before when you’re not hands-on, see “The Importance of Leading by Example”.)

 

[1] Luckily these kiosks weren’t connected to a network; upgrades were a site visit with a USB stick.

[2] Sadly there were cultural reasons for this – a topic for another day.

[3] This is debatable but I’m trying to be generous here as my expertise is mostly on the technical side of the fence.

New Zealand, Australia and Fresh ideas

Allan Kelly from Allan Kelly Associates

FreshNZ-2020-01-21-11-41.jpg

Before this weeks regular post (In defence of hierarchy) a commercial, by me, for me 🙂

In March I’m going on tour, I’m speaking at Agile on the Beach New Zealand. I hope to call into Australia on the way, and I’ve been invited down to Wellington too.

My itinerary is not confirmed yet and ideally I’ll delivery some more talks – and charge some fees to make the trip more sensible. So if you know anyone in Australia or New Zealand who would like me to deliver some training or presentation please put them in touch, allan@allankelly.net.

Which makes it a great time to introduce my Fresh Ideas workshops.

These are short (2 to 3 hours) workshops which expand on ideas you will find in my conference presentations and books. So far these are:

  • Value and the story: this includes an estimation poker exercise which is a lot of fun
  • Continuous Digital: ideal for regular readers of this blog who want to share these ideas with their colleagues.
  • Strategic capacity planning: this is largely an exercise extracted from my Strategic Product Owner course and builds on experiences with capacity planning

I’m also including two old favourites under here, maybe not entirely fresh but they fit the bill in other ways:

  • Better User Stories: extracted from my Requirements, User Stories and Backlogs workshop
  • Agile refresher: this features in my agile training and has been run a number of times as a stand alone exercise, it is a lot of fun and a way to start a conversation about getting better

As always, if you are interested in any of these get in touch – especially if you are in New Zealand or Australia.

The post New Zealand, Australia and Fresh ideas appeared first on Allan Kelly Associates.

In defence of hierarchy

Allan Kelly from Allan Kelly Associates

Heirarchy-2020-01-21-13-16.jpg

Hierarchy, it is one of those topics which provokes a reaction.

There are many in the agile community who believe hierarchy is a bad thing. Teams – and whole organizations are better off without hierarchy. It is simply(!) a case of finding better ways of organizing which don’t involve hierarchy.

Then there are those who acknowledge that hierarchy has been with us for millennia and find it hard to imagine how anyone could organise without it.

As a general rule the supporters of agile come down against hierarchy. I’ve been known to rail against hierarchy myself but at the same time I’ve long had my doubts that it is possible to remove hierarchy altogether. Rather I aim for less hierarchy and greater independence rather than the abolition of all hierarchy. I tend to keep this view to myself because a) it is subtle and b) I suspect I’ll loose a lot of agile street cred if people know.

So when I heard about a book called Hierarchy by John Child I immediately bought a copy. This is no easy read – Child is a Professor and so the style is academic rather than pop-psychology business blockbuster. So far I’m about half way through but the book raises many interested point that I’m still thinking through.

In particular Child highlights a number of benefits of hierarchy which deserve attention and I think are too often overlooked. Right now I want to capture my thoughts so far before I get into the arguments against hierarchy. So…

The benefits of hierarchy to an organization:

  • Hierarchy has benefits were there is regulation
  • Hierarchy helps large enterprises bring structure to their activities
  • Hierarchy tend to develop in all societies whether they are intended or not, therefore imposing a hierarchy gives an organization a chance both to impose the order they want – and choose the leaders – and rather than accepting the hierarchy and leaders that emerge.
  • Hierarchy allows for succession planning
  • Reduce illegal behaviour: maybe not a big issue in your twenty-first century office but this can be an issue. It can also be a particular issue when people rise to the top of a hierarchy and find the organizational controls they had before are gone: they have nobody to report to. Could this explain some of the sexual and business mis-conduct that has been in the media in recent times?

Benefits to the individual:

  • Hierarchy creates psychological safety, individuals know where they fit in and what is expected of them. They know who they need to pay attention to.
  • Hierarchy reduces cognitive load and fear, in part because of the psychological safety it creates.
  • Hierarchy provides a career model: people know what advancement in the organization looks like and those who want advancement can map out a course to achieve that.

Benefits to a team:

  • Hierarchy creates clear areas of responsibility which enables team members and creates focus within the team.
  • Hierarchy provides team members with a structures and helps them accept their position. On the whole people are accepting of hierarchy and accept the position. (Perhaps because they also know how they can advance their position.)
  • Hierarchy allows a mix of strong and weak personalities to work together. While hierarchy can be used by powerful formal leads to suppress weaker staff hierarchy cuts both ways. Team decision making can be improved when hierarchy facilities equitable discussion between strong personalities.
  • Formal hierarchy, with formal leaders, actually puts a responsibility on the leaders to ensure balance and allow weaker personalities to have their say. (While this can be abused when it is abused it should be clear to all concerned that the leader is not upholding their part of the bargain.)
  • In a recognised hierarchy the senior people have a responsibility to moderate and listen to all. Where there is no hierarchy few may feel that responsibility and a single strong voice may dominate the weak.
  • Where there is no hierarchy and multiple strong personalities the result can be conflict and disagreement. Without the hierarchy it can be hard to resolve such problems.
  • Hierarchy provides for decision makers, perhaps one, perhaps more. Having recognised decision makers can make for rapid decisions: people know who to go to and that person knows they have responsibility. Conversely, where decisions are made by consensus decision making can be slow or absent entirely.

Recently I observed a team which made most decisions by consensus. But as one strong personality rarely agreed with the others any decision they didn’t agree with became a battle field. Most team members kept quiet and nothing changed. Sometimes another strong personality would try and force the decision through but this was usually unsuccessful. Only when several other team members were prepared to take sides. As a result consensus became a recipe for not changing.

Of these arguments the ones I find most interesting are those which concern the emergence of social hierarchy. I’ve certainly seen this – even in organizations with a formal hierarchy. Emergent leadership and hierarchy can be a good thing. They can also be a bad thing.

I can immediately think of several teams I’ve worked with were one of the developers – sometimes a relatively junior one at that – emerge as leaders and the team adopted a hierarchy oriented towards that leader. On one occasion that leader was me and I like to think I brought an order and structure which was beneficial. But I’ve seen other occasions were the leader who emerged was at odds with others in the organization with the result of tension.

I expect the idea of emergent hierarchy is immediately recognisable to those schooled in emergent design and behaviour. The question becomes: should organizations accept this? Or should they try to guide it?

In the extreme should an organization fight an emergent hierarchy which conflicts with the aims and goals of the organization? And is it worth the effort?

So far I have more questions than answers. I haven’t suddenly become an advocate for hierarchy but I now recognised this is not a one sided discussion. I plan to write another blog when I get to the end of the book and have had a chance to think through the for and against arguments.

In the meantime I’d love to hear your thoughts and comments, just add them below.


Like this post? – Like to receive these posts by e-mail?

Subscribe to my newsletter & receive a free eBook “Xanpan: Team Centric Agile Software Development”

The post In defence of hierarchy appeared first on Allan Kelly Associates.

Insomnium – Norwich

Paul Grenyer from Paul Grenyer

Something my first proper girlfriend said to me has stuck with me my entire life as I disagree with it (mostly).  She said that the best way to discover a new band was to see them live first. The reason I disagree is because I get most pleasure from knowing the music I am listening to live - most of the time.

I’m a member of the Bloodstock Rock Society and their Facebook page is often a place of band discussion. Lots of people there were saying how good Insomnium are, but they didn’t do a great deal for me when I listened to them on Spotify. Then it was early 2020, I hadn’t been to a gig since Shakespears Sister in November, I fancied a night out and Insomnium were playing in Norwich. So I took a chance….

From the off they were great live and I really enjoyed it. I came to the conclusion that I must like some of their older stuff as it was the new album which hadn’t done much for me. There were lots of things I like, like widdly guitars, metal riffs and blast beats, but what really lets Insomnium down is the vocals. Death metal vocals, to a certain extent, are death metal vocals, but this guy sounded like he was singing a different song in a different band - it’s the same on the album I tried. If the vocals were more suited to the music, like there are with Wintersun, it would be even better. I also learned that Norwich City’s current start player is from the same town in Finland as the band.

The first thing I did this morning was look up which albums the setlist was from an make a list:

  • One for Sorrow
  • Across the Dark
  • Above the Weeping World
  • Shadows of the Dying Sun
  • Heart like a Grave

And then die a little inside at the prices on Amazon and eBay. I think I’ll be playing a lot of Insomnium on Spotify for the time being so I’m ready to enjoy them to the full next time.



On Two By Two – student

student from thus spake a.k.

The Baron's most recent wager with Sir R----- set him the challenge of being the last to remove a horizontally, vertically or diagonally adjacent pair of draughts from a five by five square of them, with the Baron first taking a single draught and Sir R----- and he thereafter taking turns to remove such pairs.

When I heard these rules I was reminded of the game of Cram and could see that, just like it, the key to figuring the outcome is to recognise that the Baron could always have kept the remaining draughts in a state of symmetry, thereby ensuring that however Sir R----- had chosen he shall subsequently have been free to make a symmetrically opposing choice.