Replication: not always worth the effort

Derek Jones from The Shape of Code

Replication is the means by which mistakes get corrected in science. A researcher does an experiment and gets a particular result, but unknown to them one or more unmeasured factors (or just chance) had a significant impact. Another researcher does the same experiment and fails to get the same results, and eventually many experiments later people have figured out what is going on and what the actual answer is.

In practice replication has become a low status activity, journals want to publish papers containing new results, not papers backing up or refuting the results of previously published papers. The dearth of replication has led to questions being raised about large swathes of published results. Most journals only published papers that contain positive results, i.e., something was shown to some level of statistical significance; only publishing positive results produces publication bias (there have been calls for journals that publishes negative results).

Sometimes, repeating an experiment does not seem worth the effort. One such example is: An Explicit Strategy to Scaffold Novice Program Tracing. It looks like the authors ran a proper experiment and did everything they are supposed to do; but, I think the reason that got a positive result was luck.

The experiment involved 24 subjects and these were randomly assigned to one of two groups. Looking at the results (figures 4 and 5), it appears that two of the subjects had much lower ability that the other subjects (the authors did discuss the performance of these two subjects). Both of these subjects were assigned to the control group (there is a 25% chance of this happening, but nobody knew what the situation was until the experiment was run), pulling down the average of the control, making the other (strategy) group appear to show an improvement (i.e., the teaching strategy improved student performance).

Had one, or both, low performers been assigned to the other (strategy) group, no experimental effect would have shown up in the results, significantly reducing the probability that the paper would have been accepted for publication.

Why did the authors submit the paper for publication? Well, academic performance is based on papers published (quality of journal they appear in, number of citations, etc), a positive result is reason enough to submit for publication. The researchers did what they have been incentivized to do.

I hope the authors of the paper continue with their experiments. Life is full of chance effects and the only way to get a solid result is to keep on trying.

ACCU Conference 2018

Products, the Universe and Everything from Products, the Universe and Everything

We had an absolute blast at this year's ACCU Conference and if you were there we imagine you did too.

For us the highlight had to be the launch of #include <C++>, a new global, inclusive, and diverse community for developers interested in C++.

@include_cpp logo
The #include <C++> community logo.

This is an initiative that has been brewing for a while, and one we are very happy to be a part of. Above all else #include <C++> is designed to be a safe place for developers irrespective of their background, ethnicity, gender identity or sexuality. The group runs a Discord server which is moderated to ensure that it remains a safe space and which you are welcome to join.

On the technical front, one unexpected highlight of the conference was Benjamin Missel's wonderful short talk on writing a C compiler for a BBC Micro, during which he demonstrated SSHing into a BBC Model B through the serial port!



Most conference sessions were recorded so even if you weren't there you can still watch them.

See you at ACCU 2019!

ACCU Conference 2018

Products, the Universe and Everything from Products, the Universe and Everything

We had an absolute blast at this year's ACCU Conference, and if you were there we imagine you did too.

For us the highlight had to be the launch of #include <C++>, a new global, inclusive, and diverse community for developers interested in C++.

#include C++ logo

This is an initiative that has been brewing for a while, and we're very happy to be a part of. Above all else #include <C++> is designed to be a safe place for developers irrespective of their background, ethnicity, gender identity or sexuality. The group runs a Discord server which is moderated to ensure that it remains a safe space and which you are welcome to join.


On the technical front, one unexpected highlight of the conference was Benjamin Missel's wonderful short talk on writing a C compiler for a BBC Micro, during which he demonstrated SSHing into a BBC Model B through the serial port!

Most conference sessions were recorded so even if you weren't there you can still watch them.

See you at ACCU 2019!

ACCU Conference 2018

Products, the Universe and Everything from Products, the Universe and Everything

We had an absolute blast at this year's ACCU Conference, and if you were there we imagine you did too.

For us the highlight had to be the launch of #include <C++>, a new global, inclusive, and diverse community for developers interested in C++.

#include C++ logo

This is an initiative that has been brewing for a while, and we're very happy to be a part of. Above all else #include <C++> is designed to be a safe place for developers irrespective of their background, ethnicity, gender identity or sexuality. The group runs a Discord server which is moderated to ensure that it remains a safe space and which you are welcome to join.


On the technical front, one unexpected highlight of the conference was Benjamin Missel's wonderful short talk on writing a C compiler for a BBC Micro, during which he demonstrated SSHing into a BBC Model B through the serial port!

Most conference sessions were recorded so even if you weren't there you can still watch them.

See you at ACCU 2019!

ACCU Conference 2018

Products, the Universe and Everything from Products, the Universe and Everything

We had an absolute blast at this year's ACCU Conference, and if you were there we imagine you did too. For us the highlight had to be the launch of #include <C++>, a new global, inclusive, and diverse community for developers interested in C++.
#include C++ logo
This is an initiative that has been brewing for a while, and we're very happy to be a part of. Above all else #include <C++> is designed to be a safe place for developers irrespective of their background, ethnicity, gender identity or sexuality. The group runs a Discord server which is moderated to ensure that it remains a safe space and which you are welcome to join.
On the technical front, one unexpected highlight of the conference was Benjamin Missel's wonderful short talk on writing a C compiler for a BBC Micro, during which he demonstrated SSHing into a BBC Model B through the serial port! Most conference sessions were recorded so even if you weren't there you can still watch them. See you at ACCU 2019!

ResOrg 2.0.7.27 has been released

Products, the Universe and Everything from Products, the Universe and Everything

ResOrg 2.0.7.27 has just been released. This is a maintenance update for ResOrg 2.0, and is compatible with all ResOrg 2.0 licence keys.

The following changes are included:

  • Fixed a bug in the Symbols Display which could cause some "OK" symbols to be incorrectly shown in the "Problem Symbols Only" view.

  • Corrected the upper range limit for control symbols from 28671 (0x6FFF) to 57343 (0xDFFF).

  • ResOrg binaries are now dual signed with both SHA1 and SHA256.

  • Added support for Visual Studio 2017.

  • Corrected the File Save Dialog filters used by the ResOrgApp "File | Export" command.

  • The ResOrgApp "File | Export", "File | Save", "File | Save As" and "File | Properties" commands (which apply only to symbol file views) are now disabled when the active view is a report.

  • Fixed a crash in the Symbol File Properties Dialog.

  • Fixed a typo on the Symbol File "Next Values" page.

  • Various minor improvements to the installer.

Download ResOrg 2.0.7.27

ResOrg 2.0.7.27 has been released

Products, the Universe and Everything from Products, the Universe and Everything

ResOrg 2.0.7.27 has just been released. This is a maintenance update for ResOrg 2.0, and is compatible with all ResOrg 2.0 licence keys.

The following changes are included:

  • Fixed a bug in the Symbols Display which could cause some "OK" symbols to be incorrectly shown in the "Problem Symbols Only" view.
  • Corrected the upper range limit for control symbols from 28671 (0x6FFF) to 57343 (0xDFFF).
  • ResOrg binaries are now dual signed with both SHA1 and SHA256.
  • Added support for Visual Studio 2017.
  • Corrected the File Save Dialog filters used by the ResOrgApp "File | Export" command.
  • The ResOrgApp "File | Export", "File | Save", "File | Save As" and "File | Properties" commands (which apply only to symbol file views) are now disabled when the active view is a report.
  • Fixed a crash in the Symbol File Properties Dialog.
  • Fixed a typo on the Symbol File "Next Values" page.
  • Various minor improvements to the installer.

Download ResOrg 2.0.7.27

ResOrg 2.0.7.27 has been released

Products, the Universe and Everything from Products, the Universe and Everything

ResOrg 2.0.7.27 has just been released. This is a maintenance update for ResOrg 2.0, and is compatible with all ResOrg 2.0 licence keys.

The following changes are included:

  • Fixed a bug in the Symbols Display which could cause some "OK" symbols to be incorrectly shown in the "Problem Symbols Only" view.
  • Corrected the upper range limit for control symbols from 28671 (0x6FFF) to 57343 (0xDFFF).
  • ResOrg binaries are now dual signed with both SHA1 and SHA256.
  • Added support for Visual Studio 2017.
  • Corrected the File Save Dialog filters used by the ResOrgApp "File | Export" command.
  • The ResOrgApp "File | Export", "File | Save", "File | Save As" and "File | Properties" commands (which apply only to symbol file views) are now disabled when the active view is a report.
  • Fixed a crash in the Symbol File Properties Dialog.
  • Fixed a typo on the Symbol File "Next Values" page.
  • Various minor improvements to the installer.

Download ResOrg 2.0.7.27