ROCking Innovation

My engineering thesis was all about diagnosing irregular heartbeats. The problem almost seems easy – I was given data for about 110,000 heartbeats, each one already diagnosed by a team of doctors as either ‘normal’ or ‘irregular’. The aim was to create a computer program that correctly identifies each type.

It turns out there’s a really simple way to do this without any knowledge of computers or medicine. Diagnose everything as normal, and you’ll be 100% correct on the normal beats. There are far more normal beats than irregular ones, so you end up with a fabulous success rate of almost 80%!

The slight drawback is that you’re 100% guaranteed to be wrong on the irregular beats. In an unfortunate coincidence, doctors and patients seem to care more about the not-so-normal ones. Quite a lot more, in some cases.

It’s not only about how well you identify successful outcomes, but you also have to look at how well you reject potential failures. For the visually minded (or anyone with a consulting background), this can be conveniently illustrated in a 2×2 matrix:

If you measure at least two of the boxes (usually the ticks), it becomes clear that this simple system actually adds no value. In other words, the success rate is simply the same as the percentage of normal beats. Which looks great if that percentage is high, and terrible if it’s low.


So how does this apply to your innovation agenda? At some point you will have to choose between competing projects, advancing some and rejecting others. Ideally you only want to take on projects that are likely to be successful. It is common to measure success rates, because that’s what drives business growth.

A well designed selection process also has a good rejection rate. If you only measure success, you run the risk that the selection process hasn’t added any value. In fact, it could be limiting potential by rejecting really great projects. You need to be measuring rejection rates as well to get the full picture. Yes, correctly identifying bad ideas isn’t as profitable as picking winners. But it does mean you know what doesn’t work. Which can be valuable down the track.

ROC Curves

It turns out that during WWII they faced a very similar problem using radar to identify enemy aircraft. Plotting successes and failures generates what is now known as a ROC curve. A straight (dotted) line means that the system didn’t actually tell you anything new. The curved (blue) line tells you that the system is much better, correctly selecting many of the successes, and also rejecting many of the failures.

Clever mathematicians even came up with a single number to tell you exactly how much better the blue line is than the dotted one, but in the interests of being able to communicate with a wide audience (i.e. people other than clever mathematicians) it’s probably better to stick to two percentages for now.

Measuring successful outcomes is the starting point for selling innovation as the driver of business growth. That allows you to point to hard financial benefits and also gives you great stories to tell. Even though you might not want to highlight them as loudly, measuring how well you identify potential failure is just as important if you want to understand how to improve innovation outcomes over time.

Taking a more balanced approach also allows your personal brand to expand beyond the relentless sales pitch that is so often required to champion innovation. Shifting your brand towards a ‘considered expert’ not only helps increase your reach, but it ultimately allows your ideas to be accepted by a much broader audience.