Sir Arthur Eddington, an English astrophysicist, told a short story involving a scientist studying fish by pulling them up with nets. After checking all the fish hauled up, the scientist concludes that there is a minimum size of fish in the sea. But the fish seen were determined by the size of the holes in the net, the smaller ones having slipped through, unmeasurable. The instrument you use affects what you see. Or as Richard Hamming puts it: "You get what you measure."

This analogy provides a nice concrete example of a phenomena that affects us routinely in more subtle ways. What and how we choose to measure affects the conclusions we draw. So, a website may easily measure sales and bounce rate for its pages, while things like trust, authority or satisfaction, which may be more significant longer-term metrics, go unmeasured.

"There is always a tendency to grab the hard, firm measurement, though it may be quite irrelevant as compared to the soft one which in the long run may be much more relevant to your goals. Accuracy of measurement tends to get confused with relevance of measurement, much more than most people believe. That a measurement is accurate, reproducible, and easy to make does not mean it should be done, instead a much poorer one which is more closely related to your goals may be much more preferable. For example, in school it is easy to measure training and hard to measure education, and hence you tend to see on final exams an emphasis on the training part and a great neglect of the education part."

What gets measured gets better (an earlier take on this)

Quote from Chapter 29, You get what you measure, of Richard Hamming's, The Art of Doing Science and Engineering: Learning to Learn.

If you happen to be in the business of doing research or science Richard Hamming's Bell Lab's talk, You and Your Research (pdf), is an excellent read.

You get what you measure

There are lots of questions you can ask of data. I like this model as it gives a framework for the different types of questions you might ask. It was introduced to me as the data science pyramid with descriptive analytics — measuring what’s happening — being the foundation on which others are built. You don’t have to go in order, but it’s a nice approach to consider how the questions you’re asking of your data might evolve.

To take our school Summer fête as an example, for a long time takings were all in cash and put into one bucket at the end for counting. You end up with one number for how much you raised. But you might move to collecting more descriptive data on the takings for each stall — whether the bar earns more than the tombola, for example. You then might be able to ask diagnostic questions like figuring out why one fête earned more than another with a different mix of stalls. If you know how many people came you might be able to make some predictions on the takings of a future fête based on number of people and stall types. And if you get fancy you might start to use your data to figure out how you could raise more money for the school in future.

There are many ways to slice and dice data and the questions you ask, but I like the clarifying simplicity of this one.

As far as I know, the framework is the Analytics Ascendancy model, or Analytics Value Escalator, or other such business sounding name from Gartner.

Analytics maturity

Pitched as the one number you need to grow your business, the Net Promoter Score (NPS) is a way to find out from your customers how many love what you’re doing enough to tell their friends vs how many are ambivalent or negative about your business. The resulting number, your NPS, ranging from +100 to -100 has been shown to be an indicator of companies’ future success.

To calculate your NPS ask your customers how likely they would be to recommend your business to a friend—or variations thereof—and give them a scale of 0 (not likely) to 10 (very likely) to answer with. Those who answer 9 or 10 are your biggest fans or Promoters. Those who answer 7 or 8 are passively satisfied—they like your business but are unlikely to shout about it. And those answering 6 or below are your Detractors. If they talk about your business it could well be negative. Your NPS is your % of Promoters minus the % of your Detractors. For example, if you received 100 responses, 35 as 9s or 10s, 37 as 7s and 8s, and 28 as 0–6s then your NPS would be 35 minus 28 = 7.

Keep track of your NPS over time and you’ll get a sense of how your business is doing. Follow up with qualitative feedback, for instance, “Why did you give this rating?”, and get a recipe for improving your business. Simple.

Learn more in the original Harvard Business Review article, “The One Number You Need to Grow” by Frederick F. Reichheld.

Net Promoter Score

Kind of a Goodhart’s law with more explanation, Campbell’s law, rather pessimistically suggests:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

It pretty much seems to work with everything, unfortunately. If you measure the performance of an employment office by the number of cases handled you may get quick but ineffective placements and interviews. If, instead, you measure them on number of persons placed in jobs you may unintentionally drive a focus on placing the simplest cases at the expense of those who most need the service.

When customer support is measured by number of tickets dealt with I’ve heard of writing quick answers that don’t fully solve a question in the knowledge that the customer will soon come back with another ticket to solve.

When policing effectiveness is judged by crimes reported and the number of unsolved cases you may get both under reporting of crimes, and a form of plea bargaining where a criminal may be encouraged to confess to extra crimes — therefore fewer unsolved cases — in exchange for more lenient sentences. Campbell notes that:

It seems to be well documented that a well-publicized, deliberate effort at social change–-Nixon’s crackdown on crime–-had as its main effect the corruption of crime-rate indicators, achieved through underrecording and by downgrading the crimes to less serious classifications.

Measuring schools on attendance encourages creative reasons for children being taken out of school. Assessing learning can quickly become teaching to, and cramming for, the test. When reviews come to dominate purchase decisions pressure increases for fake reviews.

And then there’s elections and voting…

Campbell, Donald, Assessing the Impact of Planned Social Change, Dec 1976. (pdf)

HT: Benjamin Baker who shared it with me when I posted about Goodhart. Also, for alternative sides to what gets measured gets better there’s Seth Godin’s take on Campbell’s law: Speedometer confusion, and Scott Carter’s pithier paraphrasing of Campbell’s law: “What gets measured, gets corrupted.”

Campbell’s law

The story goes something like this. Back in colonial India the top Brit in charge decided there were too many cobras around Delhi. To reduce the population they put in place a cash reward, or bounty, for anyone who brought in a dead cobra. The intention was clear. Legend has it that people did bring in the cobras reliably because some enterprising souls had started breeding cobras for the very purpose of getting the bounty. When the authorities realised this they scrapped the scheme, the cobra farms closed and the bred cobras were released into the wild significantly increasing the cobra population by a few orders of magnitude. Hence, the cobra effect: when a well-intentioned measure can have the opposite effect to that desired.

It’s not clear how true this story is, but the cobra effect has been demonstrated many times. The Freakonomics podcast on the cobra effect discusses some excellent examples (with references), including:

A bounty on rats in French colonial Hanoi — where the bounty was earned by bringing in a rat tail — which went the same way as the cobra story

Removing wild pigs at Fort Benning, Georgia, at $40 a pigs tail that didn’t make a dent. It could be the hunters would bait the pigs with all sorts of tempting scraps. Many pigs learned to avoid the hunters and instead bred like crazy while well-fed on scraps.

The capital of Colombia, Bogotá, tried to reduce traffic congestion by restricting each car from driving one day during the week. The result: people started buying more than 1 car and did more driving on weekends.

Publication of proposed areas of habitat to be protected for endangered species meant land developers responded by building in those areas before the land was protected.

People are smart. Incentives work, in the sense that you tend to get what you incentivise. So be careful what you are incentivising.

Also see: Goodhart’s Law and What gets measured gets better.

The cobra effect

This is the fascinating observation that it’s not straightforward to say how long a coastline is. If you were to measure the coastline of a country by using a ruler on a globe, you would come out with a vastly different number than if you were to pace around the edge. The closer you look, the more wiggles and squiggliness you come across and instead of converging on a more accurate length, the coastline just keeps getting longer. The smaller your ruler, the longer it gets.

This was originally spotted, incredibly, in the 1950s, by an Englishman, Lewis Richardson, when trying to check a theory he had that the likelihood of war between countries depended on the length of their shared borders. Remarkably, he found that the quoted lengths of borders varied significantly. While measuring on maps at different scales he saw that the smaller scale map he used, or the smaller the width of his calipers he was measuring with, the length systematically increased. When looking at coastlines, instead of borders, some countries had wigglier coasts and so the length increased at a faster rate with the scale — for instance, Norway’s coastline, with it’s crinkly fjords, increases faster than Britain’s, which in turn increases faster than South Africa’s, as he zoomed in. The rate of this increase later became known as its fractal dimension.

Long after Richardson’s research, Benoit Mandelbrot published a paper How Long Is the Coast of Britain? Statistical Self-Similarity and Fractional Dimension that discussed how the wiggliness of something like a coastline at one scale can be repeated at smaller and smaller scales. The work led to the later term fractals. Many other things exhibit fractal-like behaviour such as river networks, borders, brains, frequencies, lightning or even the stock market.

More mindbending mapping, and the difference between Great Britain, the UK and the British Isles.

There’s a super section on this in Scale, by Geoffrey West.

The coastline paradox

In other words, if you pick a measure to assess people’s performance, then we find a way to game it. I like the illustration of a nail factory that sets number of nails produced as their measure of productivity and the workers figure out they can get tons of tiny nails. And, if they switch it to the weight of nails made, they get a few giant heavy nails. Or perhaps the story of measuring fitness by steps from a pedometer only to find they get put on the dog.

Some strategies for helping this are to try and find better, harder to game measures, assess with multiple measures, or to allow a little discretion. More detail in this nice little article.

I also liked an idea I read in Measure What Matters of pairing a quantity measure with a quality measure, for example, assessing both number of nails and customer satisfaction of the nails.

Goodhart’s Law: when a measure becomes a target, it cease to be a good measure.

The clarity of the night sky. A low number means clear, black and amazing stars. Higher numbers mean gradually more urban environments with increasing light pollution. At 9 you’ll barely spot a star being pretty much in Piccadilly Circus or Times Square.

More detail needed? John Bortle, Light Pollution And Astronomy: The Bortle Dark-Sky Scale, Sky and Telescope, July 18 2006.

Bortle Scale

The length of your arms relative to your height. Typically:

Da Vinci’s classic Vitruvian man has an arm span equal to his height and therefore an ape index of 1. An ape index greater than 1 (arm span longer than height) seems to be helpful for rock climbers, swimmers (Michael Phelps = 1.06), boxers, professional basketball and goalkeeping.

It’s also known as the gorilla index though I couldn’t actually find a measure for gorillas remarkably. A quick calculation shows for an average male gorilla at 1.75m tall and arm span of 2.45m that the gorilla index for a real gorilla would be = 2.45 / 1.75 = 1.4. I’d definitely fancy the gorilla in rock-climbing and boxing…

It’s sometimes calculated as arm span minus height, that would give gorillas a ridiculous: +70

I have not been able to figure out if your ape index changes from a child to adult though it seems like it might.

Ape index

A microlife is thirty minutes of your life expectancy. You can use it up by chilling for 30 minutes or you could start to speed through them by smoking a few cigarettes or having a few drinks.

A simple way to bring difficult-to-grasp long-term, slow-burn risks into a unit that can motivate you and a handy way to compare risks.

Even better, you gain a microlife with 2-3 cups of coffee. Win!

Interesting? See Microlives: A lesson in risk taking, by Cambridge University Professor of risk David Spiegelhalter. Or see his 15 min TED talk: Motorbikes, Terrorism, Heart Attacks, Sausages: Professor David Spiegelhalter at TEDxOxbridge, or the full paper.

Microlife: The unit of life

A micromort is a one in one million chance of sudden death during the activity. You get a day of regular living for one micromort (though clearly that probably depends on what your average day is). Turns out you get a lot further per micromort by car, and that motorways are by far the safest form of road. You’ll experience 7 micromorts from running a marathon and 39,427 from climbing Everest.

Introduced by Stanford Professor, Ronald Howard.

Micromort: The unit of death

Tricky topics - validity trickier than reliability. At it’s core I believe reliability is about getting the same results given the same conditions and validity is about measuring what you intend to measure. Harder than it sounds.

Understanding reliability and validity

In both a “great! we’re improving kind of way,” and also a “teaching to the test” kind of way. From Richard Hamming’s, The Art of Doing Science and Engineering.

Sketchplanations tagged with measurement