Completion Isn't Contribution

I think the list of things you shipped this month is the most flattering page in any marketing report, and one of the least useful.

You know the page. The campaign went out on time. The new landing page is live. Twelve posts published, two emails sent, the pricing test called with a winner. Every line has a check mark next to it.

None of those check marks tells you whether anything changed for the customer. They tell you the work got finished, and sometimes that it passed a test. Those feel like the same thing. They aren't.

You've probably read the post about dashboards lying before. This isn't quite that one. The part it usually skips is why a test can pass honestly and still point you the wrong way, and why the people who built the work are the worst placed to notice.

Completion isn't contribution. A test that passes only tells you about the test.

The clearest case I know of a team getting that wrong is one almost everyone has heard of. Most people just remember the wrong half.

April 23, 1985

On April 23, 1985, the Coca-Cola Company changed the formula of Coca-Cola. The drink was almost a hundred years old. The company announced a sweeter version and stopped making the original.

People were furious. The company took calls of protest by the thousands. Shoppers stocked up on the old cans. On July 11, 79 days after the announcement, Coca-Cola brought the original back and called it Coca-Cola Classic.

That's the version most people know. The obvious reading is that the company made a worse drink and got punished for it.

It won

It didn't make a worse drink, at least not by the measure it used.

Before the launch, Coca-Cola ran taste tests with nearly 200,000 consumers. The new formula beat the original Coke. It also beat Pepsi. By the company's own test, it was the better cola.

There was a warning in the data. When testers were told the new flavor would replace Coke, about 10 to 12 percent said they were angry about it and might stop drinking Coke altogether. That finding got downplayed.

The test had flaws of its own. By one account, only about 20 percent of the taste tests used the final formula. But even a perfect taste test measures one thing, which is how people react to a sip. But nobody buys a sip of Coke. They buy a drink they've had their whole lives, from a brand that meant something to them. The taste test couldn't see any of that, so it scored the new formula a winner.

And to be fair to Coca-Cola, once the product was out, they did listen. The calls came in, the company read them, and it reversed in under three months. That part worked. The expensive mistake came earlier, when a passing score on a narrow test was allowed to stand in for the question that actually mattered: would people still want this?

That's the trap. The test was supposed to tell them whether customers would like the new Coke. Somewhere along the way, winning the test became the goal. When a measure turns into the target, it stops telling you much about the thing it was meant to track.

I wrote about a cousin of this in the Perception Engine: research answers the questions you know how to ask, not always the ones your customer is living with.

Why the people who built it can't see it

Think about the last launch you defended in a meeting. How much of your confidence came from what customers did afterward, and how much came from the fact that it was done?

There's a reason that question is uncomfortable, and it has to do with the people who made the work.

In 2012, three researchers, Michael Norton, Daniel Mochon, and Dan Ariely, ran a set of studies where people built things: IKEA boxes, origami, Lego sets. Then they asked the builders what their creations were worth. People valued their own amateur work about as highly as work made by experts, and they expected other people to see it the same way.

The detail that stays with me is when that extra love showed up. It only appeared when people finished. When participants built something and then took it apart, or never got to complete it, the effect went away.

So the love doesn't come from the effort. It comes from finishing.

The studies measured individual people with their own projects. Nobody has shown the same thing happens to a whole team, and I don't want to pretend otherwise. But every team is made of people who each built their own piece. The copywriter wrote the page. The analyst designed the test. The strategist picked the angle. Each of them finished something, and each of them has a reason to like how it turned out.

"Shipping fast is how you learn"

This is where someone pushes back, and they have a point.

Speed matters. You learn more from a real launch than from another month of planning. A lot of teams would be better off shipping more and debating less.

But shipping only teaches you something if something comes back. The launch is the question. What customers do afterward is the answer. If nothing is set up to catch that answer, shipping fast is just guessing fast. You get more check marks and the same amount of knowledge.

There's a line I keep coming back to here: fix the loop before you fix the product. If the path from what customers do to what the team decides is broken, a better product won't help, because nobody will find out it's better.

The man who built the dashboard

You've probably sat in a meeting where a dashboard was the argument. The numbers were green, so the work was good, and nobody asked what the numbers were measuring.

The clearest warning about that I know came from hospitals, not marketing. In 1966, a physician named Avedis Donabedian published a way to judge the quality of medical care. Look at the structure, the process, and the outcomes. Before that, there wasn't a common way to tell whether a hospital was doing right by its patients. His framework became the backbone of how health care measures itself.

Then the measuring kept growing, and that part stays with me. More metrics, more reports, more frameworks built on top of frameworks. Over time a lot of the effort went into checking the process, and the plain question underneath, whether this patient is getting good care, got harder to see.

In an interview he gave shortly before he died in 2000, published the next year, Donabedian said, "Ultimately, the secret of quality is love." And then: "If you have love, you can then work backward to monitor and improve the system."

The man who built the dashboard put the dashboard second. The process measures were never the point. The patient was.

That's the same gap as the taste test and the check marks. A dashboard can tell me a piece performed. It can't tell me why it mattered to the person who needed it. If I let the measurable stand in for the meaningful, I'll end up doing work that scores well and means nothing, and I'll have a chart to prove it.

Name what the test can't see

I don't have a launch of my own to walk you through here, and I'm not going to invent one. What I can give you is the check I'd run on the next one.

Before you trust a passing result, write down three things.

What the test actually measured. Be literal. "The variant got more clicks on Start Trial" is a measurement. "The new page works" isn't.

What the test can't see. Every test has a boundary. A click test can't see whether the people clicking ever pay. A survey can't see what people do when nobody's asking. A taste test can't see a hundred years of habit.

The one customer behavior that would tell you it's working, and when you'll check it. Not a new metric to report. One thing a customer does. For the pricing page, that might be how many of those new trials turn into paying accounts after thirty days. For a campaign, it might be whether the people who responded come back and buy again.

Then put that date on the calendar and look.

This isn't meant to be another report. It's one line added to the report you already send, and one behavior per launch, not a new dashboard. If it turns into more measuring, it's become the problem it was supposed to catch.

If you can, have someone who didn't build the thing do the looking. It's not that the team can't be trusted. It's that finishing it is exactly what makes it hard to see clearly.

This won't have Coca-Cola's drama. Your pricing page isn't a national scandal, and a weak campaign doesn't get you on the evening news. The drama doesn't transfer. The trap does. Most of the time, a passing test is standing in for a question nobody wrote down.

I call this the Shipping Illusion. It's a name for where teams end up, not an explanation of every way they get there. It's still worth having a name for it, because once you can name it you can check for it.

Finishing the work was never the hard part. Knowing whether it did anything is.

~ Charles