Confidently Wrong

I don't trust how sure I feel anymore. I have the numbers for why.

For a long time I treated certainty the way most of us do. If I felt strongly about a call, that feeling counted as information. The surer I was, the less I thought I needed to check.

Then I wrote my calls down, and the record told me something my own head never had. You can't see the shape of your own blind spot by looking harder. You need something outside your head to show it to you.

"We know our customer"

You've probably been in the room where this plays out.

A team is planning the next campaign. Someone has been with the account for years. They know the customer, they say, and nobody argues, because they've been right before. The research readout comes back and it matches what everyone expected. The plan goes ahead with a lot of confidence behind it.

It feels like the experienced people in the room see what the rest can't. That's what experience is supposed to buy you.

Most of the time nobody checks whether that confidence has earned its place. There's no record to check it against.

Six months of writing it down

So I made one. For six months I tracked my decisions. For each one I noted what I decided, how confident I felt, and what I predicted would happen. Then I went back and checked.

My "90% confident" calls were right 58% of the time.

The pattern was clear once I could see it. I was overconfident on decisions about people and underconfident on technical ones. I rushed choices I could have reversed, and I agonized over ones that barely mattered.

None of this was visible while I was living it. All of it was obvious in the journal.

That's the part that stayed with me. I wasn't careless, and I wasn't lying to myself on purpose. The feeling of being sure was real every time. It just wasn't measuring what I thought it was measuring.

It also isn't just me. In 1977, Sarah Lichtenstein and Baruch Fischhoff asked people general-knowledge questions, had them pick an answer, and asked how sure they were. People who said they were 65 to 70 percent sure were right about half the time, and the harder the questions got, the bigger the gap. The same year, the two of them and Paul Slovic ran five studies with 528 paid volunteers. People were consistently overconfident, and sure enough to bet money on it.

That result has been repeated for decades. What my journal added was the part the studies can't give you: which of my calls it applied to. That's the useful part, and you only get it from your own record.

Looking right at it

You've probably heard of the invisible gorilla. In 1999, psychologists asked people to watch a short video of students passing a basketball and to count the passes. Partway through, a person in a gorilla suit walks through the middle of the game. A lot of viewers never see it.

In 2013, researchers ran a version of it on radiologists, people whose whole job is to look at medical scans and find the thing that shouldn't be there. They asked 24 radiologists to search five lung CT scans for nodules, the small spots that can be early cancer, and tracked their eyes while they did it. Each radiologist had up to three minutes per scan, scrolling through the slices the way they would at work.

In the last scan, the researchers inserted a picture of a gorilla, 48 times the size of the average nodule. It faded in and out over five slices, placed right next to a nodule, so anyone checking that nodule had it in view.

Twenty of the 24 radiologists didn't report it.

The eye tracking showed something worse. Of the 20 who missed it, 12 looked directly at the spot where the gorilla was. On average, their eyes rested on it for a little over half a second. It never registered.

And it wasn't hard to see. After the test, when the researchers asked whether they'd noticed anything unusual and showed them the scan, all 24 saw the gorilla.

This isn't a one-off, either. The same paper points to an earlier study where roughly 60% of radiologists failed to notice that a chest x-ray was missing a collarbone.

These are experts doing exactly what they're trained to do. The researchers don't settle why the gorilla got missed. My read is that the training that tells a radiologist where to look for cancer also shapes what doesn't get noticed, and nothing tells them what it left out.

Why the readout agrees with you

My journal and those scans aren't the same kind of mistake. One is about how sure I felt. The other is about what a trained eye let through. What they share is a filter that decides what reaches you and never tells you what it dropped.

Now go back to the room where the team knows its customer.

What follows is my read, not something those studies measured. Going from one person's blind spot to a whole team's is an analogy, and I want to be clear about that.

The clearest public version of this I know didn't come from marketing. In October 2008, in the middle of the financial crisis, Alan Greenspan sat in front of a congressional committee. He had run the Federal Reserve for nearly two decades. A congressman asked whether his ideology had pushed him into decisions he regretted.

Greenspan said an ideology "is a conceptual framework with the way people deal with reality. Everyone has one." Then he said he'd found a flaw in it, though he didn't know "how significant or permanent it is." The part that stays with me is what came next: "I was shocked, because I had been going for 40 years or more with very considerable evidence that it was working exceptionally well."

Forty years of evidence, and the evidence was real. The way I read it, the framework was also deciding what counted as evidence. Four decades of confirmation made it feel more solid, right up to the point where it broke.

Here's what I'd expect to happen on a marketing team. The team's picture of the customer decides which questions get asked. The research gets built around those questions, so the answers come back in the shape of the picture. The readout agrees with the team, and the team gets more certain.

You can see how it happens without anyone doing anything wrong. The survey offers answer choices the team wrote, so the customer picks from the team's categories. The interviews go to the customers the team already knows how to reach. The segments in the readout are the segments from last year's plan. Every piece of that is reasonable. Together, they make it hard for the research to say something the team hasn't already thought.

Each round of agreement makes the picture tighter. It doesn't make it more right. Certainty tells you how tightly your picture holds together. It doesn't tell you whether it matches the customer.

"Isn't experience the whole point?"

This is where someone with twenty years in the business pushes back, and they should.

Experience does matter. In the same 2013 study, the researchers also tested 24 people with no medical training. None of them saw the gorilla. The radiologists did a little better, and they were far better at the task they'd actually trained for, which was finding nodules.

So the same study is doing two jobs here, and both are true. Experience sharpens the work it was built for. It doesn't show you the frame it was built inside.

My own record cut both ways, too. I was overconfident about people and underconfident about technical calls. Writing things down didn't only knock me down a peg. It told me where I'd been too timid.

Write it down before you know

You're taking my word for a record you can't see. The post I wrote at the time didn't say how many 90% calls there were, and I won't guess now. I know how that sounds in a piece arguing that a record beats a feeling. It's the best reason I can give you to keep your own.

It doesn't need to be fancy. Before the result is in, write down what you expect and how sure you are. Then check back when the answer arrives.

Here's what that looks like for a research readout. The day before it lands, write three things you expect it to say. Put a number next to each: how sure you are, as a percentage. Add one line on what result would make you change the plan. When the readout arrives, mark each call right, wrong, or didn't come up. Don't explain the misses yet. Just mark them.

Do the same before a launch. Write the number you expect and the range you'd bet on. Write it where you can't quietly edit it later.

It isn't free. It takes a few minutes before every readout and every launch, the payoff takes months, and a wrong call on paper is harder to shrug off than one you only felt. I'd keep it to yourself at first. A record other people are watching tends to fill up with safe numbers.

After twenty or thirty of these, sort them by how sure you were. Look at the calls you marked 90 percent. Were they right nine times out of ten? Mine weren't.

Twenty or thirty calls is still a small pile, spread across different levels of confidence. Read it for the direction of the gap, not a verdict, and keep going.

One miss won't tell you much. The pattern across twenty or thirty of them will. That's where the blind spot shows up, in the calls you were surest about.

I wrote about a version of this in Proof of Transfer: a document with nothing to check it against makes people stop asking questions. The same is true of your own judgment.

I still trust a record over a confident veteran. Including when the veteran is me.

The thing that shows you your blind spot was never going to be your own eyes. It's a page you wrote before you knew the answer.

~ Charles