Rob's Notes 15: Metrics Traps and Cognitive Biases

How we go to surprising lengths to justify our past decisions rather than admit mistakes & what it means for metrics

I’m going back through some old writings. A decade ago I read a book that had a big impact on me. In "Mistakes Were Made (But Not By Me),", Carol Tavris and Elliot Aronson explore how cognitive dissonance shapes our decision-making. Their central insight—that we will go to surprising lengths to justify our past decisions rather than admit mistakes—has profound implications in many spheres of life, including on how businesses measure and manage performance.

Tavris and Aronson describe cognitive dissonance as that uncomfortable feeling when we encounter information that contradicts our self-image or prior choices. As they note, "The engine that drives self-justification is the need to reduce dissonance between conflicting cognitions." In business, this manifests most powerfully in our choice of metrics.

When faced with evidence that our strategy might be failing, we have two options:

1. Acknowledge the mistake and change course (painful)

2. Find metrics that justify our current path (easier)

This dynamic creates what we might call "The Metrics Trap": selecting measurements that confirm our biases while dismissing those that challenge our assumptions.

The Pyramid of Choice

Tavris and Aronson describe how initial small decisions can lead down a "pyramid of choice," where each subsequent decision serves to justify the previous ones. In metrics, this often looks like:

1. Choosing an initial metric that supports our strategy

2. Building reporting systems around that metric

3. Creating incentive structures based on that measurement

4. Dismissing contradictory data as "noise"

5. Eventually reshaping the entire organization around potentially flawed measurements

Nokia's descent from market leader to obsolescence illustrates this pyramid perfectly. Each decision to focus on traditional phone metrics (market share, hardware margins) made it harder to acknowledge the fundamental shift in how people used mobile devices. As Tavris and Aronson write, "Self-justification is not the same thing as lying or making excuses... It's more like a protective shield that allows us to sleep at night."

Breaking Free from the Metrics Trap

1. Implement Metric Balancing

Create dashboards that pair growth metrics with corresponding health metrics:

• User growth alongside user retention

• Revenue growth alongside unit economics

• Feature shipping velocity alongside feature adoption

• Sales bookings alongside implementation success

2. Challenge Metric Selection

Regularly review which metrics drive major decisions:

• Question whether current metrics truly reflect business health

• Look for blind spots in measurement frameworks

• Create forums where metric selection can be debated

• Reward teams for surfacing concerning metrics

3. Build Reality-Check Mechanisms

Institute processes that force confrontation with uncomfortable data:

• Monthly review of "warning signal" metrics

• Regular audit of dismissed or downplayed measurements

• Cross-functional metric review sessions

• Customer feedback integration into metric selection

As we saw earlier, Nokia focused on market share metrics while missing fundamental shifts in user experience preferences. But there are plenty of other examples: 

• Blockbuster emphasized store revenue while ignoring early warning signs in customer behavior trends

• Yahoo tracked portal engagement while missing the larger shift toward search-driven discovery

• Kodak measured film sales growth while dismissing digital photography adoption metrics

Wells Fargo's account scandal represents perhaps the most dramatic example of metrics gone wrong in banking history. The bank's intense focus on "new accounts per customer" as a key metric led to the creation of an elaborate cross-selling machine. Management built detailed reporting systems and incentive structures around account creation, celebrating and rewarding aggressive cross-selling while systematically downplaying or ignoring fraud warnings. The pressure to maintain growth in this metric eventually led to the creation of millions of unauthorized accounts. Each layer of success metrics made it increasingly difficult for executives to question the fundamental strategy, leading to a massive scandal that severely damaged the bank's reputation and resulted in billions in fines. Wells Fargo by some measures is one of the 3 big companies that Americans like the least.

Each of these companies fell victim to Tavris/Aronson’s "confirmation bias squared"—not only seeking information that confirmed their existing beliefs but actively creating measurement systems that supported those beliefs.

Moving Beyond Self-Justification

The goal isn't to eliminate bias—that's impossible—but to build systems that help us recognize and correct for it. In metrics, this means:

1. Acknowledge the Bias:

   • Accept that metric selection is inherently subjective

   • Regularly question whether current metrics serve the business

   • Create processes for surfacing and discussing metric bias

2. Balance the Dashboard:

   • Pair growth metrics with health metrics

   • Include both leading and lagging indicators

   • Measure both positive and potentially negative trends

3. Create Safety for Truth:

   • Reward identification of concerning metrics

   • Build processes for regular metric review

   • Create forums where metric selection can be challenged

Understanding cognitive dissonance and self-justification allows us to build better measurement systems. By acknowledging our natural tendency to justify past decisions, we can create frameworks that help us see both comfortable and uncomfortable truths about our performance.

The most dangerous lies are the ones we tell ourselves. By building systems that help us confront these self-deceptions, we can create more resilient and adaptable organizations. Rather than viewing metrics as absolute truth, we should see them as tools for learning—tools that are most valuable when they challenge our assumptions rather than confirm them. As Tavris and Aronson remind us, "The ability to admit mistakes... is a strength, not a weakness."