Rob's Notes 15: Metrics Traps and Cognitive Biases
How we go to surprising lengths to justify our past decisions rather than admit mistakes & what it means for metrics
I’m going back through some old writings. A decade ago I read a book that had a big impact on me. In "Mistakes Were Made (But Not By Me),", Carol Tavris and Elliot Aronson explore how cognitive dissonance shapes our decision-making. Their central insight—that we will go to surprising lengths to justify our past decisions rather than admit mistakes—has profound implications in many spheres of life, including on how businesses measure and manage performance.
Tavris and Aronson describe cognitive dissonance as that uncomfortable feeling when we encounter information that contradicts our self-image or prior choices. As they note, "The engine that drives self-justification is the need to reduce dissonance between conflicting cognitions." In business, this manifests most powerfully in our choice of metrics.
When faced with evidence that our strategy might be failing, we have two options:
Acknowledge the mistake and change course (painful)
Find metrics that justify our current path (easier)
This dynamic creates what we might call "The Metrics Trap": selecting measurements that confirm our biases while dismissing those that challenge our assumptions.
The Pyramid of Choice
Tavris and Aronson describe how initial small decisions can lead down a "pyramid of choice," where each subsequent decision serves to justify the previous ones. In metrics, this often looks like:
Choosing an initial metric that supports our strategy
Building reporting systems around that metric
Creating incentive structures based on that measurement
Dismissing contradictory data as "noise"
Eventually reshaping the entire organization around potentially flawed measurements
Nokia's descent from market leader to obsolescence illustrates this pyramid perfectly. Each decision to focus on traditional phone metrics (market share, hardware margins) made it harder to acknowledge the fundamental shift in how people used mobile devices. As Tavris and Aronson write, "Self-justification is not the same thing as lying or making excuses... It's more like a protective shield that allows us to sleep at night."
Breaking Free from the Metrics Trap
1. Implement Metric Balancing
Create dashboards that pair growth metrics with corresponding health metrics:
User growth alongside user retention
Revenue growth alongside unit economics
Feature shipping velocity alongside feature adoption
Sales bookings alongside implementation success
2. Challenge Metric Selection
Regularly review which metrics drive major decisions:
Question whether current metrics truly reflect business health
Look for blind spots in measurement frameworks
Create forums where metric selection can be debated
Reward teams for surfacing concerning metrics
3. Build Reality-Check Mechanisms
Institute processes that force confrontation with uncomfortable data:
Monthly review of "warning signal" metrics
Regular audit of dismissed or downplayed measurements
Cross-functional metric review sessions
Customer feedback integration into metric selection
As we saw earlier, Nokia focused on market share metrics while missing fundamental shifts in user experience preferences. But there are plenty of other examples:
Blockbuster emphasized store revenue while ignoring early warning signs in customer behavior trends
Yahoo tracked portal engagement while missing the larger shift toward search-driven discovery
Kodak measured film sales growth while dismissing digital photography adoption metrics
Wells Fargo's account scandal represents perhaps the most dramatic example of metrics gone wrong in banking history. The bank's intense focus on "new accounts per customer" as a key metric led to the creation of an elaborate cross-selling machine. Management built detailed reporting systems and incentive structures around account creation, celebrating and rewarding aggressive cross-selling while systematically downplaying or ignoring fraud warnings. The pressure to maintain growth in this metric eventually led to the creation of millions of unauthorized accounts. Each layer of success metrics made it increasingly difficult for executives to question the fundamental strategy, leading to a massive scandal that severely damaged the bank's reputation and resulted in billions in fines. Wells Fargo by some measures is one of the 3 big companies that Americans like the least.
Each of these companies fell victim to Tavris/Aronson’s "confirmation bias squared"—not only seeking information that confirmed their existing beliefs but actively creating measurement systems that supported those beliefs.
Moving Beyond Self-Justification
The goal isn't to eliminate bias—that's impossible—but to build systems that help us recognize and correct for it. In metrics, this means:
Acknowledge the Bias:
Accept that metric selection is inherently subjective
Regularly question whether current metrics serve the business
Create processes for surfacing and discussing metric bias
Balance the Dashboard:
Pair growth metrics with health metrics
Include both leading and lagging indicators
Measure both positive and potentially negative trends
Create Safety for Truth:
Reward identification of concerning metrics
Build processes for regular metric review
Create forums where metric selection can be challenged
Understanding cognitive dissonance and self-justification allows us to build better measurement systems. By acknowledging our natural tendency to justify past decisions, we can create frameworks that help us see both comfortable and uncomfortable truths about our performance.
The most dangerous lies are the ones we tell ourselves. By building systems that help us confront these self-deceptions, we can create more resilient and adaptable organizations. Rather than viewing metrics as absolute truth, we should see them as tools for learning—tools that are most valuable when they challenge our assumptions rather than confirm them. As Tavris and Aronson remind us, "The ability to admit mistakes... is a strength, not a weakness."