Data-Informed, Not Data-Driven: What's the Difference?
Everyone claims to be 'data-driven.' But the best product decisions I've made came from using data as input, not as the driver. Here's the distinction that changed my thinking.
Early in my PM career, I worshipped data. A/B tests decided everything. Metrics were gospel. If the numbers said ship it, we shipped it.
Then I shipped a feature that improved conversion by 15%—and destroyed customer trust.
The difference matters
Data-driven: The data determines the decision. If the A/B test wins, you ship. If the metrics say X, you do X. Data is the decider.
Data-informed: The data informs the decision, but human judgment remains. You consider context, long-term effects, qualitative signals, and things that can't be measured. Data is an input, not the answer.
When data-driven goes wrong
That conversion-boosting feature? It added friction to a flow that made users click "I agree" to terms they didn't understand. Conversion went up because we made it harder to opt out.
Three months later: support tickets increased 40%, churn among that cohort was 2x higher than average, and we had to revert.
The data said "ship it." The data was wrong—not because the numbers were wrong, but because the numbers didn't capture what mattered.
How I use data now
1. Start with why the metric matters: If conversion is the goal, ask why. Is it engagement? Revenue? Retention? The underlying goal might not align with the direct metric.
2. Look for lagging indicators: Short-term metrics don't tell the full story. Track cohort behavior over time. Today's "win" might be tomorrow's churn.
3. Include qualitative data: Customer interviews, support tickets, sales feedback. These are data too—just not numerical. They often catch what metrics miss.
4. Develop judgment about the data itself: Some tests don't have valid sample sizes. Some metrics are gameable. Some results are noise. Learn to question the data, not just act on it.
5. Be willing to override: When your gut, customer feedback, and long-term strategy all say "don't ship this," the A/B test is probably missing something. Have the courage to trust human judgment.
Data is a tool. A powerful one. But like any tool, it's only as good as the person wielding it.