Goodhart's Law Is Real
Lutke: Goodhart's law is real. The moment a metric becomes a goal, it's no longer a useful metric. Why? Because no metric by itself is a complete heuristic for a complex business. There's a million different tensions in a company, and you can't keep them all in harmony by optimizing for one thing.
It's true that we don't have KPIs and we don't have OKRs — at least not in the sort of Silicon Valley sense. But we are extremely data-informed. We've invested enormous amounts of money and time into systems that give us basically everything at our fingertips.
The moment a metric becomes a goal, it's no longer a useful metric. We don't have KPIs. But we are extremely data-informed.
80% of the Value Space Is Unquantifiable
Lutke: Everyone competes for everything highly quantifiable because it's fun — it's like a game. You tweak a number and 0.1 more is better than 0.1 less. Immediate gratification.
But I think the overlap of the most valuable things you can do with a product and the things that happen to be fully quantifiable is maybe 20%. Which leaves 80% of the value space untouchable by people who only look at quantifiable things.
The overlap of most valuable things and fully quantifiable things is maybe 20%. That leaves 80% untouchable.
Taste, Quality, Passion, Love
Lutke: Shopify is comfortable with unquantifiable things such as taste, quality, passion, love — hate. The strong emotions that people have. The deep satisfaction that a craftsperson feels when they've done a job well is actually a better proxy, if you allow it to be, than whether the unit tests pass.
Are you representing your brand? That's an unquantifiable question. Are you proud of the thing that you have built? There needs to be more acceptance in businesses of unquantifiable things.
Are you proud of the thing you built? That's an unquantifiable question. There needs to be more acceptance of that.
A Cockpit for a Pilot
Lutke: We think about it as a cockpit for a pilot. The decisions are still made by pilots. There's an extremely sophisticated rollout system in Shopify that forever holds out and correlates everything with everything in every experiment.
It's a bit of a nuanced conversation. Maybe slightly less good on a fortune cookie — saying Shopify doesn't do OKRs or doesn't do metrics. It's actually just that the metrics take a support function. We often defer to less quantifiable things.
We think about it as a cockpit for a pilot. The decisions are still made by pilots.