Natalie Hanson, Ph.D.

From Vanity Metrics to Business Outcomes

(and Why “Musk Metrics” Are the Worst)

From the early days of my work in User Experience, I realized that people’s lack of understanding about the field meant I was going to have to speak in their terms about the impact my team and our services were making.

That quickly led me to another realization: very few people in UX actually understood the businesses they were trying to change – and that was just as problematic. Without understanding the business, it’s impossible to identify and drive meaningful change in its metrics.

Metrics that Matter

As part of a keynote I gave in 2018, I put together this slide to spark a conversation within the UX community on how we hold ourselves accountable to the businesses we serve.

Slide presenting key metrics in User Experience, divided into three categories: Business Impact, End User Behavior, and Usage, with bullet points under each category outlining relevant questions for assessment.

The talk track went something like this:

At the bottom of the grid are classic IT metrics. These often struggle to get the attention of the business because they’re technical – think number of concurrent users, number of logins, server uptime. Metrics like these are useful for infrastructure capacity planning, but these “vanity” metrics are rarely useful for business decision-making.
UX has its own version of this problem. We collect metrics that are deeply meaningful to us – like the SUS score (a measure of customer satisfaction) – but business leaders don’t know what to do with them. By contrast, they’ll act on something like NPS because they understand its connection to loyalty and revenue.
And then, at the top of the ladder, we find the real reason we show up to work every day – business outcomes. Metrics like customer retention, annual recurring revenue (ARR), cost of customer acquisition (CAC), and customer satisfaction (CSAT).

The structure of the slide was deliberate. All of our efforts should ladder up to business outcomes. If they don’t, we’re framing data in ways that only make sense to us, and not in ways that are useful or actionable to the business.

The biggest risk for functional leaders? Hiring talent that isn’t invested in understanding the business. Because it doesn’t matter if you create a beautiful algorithm or a beautiful interface – if you don’t understand the business intent, you’re unlikely to create a solution that drives lasting impact.

Elon Declares an Early Victory

I was prompted to revisit this recently because of a tweet by Elon Musk that Guy Allen reposted on LinkedIn.

Bar chart comparing iOS app updates for various AI chatbot applications over two weeks, highlighting Grok with the highest number of updates.

Musk was claiming victory in the AI model race based purely on the volume of new releases, and I couldn’t help but ask: what’s worse than vanity metrics? The answer, in this new AI world, might be equating volume with success.

So I (temporarily) added a new category to my metrics slide – one step lower than vanity metrics: Musk Metrics.

Slide titled 'Metrics that Matter' outlining different categories of metrics including Business Impact, End User Behavior, Usage, and Musk.

Musk Metrics celebrate activity (like model releases) as if it were impact, which is like saying water is wet.

More Code Is Not Better

Soon after I saw Musk’s claim, I came across a study by Faros AI, summarized by Mallun Yen on LinkedIn.

The study reviewed the work of 1,255 teams and more than 10,000 developers using AI coding assistants. At first, the numbers look impressive:

+21% tasks completed
+98% pull requests (formal request for code changes) merged

A bar graph displaying average percentage change in quality metrics related to AI adoption, showing an increase in PR size by 154.7% and bugs per developer by 9.1%, with positive and negative changes indicated. — The Faros AI report – https://www.faros.ai/blog/ai-software-engineering

But the story doesn’t end there:

+9.1% more bugs created per developer
+154% increase in pull request size
+91% longer review times

The most important finding was at the organizational level: AI coding tools had weak or nonexistent correlations with actual delivery improvements.

Why? Because while individual coding tasks may be faster, organizational bottlenecks remain. As the report succinctly put it:

“Coding may be faster. Delivery is not—yet.”

In short, as Yen so aptly summarized – “More code is not better.”

The Bigger Lesson

Whether we’re talking UX, IT, or AI-assisted engineering, the pattern is the same – metrics that focus inward are not enough. The only metrics that matter are those connected to business outcomes.

Because no matter what your craft is – whether design, data science, or engineering – we are all working in service of business outcomes, and along with that come metrics that define real change and impact.

What’s Next

I’ll be revising my original “Metrics that Matter” framework this week based on other news I’ve been reading. I’ll share more on that evolution in my next post – stay tuned.

In the meantime, what’s your experience with measuring real business impact (or falling into the “vanity metrics” trap)? Share your story in the comments – I’d love to hear your war stories!