Metrics guide a huge number of decisions that we make on a daily basis, and rightly so: the world is complicated. There is, however, a metric antipattern that I want to highlight here: the clever metric.
Generally the cleverness enters because it's tempting to try to capture the complexity of the world in the definition of our metrics, which, after all, are supposed to give us a guidepost that is independent of the passions of the day. This is not a bad thing. In a good metric, we have a principled and straightforward way to determine if on any given day we're doing the right thing or the wrong thing.
The trap is when we try to capture too much complexity. When the value of a metric is difficult to understand, what began as a way to distill complexity can become a permission structure for dismissing it. An example of this is the current state of political polls. If you're curious about the question "who is going to vote for whom", you can find just about any answer to that question that you'd like. The underlying dynamics of political polls are so incredibly complicated that it's just not possible to understand at a lay level, but we all try. So when we're wrong (remember 2016?), it's easy to dismiss because understanding the details of polling is just so hard.
Here are a couple of questions about metrics that should, in my view, have straightforward answers.
- Is it possible to say something about what the value of the metric should be?
- Is the value of the metric going up or down an unambiguous signal?
- Is the value of the metric something that you can actually do something about?
Put simply, your metrics, and their definitions, should be working for you, not creating a structure by which they can be ignored.