On Key Performance Indicators

You’ve got a company, or an organization, or even just in everyday life, and you have goals:  you want to grow, you want to learn, you want to educate.  But how do you quantify those goals?  From a business perspective, how do you build successful metrics or Key Performance Indicators (KPIs)* when you want to quantify how well?

This came up recently when working on the nonprofit Louisville Makes Games organization, and it’s something I deal with at work a lot, so I thought I’d collect all my thoughts in one place.

First, of course, it’s not easy.  There are books, and books, and books on the topic.  There’s even a book for Dummies:

So to boil it down to blog post size you should remember that there’s a lot more out there. That said, here’s some key wickets, in my characteristically non-terse fashion:

KPIs need to be well defined and measurable.  This seems obvious, but it’s really one of the most frequent mistakes I encounter.  The important meaning is that every time you make a measurement it is comparable to previous measurements. You run the same calculation against very similar inputs.  We ran a large data quality process for a department of defense client, and they wanted to know, naturally, if their data quality was improving or diminishing over time.  The difficulty we enountered was in having apples-to-apples comparisons because, at the same time we were measuring quality, we were constantly expanding the scope of what we measured.  If 1% of the data (there was a lot of it) was 80% clean in month 1 and 2% of the data was 85% clean in month two, was there really a 5% improvement, or was the newly added data just more clean to begin with?  It’s important to have apples-to-apples comparisons.  Keywords like “reproducibility” and “repeatability” are common, and entire management areas like Six Sigma keep this as a core principle.  Another way to put it is that you must understand your baseline, and ensure that your KPI properly accounts for it.  For this reason some people recommend not using metrics like customer happiness, because it’s too subjective, although I find it too important to ignore and there are few good alternatives (although repeat-customer numbers and customer referrals may be stronger more quantifiable metrics). Here’s a fun example from an article about How to select KPIs for your dairy that actually came up at work today:

For example, if you use milk fever as an indicator of fresh-cow management, then you can define the indicator as “the number of multiparous animals that develop clinical or subclinical signs of hypocalcemia.” That tells everyone what the indicator is.

I don’t really know what that means, but it sounds specific.  Of course it doesn’t sound like a KPI either; “number of [] animals” sounds like a metric…

Don’t confuse Metrics and KPIs.  A metric is anything you can measure, really… the term is used in the dictionary sense of “a system or standard of measurement”.  Personally I subscribe to the “measure everyting” school of thought — with big data storage dropping in price, capturing metrics for analysis later is cheap and easy.  These are important, but they’re rarely the same as a KPI, which is more of a business term with some nuanced meaning.  A good KPI will combine related business metrics into a single number, to dissuade gaming the underlying metrics.  A common industry example of this is help desk performance (that link is a great example of thinking through KPIs by the way, in much more detail).  If you push to measure productivity by the length of a customer call, then customers frequently call back or have low satisfaction.  If you ignore call length then customers get more satisfaction, but costs go way up, and there may be significant waste (time spent bantering or what have you).  Good KPIs are often ratios or products of standalone metrics, to balance the positive effects of a decision against any potential costs or negative effects.  If your goal is to grow efficiently then a growth/cost KPI may make sense.  If you want to increase attendance but not at the cost of satisfaction, you may multiply the percent changes (attendance%increase * satisfaction%increase [or decrease!]).  And don’t limit yourself to two metrics; daisy chaining products of ratios can be very powerful, but you need to take a good look at the math so that no one metric dominates a KPI when it shouldn’t.  Normalizing metrics to percentages or fixed ranges (such as 1-5 in surveys) can help.  Almost no KPI should be JUST a metric, because improving anything almost always comes at a cost of something else, and those facts should be balanced.  It’s possible to do that with multiple metrics on a balanced scorecard.  Yes, you will find exceptions.  That’s ok.

Align your KPIs with your goals.  You’d think this is a no-brainer too, but it’s harder than you may think.  For one thing, people frequently expect KPIs to constantly improve, but many good KPIs can be balanced so that either staying constant isn’t bad — even preferred — or improving them becomes an exponentially difficult affair; it’s harder to grow the bigger you are.  This is particularly true for the first few periods of a new KPI because there’s often low hanging fruit that’s easy to improve.  To take a real-world example, take an insurance company who had KPIs based on a denominator of “per Agent”, such as new policies per agent or total customers per agent.  As the number of agents grows, the total number of policies goes up, but hiring could easily in any given month outstrip sales growth dragging the KPI down.  Also, many agents actively leave sales to take on management or training duties, so positive growth can pull a KPI down.  To get around this, the definiton of agent could be tweaked so that agents with less than a few months were excluded, and agents who were really managers could be reclassified, or the KPI can be reformulated to account for time-spent-on-task, if that can be measured.

Many people require that a KPI include a target, or at least insist on more delineation between metrics, KPIs, targets, objectives, and maybe other things.  Here’s an article that focuses on targets being Real, Relevant and Robust.  Yay alliteration.  For me, it’s more a combination.  If you’re doing six-sigma work, then a constant improvement to that 99.9999% target is specific and required, but for a lot of business metrics it’s a balancing act — setting a target is fine, but understanding how your business ebbs and flows in different areas may be more important than a tunnel-vision approach to fixed numbers.

Marketing “coverage” metrics often fall into this trap that poses as a general 80/20 rule… it’s fairly easy to identify or target the majority (i.e. 80%) of a market segment, but the last 20% is far more difficult.  A KPI can either reflect that directly (decaying target growth, or target the rate of the rate of growth or something), or more easily it just needs to be interpreted intelligently, which brings up another point:

Base your analysis in math and statistics.  I can’t overemphasize this; there are many bad KPIs out there that may not be poorly formed, just poorly interpreted.  It’s good, for example, to be able to measure a metric at different levels — enterprise, organizational, department, branch, geographic area, whatever.  But if you split a KPI that’s effective over large numbers into small components, it may lose effectiveness.  There are lots of basic rules of thumb to follow for analysis which should be kept in mind.  Statistics has the “rule of 30” which points out that in a sample population of size < 30, a single random issue starts to reach a level of statistical significance (in the p-value world), so make sure that every component of a KPI measures at least 30 things, preferably more.  Understand the statistical distribution and establish bayesian priors for your measurements and targets.  Ask a statistician if you need help.

Keep it simple (stupid).  You can build a thousand metrics in the smallest workplace, but putting anyone in charge of more than two or three KPIs is usually overwhelming.  This isn’t just about quality, but about visibility and control as well.  Aligning people with KPIs can make sense, but it can be overdone — don’t live and die by the numbers.  Done right measurement can be incentivising and motivating… done wrong it can demotivate and inspire corner cutting and unintended behavior.

QUESTION: Where Can I Find Industry-Standard KPIs?

http://www.aicpa.org/InterestAreas/FRC/AccountingFinancialReporting/EnhancedBusinessReporting/DownloadableDocuments/Industry%20Key%20Performance%20Indicators.pdf

http://www.industryweek.com/lean-six-sigma/five-rules-selecting-best-kpis-drive-operational-improvement

There are some good quotes in here:  http://www.industryweek.com/kpi-best-practices

This list is terrible:  https://www.linkedin.com/pulse/20130905053105-64875646-the-75-kpis-every-manager-needs-to-know

And this is a great teardown of a single considered KPI: http://www.industryweek.com/blog/oee-useful-key-performance-indicator