Measuring Success - A Practical Framework for Key Performance Indicators

Written by Matthew K. Brown, MBA, PE

You are preparing to kick off a pilot and the project manager asks, "What should our KPIs be?" Everyone begins sharing opinions from their respective experience. One idea is shot down because the data isn’t readily available; the next universally dismissed as too broad. 

The PM asks everyone to write down their ideas on a post-it note. You go around the room to share ideas. Everything mentioned is about the progress of implementation with no mention of impact or delivering on the original goal of the project… Sound familiar?

KPIs are perhaps one of the most misunderstood and abused tools in business for managing products, programs, and projects. In the best cases, they provide insight into progress, drive alignment in activities, and connect outcomes to business value. At worse, they create adverse incentives, sow disconnected efforts, and generate confusion. They are often molded out of convenience, biased to show some sort of progress regardless of added value, and rarely tracked long enough before choosing other measurements to avoid the appearance of failure or latch on to a flavor-of-the-month message.

From project management to program implementation to product development, rarely is the process of selecting KPIs given the due process needed to create meaningful measurements. Dozens, if not hundreds of books have been written on the subject from the thoughtful "Key Performance Indicators" by David Parmenter, to the anecdotal "Measure What Matters" by John Doerr. So, why can it be so hard? It doesn’t have to be.

My work in innovation, particularly developing data science products, has led me to develop a practical and methodical approach that we use for our team to develop KPIs. But first, let’s establish what a KPI is:

Key Performance Indicators (KPIs) are select metrics that represent the status of or progress toward an objective or strategy tracked over a duration of time.

Now, there are certainly many other alternative definitions, but this one is loaded with important information:

• Without an objective or strategy, there is nothing to measure.
• All KPIs are metrics, but not all metrics are KPIs; only select metrics that are representative of success actually matter. Tracking too much can lead to misalignment, distractions, and false representations. Tracking too little can lead to myopic focus and adverse incentives.
• KPIs are tracked over time and should only be modified if they do not adequately reflect the objective, the objective has changed, or the objective is complete.

Let’s dive into the framework:

As general guidance, a set of KPIs should satisfy three criteria:

• Appropriate – Do the metrics capture performance as it relates to the value proposition? Do the metrics align with corporate, business, and stakeholder strategies?
• Adequate – Are the metrics S.M.A.R.T.? Can we acquire the data and perform the calculations?
• Accepted – Have key stakeholders agreed that these are the metrics that should and will be used as KPIs? Has accountability been assigned for each KPI?

This guidance asks you to consider that metrics lack meaning if they are created out of context, if they cannot be measured, or they cannot generate action.

A key point of contention I have witness in developing metrics revolve around performance versus impact. We need be able to recognize and distinguish the two in a discussion:

• Performance – How are we doing day to day? When do we have a problem that needs to be addressed?
• Impact – How are we contributing the company’s strategy? Are we making a difference?

Both are important. However, some stakeholders will prefer one of the other. For example, CEOs tend to care about impact. Project Owners might care more about performance. They are not mutually exclusive and should be considered in tandem.

Step 1: Establish the Critical Success Factors  (60 - 90 Minute Group Session)

Critical Success Factors (CSFs) can be a fuzzy term, but I think the Wikipedia community has done the best job yet to create an understanding them: (https://en.wikipedia.org/wiki/Critical_success_factor)

• Critical Success Factors are elements that are vital for a strategy to be successful.
• A Critical Success Factor drives the strategy forward, it makes or breaks the success of the strategy (hence “critical”).
• Strategists should ask themselves “Why would customers choose us?”  The answer is typically a Critical Success Factor.

Practically, I also like to include:

• CSFs can use relative descriptors such as more, less, better, faster, cheaper, etc…
• Specific measures are not CSFs unless there are external drivers so specific and precise that they cannot be avoided. (ex. federal regulations)

These success factors are likely to be different among stakeholders, which leads us to the exercise for building the Stakeholder Critical Success Factor Matrix:

1. Have someone restate the strategy, objectives, and scope. Before anything else, everyone needs to be on the same page. If there is confusion or disagreement among stakeholders, don’t go any further. Use the time to shore up the strategy and/or reschedule KPI development for another day.
2. Determine the primary stakeholders. For example, in developing a product at a healthcare operator, we might consider: Patients, End-Users, Facility, Product Team, and Parent Company.
3. List the critical success factors from the perspective of each stakeholder. Two categories should be considered: success factors for process and for outcomes.

This exercise should be done as a group with key stakeholders. The purpose is to surface everyone’s interest, many of which likely overlap. Those overlapping success factors are clues for finding the right KPIs.  

Step 2: Consolidate the Critical Success Factors   (Management Activity)

There may be a few success factors or there may be many. You will need to digest them into mutually exclusive Critical Success Factors that can be refined; ideally up to five, but use your discretion. This can be done as a group activity, but I would highly recommend working directly with the manager of the project to refine the list.

Step 3: Step Away

Seriously. Leave it alone for a day or two. Staring at this problem for too long without a break will make your eyes cross. Take a break, come back, and review what you have. If everything looks reasonable, it’s time for the next step. If something doesn’t seem to fit, revisit it. Remember, we want success to be measured appropriately, adequately, and in an accepted manner.

Step 4: Create a Performance Indicator Matrix   (60 - 90 Minute Group Session)

Reconvene your group from the first session. Consider the first half of the meeting a brainstorming session. For each consolidated Critical Success Factor, brainstorm several metrics that could be direct representations or proxies for the CSF.  (Note: A given metric could be used for multiple CSFs.)

As ideas are generated, ignore the S.M.A.R.T criteria for the moment, we just want the ideas. Also, ignore whether the data is readily available or not. Often, people will bias their choices to information that has been used in the past or data they are familiar with. This is not necessarily bad, but it can constrain thinking and lead to less than optimum ideas. Metrics determine the data that needs to be collected, not the other way around.

Notice that I have not mentioned the S.M.A.R.T. method up until this point. The method has a bad habit of leading people into traps. S.M.A.R.T. is a set of criteria to apply to metrics, not a framework for developing KPIs. But now that we have a list of metrics, we can apply these criteria.

Discuss the metrics that have been listed and use the S.M.A.R.T. criteria to narrow down your options. Remember, just because data is not readily available does not mean it cannot be measured.

Step 5: Select the Key Performance Indicators  (Group Session Continued)

By this point, the group has discussed metrics in a structured way that has surfaced the priorities and interests of each stakeholder and how it aligns with the strategy and objectives of the project. As a group, select the 3 to 5 metrics that best represent the success of the project. Allow for expected debate, but by now several metrics will likely make obvious candidates. If are still significant challenges to landing on agreed KPIs, problems beyond the project itself or priorities that are completely at odds may need to be addressed.

Step 6: Assign Targets and Accountability

KPIs are created to understand the progress and success of a project. To be actionable, there must be some target or end goal in mind. Targets may be well understood and well known. Sometimes it may take a baseline period to collect data. Regardless, the word “performance” assumes some comparison to a standard or target. Without a target, you have a metric that presents information, but not a KPI.

So, what happens when a KPI is not meeting a target? Who is responsible for fixing a problem if one exists? Who should inquire into the variance if appropriate? All of these questions should have an answer. For a KPI to be more than information, but a call to action, accountability must be established. For each KPI you will need to assign an appropriate person to take ownership.

Step 7:  Update Frequently and Transparently  

You have your KPIs, now what? Pay attention to them regularly and make them readily available to everyone on the team. KPIs are a tool for communicating the progress and success of your project. If not updated regularly, or hidden from view, they are ineffective and useless.

At some point, you may find the a KPI is no longer necessary, turned out to be a poor measure of success, or has been achieved. Make changes when necessary and adapt to the strategy of your project. Just remember that KPIs are only useful if they are measured over time and that frequent changes obscure the performance of a project.

Final Thoughts

This is only one approach to solving the challenge of developing meaningful Key Performance Indicators for products, programs, and projects. There are numerous other methods and resources that may be more helpful for your needs. But, regardless of your situation, give KPIs their due thoughtful process.

Matthew Brown has an MBA from Vanderbilt’s Owen Graduate School of Management and is a licensed Professional Engineer with industry experience in electric utilities, manufacturing, e-commerce, and healthcare. He is currently a member of HCA Healthcare’s flagship data science team in Nashville, Tenn., focused on developing innovative digital products for improving healthcare operations, clinical outcomes, and care coordination for 185 hospitals across the US and UK.