BI Tools: Figuring Out What to Measure

Metrics are great. We all know that we should be measuring things and generating reports with pretty charts right? There are numerous books on the topic of metrics for IT, tools that can generate reports with graphs and so on.

The problem is that once all of that work is done and the reports are created, are they really worth anything?

One of my favorite stories is of a consultant working with an organization who questioned the need for a thick management report produced each month on high quality paper, bound and presented to management. There were several people working to compile the report and then the costs of printing, binding and so on. It definitely looked impressive and had a cost to match.

Nobody who produced the report was sure if the intended audience of managers were actually reviewing it. To subtly answer the question, they decided to stop publishing the report and see if anyone spoke up. After several weeks they received an inquiry from a senior manager about the report. Of course, everyone was excited that someone had missed the report because surely it must be adding value and they had not been wasting their time and the organization’s money.

They set up a meeting with the senior manager and asked him how he used the report. He sheepishly admitted that he didn’t read the report but his daughter really enjoyed drawing on the fancy paper. Needless to say, they stopped printing the report after that.

It’s a great story and where I originally encountered it I no longer remember but the message is very important—produce reports to meet the requirements of stakeholders, not because you can.

When a manager or executive reads a report, they need to have the information they require to make concise decisions. These people are typically very pressed for time so long reports with answers hidden somewhere in the pages are less likely to be read or even resented. However, a focused report that is just a page or two long that has the proper analysis and presentation is a welcome relief.

What to Measure, When

Asking managers for requirements can be a challenge and people are sometimes asked to recommend metrics. What is useful to bear in mind is that process adoption tends to follow a lifecycle. Metrics aren’t just for reporting as they drive behavior as well. Once employees understand how they are being measured, they typically adjust their activities accordingly.

When a process is first introduced, it helps to track metrics that demonstrate compliance to the process. What you are seeking to understand and influence is adoption of the new process. Metrics might revolve around process exceptions detected during audits, detective controls in production that demonstrate processes were not followed, etc.

Another class of metrics to look at early on relate to effectiveness. In other words, are the outcomes of the process what were expected? If a process is supposed to increase availability, then is it really doing that? The metrics here revolve around the desired outcome. Look for metrics that are meaningful not just to IT but the business as well. A good example is availability. This is an opportunity to look not just at uptime percentages but also sales lost due to unplanned downtime, and so on.

As the process adoption increases and matures, then efficiency becomes more relevant. In this case we are looking at the ratio of inputs to outputs. A good example is cost versus benefit. Relative to the costs we are incurring for the process, are the benefits worthwhile?

As the process stabilizes economy enters in. In this case, we are measuring the costs over time and looking for trends. We may also compare the costs of the current solution to other potential solutions to determine if there is a more economical means of achieving an objective.

In most cases, no single metric can tell the entire story. It is worthwhile to review and select metrics based on their ability to corroborate what another metric is showing. Sometimes over-emphasis on a single metric can have negative results. Some groups used to emphasize the first call fix rate metric. With this metric you are measuring the number of calls that are resolved on the first call to the service desk.

These groups were using this metric to assess service desk personnel performance. The net effect was that first-call-fix-rates increased but customer satisfaction went down. This happened because the operators would hold on to the call for extended periods of time trying to solve the incident versus appropriately escalating the call to the next level of support. By switching the measure to be a combination of metrics they eliminated the unwanted behavior and obtained better visibility to performance.

Metrics and reports can definitely help organizations improve. To get the most out of them we need to select metrics and reporting methods with the requirements of each reader in mind. We need to deliver information that these people can then use to make appropriate decisions. At the same time, we also want to select the right metrics to drive desired behaviors. With careful planning and implementation meaningful information is possible versus just making another fancy coloring book.

This article appears courtsey of ITSMWatch.com.

George Spafford is a long-time industry consultant and contributor to CIOUpdate.com and ITSMWatch.com. He can be reached at [email protected].