Quality of Experience is the Measure of IT Value

Are you measuring the right things for your e-services? Many CIOs aren’t, and even those who claim to measure IT value and report on e-services understand the concepts poorly. Why is this important? Because customer perception matters and QoE is the most significant single factor in a real-world evaluation of the user experience and, therefore, the key to your success.

Simply put, if you are not measuring satisfaction across the five dimensions of a service from the point of view of customers then you are not measuring service (and you have only vague predictions about what your service experience will be like).

While all physical products deliver a service at some level, services are not the same as durable goods or products. Service quality is based on the experience of consuming ― QoE. Product quality is based on ensuring physical characteristics are as designed ― QoS. Here is an easy way to know if you (or someone you know) understands service QoE:

  • If you don’t know that customer satisfaction and quality are not the same; if you can’t explain the relationship between satisfaction and quality; or if you don’t know about the five dimensions of service quality, then you are not measuring QoE.

  • If you measure technical operational status like page loads, response times or availability then you are not measuring QoE. Instead, you are trying to predict satisfaction with regard to a single dimension of service quality ― QoS.

  • If you ask customers if they are satisfied with your technical support then you are not measuring service quality you are measuring satisfaction with one part of one dimension of a service.

Services have several inherent characteristics that make them different from goods, not the least of which is intangibility: You cannot capture, store, hold or otherwise examine a service as you can a physical product. The act of producing a service is a performance that marshals all the systems (people, process, technology, etc.) it takes to deliver that service.

While you can make sure a product meets quantitative criteria before it’s consumed, a service is produced and consumed simultaneously. Therefore all you can do inside the provider is prepare for its production, and all your internal measurements are nothing more than predictions of what you think the customer QoE will be.

An easy analogy and example of a service and QoE is a musical concert; think about your favorite performer. If you quantitatively measure the bands instruments (the drums, guitars, horns, etc.) to make sure they are tuned and positioned properly, you are setting the stage for a quality performance. But the performance has not yet occurred, so all you are doing is preparing. Moreover, even with all the instruments tuned (QoS) the performance (QoE) isn’t always of high quality, is it? We have all seen (and heard) a performer’s “off-night.”

You simply cannot measure service quality until after its delivery and consumption, and your measurement must be from the point of view of the customer. Anything else is not measuring service QoE, it is measuring QoS and the two are not the same.

Unlike manufacturing quality, which is loosely defined as quantitative conformance to specifications, service quality is, well, qualitative. Quantitative refers to measuring the amount of something. When you measure the “uptime” of a transmission link, application or server, then you are making a quantitative measurement. QoS refers to these quantitative and possibly predictive measurements but not to measuring service quality however.

QoE refers to measuring distinctive characteristics or dimensions of something or someone. Qualitative research aims to gather an understanding of human behavior and the reasons for that behavior. Qualitative research means the why and how of decision making, not just what, where, when (quantitative). When you measure the perceptions of satisfaction ― the opinions formed by the user while consuming the service ― then you are starting to measure QoE.

The trap that most service provider organizations fall into is to measure the technical operational systems and equate “uptime”, “capacity” and other technical aspects of service production to QoE. This is wrong on a number of fronts, not the least of which is to miss completely the concepts of the human-centric elements of service production and consumption – QoS does not equal QoE. Trying to call operational status quality also fails because it does not take into account that service quality can only be ascertained after its production and subsequent consumption, and only from the point of view of the customer.

How to measure QoE

You need to measure satisfaction across the five dimensions of a service. If you satisfy customer expectations for each of the five dimensions that you have achieved an acceptable QoE. You measure service quality retroactively to its production and consumption, and you have to ask your real customers their opinions. Note (and this is important) that you must use a valid method for qualitatively assessing customer perception. Simple coming up with a list of things you (or your team) thinks is interesting is most likely not going to cover the five dimensions. There are numerous proven systems for measuring service quality, use them, don’t roll your own.

Measuring QoS technical operational metrics is part of the answer. Measuring the human QoE elements of the service is also required. You can set the stage for quality through strategy, design, transition and operation of the service. Process, technology, people and other service assets must be properly positioned, trained and ready.

How to report (and use) QoE

Service quality should be reported to customers according to satisfaction across the five dimensions, on other words, in QoE terms. For each dimension, there are several levels relating to satisfaction: expected, perceived, and desired. QoE (the sum of dimensional satisfaction) is easy to understand when plotted as a zone spanning expected (lower level) desired (upper level) and perceived (actual.)

If service delivery falls in the zone between expected and desired, the customer is satisfied with that dimension. If all dimensions are “in the zone” then service quality is acceptable. Anything “out of the zone” indicates a mismatch and an area for the service provider to investigate.

Most service provider organizations produce reports that are one-way: the provider prints it and the customer ignores it. Few reports relating to service quality actually get utilized in any meaningful way. However, service QoE reports as described above are understandable, in layperson terms, and actionable from both customer and provider perspective.

It is easy to map service dimensional failures into provider organizations since there is a direct correlation to gaps in service delivery indicated by quality failures and common organizational functional areas and processes. Now customers cannot only explain their needs for a service, but their service provider can take action on those needs.

Hank Marquis is practice leader for Business Service Management at Global Knowledge. You can reach Hank at [email protected].