So, with this in mind here are a few tips from two survey experts on how to interpret results:
Frank Newport, editor and chief of the Gallup Poll, perhaps the most easily recognizable survey organization in the country, first suggestion is going to a subject matter expert just as you would to decipher financial data or your taxes.
But, since most companies probably don’t have such an individual on staff, the next best step is to be a little skeptical until you uncovered a few basic facts like who conducted the survey and for who paid for it?
Reputation is key here. Look for surveys conducted by analyst houses you know and respect then ask yourself ‘What is the company’s track record? Have they been historically accurate?’ “Consider the source,” says Newport.
Rob Daves, director of Strategic and News Research for the Minneapolis-St. Paul Star-Tribune and director of the paper’s annual, state-wide Minnesota Poll, suggests viewing statistics with an eye towards the survey’s sample, i.e. who was polled, how was it conducted and when?
“Question wording, and then the definition of the population under study, that’s what I’m talking about,” says Daves.
Surveys commission by groups or corporations (a common IT industry practice) should be viewed with care. If the results don’t match what the sponsoring organization is looking for, results could be withheld. Daves has seen this first hand more than once.
Also, look for context: a number standing alone is just that. Numbers need to be viewed against the backdrop of who participated in the survey and why they participated … the last is a question you will probably never get answered but a good one, along with many others, to ponder before you accept results at face value.
While useful tools, survey’s are just that, tools, says Greenbaum. A survey is only as good as its methodology. So, the best advice is compare the results of any survey against the backdrop of your own experience and then do a little digging. Get the questions and read them, and then see if you can talk to some of the survey’s participants to get their take on the results.
“These surveys should be guideposts, but you need a better map than just a collection of surveys” to base decisions on, says Greenbaum.
Want to discuss the issues raised in this story? Take it over to our IT Management Forum.