Tackling Data Quality

“This is the problem we are having in corporations is we have never learned to gather data in one place, clean it and distribute it and we are learning that,” said Abai.

But, today, that is changing. Service-orientated architectures (SOA) and compliance with Sarbanes Oxley (SOX) are forcing companies and IT departments to use data from a shared resource since SOA is decoupling applications from databases and SOX is forcing companies to become more aware of importance for getting their numbers straight.

To begin the remediation process, the first step is to sit down with all concerned parties and decide things like whose data is correct, whose metadata best describes that data in question, definitions of terms like “customer”, and so on.

Not an easy task by any stretch but until completed all the automation in the world will not get you past the core of the problem: people doing what people do to make their lives and work easier.

“That’s the thing,” said Russom, “at some level, a lot of companies stall because they get into arguments. People will argue about ‘Well, how do we define customer? And those (issues) have to be resolved somehow before you really want to tackle moving data around.

Sometimes people think, ‘Well, it’s data so we must be able to hurl a technical solution at the problem. But really I’m here to tell you, the technical thing is half of it. Various types of actions taken by people is the other half.”

Of course, since this is an old problem dating back to mainframe days, there is another option, said Russom: Do nothing. If it ain’t broke, don’t fix it.

“Sometimes the data is in terrible condition but the business can tolerate it. If you can tolerate the bad condition of data, don’t fix it.”