Taming BIG Data: Taking back control, Part 5 – Maintenance

New technology can be assessed and leveraged for the appropriate place alongside traditional data base management systems. Hadoop is a good example of how open source is changing the face of technology. Open source allows developers worldwide to vet and create better products quicker. Companies with proprietary frameworks and code used to manage Big Data are now faced with a decision to either open up their technology or face the potential lost in market share.

HPCC is one such example. Established companies like Lexis Nexis are well positioned to develop mainstream frameworks and applications that can compete with the likes of new open source technology like Hadoop. One advantage is that such established companies have a long history of securely dealing with vast volumes of data quickly (i.e., 35,000 data sources, 5,000 transactions per second in Lexis/Nexis’ case) and already have a large customer base.

Keeping an educated workforce is imperative in maintaining a competitive advantage in this market. Understanding if new technology will replace or complement existing technology and how the solution fits with the organization’s security model are two major factors. Team members who understand both the new technology and existing business technical infrastructure are best positioned to assess and make the appropriate recommendation. A staff development model with a balanced mix of expertise and training required is an important outcome for the technical workgroup members.

EIM workgroup

The EIM program management workgroup members are responsible for:

  1. Identifying the best resources who understands maintenance and training development;
  2. Developing the facilitated session agendas and outcomes; and
  3. Finalizing the specific deliverables (i.e., policy and procedure manual, stewardship model, company standard toolset and architecture list, period review schedule, etc.).

The EIM program management workgroup members have three main responsibilities: identifying resources, developing sessions, and finalizing deliverables.

This group will facilitate sub-team meetings with human resources for both the functional and technical workgroup members. During these meetings, the goal is to align individual performance measures with corporate and EIM program goals, to minimize risk and enhance the acceptance of corporate EIM program objectives, and implementation.

The maturity of a company’s information management program is indicative of its state of readiness to take on Big Data. Industry leaders in the information management space refer to Big Data as not only data that presents unique opportunities for a company due to the vast amount of data, but also includes other factors, such as the speed of capturing and preparing the data for analysis, and the range of sources and types of data.

Unstructured data, call detail, high-traffic website log files, system log files, and data that may produce terabytes or petabytes of data in a single feed, or data that is machine-generated are typical examples of Big Data.

Many companies are not in a position to take advantage of Big Data given constraints in their existing infrastructure, governance, architecture and supporting policies and procedures. An EIM program is even more critical for companies that rely on Big Data strategies since the information maturity approach allows the company to be best positioned for handling the various frameworks and technology required to be successful in efficiently processing data into usable information. The main goal is to facilitate a process by which decisions are made in the best interest of the company (not an individual or division) through a well devised governance model.


Information management will continue to flourish and evolve with technology as companies become more dependent on information to sustain a competitive advantage with competitors. The Taming Big Data series provides you with one approach to gaining control of your information by leveraging information lifecycle management techniques.

Regardless of which approach you use, it is imperative that leaders of your organization continue looking at viable solutions that keeps your company relevant and maintains a competitive advantage through the formal development of an enterprise wide information management program.

Finally, continue the discussions with peers in other organizations, read articles and books that allow you to understand enterprise information management and make yourself relevant by contributing. Collaborative discussions will allow you to develop the best solutions as your company matures.

Stephen Boschulte MBA, PMP, CISSP, is a senior information management strategist. He has over 14 years’ experience working with more than 20 Fortune 500 companies and small organizations. Mr. Boschulte is the author of “A Practical Guide for Implementing an EIM Program” and can be reached at [email protected].