Taming BIG Data: Taking back control, Part 5 – Maintenance

The final chapter of the Taming BIG Data series is about taking back control of the ever increasing growth of information and underlying data through maintenance standards. The approach suggested by the series of articles is to try to simplify the enterprise information management (EIM) program by leveraging the information lifecycle management areas which include usage, creation, retention, availability, and maintenance.

An EIM program allows a company to provide accurate, consistent information to all of its resources (employees, computer databases, etc.), allowing them to perform their jobs more effectively. A key objective of the EIM program is to transform a vast amount of information collected every day into a strategic advantage. The primary EIM subcomponents referenced during this phase include information governance, communication and information quality.

Over the last four months, our journey included understanding how information assets are used, created, retained, and accessed (see Taming Big Data-Part I: Usage; Part II: Creation; Part III: Retention; Part IV: Availability). The maintenance phase defines the tools and technology that are used to cleanse, transform, load, monitor, and secure the information assets. Supporting policies and procedures are developed to obtain a controlled environment where changes can be managed effectively.

Some companies may prefer to start with this phase, the maintenance phase, first and then move to one of the other phases. This phase is discussed last because companies often go through a cultural shift through the first four phases. This cultural shift provides two main benefits. First, the company more clearly understands the information assets, requirements, and priorities. Second, the collaboration between business and technical groups matures to a point where everyone understands the process of working together to determine what is best for the company — a theme that runs through each phase of the Taming BIG Data series.

The approach identified here allows the company to start focusing on more independent activities (i.e., usage requirements, etc.) and building up toward a fully repetitive and complete process that is incorporated into the fabric of the company via the governance framework. It is critical that the business, technology and EIM program management resources work together to implement a solution that align to the corporate goals. Collaboration is required to minimize risk.

Below you will find the role each management resource plays in the company.

Roles and responsibilities

Business team members: The business workgroup members understand the company’s core competencies and can best assess the needs of the organization.

Technology team members: The technical workgroup members support the business by providing tools that can be used by a company to effectively perform their jobs.

EIM program management: EIM program management structures the project, defines the goals and outcomes, facilitates the workgroup interactions and formally submits recommendations to the governance review board.

Collaboration is most effective when individuals are clear with their workgroup roles and the goals of the initiative. During this final phase, the company focuses on studying and maturing the business process management and technical maintenance procedures used to manage important information assets.

Business workgroup

The business workgroup members are responsible for:

  1. Defining/Revising the interaction model for change management and issue resolution;
  2. Clarifying stewardship roles (data owners and other business resources);
  3. Creating/Updating the corporate policy and procedural manual; and
  4. Determining the communication and training requirements.

During the final phase of the program, business workgroup members are responsible for defining the change management and issue resolution interaction model. Stewardship roles, including the roles of the data owners and other business resources, should be clarified. A critical and often overlooked component for these team members is working with the human resource (HR) group.

Working with the HR group will ensure that the objectives of the stewardship roles are incorporated into the annual evaluation process to facilitate alignment of the EIM goals with the company’s goals. These members will prepare recommended changes to corporate policy and procedure manuals, and work with the training and communication liaisons to identify the training requirements for various groups. This will minimize risk by educating the company resources and providing the reference documentation needed to help the program succeed.

Technical workgroup

The technical workgroup members are responsible for:

  1. Developing the list of standard tools and architectural models;
  2. Defining the technical data stewards and clarify roles;
  3. Creating/Updating the corporate procedural manual; and
  4. Determining technical training and communication requirements.

The technical workgroup members have the heaviest workload in this phase. This phase is all about finalizing the enterprise-wide standards and architectural models that will be used moving forward. They also review the transition plan to move from a current state architecture to the new recommended state and establish the periodic architecture review in this phase.

Technical stewardship roles are clarified and procedure manuals are updated to provide supporting reference information that aligns to the corporate objectives. Like the business workgroup members, the technical workgroup members should also meet with HR to incorporate performance metrics that aligns the individual professional goals to the corporate goals.

The technical workgroup member’s standardization of toolsets may be challenging depending on the number of tools currently being used throughout the organization. This is one area where tool categorization may be helpful. Based on the information requirements developed from previous phases, team members are better positioned to understand, assess and recommend frameworks, architecture, and specific vendor tools. A training plan is often required to educate resources on new advances in technology.

Educating people on the latest technology and having a research/development group to consolidate and disseminate information to both technical and functional resources is important. This allows for fruitful discussion to take place on the strategic development of new solutions specific to the company. This is particularly important as the company seeks to adopt new technology that addresses the organization’s Big Data.

One only needs to perform a search on the internet to find many of the solutions available that addresses Big Data. Solutions such as Hadoop, HPCC from Lexis/Nexis, Cassandra, Condor, HaLoop, MapRejuice, Twitter’s Storm, and Stratosphere are just a few of the technologies providing solutions for the open source community.