Lean is a hot topic in IT circles these days. The primary trigger for increased attention was the global recession, but a more fundamental force at work is that IT as a practice area has matured to the point where it can truly leverage manufacturing techniques, which is where the lean management system was perfected. Furthermore, the increased complexity of IT systems and fragmentation of data is driving the need for effective integration techniques to new heights.
Lean is not rocket science. In fact, it is largely common sense — which is the secret to its effectiveness. If you are trying to integrate systems across disparate functions and across organizations, you will have a hard time establishing and sustaining integrated solutions unless the underlying principles and methods are ones that everyone can easily understand and adopt.
While Lean is not difficult, it does require a paradigm shift; a different way of thinking and acting. To change behavior, you need to stop doing some things and start doing other things. Let’s take a look at some of the practices which are in common use across organizations that are distinctly not lean. In other words, these are things you need to stop doing.
Seven deadly sins
The seven deadly sins are practices that defy the laws of effective integration. I first wrote the integration laws in 2002, and since they are just as valid today, they are included in my most recent book about Lean Integration. These laws should not be confused with the High Performance Computing and Communication Act of 1991 which resulted in the National Information Infrastructure, or the Enterprise Integration Act of 2002, or the U.S. National Intelligence Strategy for sharing information, which is derived from the Intelligence Reform and Terrorism Prevention Act of 2004. The laws I described are not acts of congress, but rather laws of nature like gravity and electromagnetism.
Integration “laws” are a way of thinking about the fundamental drivers of integration. The laws reflect the reality of dealing with systems of systems that emerge at the enterprise level. They represent the reality of the real world of IT just like the laws of physics describe characteristics of the real world. An effective integration strategy must not conflict with the integration laws since ignoring them will likely add to the list of failed projects. For a brief description of the laws, pick up a copy of my bookLean Integration: : An Integration Factory Approach to Business Agility or visit this blog for more information. In the meantime, here is the list:
- Law No. 1: The whole is greater than the sum of its parts.
- Law No. 2: There is no end-state.
- Law No. 3: There are no universal standards.
- Law No. 4: Information adapts to meet local needs.
- Law No. 5: All details are relevant.
The question you may be asking is “What practices should my organizations avoid to ensure we don’t get trapped in the pitfalls of these integration realities?” Here now are the seven deadly sins that are the opposite of lean integration practices:
1. Point-to-Point integration: This practice looks very seductive. The typical thinking is “I just need to exchange information between these two systems. If I treat everything else as external and irrelevant, I can safely ignore them and build a custom integration fast and cheap.” The problem with this practice is that it ignores the first law, The whole is greater than the sum of the parts, and will result in an unmanaged integration hairball 100% of the time and, in the long run, is neither fast nor cheap as organizations build hundreds and thousands of connections.
Point-to-point integrations add unnecessary cost and complexity, and destabilize the enterprise wide system-of-systems to the point where production surprises are common occurrences and system changes take months or years rather than days or weeks.
2. Losing sight of the customer: This deadly sin occurs when teams focus internally on their activities. While their “customer” may be another internal team or function, they forget that they are serving someone. This is where the over-the-wall syndrome comes from, with thinking that goes something like, “I finished writing the requirements and sent the document to the development group, so my job is done. Besides, the next release cycle has started and the schedule demands that I start the next set of requirements.”
By contrast, Lean acknowledges Law No. 5: All details are relevant, by demanding that every person or group think of the next person in the value chain as their customer and to ask, “Did I deliver everything they needed to get their job done?” An effective metric for this is the percentage of deliverables which make it through an entire project life-cycle without being reworked. When most organizations start measuring first-time-through-percentage, the result is typically 0%. But after a while, people get the message and change their behavior to understand the true needs of their internal customers.
3. Documentation using Word: The problem with most office productivity tools is that they are totally unstructured which means there is lots of room for interpretation and miscommunication (as per Law No. 2, Information adapts to meet local needs). This is the root cause of the ubiquitous project binder gathering dust on the shelf. And just because all the unstructured documents are stored in a central repository, that doesn’t necessarily make it easier to find or interpret the information you are looking for.
General desktop productivity tools can and should be used for documenting perishable information, but anything that needs to survive beyond the end of the project should be stored and maintained in a structured repository with formalized meta-data which describes, and constrains, the meaning and relationships of the data.
4. Agile development without standards: Why is it that NASCAR teams can change tires and refuel a car in seconds? Key ingredients are standard tools, well-defined repeatable processes, and continuous improvement through practice and learning. Agile software development methods are also intended to deliver fast results, but if the investment isn’t made in establishing standard tools, consistent data models, and repeatable processes, then the result is the continued propagation of an unmanaged hairball of integration points. As per Law No. 3: There are no universal standards, the tricky part is figuring out what to standardize without adding unnecessary bureaucracy. Lean Integration provides some powerful techniques to achieve the right balance.
5. Independent teams re-inventing the wheel: While the data being exchanged in each of the thousands of integration points in an enterprise is unique, the reality is that there are a relatively small number of patterns that have common solutions. If teams are working independently in silos, the patterns will be implemented from scratch each time which drives up the initial development cost, and more importantly, the ongoing maintenance burden. With middleware tools that have had 15 years to mature and integration competency practices that are well-established and proven, there really is no excuse any longer for organizations to allow individuals to custom-craft each integration point.
6. Relying on integration tests to find quality issues: Large-scale integration test cycles are a deadly sin since they are a self-fulfilling prophecy. The thinking starts with the idea that, “Integration testing is complex and expensive, so we should batch up a large number of changes to test them all at once. And since we always find a large number of defects, we better allow for several test cycles.”
This of course breeds the mindset that, “Let’s get the requirements and coding done quickly. We can always fix during integration test.” Lean integration on the other hand focuses on building quality in rather than inspecting quality in. In other words, ensuring that each step of the value chain is done correctly in the first place to minimize surprises in testing. The result is a major paradigm shift and a move to many small production releases rather than a few large releases which in turn increases organizational agility.
7. Adding complexity without systematic portfolio rationalization: It is a natural human tendency to be enamored with new things. The result of continuously adding new systems, data, and functionality in a piecemeal fashion (either organically, through acquisition, or otherwise) without also investing in simplification and consolidation is once again the unmanageable hairball. In support of Law No. 2: There is no end-state, we need to constantly be balancing the drive for delivering new functionality quickly with the discipline to rationalize the legacy environment to keep up with the changes.
Interestingly, and perhaps not coincidentally, Lean Integration offers seven principles of effective and sustainable integration which present a way to reform the practices based on the seven deadly sins. There really is a better way, and it’s not rocket science.
John Schmidt is vice president of Global Integration Services at Informatica Corp. where he advises clients on the business potential of emerging technologies and directs the company’s Integration Competency Center Practice. Previous employers include Wells Fargo Bank, Bank of America, Best Buy, American Management Systems, and Digital Equipment Corporation. He has written hundreds of articles on Systems Integration, Enterprise Architecture and Program Management, is a frequent speaker at industry conferences, and served as Director and Chairman of the Integration Consortium from 2002-2009. John co-wrote the first book about ICCs in 2005, Integration Competency Center: An Implementation Methodology, and followed it up in 2010 with Lean Integration: An Integration Factory Approach to Business Agility.