SOA, The (Business) Process Layer, and The System Architecture

As I discussed in my posts The Paradigm Shift of Service Oriented Architecture and SOA in a Rapid Implementation Environment, there is a significant cultural/business/technical shift in thinking when an organization starts to migrate from a fragmented, monolithic IT architecture, or Object Oriented/Web Service using code development using COTS or custom developed applications to Composite Applications within an SOA.  This key shift is formal linking of the IT application with the business process instead of the functions that make up the process.  I discuss several of the advantages in Assembling Services: A Paradigm Shift in Creating and Maintaining Applications.

Process and The Assembly Line

While it may seem like much of a change to link the application to the process instead of the function, consider this concept in the context of an assembly line.  An assembly line is the tooling enabling and supporting the process of assembly.  In turn, a process is "a set of activities ordered to achieve a goal", the goal or operational mission for the assembly line is to assemble the product with no defects.  I suspect that all manufacturing engineers would agree with that goal, but understand that Murphy's law works on all assembly lines, so they attempt to minimize the number of defects, still... On an assembly line, as much as is feasible, all of the activities and their enabling and supporting tooling works in concert with one another, orchestrated by the manufacturing engineering team.  This team spends significant time and effort measuring the flows through the assembly line to ensure, a) that the line produces products with as few defects as possible, while b) producing those products as cost efficiently as possible (e.g., minimizing the work in process [WIP]), and c) meeting the constantly changing demand with as little inventory as possible (which constitutes operational agility for the assembly line).  Additionally, the manufacturing engineers want the tooling to be agile, so that as the components of the product change, with changes in customer's requirements (and wants), the cost of retooling is minimized.

Process and IT

Contrast that with the way organizations engineer IT.  From its inception as supporting single functions, like accounting and payroll, with card-based semi-mechanical "electronic computers" in the late 1950s, IT has focused on supporting individual functions.  By the early 1980s the result was apparent to anyone that developed or used applications; they were information silos, that is, the tooling supporting each function had associated data, but there were no interfaces between functions to enable activity 2 in the process to share its data with activity 3, for example.  During this time, I worked at a major university.  In one Dean's office, that I supported, the Dean's administrative assistant had five terminals on her desk.  Why?  For two reasons; she needed data from five applications to perform her job of supporting the Dean, and she needed output from one application to serve as input to another--she was, in fact, the interface between the functions.

Since this situation was intolerable and expensive for operating the functional applications, the developers/coders created interfaces as the customers identified the need (and had funding to provide for it).  Consequently, the application layer functional design went from a series of virtually unconnected silos of data and code to something approaching a picture of a bowl of spaghetti.  In one study I did measuring the cost of maintenance of each link in this "fragmented" architecture, I found that for one small but business critical process for one of the business units, the cost was approximately $100/month/link.  Considering that even in mid-sized organizations, if half the maintenance cost of what I found is representative (i.e. $50 per link per month), then even a mid-sized organization would be spending a significant portion of its IT budget on link maintenance.  This cost created great pressure by finance engineering on IT management to become more cost efficient. 

Again, manufacturing engineering provided the concept, Materials Resource Planning (MRP).  The MRP systems interlinked all of the data systems and IT functions supporting the manufacturing floor by integrating them into a single application, thereby eliminating all of the linkages among functions.  This concept was expanded into MRPII, and then into the Enterprise Resource Planning (ERP) systems, like the SAP and Oracle tools.  The architecture is call Monolithic because all of the functions are tightly coupled in a single application.

Within a couple of years system users found several issues with ERP systems.

• The ERP systems are difficult to tailor to support the organization's processes.  Typically, the suppliers of ERP systems build the functions to a group of standards that are used by most industries or most organizations within an industry.  If for whatever reason, a) the enterprise that is implementing the ERP does not have a process that uses those standards, then either the enterprise must change its process--unlikely--b) it must attempt to tailor the ERP system to match its process, or c) some combination of the two.  Generally, none of the alternatives are particular effective or popular in large organizations with a diverse product mix.  The result is that either the initiative fails completely or that the personnel of all of the business lines work much harder and longer (and possibly create "shadow systems") to overcome the limitations imposed by the ERP system.
• The Systems are difficult to update.  Frequently, the software supplier implements the next version of a package  the equivalent of installing, configuring, and tailoring the original system.  The reason is that they have updated the technology, perhaps the functional architecture (design) sufficiently, that any bolt-ons, add-ins, or other tailoring will not work with the new version.
• The Systems are operationally inflexible.  That is, they are not agile; able to respond successfully to unexpected challenges and opportunities.  Because the tailoring required for an ERP system can be major, the costs and time involved becomes a significant item in the organization's budget.  Consequently, the time to change becomes long and the cost of change becomes high.  This leads to the problem that IT systems have become an impediment to change rather than an enabler, as studies by Gartner and Forester have shown.

The advent of Object Oriented Programming and its successor, Web Services, enabled organizations to partially address the issues above.  As discussed in my post SOA and a Rapid Implementation Environment, OOD design enables a developer to treat an application as a set of objects.  If these objects use Web Services Standards (either WS or JSR) for their interfaces, then they are Web Services.  In effect, this transformed an application into a set of components (classes, or Web Service components) that could be lashed together to create the application.  The advantage of the OO software architecture is that each of the objects is a small code module that could be updated or replaced with new technology and as long as the objects external interfaces and the resulting functions remained the same, the application would operate in the same manner.  This change met two of the three issues noted above; updating of the application's technology is possible without a complete re-installation and configuration of the application and initial tailoring of the application is much more feasible.

While this tailoring of an application made up of objects or Web Service components does address the agility issue, it is only in a very minor sense.  Since Object Oriented Architecture only enables an implied workflow, rather than making it explicit and independent of the Service Components, it would require reprogramming to change in response to unexpected challenges and opportunities.

Process, SOA, and the System Architect

 As discussed in my post SOA and a Rapid Implementation Environment, SOA formally separates the process flow from the functions.  This formal separation means that the organization can have its transformation teams restructure and order the functions without reprogramming of the functions themselves.  Instead, they can simply be reassembled (see my post Assembling Services: A Paradigm Shift in Creating and Maintaining Applications).

Formally and explicitly assembling Composite Applications using orchestration and choreography enables the System Architect to model the organization's (business) process for both effectiveness (in achieving the organization's mission) and cost efficiency and then to model the Composite Application linked with the process to determine its ability to optimally enable and support that process in both an effective and cost efficent manner.  Consequently when the organization's processes require change in response to changes in the organization's mission, strategies, the organization's environment, or technology, the System Architect, together with the rest of the transformation team may only need to change the process flow Component,  may need to add or update certain Service Components supporting functions, or both.  Still, this will make much more agile support for the processes.  This links the SOA-based Composite Application directly to the organization's process.

There is an additional benefit to separating the process flows from the Service Components.  When the organization's automated business rules change in response to changes in policies and standards (which in turn should be in response to changes in Governance), the System Architect can very quickly evaluate the change by modeling the process and Composite Application to understand the consequences of the change.  In the best situation, the System Architect would model all affected applications before any proposed change in policies, standards, or business rules is put into place because this would enable both the Governance and Policy Management teams to understand the consequences and results of the proposed change before those changes become unmanageable negative externalities.

Obviously, this is a significant shift in the organizational culture.