SOA Architects: "Don't Bury the Business Rules in the Database"

Why Rules Get Buried in Databases

Even though the Millennium Bug was first noted in 1982 (or perhaps earlier), it was not widely recognized until the mid-1990s.  Fundamentally, the "millennium bug"was caused by the need to minimize the amount of data storage during the 1960s and 1970s and the lack of understanding of System Architecture earlier.

Fragmented Architecture

Initially, the business logic was hard coded into the programming.  Partially, this was due to the fact that in the 1950s and early 1960s, computers only assisted in supporting the mathematical (e.g., accounting) functions of an organization.  So, it was rather straightforward to code in the Business Rules (the if-then-else, as I discuss in Types of Business Rules).  However, as the System Developers attempted to integrate these functions and as the organization's leaders attempted to respond to a changing organizational environment and to changing governmental regulations, they found that changing the enabling and supporting IT systems difficult and expensive (And many large organization's still do, since they are still dependent on millions of lines of code written in the 1960s and early 1970s).  A great part of the reason for this lack of agility was and is the hard-coded business rules in the program's logic--some of which is documented only in that logic.

Monolithic Architecture

In the early 1970s, as disk storage became somewhat less expensive, Relational Data Bases (RDBs) and Relational Data Base Management Systems (RDBMS) started to replace earlier data storage systems.  The RDBMS enabled developer to map data elements into tables with relationship among all the data.  This mapping essentially converts data into information (see the Blog's glossary page).

The advent of the RDBMS was one of the technical enablers of the MRP, MRPII, and ERP systems, built using Monolithic Architecture.  For example the SAP R3 system could not exist without an associated Oracle or other RDBMS system.  The reason is that the Monolithic Architecture is superior to a Fragmented Architecture because it reduces the number of interfaces that have to be maintained among programs to nearly zero--and as discussed in SOA, The (Business) Process Layer, and The System Architecture I found that the cost per interface was $100 per month.  When you multiple that by the 1000s or 10,000s of interfaces required in a large organization with a Fragmented Architecture, the apparent cost efficiency is enough to delight any finance engineer, be they the CFO or any of his or her minions.

Once the RDBMS systems took hold, the Database Designers (DBD) and coders found that the easiest point to validate the data input was just before data insertion into the tables.  The RDBMS suppliers responded with triggers, (if-then-else logic within the RDBMS).  These triggers rejected data that was out of conformance with some standard; which is the essence of a Business Rule.  However, once the DBD was given this tool, he or she started creating ever more complex triggers; triggers that embody Business Rules that may have little to do with data validation.  For example, many times when data is inserted a trigger will change a flag or other indicator within a set of metadata (data about the data).  The flag may then change the course of the function or business process.  This type of trigger buries the Business Rules in the database.

The problem with an application built on a Monolithic Architecture is the inflexibility of the entire system, as I've discussed in The (Business) Process Layer, and The System Architecture and several other posts.  One key technical element of this inflexibility is the System Architect and SME's inability to change or tailor the data structures because the application is dependent on a single data structure (as well as a standard process, etc.).  When the triggers in the RDBMS include a significant number of Business Rules, this increases the inflexibility because the detailed designer/coder/SME may have great difficulty in even locating the Business Rule.  Therefore, he or she must write an add-on or bolt-in to modify the Business Rule, with all of the inherent configuration management problems that creates.  Further, even if the detailed designer can find the proper trigger to modify, ERP systems do not usually identify all of the functions that use the data or the trigger, so there is a significant risk of induced defects, defects in a secondary process caused by the rules change to support the primary process or function of the transformation effort.

Service Oriented and Other Agile Architectures

As I discussed in  The (Business) Process Layer, and The System Architecture, and several other posts, Service Oriented Architecture (SOA) and other agile processes require separation of the processes and Business Rules from the functions of the process. From the discussion, above, the reasons should be obvious.  Both good Governance and Policy Management (see A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture, and Enterprise Architecture, Chaos Theory, Governance, Policy Management, and Mission Alignment) and Agile Mission Alignment require this separation.  Without a Rules Repository, the Enteprise Architect and the System Architect are much less effective at optimizing the SOA or other agile architecture to support the organization in a constantly changing operational and technical environment.

The Prescription: an Agile Architectural Process Pattern

As I see it, the following is a prescription for transforming an organization from function-based to process-based, and the IT architecture of the enabling and supporting systems from Fragmented and Monolithic to agile and Service Oriented is to first understand and target agility, in addition to process effectiveness and cost efficiency.  Second, define or ensure that there are formal Enterprise Architecture (Mission Alignment, Governance and Policy Management) and Systems Engineering/System Architecture processes that coordination with one another.  These processes should have a three month cycle, not a year or more, to ensure requirements agility.  Third, as I described in my posts, Initially implementing an Asset and Enterprise Architecture Process and an AEAR, and Asset and Enterprise Architecture Repository, start with the output of current projects to create the AEAR and exercise the processes.  Fourth, expect that the processes will be less than optimal at the start (the learning curve, see the Superiority Principle).  The risk reduction on this is that since the process cycle is three months, instead of a year or more, learning takes place at a much greater rate.

Addendum: When to Use Triggers When Building to An Agile Architecture
I had a futher thought about triggers.  The Gartner Group recommends segmenting the complete data structure (supporting a monolithic architecture) into databases supporting the individual Service Components (e.g., Web Service).  Each of these Service Components would support either an entire or a logical portion of an organizational function.  I would suggest that triggers can be used within the databases supporting individual functions (Service Components), especially for ensuring data completeness and correctness.