Back to the Future: How We can Synergize the Current Polarized Politics While Integrating Technology

 Currently, the political situation in the United States is similar to the political chaos in the 1850s—moving toward the Civil War.  Like the Civil War, it is a war between economic architectures, “crony” capitalism and Marx’s Communism.   However, there is a better way, returning to the organizational economic architecture of the founding fathers, in particular Thomas Jefferson.

There is a second overriding issue, of which we are all aware, the change from the Age of Print to the Digital Age that is creating more cultural (and even biological) change than most people are aware of.  Right now, the smart phone is becoming the third lobe of our brains.  It remembers the phone numbers of our contacts, it enables us good asynchronous communications with them (in the form of text messages and email), it tells us the weather (current and future), it sings to us the music we want to hear, it even tells us where to drive our vehicles, and so much more.

But it also knows where we are, what we are doing, what we’re interested in, and who we are doing it with.  In the near future it will enable politicians and bureaucrats to control all our activities; that is unless advanced artificial intelligence enables the technology to take over and reduce humanity to slaves of the singularity.

Again, there is a better way.  My book, Jeffersonian Economic Architecture in the Digital Age discusses an alternative.  It starts, in part 1, with a new theory of the definition and source of economic value. In this part economic value and wealth is defined as a direct result of knowledge. An increase in knowledge increases the organization’s ability to survive and thrive.  It goes on to demonstrate that this concept and theory is very likely to be true.

Part 2 describes and discusses organizational architecture.  This architecture applies equally to individuals and nations throughout human history.  It starts with a static enterprise architecture based on the IDEF0 model.  It details the functions and components of that architecture to show how it interlinks these functions of input, process, output, control and mechanisms.  It then adds a process to the architecture using the OODA (observe, orient, decide and act) loop, to produce an organizational architecture.  Further, it shows how by using this model any organization can become more effective in what it’s producing and more cost-efficient in how it produces it.

Part 3 uses this organizational architecture model within the historical context of humanity to support the reality of organizational model.

Part 4 interlinks this organizational model with both the Declaration of Independence—mostly written by Thomas Jefferson—The US Constitution—of which he was involved—with recent technological concepts of Services Oriented Architecture and Cloud Computing to produce a political/economic architecture that allows and enables everyone the “pursuit of happiness” wherein “pursuit” means work at what they want to do and have a talent for.  It also means that there will be a great many more millionaires and very few billionaires, that is, it enables sharing the wealth rather than forcing the wealth through governmental redistribution taxes.

While it would be nice to receive royalties from the sale of this book, my purpose for writing it is my hope the people will take a serious look at the political right and left, and technology change, and consider the far-reaching consequences of not starting from the concepts in this book to transform the political/economic system without having a pyric civil war.

I would appreciate any thoughts or ideas you have on this paper or on Jeffersonian Economic Architecture in the Digital Age.

The Digital Future: Services Oriented Architecture and Mass Customization, Part 1

Part1: The Digital Future

I was challenged to forecast changes in economic systems based on both my knowledge of spatial economic systems and on my experiences with computers, data networking, and automation.  What I have come up with is a four part article on the digital future.

Since I normally tend to build to a thesis like any good engineer designing a product, instead of stating my thesis and then defending it, like lawyers, and journalists normally do, I will take a shot at the thesis of this paper first.

 

We have entered the Digital Age in which Capitalism, which describes the economic system of the Age of Print will be succeeded by Economic Services Oriented Architecture producing Mass Customization.  It will be an age where consortiums are formed small and entrepreneurial organizations to produce products, systems, and services the customer wants.  This architecture will replace the current organizational architecture of a single large organization producing a large quantity of products that “satisfice”, that is, they come somewhat close to satisfying the customer’s requirements—they suffice.

The article is constructed in five parts.  This part, Part 1, discusses the economic history of humankind based on how they have communicated and stored data and information.  I feel it’s important to provide the context for my forecast.

The Second Part is a discussion of the coming of the Digital Age based on my experiences seeing it over the past 50+ years.  I’ve found a structure in pattern in the seeming chaos of change in data and information storage.  This pattern leads me to architectural pattern changes that lead to my forecast.

The Third Part is a more detailed discussion of this new architectural pattern called Services Oriented Architecture (SOA).  I will give a couple of examples to demonstrate how SOA will work economically.

The Fourth Part will consider how SOA and the Digital Age will change an individual’s life by giving three examples.

The Fifth Part will show how converting to SOA will create changes as drastic as the changes from the feudal economic architecture to the Industrial architecture.   In this part, I will forecast the change to a number of industries.  Most of these changes are already starting to occur, though in a very minor way.

An Exceedingly Brief History of European and American Civilization

There have been four ages for humankind. 

The Age of Speech

The age of speech (verbal communication), from circa 300,000 BC to circa 6,000 BC  was when for the first time data, information, and knowledge could be transferred and store within and between generations.  This was the first time when clans and tribes formed.  And, according to archeologists, there was a glacially slow revolution from hunting and gathering and stone to agriculture and metals.  This was the economic architecture of the time.  During this period, the shaman, or priest was holder of the tribal information base.

The Age of Writing

The age of writing (written communication), from circa 6,000 BC to 1455 AD, was when data, information, and knowledge could be more accurately transferred longer distances and stored for much longer time periods (in fact, there are documents and records over this entire period).  Political institutions increased from tribes migrating all over the landscape to settled (or at least apparently) settled city states, and then to regional and national states.  This was the second form of an economic architecture.

During this age the first known libraries and colleges formed; for example, the library and museum (college/research center) at Alexandria.  And again, more than 600 years later, after the various barbarian tribal invasions sent Europe back to the talking age, (during the dark ages) up to 900 AD when Carolus Magnus (or Charlemagne) manage to very slightly reintroduce writing and then colleges were formed in what is now Italy.

The Age of Printing

 The age of printing, (printed communications), from 1455 AD to between circa 1942 to 1992, data, information, and knowledge, became much more readily available to humankind.  Thanks, in large part to Martin Luther insistence that everyone should be able to read the Bible, Northern Europe learned to read and read ideas and concepts that were not part of the Catholic Church Doctrine.

By 1776, Adam Smith had described how wealth was created, together with the growth of engineering knowledge, and the ability of individuals to take risks and fail or succeed, Humans entered the era of Mass Production and Liberty.  This is the basic economic architecture of the Age of Printing.

Included in the mass production was mass production of education, based on a school for all teaching reading, writing, and arithmetic.  This has led to the mass production educational systems of today.

Knowledge and Wealth

You should note that with the speech, humanity grew significantly wealthier than other animal species.  The reason is that they could accumulate more and better data, information, and knowledge through speech.

With writing, humanity accumulated a much more wealth.  This wealth was exceedingly badly distributed. Nonetheless, looking at places like Pompeii, even some of the slaves could accumulate small wealth (while “the bread and circuses” form of socialism led to the eventual destruction of the Roman Empire).

With printing, a much large chunk of humanity created orders of magnitude more wealth.  The accumulation of knowledge of how the Universe works has led to mass production, which meant mass wealth.  For example, if there is a disaster now, people expect the restoration of power, water, fuel, and communications immediately; this was never true for even the “wealthiest in the age of speech, writing, or even for most of the age of print.  This demonstrates how exceedingly rich even the poor are today, when compared with the rest of human history (This is something the liberal entitlement generation has forgotten).

The Digital Age: The next Age

The next Age has begun.  It began, in a real sense with WWII.  It gestated throughout the 1950s to the mid-1960s.  I will discuss this period and beyond in the next part of this article.

ROI Versus VOI

Return On Investment

According to Investopedia (http://www.investopedia.com/terms/r/returnoninvestment.asp#ixzz1bcKSKpx9)::

"A performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a percentage or a ratio.

The Return On Investment formula:

In the above formula "gains from investment", refers to the proceeds obtained from selling the investment of interest."

 ROI is simple and straightforward, but does not enable or support an organization's ability to make investment decisions.  It is a lagging indicator; it tells you what were good and bad decisions in the past, but nothing about deciding on investments to increase the long-term effectiveness or agility of an organization, which are vital in today's "bonkers" business environment (to use Tom Peters' term), and disruptive technology environment.

One famous example demonstrates this point.  When Jeff Bezos founded Amazon.com in 1994, he knew that he needed customers to create what he envisioned as "The Walmart of the Web".  This meant that he had to decide on a mission, which was to create a highly visited website.  He then settled on at least four strategies.  First, concentrate on selling books, don't try to sell all the products initially.  Second, create a website with a superb user experience.  Third, ensure that Amazon has access to all books any of its customers could possibly want.  Fourth provide value to the customer, to the point of selling books below wholesale cost, to generate website traffic.

Value On Investment

None of these strategies are designed to produce an ROI, but especially the fourth.  In fact, the mission of creating a highly visited site does not create ROIObviously, if the vision of creating the "Walmart of the web" is realized, there will be a good deal of ROI.  But the mission to start to realize the vision does not support ROI--but it does create Value On Investment (VOI).  A decision-maker creates Value On Investment when his or her decisions increases the organization's Production Capability (after Dr. Stephen Covey), that is, the ability of an organization to be more effective at achieving its vision and mission.

Jeff executed and persevered in the mission to increase Amazon's growth to these disciplined strategies to increase Amazon's VOI, which is the point of the mission to create a highly visited website, that is, a highly visited site is much more valuable than a site that is visited very little.  In executing this mission, it did not turn a profit until 2000, much to the ire of many of its investors, but it took an increasing share of the customer base that itself has been growing at an increasing rate.  Therefore, Jeff met this mission for Amazon.

The enterprise architecture for Amazon was constantly upgraded, including software, hardware, and physical infrastructure.  The net result is that Amazon is now competing with all other major retailers for customers.

Unfortunately, too many CEOs give into the nanosecond market that expects an increasing ROI every quarter or sooner.  It is like expecting George Washington to beat the British in every battle in the Revolutionary War.  Washington lost far more often than he won, but together with the other founding fathers, he did create an enormous VOI that subsequent generations have transformed into ROI.  Likewise, NASA in the Moon Mission created VOI that Jeff Bozos, Bill Gates, Steve Jobs, and the world have profited from.

If the people of the "Occupy Wall St." movement have a point, its that there is too much focus on Wall St. for ROI and too little on VOI.  Wall St. is using the wrong metric for the wrong duration.  And they are "leading" the rest of us into bankruptcy as a result.

Short Cycle, Agile, Level of Effort efforts, and Changes in Roles and Responsibilities

In a recent post I briefly discussed the changes in roles and emphasis when a development or transformation effort changes from a waterfall (Big Bang) effort to a short cycle-agile effort.  This post will discuss the topic in more detail in terms of a Short-Cycle, Agile, Level of Effort projects and programs.

Short Cycle

A development and transformation effort is a program or project that is changing some process, component, procedure, or tooling that supports an organization's Vision, Mission, and Strategies.  There are two types of efforts, Big Bang, and Short Cycle.  Big Bang development and transformation processes are straight line processes (see my post Product Architecture Thinking Versus System Architecture Thinking) in which there series of steps from requirements identification, through design, implementation, validation, and roll out.  It delivers the product in a single delivery--one Big Bang.

Alternatively, the Short Cycle (1 to 3 months) development and transformation process is a process that delivers the functional of the total product or system in small increments through a series of short duration development or transformation cycles.  Typically, the deliverable from the first cycle (a one month cycle) is "usable", after a fashion, and is called the Initial Operating Capability (IOC) of the system. 

I've found that the IOC will consist of the initial version of the system's access control together with a minimal set of input screens with associated data stores.  The reason for this IOC functional architecture is that security of an operational system is normally a "business" requirement; it's not wise to operate an open system.  Second, you need to "put garbage in" to "get garbage out."  If the team builds a report during the first cycle with no way to insert data to support the report, then the system is not operational in any sense.  However, if you build input screens with associated data store during the first cycle, then the customer can start inserting data during the second, while the team will likely be adding to the number of input screens and developing at least one report or information display.  My experience has been that it will take the customer about a month to get enough data in to have usable reports and output displays.  This means that by the end of the second cycle the system will be producing at least a minimum ROI for the customer.  This ROI will increase with each subsequent short cycle, which delights most customers.

Agility

The Agility of an organization  is defined as "its ability to successfully respond to unexpected challenges and opportunities".  The "successfully respond" phrase has two dimensions: making the right decision and making a timely decision.  Various components of the US military have a saying  "improvise, adapt, and overcome" that embodies the concept of agility.  This has long been a tradition of the US military.  For example, the Allies won D-Day in part, because they were much more agile.  That is, the NCOs, junior officers, and field grade officers improvised new tactics, based the manpower and equipment available, to achieve the initial mission of creating a beach head, when all of the plans for the invasion were breaking down--the capture of Omaha Beach is the seminal example.  On the other hand, the German forces followed their defense plans as closely as possible.  Consequently, senior officers were asking permission to move units to defend the coast.  By the time Hitler, hundreds of miles away, made the decision to release the panzer units, the Allies had enough units ashore to defend the beach head. 

The same definition of Agility can be used for a development or transformation process.  If a process is founded on the assumption that "All of the customer's System Requirements are known up front" then I would suggest that the process is rigid and inflexible--a totally brittle process.  That is, if new customer System Requirements are discovered as the development or transformation team is designing and implementing the system, then a formal Engineering Change Order (ECO) process (with contractual changes) is required.  All of this "administriva" costs time and resources that the team could otherwise use to create more of the product or system the customer wants.  Consequently, the customer will identify only those new requirements that will make the product or system completely unusable if not met.  This leads to poor systems and unhappy customers.  This is in contrast, agile processes based on "Not all the customer's System Requirements are known up front" so as to support the inclusion of new requirements as the system develops.

Level of Effort

Short cycle, agile development and transformation efforts require using the Level of Effort (LOE) pattern of contractual management rather than any other type.

What is a Level of Effort Project or Program?

A Level of Effort (LOE) program or project is one where the budget for the effort is spent uniformly across the duration of the effort.  Therefore, the amount of development or transformation work is uniform across the effort's duration.  This differs from the Big Bang style of development in that, normally, the Big Bang starts with a small team performing the initiation tasks, builds up the effort during detailed design and component and assembly verification and reduces the effort during post-roll out product or service support.  If the product has too many defects the PM as the ability to add personnel and use up the budget faster.

With a LOE project or program, the PM does not have that ability.  Instead, if the design team has under estimated the complexity or risk in meeting a requirement, they have the ability to refactor the requirement into two or more requirements.  They can then complete the first of these during the current segment and the others during later segments.  The fact that I'm using the term segment implies that I think that an LOE effort can only be successfully used with short cycle efforts; each segment being a cycle.  So, in short cycle, agile, LOE efforts, the System Requirements are the variable, while the budget and schedule are held as constants.

Why is it important?

Currently most efforts are still in the Big Bang style because most formal processes are of that type.  These processes include all of the required intermediate artifacts like a project plan, schedule, PDR, CDR, EWBS, IWBS, documentation for PMRs, ECOs, weekly status reports, and so on.  The reason I call these intermediate artifacts (or work products) is that they have no lasting value for the customer.

For example, since LOE projects and programs produces a uniform amount of output for each equal segment of these development or transformation efforts, there is little need for metrics like "Earned Value".  This is particularly true in short cycle development and transformation efforts where the customer can see and use the work products.  If properly performed at the end of each cycle, be it monthly or every 3 months, either the IOC version or an upgrade is rolled out.  It can be rolled out into the operational environment, or in some cases into a preproduction environment (an environment where the customer can use the new system in parallel with the old system).  Since the customer can use the IOC or upgraded system, what is the purpose of metrics like the "Earned Value" metrics (since the goal of the EV metrics is to show progress)?

Changes in Roles and Responsibilities of the Team

If, as I've experienced, there is little need for most intermediate artifacts supporting the PM procedures and methods because there is little need for those procedures and methods, then there must be changes in the roles and responsibilities of the program or project's team using a short-cycle, agile, LOE development or transformation process.

First, the process is short-cycle (one to three months) and agile ("not all the requirements are known up front"), so the process must have these two characteristics.  Having these two characteristics, immediately reduces the role of the PM, as discussed above.  No longer are all of the PM procedures  for a PDR, CDRs, EWBS, IWBS, PMRs, ECOs, weekly status reports, earned value reports, and so on necessary; in fact, they are counterproductive in that they reduce the effectiveness and cost efficiency of the effort.

Second, the emphasis is on creating or transforming a product or system to meet the customer's highest priority System Requirements, which may or may not be known at the start of the effort.  The last clause in the prior sentence is the high-level capability statement for agility.  This has two consequences.

• There is a requirement for capturing new requirements in every cycle of the effort.  This is within the role of the Systems Engineer.
• There is a requirement for the customer to prioritize the complete set of requirements at the start of each cycle.

These requirements indicates that the Systems Engineer's responsibility in identifying and managing requirements has increasing importance to any development or transformation effort.

[Sidebar:  There are studies to indicate that in software development efforts 10 to 20 percent of the software developer's effort is spent on functions not called for in the requirements.  Frequently, the customer asked for these to be removed, which costs more time and resources; and none of which produces value for the customer.  But this isn't only a problem for software developers today. A famous example is that at the start of WWII, the M-3 tank came rolling out of the factory with a police siren. No one could explain why since it wasn't in the requirements.  So, having the entire effort focus on the requirements will frequently greatly reduce costs of design and development as well as program management.]

Third, the development of functions or services are tied directly to the Functional Requirements, which link through a traceability matrix to the customer's System Requirements (see my post Types of Requirements for definitions) [though, in some smaller software development efforts, this level of decomposition may not be necessary].  Since, by the definition of a requirement, it must have a metric for when it is met, all verification and validation procedures and methods must be traceable to these metrics.  Additionally, with short-cycles the risk that the designers/developers/implementers will induce defects greatly increases.  An Induced Defect is one that is created in the process of a roll out or update of a system.  The key procedure to reduce the number of induced defects is regression testing--performing all of the same verification and validation procedures and methods used on prior releases.  Within short-cycle and agile processes, linking the V&V procedures and methods to the requirements and regression V&V emphasize both the importance of the requirements--therefore the requirements management system--and the role and responsibilities of the Systems Engineer.

Fourth, because, as discussed above, short-cycle, agile processes require LOE program management, the role of the PM is diminished further.  No longer can the PM "control" the effort by adding to reducing resources to a particular task.  Instead, the Systems Engineer must decide in each cycle how many of the highest priority requirements the team can meet during the next cycle.  Consequently, the PM is really only responsible for working with the suppliers and consultant (both external and internal to the organization), ensuring that the processes, procedures, and methods of the short-cycle, agile process are properly executed, and reporting results to higher management.

These changes in roles and responsibilities are a dramatic shift in the cultural of development and transformation.

Transformation Benefits Measurement, the Political and Technical Hard Part of Mission Alignment and Enterprise Architecture

Pre-Ramble

This post will sound argumentative (and a bit of Ranting--in fact, I will denote the rants in color.  Some will agree, some will laugh, and Management and Finance Engineering may become defensive), and probably shows my experiences with management and finance engineering (Business Management Incorporated, that owns all businesses) in attempting benefits measurement.  However, I'm trying to point out the PC landmines (especially in the Rants) that I stepped on so that other Systems Engineers, System Architects, and Enterprise Architects don't step on these particular landmines--there are still plenty of others, so find your own, then let me know.

A good many of the issues result from a poor understanding by economists and Finance Engineers of the underlying organizational economic model embodied in Adam Smith's work, which is the foundation of Capitalism.  The result of this poor understanding is an incomplete model, as I describe in Organizational Economics: The Formation of Wealth.

Transformation Benefits Measurement Issues

As Adam Smith discussed in Chapter 1, Book 1, of his Magna Opus, commonly called The Wealth of Nations, a transformation of process and the insertion of tools transforms the productivity processes.  Adam Smith called the process transformation "The division of labour", or more commonly today, the assembly line.  At the time, 1776, where all industry of "cottage industry" this transformation Enterprise Architecture was revolutionary.  He did this using an example of straight pin production.  Further, he discussed that concept that tooling makes this process even more effective, since tools are process multipliers. In the military, their tools, weapons, are "force multipliers", which for the military is a major part of their process. Therefore, both transformation of processes and transforming tooling should increase the productivity of an organization.  Productivity is increasing the effectiveness of the processes of an organization to achieve its Vision or meet the requirements of it various Missions supporting the vision.

The current global business cultural, especial finance from Wall St. to the individual CFOs and other "finance engineers", militates against reasonable benefits measurement of the transformation of processes and insertion and maintenance of tools.  The problem is that finance engineers do not believe in either increased process effectiveness or cost avoidance (to increase the cost efficiency of a process).

Issue #1 the GFI Process

Part of the problem is the way most organizations decide on IT investments in processes and tooling.  The traditional method is the GFI (Go For It) methodology that involves two functions, a "beauty contest" and "backroom political dickering".  That is, every function within an organization has its own pet projects to make its function better (and thereby its management's bonuses larger).  The GFI decision support process is usually served up with strong dashes of NIH (Not Invented Here) and LSI (Last Salesman In) syndromes.

This is like every station on an assembly line dickering for funding to better perform its function.  The more PC functions would have an air conditioned room to watch the automated tooling perform the task, while those less PC would have their personnel chained to the workstation, while they used hand tools to perform their function; and not any hand tools, but the ones management thought they needed--useful or not.  Contrast this with the way the Manufacturing Engineering units of most manufacturing companies work.  And please don't think I'm using hyperbole because I can cite chapter and verse where I've seen it, and in after hours discussions with cohorts from other organizations, they've told me the same story.

As I've discussed in A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture, the Enterprise Architect and System Architect can serve in the "Manufacturing Engineer" role for many types of investment decisions.  However, this is still culturally unpalatable in many organizations since it gives less wiggle room to finance engineers and managers.

Issue #2 Poorly Formalized Increased Process Effectiveness Measuring Procedures

One key reason (or at least rationale) why management and especially finance engineers find wiggle room is that organizations (management and finance engineering) is unable (unwilling) to fund the procedures and tooling to accurately determine pre- and post-transformation process effectiveness because performing the procedures and maintaining the tools uses resources, while providing no ROI--this quarter. [Better to use the money for Management Incentives, rather than measuring the decisions management makes].

To demonstrate how poorly the finance engineering religion understands the concept of Increased Process Effectiveness, I will use the example of Cost Avoidance, which is not necessarily even Process Effectiveness, but is usually Cost Efficiency.  Typically, Cost Avoidance is investing in training, process design, or tooling now to reduce the cost of operating or maintaining the processes and tooling later. 

[Rant 1: a good basic academic definition and explanation cost avoidance is found at http://www.esourcingwiki.com/index.php/Cost_Reduction_and_Avoidance.  It includes this definition:

"Cost avoidance is a cost reduction that results from a spend that is lower then the spend that would have otherwise been required if the cost avoidance exercise had not been undertaken." ]

As discussed in the article just cited, in the religion of Finance Engineering, cost avoidance is considered as "soft" or "intangible".  The reason finance engineer cite for not believing cost avoidance number is that the "savings classified as avoidance (are suspect) due to a lack of historical comparison." 

[Rant 2: Of course Cost Reduction Saving is like that of avoiding a risk (an unknown) by changing the design is not valid, see my post The Risk Management Process because the risk never turned into an issue (a problem).] 

This is as opposed to cost reduction, where the Finance Engineer can measure the results in ROI.  This makes cost reduction efforts much more palatable to Finance Engineers, managers, and Wall St. Traders.  Consequently, increased cost efficiency is much more highly valued by this group than Increased Process Effectiveness.  Yet, as discussed above, the reason for tools (and process transformations) is to Increase Process Effectiveness.   So, Finance Engineering puts the "emphassus on the wrong salobul".

They are aided an abetted by (transactional and other non-leader) management.  A discussed recently on CNBC Squawk Box, the recent the CEOs of major corporations cite for their obscenely high salaries is that they make decisions that avoid risk. 

[Rant 3: Of course this is ignoring the fact that going into and operating a business is risky, by definition; and any company that avoids risk is on the "going out of business curve".  So most executives in US Companies today are paid 7 figure salaries to put their companies on "the going out of business curve"--interesting]

However, Cost Avoidance is one of two ways to grow a business.  The first is to invent a new product or innovate on an existing product (e.g., the IPAD) such that the company generates new business.  The second, is to Increase Process Effectiveness. 

Management, especially mid- and upper-level management, does not want to acknowledge the role of process transformation or the addition or upgrade of tooling as increasing the effectiveness of a process, procedure, method, or function.  The reason is simple, it undermines the ability for them to claim it as their own ability to manage their assets (read employees) better and therefore "earn" a bonus or promotion.  Consequently, this leaves those Enterprise and System Architects always attempting to "prove their worth" without using the metric that irrefutably prove the point.

These are the key cultural issue (problems) in selling real Enterprise Architecture and System Architecture.  And frankly, the only organizations that will accept this cultural type of change are entrepreneurial, and those large organization in a panic or desperation.  These are the only ones that are willing to change their culture.

Benefits Measurement within the OODA Loop

Being an Enterprise and an Organizational Process Architect, as well as a Systems Engineer and System Architect, I know well that measuring the benefits of a transformation (i.e., cost avoidance) is technically difficult at best; and is especially so, if the only metrics "management" considers are financial. 

Measuring Increased Process Effectiveness

In an internal paper I did in 2008, Measuring the Process Effectiveness of Deliverable of a Program [Rant 4: ignored with dignity by at least two organizations when I proposed R&D to create a benefits measurement procedure], I cited a paper: John Ward, Peter Murray and Elizabeth Daniel, Benefits Management Best Practice Guidelines (2004, Document Number: ISRC-BM-200401: Information Systems Research Centre Cranfield School of Management), that posits four types of metric that can be used to measure benefits (a very good paper by the way).

1. Financial--Obviously
2. Quantifiable--Metrics that organization is currently using to measure its process(es) performance and dependability that will predictably change with the development or transformation; the metrics will demonstrate the benefits (or lack thereof).  This type of metric will provide hard, but not financial, evidence that the transformation has benefits.  Typically, the organization knows both the minimum and maximum for the metric (e.g., 0% to 100%).
3. Measurable--Metrics that organization is not currently using to measure its performance, but that should measurably demonstrate the benefits of the development or transformation.  Typically, these metrics have a minimum, like 0, but no obvious maximum.  For example, I'm currently tracking the number of pages accessed per day.  I know that if no one reads a page the metric will be zero.  However, I have no idea of the potential readership for anyone post because most of the ideas presented here are concepts that will be of utility in the future. [Rant 5: I had one VP who was letting me know he was going to lay me off from an organization that claimed it was an advance technology integrator that "he was beginning to understand was I had been talking about two years before"--that's from a VP of an organization claiming to be advanced in their thinking about technology integration--Huh....]  Still, I have a good idea of the readership of each post from the data,  what the readership is interested in and what falls flat on its face.  Measurable metrics will show or demonstrate the benefits, but cannot be used to forecast those benefits.  Another example is of a RAD process I created in 2000.  This process was the first RAD process that I know of, that the SEI considered as Conformant; that is, found in conformance by an SEI Auditor.  At the time, I had no way to measure its success except by project adoption rate (0 being no projects used it).  By 2004, within the organization I worked for, that did several hundred small, medium, and large efforts per year, over half of them were using the process.  I wanted to move from measurable to quantitative, using metrics like defects per roll out, customer satisfaction, additional customer funding, effort spent per requirement (use case), and so on, but "the management considered collecting this data, analyzing and storing it to be an expense, not an investment and since the organization was only CMMI level 3 and not level 4, this proved infeasible.   [Rant 6: It seems to me that weather forecasters and Wall St. Market Analysts are the only ones that can be paid to use measurable metrics to forecast, whether they are right wrong, or indifferent--and the Wall St. analysts are paid a great deal even when they are wrong.]
4. Observable--Observable is the least quantitative, which is to say the most qualitative, of the metric types.  These are metrics with no definite minimum or maximum.  Instead, they are metrics that the participants agree on ahead of time--requirements? (see my post Types of Requirements.)  These metrics are really little more than any positive change that occurs after the transformation.  At worst they are anecdotal evidence.  Unfortunately, because Financial Engineers and Managers (for reasons discussed above) are not willing to invest in procedures and tooling for better metrics like those above, unless they are forced into it by customers, (e.g., requiring CMMI Level 5), Enterprise Architects, System Architects, and Systems Engineer must rely on anecdotal evidence, the weakest kind, to validate the benefits of a transformation.

Metric Context Dimensions

Having metrics to measure the benefits is good, if and only if, the metrics are in context.  In my internal paper, Measuring the Process Effectiveness of Deliverable of a Program, which I cited above, I found a total of four contextual dimensions, and since I have discovered a fifth.  I give two, to illustrate what I mean.

In several previous posts I've used the IDEF0 pattern as a model of the organization (see Figure 1 in A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture in particular).  One context for the metrics is whether the particular metric is measuring improvement in the process, the mechanisms (tooling), or in the control functions; a transformation may affect all three.  If it affects two of the pattern's abstract components or all three, the transformation may affect each either by increasing or decreasing the benefit.  Then the Enterprise Architect must determine the "net benefit."

The key to this "net benefit" is to determine how well the metric(s) of each component measures the organization's movement or change in velocity of movement toward achieving its Vision and/or Mission.  This is a second context.  As I said, there are at least three more.

Measuring Increased Cost Efficiency

While measuring the Benefits that accrue from a transformation is difficult (just plain hard), measuring the increased cost efficiency is simple and easy--relatively--because it is based on cost reduction, not cost avoidance.  The operative word is "relatively", since management and others will claim that their skill and knowledge reduced the cost, not the effort of the transformation team or the Enterprise Architecture team that analyzed, discovered, and recommended the transformation.  [Rant 7: More times than I can count, I have had and seen efforts where management did everything possible to kill off a transformation effort, then when it was obvious to all that the effort was producing results "pile on" to attempt to garner as much credit for the effort as possible.  One very minor example for my experience was that in 2000, my boss at the time told me that I should not be "wasting so much time on creating a CMMI Level 3 RAD process, but instead should be doing real work."  I call this behavior the "Al Gore" or "Project Credit Piling On" Syndrome (In his election bid Al Gore attempted to take all the credit for the Internet and having participated in its development for years prior, I and all of my cohorts resented the attempt).  Sir Arthur Clarke captured this syndrome in his Law of Revolutionary Development.

"Every revolutionary idea evokes three stages of reaction. They can be summed up as:
–It is Impossible—don’t Waste My Time!
–It is Possible, but not Worth Doing!
–I Said it was a Good Idea All Along!"]

Consequently, "proving" that the engineering and implementation of the transformation actually reduced the cost, and not the "manager's superior management abilities" is difficult at best--if it weren't the manager's ability, then why pay him or her the "management bonus" [Rant 8: which is were the Management Protective Association kicks in to protect their own].

The Benefits Measurement Process

The two hardest activities of Mission Alignment and Implementation Process are Observe and Orient, as defined within the OODA Loop (see A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture for the definitions of these tasks or functions of the OODA Loop).  To really observe processes the results and affects of a process transformation requires an organizational process as described, in part, by the CMMI Level 4 Key Practices or some of the requirements of the ISO 9001 standards.

As usual, I will submit to the reader that the keys (culturally and in a business sense) to getting the the organization to measure the success (benefits) of its investment decisions and its policy and management decisions is twofold.  The first high-level activity is a quick, (and therefore, necessarily incomplete) inventory of its Mission(s), Strategies, Processes and tooling assets.  As I describe in Initially implementing an Asset and Enterprise Architecture Process and an AEAR, this might consist of documenting and inserting the data of the final configuration of each new transformation effort as it is rolled out into an AEAR during an initial 3 month period; and additionally inserting current Policies and Standards (with their associate Business Rules) into the AEAR.  Second, analyze the requirements of each effort (the metrics associated with the requirements, really) to determine the effort's success metrics.  Using the Benefits Context Matrix determine where these metrics are incomplete (in some cases), over defined (in others), obtuse and opaque, or conflicting among themselves.  The Enterprise Architect would present the results of these analyses to management, together with recommendations for better metrics and more Process Effective transformation efforts (projects an programs).

The second high-level activity is to implement procedures and tooling to more effectively to efficiently observe and orient the benefits through the metrics (as well as the rest of the Mission Alignment/Mission Implementation Cycles).  Both of these activities should have demonstrable results (an Initial Operating Capability, IOC) by the end of the first 3 month Mission Alignment cycle.  The IOC need not be much, but it must be implemented, not some notional or conceptual design.  This forces the organization to invest resources in measurements of benefits and perhaps, in which component the benefits exist, control, process, or mechanisms.

Initially, expect that the results from the Benefits Metrics to be lousy for at least three reasons.  First, the AEAR is skeletal at best.  Second, the organization and all the participants, including the Enterprise Architect have a learning curve with respect to the process.  Third, the initially set of benefits metrics will not really measure the benefits, or at least not effectively measure the benefits. 

For example,I have been told, and believe to be true, that several years ago, the management of a Fortune 500 company chose IBM's MQSeries as middleware, to interlink many of its "standalone" systems in its fragmented architecture.  This was a good to excellent decision in the age before SOA, since the average maintenance cost for a business critical custom link was about $100 per link per month and the company had several hundred business critical links.  The IBM solution standardized the procedure for inter-linkage in a central communications hub using an IBM standard protocol.  Using the MQSeries communications solution required standardized messaging connectors.  Each new installation of a connector was a cost to the organization.  But, since connectors could be reused, IBM could right claim that the Total Cost of Ownership (TCO) for the inter-linkage would be significantly reduced. 

However, since the "benefit" of migrating to the IBM solution was "Cost Reduction", not Increased Process Effectiveness [RANT 9: Cost Avoidance in Finance Engineering parlance], Management and Finance Engineering (Yes, both had to agree), directed that the company would migrate its systems.  That was good, until they identified the "Benefit Metric" on which the management would get their bonuses.  That benefit metric was "The number of new connector installed".  While it sounds reasonable, the result was hundreds of new connectors were installed, but few connectors were reused because the management was not rewarded for reuse, just new connectors.  Finance Engineering took a look at the IBM Invoice and had apoplexy!  It cost more in a situation where they had a guarantee from the supplier that it would cost less [RANT 10: And an IBM guarantee reduced risk to zero].  The result was that the benefit (increased cost efficiency) metric was changed to "The number of interfaces reusing existing connectors, or where not possible new connectors".  Since clear identification and delineation of metrics is difficult even for Increased Cost Efficiency (Cost Reduction), it will be more so for Increased Process Effectiveness (Cost Avoidance).

Having effectively rained on every one's parade, I still maintain that with the support of the organization' s leadership, the Enterprise Architect, can create a Transformation Benefits Measurement procedure with good benefit (Increased Process Effectiveness) metrics in 3 to 4 cycles of the Mission Alignment Process.  And customer's requiring the suppliers to follow CMMI Level 5 Key Practices, SOA as an architectural pattern or functional design, together with Business Process Modeling, and Business Activity Monitoring and Management (BAMM) tooling will all help drive the effort.

For example, BAMM used in conjunction with SOA-based Services will enable the Enterprise Architect to determine such prosaic metrics as Process Throughput (in addition to determining bottlenecks) before and after a ttransformation. [RANT 11: Management and Finance Engineering are nearly psychologically incapable of allowing a team to measure a Process, System, or Service after its been put into production, let alone measuring these before the transformation.  This is the reason I recommend that the Enterprise Architecture processes, Like Mission Alignment be short cycles instead of straight through one off processes like the waterfall process--each cycle allow the Enterprise Architect to measure the results and correct defects in the transformation and in the metrics.  It's also the reason I recommend that the Enterprise Architect be on the CEO staff, rather that a hired consulting firm.] Other BAMM-derived metrics might be the cost and time used per unit produced across the process, the increase in quality (decreased defects), up-time of functions of the process, customer satisfaction, employee satisfaction (employee morale increases with successful processes), and so on.  These all help the Enterprise Architect Observe and Orient the changes in the process due to the transformation, as part of the OODA Loop-based Mission Alignment/Mission Implementation process.

Governance, and Policy Management Processes: the Linkage with SOA

Business Rules and Process Flow

In a recent post, A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture, I briefly described the Governance and the Policy Management processes within the context of the IDEF0 Model of the organization and the OODA Loop process pattern.  One function of the Policy Management process is to create "business" (organizational) rules and measurably instantiate the policies and standards.  Once created, these rules can be automated since they turn out to be the "if-then-else" statements in the process logic (for more on this see Types of Business Rules).  As discussed in the post, there are at least three types of Business Rules, knowledge rules, event rules, and value rules.  Each of these rules is an attributed metric, that is, it measures what is going on or changing and compares that with some limit or standard.  This makes the rules relatively easy to automate.

SOA and Process Flow

Many individuals and organizations still confuse Service Oriented Architecture (SOA) with Web Services on either the Internet (the Public Cloud) or an intranet (a Private Cloud).  SOA is a much larger change in IT architecture than that (i.e., "A gaggle of Web Services does not a SOA make").  As I discussed in SOA, The (Business) Process Layer, and The System Architecture , one key difference is that SOA formally separates the process flow from the functions of the process.  This difference makes it feasible to enforce a continuously changing set of rules while the system is operating rather than updating or developing new applications then rolling them out.  Consequently, this is one of the reasons that SOA-based systems are very agile and one of the reasons that good Governance and Policy Management processes are so important to the success of SOA-based systems as Gartner Group and other studies of SOA implementations have found. 

Additionally, it is the reason that both the Orchestration and Choreography engines of the SOA-supporting infrastructure must be directly linked to a Rules engine, which uses a Rules Repository (within the AEAR) since both control the process flow of the Composite Application.  In this configuration any changes in a rule will immediately affect all processes using the rule (so rules modeling, verification, and validation is an imperative as functions of the Policy Management Process or the Services will end up in a complete).

Where Rules Apply
There is yet another complication in this rules management discussion, relative to SOA, System, and Enterprise Architecture, that is some policies and standards apply during the development or transformation process (Design Time) and other apply during Service operation (Operational).  All Design Time and Operational Policies and Standards are Design Constraints (see Why Separate Design Constraints from the Customer's System Requirements? for a discussion of Design Constraints).  Both the Systems Engineer and the System Architect must ensure that these requirements are identified and met.

Design Time

Design Time policies and standards divide into two types, those that effect how the designer and developers implement the Composite Application and those that effect the resulting Composite Application.  For example, in a SEI CMMI Level 5 organization, developing a Composite Application following the dictates of the CMMI Level 3 process is a must, but, whether or not the Composite Application has two-factor authentication is not a concern.  However, if the organization has an organizational standard that all applications will have two-factor authentication will directly affect the resulting Composite Application.  So the first example standard effects "how the Composite Application is built" while the second dictates "what is built".  Good process engineering and management, systems engineering, and system architecture will ensure that all of these requirements are met.

Operational

Operational Policies and Standards are not IT-based, but "business" based for whatever the "business" of the organization is.  They are the attributes of the "If-then-else" branching clauses in the "business" process flow, which is instantiated in the code.  While the branching statements themselves will, generally (See the sidebar for the caveat), remain the same, the attributes, as instantiated by the rules can and will change.  This makes Verification and Validation of the Service (Composite Application) process flows during the development or transformation process difficult, at best.  This is the reason for the modeling, verification, and validation of the Services affected by the rules change before it is roll into the operational environment.

Sidebar
In a four part paper in the Journal of Enterprise Architecture, I envisioned the Self Adapting Service Architecture in which both the form and attributes of the branching statements adapt to changes as part of self-rules setting (or information abstraction) process (see my post The Hierarchy of Knowledge and Wisdom for the definitions).  A prototype of such a system is the IBM's Watson computer that won at Jeopardy over several Jeopardy champions.  As I suggested in the Fourth part of the paper, I would not expect such systems to be operational before 2021.
End of Sidebar

The Linkage
From this brief discussion, I think I've made clear that there is a close linkage between SOA and the Policies and Standards set by the Governance and Policy Management processes.  Getting this linkage in place will require the teaming of the leadership, Enterprise Architect, management, Systems Engineers, System Architect, and Subject Matter Experts (SMEs); this is a significant cultural shift throughout the organization, but organizations that achieve it will have a major competitive advantage over those that don't.

SOA Architects: "Don't Bury the Business Rules in the Database"

Why Rules Get Buried in Databases

Even though the Millennium Bug was first noted in 1982 (or perhaps earlier), it was not widely recognized until the mid-1990s.  Fundamentally, the "millennium bug"was caused by the need to minimize the amount of data storage during the 1960s and 1970s and the lack of understanding of System Architecture earlier.

Fragmented Architecture

Initially, the business logic was hard coded into the programming.  Partially, this was due to the fact that in the 1950s and early 1960s, computers only assisted in supporting the mathematical (e.g., accounting) functions of an organization.  So, it was rather straightforward to code in the Business Rules (the if-then-else, as I discuss in Types of Business Rules).  However, as the System Developers attempted to integrate these functions and as the organization's leaders attempted to respond to a changing organizational environment and to changing governmental regulations, they found that changing the enabling and supporting IT systems difficult and expensive (And many large organization's still do, since they are still dependent on millions of lines of code written in the 1960s and early 1970s).  A great part of the reason for this lack of agility was and is the hard-coded business rules in the program's logic--some of which is documented only in that logic.

Monolithic Architecture

In the early 1970s, as disk storage became somewhat less expensive, Relational Data Bases (RDBs) and Relational Data Base Management Systems (RDBMS) started to replace earlier data storage systems.  The RDBMS enabled developer to map data elements into tables with relationship among all the data.  This mapping essentially converts data into information (see the Blog's glossary page).

The advent of the RDBMS was one of the technical enablers of the MRP, MRPII, and ERP systems, built using Monolithic Architecture.  For example the SAP R3 system could not exist without an associated Oracle or other RDBMS system.  The reason is that the Monolithic Architecture is superior to a Fragmented Architecture because it reduces the number of interfaces that have to be maintained among programs to nearly zero--and as discussed in SOA, The (Business) Process Layer, and The System Architecture I found that the cost per interface was $100 per month.  When you multiple that by the 1000s or 10,000s of interfaces required in a large organization with a Fragmented Architecture, the apparent cost efficiency is enough to delight any finance engineer, be they the CFO or any of his or her minions.

Once the RDBMS systems took hold, the Database Designers (DBD) and coders found that the easiest point to validate the data input was just before data insertion into the tables.  The RDBMS suppliers responded with triggers, (if-then-else logic within the RDBMS).  These triggers rejected data that was out of conformance with some standard; which is the essence of a Business Rule.  However, once the DBD was given this tool, he or she started creating ever more complex triggers; triggers that embody Business Rules that may have little to do with data validation.  For example, many times when data is inserted a trigger will change a flag or other indicator within a set of metadata (data about the data).  The flag may then change the course of the function or business process.  This type of trigger buries the Business Rules in the database.

The problem with an application built on a Monolithic Architecture is the inflexibility of the entire system, as I've discussed in The (Business) Process Layer, and The System Architecture and several other posts.  One key technical element of this inflexibility is the System Architect and SME's inability to change or tailor the data structures because the application is dependent on a single data structure (as well as a standard process, etc.).  When the triggers in the RDBMS include a significant number of Business Rules, this increases the inflexibility because the detailed designer/coder/SME may have great difficulty in even locating the Business Rule.  Therefore, he or she must write an add-on or bolt-in to modify the Business Rule, with all of the inherent configuration management problems that creates.  Further, even if the detailed designer can find the proper trigger to modify, ERP systems do not usually identify all of the functions that use the data or the trigger, so there is a significant risk of induced defects, defects in a secondary process caused by the rules change to support the primary process or function of the transformation effort.

Service Oriented and Other Agile Architectures

As I discussed in  The (Business) Process Layer, and The System Architecture, and several other posts, Service Oriented Architecture (SOA) and other agile processes require separation of the processes and Business Rules from the functions of the process. From the discussion, above, the reasons should be obvious.  Both good Governance and Policy Management (see A Model of an Organization's Control Function using IDEF0 Model, The OODA Loop, and Enterprise Architecture, and Enterprise Architecture, Chaos Theory, Governance, Policy Management, and Mission Alignment) and Agile Mission Alignment require this separation.  Without a Rules Repository, the Enteprise Architect and the System Architect are much less effective at optimizing the SOA or other agile architecture to support the organization in a constantly changing operational and technical environment.

The Prescription: an Agile Architectural Process Pattern

As I see it, the following is a prescription for transforming an organization from function-based to process-based, and the IT architecture of the enabling and supporting systems from Fragmented and Monolithic to agile and Service Oriented is to first understand and target agility, in addition to process effectiveness and cost efficiency.  Second, define or ensure that there are formal Enterprise Architecture (Mission Alignment, Governance and Policy Management) and Systems Engineering/System Architecture processes that coordination with one another.  These processes should have a three month cycle, not a year or more, to ensure requirements agility.  Third, as I described in my posts, Initially implementing an Asset and Enterprise Architecture Process and an AEAR, and Asset and Enterprise Architecture Repository, start with the output of current projects to create the AEAR and exercise the processes.  Fourth, expect that the processes will be less than optimal at the start (the learning curve, see the Superiority Principle).  The risk reduction on this is that since the process cycle is three months, instead of a year or more, learning takes place at a much greater rate.

Addendum: When to Use Triggers When Building to An Agile Architecture
I had a futher thought about triggers.  The Gartner Group recommends segmenting the complete data structure (supporting a monolithic architecture) into databases supporting the individual Service Components (e.g., Web Service).  Each of these Service Components would support either an entire or a logical portion of an organizational function.  I would suggest that triggers can be used within the databases supporting individual functions (Service Components), especially for ensuring data completeness and correctness.

SOA, The (Business) Process Layer, and The System Architecture

As I discussed in my posts The Paradigm Shift of Service Oriented Architecture and SOA in a Rapid Implementation Environment, there is a significant cultural/business/technical shift in thinking when an organization starts to migrate from a fragmented, monolithic IT architecture, or Object Oriented/Web Service using code development using COTS or custom developed applications to Composite Applications within an SOA.  This key shift is formal linking of the IT application with the business process instead of the functions that make up the process.  I discuss several of the advantages in Assembling Services: A Paradigm Shift in Creating and Maintaining Applications.

Process and The Assembly Line

While it may seem like much of a change to link the application to the process instead of the function, consider this concept in the context of an assembly line.  An assembly line is the tooling enabling and supporting the process of assembly.  In turn, a process is "a set of activities ordered to achieve a goal", the goal or operational mission for the assembly line is to assemble the product with no defects.  I suspect that all manufacturing engineers would agree with that goal, but understand that Murphy's law works on all assembly lines, so they attempt to minimize the number of defects, still... On an assembly line, as much as is feasible, all of the activities and their enabling and supporting tooling works in concert with one another, orchestrated by the manufacturing engineering team.  This team spends significant time and effort measuring the flows through the assembly line to ensure, a) that the line produces products with as few defects as possible, while b) producing those products as cost efficiently as possible (e.g., minimizing the work in process [WIP]), and c) meeting the constantly changing demand with as little inventory as possible (which constitutes operational agility for the assembly line).  Additionally, the manufacturing engineers want the tooling to be agile, so that as the components of the product change, with changes in customer's requirements (and wants), the cost of retooling is minimized.

Process and IT

Contrast that with the way organizations engineer IT.  From its inception as supporting single functions, like accounting and payroll, with card-based semi-mechanical "electronic computers" in the late 1950s, IT has focused on supporting individual functions.  By the early 1980s the result was apparent to anyone that developed or used applications; they were information silos, that is, the tooling supporting each function had associated data, but there were no interfaces between functions to enable activity 2 in the process to share its data with activity 3, for example.  During this time, I worked at a major university.  In one Dean's office, that I supported, the Dean's administrative assistant had five terminals on her desk.  Why?  For two reasons; she needed data from five applications to perform her job of supporting the Dean, and she needed output from one application to serve as input to another--she was, in fact, the interface between the functions.

Since this situation was intolerable and expensive for operating the functional applications, the developers/coders created interfaces as the customers identified the need (and had funding to provide for it).  Consequently, the application layer functional design went from a series of virtually unconnected silos of data and code to something approaching a picture of a bowl of spaghetti.  In one study I did measuring the cost of maintenance of each link in this "fragmented" architecture, I found that for one small but business critical process for one of the business units, the cost was approximately $100/month/link.  Considering that even in mid-sized organizations, if half the maintenance cost of what I found is representative (i.e. $50 per link per month), then even a mid-sized organization would be spending a significant portion of its IT budget on link maintenance.  This cost created great pressure by finance engineering on IT management to become more cost efficient. 

Again, manufacturing engineering provided the concept, Materials Resource Planning (MRP).  The MRP systems interlinked all of the data systems and IT functions supporting the manufacturing floor by integrating them into a single application, thereby eliminating all of the linkages among functions.  This concept was expanded into MRPII, and then into the Enterprise Resource Planning (ERP) systems, like the SAP and Oracle tools.  The architecture is call Monolithic because all of the functions are tightly coupled in a single application.

Within a couple of years system users found several issues with ERP systems.

• The ERP systems are difficult to tailor to support the organization's processes.  Typically, the suppliers of ERP systems build the functions to a group of standards that are used by most industries or most organizations within an industry.  If for whatever reason, a) the enterprise that is implementing the ERP does not have a process that uses those standards, then either the enterprise must change its process--unlikely--b) it must attempt to tailor the ERP system to match its process, or c) some combination of the two.  Generally, none of the alternatives are particular effective or popular in large organizations with a diverse product mix.  The result is that either the initiative fails completely or that the personnel of all of the business lines work much harder and longer (and possibly create "shadow systems") to overcome the limitations imposed by the ERP system.
• The Systems are difficult to update.  Frequently, the software supplier implements the next version of a package  the equivalent of installing, configuring, and tailoring the original system.  The reason is that they have updated the technology, perhaps the functional architecture (design) sufficiently, that any bolt-ons, add-ins, or other tailoring will not work with the new version.
• The Systems are operationally inflexible.  That is, they are not agile; able to respond successfully to unexpected challenges and opportunities.  Because the tailoring required for an ERP system can be major, the costs and time involved becomes a significant item in the organization's budget.  Consequently, the time to change becomes long and the cost of change becomes high.  This leads to the problem that IT systems have become an impediment to change rather than an enabler, as studies by Gartner and Forester have shown.

The advent of Object Oriented Programming and its successor, Web Services, enabled organizations to partially address the issues above.  As discussed in my post SOA and a Rapid Implementation Environment, OOD design enables a developer to treat an application as a set of objects.  If these objects use Web Services Standards (either WS or JSR) for their interfaces, then they are Web Services.  In effect, this transformed an application into a set of components (classes, or Web Service components) that could be lashed together to create the application.  The advantage of the OO software architecture is that each of the objects is a small code module that could be updated or replaced with new technology and as long as the objects external interfaces and the resulting functions remained the same, the application would operate in the same manner.  This change met two of the three issues noted above; updating of the application's technology is possible without a complete re-installation and configuration of the application and initial tailoring of the application is much more feasible.

While this tailoring of an application made up of objects or Web Service components does address the agility issue, it is only in a very minor sense.  Since Object Oriented Architecture only enables an implied workflow, rather than making it explicit and independent of the Service Components, it would require reprogramming to change in response to unexpected challenges and opportunities.

Process, SOA, and the System Architect

 As discussed in my post SOA and a Rapid Implementation Environment, SOA formally separates the process flow from the functions.  This formal separation means that the organization can have its transformation teams restructure and order the functions without reprogramming of the functions themselves.  Instead, they can simply be reassembled (see my post Assembling Services: A Paradigm Shift in Creating and Maintaining Applications).

Formally and explicitly assembling Composite Applications using orchestration and choreography enables the System Architect to model the organization's (business) process for both effectiveness (in achieving the organization's mission) and cost efficiency and then to model the Composite Application linked with the process to determine its ability to optimally enable and support that process in both an effective and cost efficent manner.  Consequently when the organization's processes require change in response to changes in the organization's mission, strategies, the organization's environment, or technology, the System Architect, together with the rest of the transformation team may only need to change the process flow Component,  may need to add or update certain Service Components supporting functions, or both.  Still, this will make much more agile support for the processes.  This links the SOA-based Composite Application directly to the organization's process.

There is an additional benefit to separating the process flows from the Service Components.  When the organization's automated business rules change in response to changes in policies and standards (which in turn should be in response to changes in Governance), the System Architect can very quickly evaluate the change by modeling the process and Composite Application to understand the consequences of the change.  In the best situation, the System Architect would model all affected applications before any proposed change in policies, standards, or business rules is put into place because this would enable both the Governance and Policy Management teams to understand the consequences and results of the proposed change before those changes become unmanageable negative externalities.

Obviously, this is a significant shift in the organizational culture.

SOA in a Rapid Implementation Environment

"The farther backward you can look, the farther forward you can see. "

Winston Churchill

Assertion

SOA is particularly well suited to Rapid Implementation and Rapid Transformation, which is an outgrowth of RAD Processes. 

I've divided this post into two sections.  First, a brief history of the trends  in software coding and development, I've experienced; then, a discussion of how these trends will be amplified using Rapid Application Implementation processes, coupled with SOA.

Background

In my post The Generalized Agile Development and Implementation Process, I indicate that in my experience (that is over 40 years of experience (boy, time goes fast)), RAD software implementation process produces more effective and cost efficient software than any other method.  There have been many changes in that time;  binary to machine language, from machine languages to assemblers, from assemblers to procedural languages, from procedural languages to structured code, from structured code to object oriented languages, from object oriented to web services, which have been extended to service oriented. 

Up to Object Oriented, all of these changes were developed to enhance the cost efficiency of creating and maintaining code.  For example, going from coding in an assembler to coding in FORTRAN 1 enabled coders to construct an application much faster by using a single statement to insert a standard set of assembler code into the object deck of the application.  For example, no longer did the coder have to rewrite all of the assembler instructions for saving data to given location on a disk--the instruction together with disk utilities took care of ensuring there was adequate space, writing the data, and the associated "housekeeping" activities.  The downside to this change was that the object code was much larger, requiring more CPU cycles and memory.  Consequently, there were (and are) applications that still require coding in an assembler (e.g., real time flight controls).

The second evolutionary transformation of coding was from procedural to structured.  Due, simply, to the limitations in CPU cycles and memory in early computers and the programming culture of assembler where code was reused by branching into a section and then to another to minimize the number of separate instructions.  With increasing computing power and the implementation of procedural languages, like FORTRAN and COBOL, the branching of code continued; to the point that maintenance of the code became infeasible in many cases, depending on the programmer.  One way that many programmers, including me, used to avoid the issue way to use a modular IT system architecture--that is using callable subroutines as part of the code.  In some cases the majority of the code was in subroutines with only the application work flow as part of the main--Sound familiar?

Anyway, this concept was formalized with the third evolutionary transformation, from procedural to structured (also known initially as Goto-less programming).  This was a direct assault on programmers that used unconditional branching statements with great abandon.  The thought here was to reduce the ball of tangle yarn logic found in many programs to increase the code readability and therefore maintainability.  That's when I first heard the saying, "Intel giveth, structured programming taketh away", meaning that structured code use a great deal more computing power.  At this point, compiler suppliers started to supply standardized sub-routine libraries as utilities (a current example of this type of library is Microsoft's Dynamic Link Library (DLLs)).  Additionally, at this time Computer Aided Software Engineering (CASE) tools began to come on the market.  These tools aided the code developer and attempted to manage the software configuration of the application.

The fourth transformation was from structure programming to object oriented development.  Object Oriented Development (OOD) is not a natural evolution from structured programming.  Instead, it's a paradigm shift in the programming concept--notice the shift in terminology from "programming" to "development".  OOD owes as much to early simulation languages as coding.  These languages, like Simscript, GPSS, GASP, all created models of "real-world" things (i.e., things with names or objects).  Each of these model things was programmed (or parametrized) to mimic the salient characteristics of a "real-world" thing.  OOD applied this concepts of objects with behaviors interfacing with each other to "real-world" software.  So that, for example, a bank might have a class of objects called customers.  Each object in the class represents one customer.  The parametrize attributes of each customer object would include the customer's name, account number, address, and so on.  Each of these customer objects has "behaviors" (methods) that mimic, enable and support what the customer wants and is allowed to do.  For example, the customer object will have deposit, withdrawal, and inquiry methods.  The interaction of the input and output methods, together with the objects and attributes, create the application.

A key problem in creating Object Oriented applications is with the interfaces between and among methods supporting the interaction among objects.  OOD did not have standards for these interfaces so that sharing objects among applications was hit or miss.  Much time and effort was spent in the 1990s on the integration among object orient application objects.

To resolve this interface, application developers applied the concepts of HTML so successfully use by web developers.  HTML is based on the Standard Generalized Markup Language (SGML), developed, in part, by the aerospace industry and the Computer Aided Logistics Supporting initiative to support the inter-operation of the publishing of technical documentation.  And this industry had a lot of it.  At the time, (the late 1980s) a Grumman Corporation technical document specialist reported that when the documentation for all aircraft on an aircraft carrier was initially loaded on, the carrier sank 6".  The SGML technical committee came up with standard tagging and standardized templates for document interchange and use.  Web development teams latched on to the SGML concepts to enable transfer of content from the web server to the web browser.

Likewise, those seeking to easily interlink objects were looking for standardizing the interface and its description.  The consequence of this process research was XML.  The application of this standard to objects creates Web Service style Service Components.  These components have a minimum of three characteristics that enable them to work as Web Service style Service Components:

1. They persistent parametrized data mimicking the characteristics of some "real-world" object
2. They have methods that create the object's behavior and
3. They have interfaces adhering to the XML standard.

Having a great many Web Services is goodness, but it could turn the Internet into the electronic equivalent of a large library with no card catalog.  The W3C and OASIS international standards organizations recognized this issue early and created to standards to help address the problem, Web Services Description Language (WSDL) and the Universal Description Discovery and Integration (UDDI).  These standards, with many additional "minor" standards are converging on a solution to the "Electronic Card Catalog for Web Services" and/or the "Apps Store for Web Services".

In summary, initially programming was about creating code in as little space as possible.  As hardware increased in power (according to Moore's Law), programming and its supporting tools focused on creating more cost effiecent code, that is, code that is easily to develop and maintain.  This led to the paradigm shift of Object Oriented Development, which began to create code that enabled and supported particular process functions and activities.  In total, this code was much more complex, requiring more CPU cycles, more memory, and more storage.  Since, by the 1990s, the computing power had doubled and redoubled many times, this was not a problem, but linking the objects through non-standard interfaces was.  The standards of Web Services provided the answer to this interfacing issue and created the requirements for another paradigm shift.

Now that Service Components (Web Services and Objects meeting other interface standards) can enable and support functions or activities of a process, organizations had a requirements for these functions to be optimally ordered and structured to support the most effective process possible, and the most cost efficient manner (as technology changed) and to be agile enough to enable the organization to successfully respond to unexpected challenges and opportunties.

Rapid, Agile, Assembly

Even a superb research library with a wonderful automated card catalog is of no value to anyone that has no idea what they are looking for.  To use another analogy; my great uncle was an "inventor".  But since he didn't what type of mechanical invention he might want to work on, he would go to auctions of mechanical parts--which meant that he had a shed sized room filled with boxes of mechanical part.  He didn't know if he would ever use any of them, but he had them just in case. 

Many organization's initial purchase of Service Components (Web Services) has been much like my uncle's acquisition of mechanical parts.  Many times, the purchase of Service Components and supporting infrastructure components from multiple suppliers creates as many issues as it resolves because later versions and releases of "the standards" are incompatible with earlier versions of the same standard, and many software suppliers interpreting the standard to better match their existing products.  The result is "the Wild West of Web Services" with supporters of one version of a standard or one product suite "gunning for" anyone that doesn't agree with them.  However, the organization IT is cluttered with the equivalent of a Service Component bone yard (which is like an auto junkyard--a place where you can go to potentially find, otherwise difficult to locate car parts.

To convert these (Service Component) parts into Composite Applications supporting the organization's processes requires a blueprint in the form of a functional design (System Architecture), an identification of the design constraints on the process and tooling (including all applicable policies and standards).  The key is the functional design and its implementation as a Composite Application using Assembly methods; see my posts Assembling Services: A Paradigm Shift in Creating and Maintaining Applications and SOA Orchestration and Choreography Comparison.

In addition to the paradigm shift from software development (coding) to Service Component assembly, there are two major changes in programming when moving from creating Web Services to creating SOA-based Composite Applications.  If the processes for these are not well thought out, or have poor enabling and supporting training and tooling, or are poorly executed, then assembling Service Components will be difficult, time and resource consuming.  This means the creating Composite Applications would be Slow Application Development (SAD???) instead of RAD.  This happens frequently when initially operating the assembly process and seems to follow the axiom that "the first instance of a superior principle is always inferior to a mature example of an inferior principle".  However, many books on the development of SOA-based applications describe processes that either are inherently heavy weight and slow and resources consuming, or that will process applications with a great many defects; and rectifying a defect in a product that is being used costs much more than rectifying it in design.  Therefore, spending the time and effort to fully develop the process and will continue to provide a genuine ROI with each use of the process--though the ROI is difficult to measure unless you've already have a poor process that measure first, or if you can measure the ROI in a simulation of alternatives.

Here is an outline of the RAD-like implementation process I would recommend.  It fits very nicely within The Generalized Agile Development and Implementation Process I discussed.  Implementing and operating/ maintaining a SOA-based Composite Application is much more than simply coding into a work flow Service Component and then executing it in production.  There are several activities required to implement or transform a Composite Application in an effective and cost efficient manner using a RAD-like implementation process.

This implementation process is predicated on at least 5 items in the organization's supporting infrastructure.

1. That the implementation team has established or is establishing an Asset and Enterprise Architecture Repository (AEAR).  This should include a Requirements Identification and Management system and repository (where all of the verification and validation (tests) are stored).
2. That the implementation team is using a CMMI Level 3 conformant process (with all associated documentation
3. That there are business process analysis (including requirements identification and modeling), and a RAD-like Systems Engineering and System Architecture process preceding the implementation process.
4. That the implementation team has access to a reasonable set of Composite Applications modeling tools that link to data in the AEAR.
5. That the organization's policies and standards are mapped into automated business rules that the process flow can use through a rules engine linked to both the Orchestration engine and the Choreography engine.

Model and Verify the Composite Application's flow Before Assembly

First, create and model the process or work flow of the Composite Application before assembling the Service Components.  The key technical differentiator of an SOA-based application from a web service or other type of application is that there are three distinct elements of the SOA application; the work flow, the Service Components (that support the activities and functions of the process), and the business rules that are derived from the organization's policies and standards.  This first activity creates the process flow and ensures that all applicable business rules are linked to the flow in the correct manner.

For those Composite Applications using Orchestration, the entire Composite Application should be statically and dynamically modeled to verify the functions using Service Components models to ensure that they are in proper order, that their interfaces are compatible, that they meet all design constraints, that the Composite Application will meet all organization's business rules (that automate the organization's internal and external policies and standards, and that the Composite Application's performance and dependability meet their requirements.

I found that with RAD, performing this type of verification activity can take from 1 hour for simple applications to two days for a complex application with complete regression testing.  Additionally, I found that both software coders/programmers and program management wanted to skip all of the verification activities that I recommended, but for somewhat different reasons. 

The reason for developers not wanting to do it, was, according to them, "Boring"; actually, it is more "programmer friendly", that is, to ignore verifying that the requirements have been met and that there are no induced defects (defects introduced in the process of updating the Composite Application's functionality), which required in a RAD process to ensure reliability.  Additionally, it's more fun to program than it is to ensure that the version verified is the version that goes into production.

The reason why program managers don't pay attention to the customer's requirements and the requirements verification is that the earned value procedures--the supposed measurement of a program's progress--is not based on measuring how many of the customer's system requirements have been met through verification and validation (V&V), but on the resource usage (cost and schedule) versus the implementer's best guess at how much of which task in the "waterfall" schedule have been completed.  Since producing the earned value is the base for payment by the customer and the PM's and management's bonuses, that is what they focus on--naturally.  Therefore, neither the implementers or management have any inducement to pay a great deal of attention to ensuring that the customer's requirements are met.

However, with a good RAD process like the Generalized Application Development and Implementation Process, the customer's system requirements are paramount (additionally, my experience has been focusing on the programmatic requirements (cost and schedule) pushes the schedule to the right (it takes longer), while focusing on ensuring the customer's requirements (functions, and design constraints) are met, pushes the schedule to the left).

Assemble the Composite Application and Verify

Second, once the work flow has been verified, then it is time to link the Service Components to it in the development environment.  The verification, in this case, is to assure that the Service Component behaviors are as expected (the components meet the component requirements), and that the components have the expected interfaces.

Verify the Sub-systems

Third, once components have been assembled into a portion of the system, or a sub-system, these too, must be verified to ensure that they enable and support the functions allocated to them from the System Architecture (functional design) and meet all applicable design constraints.  Again, this can be done in the development environment, and must be under configuration control.

System Validation

Fourth, System Validation means that the current version is evaluated against the customer's system requirements, that is, that it performs as the customer expects, with the dependability that the customer expects, and meets all of the policies and standards of the organization (as embodied in the design constraints).  Typically, this step is performed in a staging environment.  The are several reasons for doing this, as most configuration management training manuals discuss.  Among these are:

• Separating the candidate application from the developers, since the developers have a tendency to "play" with the code, wanting to tweak it for many different reasons.  These "tweaks" can introduce defects into code that has already been through the validation process.  Migrating the code from the development environment to the staging environment and not allowing developers access to the staging environment, can effectively lock out any tweaking.
• The migrating the application to the staging environment, itself, can help to ensure the migration documentation is accurate and complete.  For example, the order in which code patches are applied to their COTS product can change, significantly, the behavior of the product.
• The staging environment is normally configured in a manner identical with the production environment.  It uses the same hardware and software configurations, though from large installation, the size of the systems is generally smaller than production.  Since the code or product implementers have a tendency to "diddle with" the development environment.  Very often I've been part of efforts where the code runs in the development/assembly environment, but not production because the configuration of the two environments is very different with different releases of operating system and data base COTS software.  Having the staging environment configuration control to match the production environment obviates this problem.

The result is an application that rolls out quickly and cleanly into production.

Roll-out

Fifth, Roll out consists of releasing the first or Initial Operating Capability (IOC) or the next version of the Composite Application.  In the RAD or Rapid Application Implementation process, this may occur every month to 3 months.  If all of the preceding activities have been successfully execute, post roll out stablization will be very boring.

Advantages of SOA in a RAD/I Process
There are several advantages to using SOA in a Rapid Application Development/Implementation process including:

1. The implementer can use Service Component or assemble a small group of Service Components to perform the function of a single Use Case.  This simplifies the implementation process and lends itself to short-cycle roll-outs.
2. Since the Service Components are relatively small chunks of code (and generally perform a single function or a group of closely coupled functions) it allows the implementers at least the following actions:
1. Replace one Service Component with another technologically updated component without touching the rest of the Composite Application as long as the updated component has the same interfaces as the original.  For example, if a new developmental authentication policy is set
2. Refactoring and combining the Service Components is simplified, since only small portions of the code are touched.
3. Integration of COTS components with components that are create for/by the organization (for competitive advantage) is simplified.  While my expectation is that 90 percent or more of the time, assembling COTS components and configuring their parameters will create process enabling and supports Composite Applications, there are times when the organization's competitive advantage rests on a unique process, procedure, function, or method.  In these cases unique Service Components must be developed--these are high-value knowledge-value creating components.  It's nice to be able to assemble these with COTS components in a simple manner.
3. The Structure of an SOA-based Composite Application has three categories of components: Service Components, Process or Workflow Components, and the Business Rules.  Change any of these and the Composite Application changes.  Two examples:

1. Reordering the functions is simplified because it requires a change in the process/workflow component rather than a re-coding of the entire application.
2. If properly created in the AEAR, when an organization's policy or standard changes, the business rules will reflect those changes quickly.  Since in well architected SOA-based systems will have a rules engine tied to the rules repository in the AEAR and to the orchestration and choreography engines executing the process or work flows, any change in the rules will be immediately reflected in any Composite Application using the rule.

These two example demonstrate that a Composite Application can be changed by changing the ordering or rules imposed on the application, without ever write or revising any code.  This is the reason that SOA is so agile, and another reason it works so well with short-cycle Rapid Application Development and Implementation processes.

Tooling

In addition to good training on a well thought out Rapid Application Development and Implementation process, the implementation of the process for creating SOA-based Composite Applications requires good tools and a well architected AEAR.  While implementing the process and acquiring the tooling and instantiating the AEAR may seem like it's a major cost risk, it really doesn't need to be, (see my post, Initially Implementing the Asset and Enterprise Architecture Process and an AEAR).  Currently, IBM, Microsoft and others have tooling that can be used "out of the box" or with some tailoring to enable and support the development and implementation environment that is envisioned in this post.  Additionally, I have used DOORS, SLATE, CORE Requisite-Pro, and other requirements management tools on various efforts and can see that they can be tailored to support this process Rapid Appliction Development and Implementation process.

However, as indicated by the Superiority Principle, it will take time, effort, and resources to create the training and tooling, but the pay back should be enormuous.  I base this prediction on my experience with the RAD process that I developed in 2000, which by 2005 had been used on well over 200 software development efforts and which was significantly increasing the process effectiveness of the IT tools supporting the process, and the cost efficiency of the implementations. 

(PS--the Superiority Principle is "The first instance of a superior system is always inferior to a mature example of an inferior system").