The Digital Future: Services Oriented Architecture and Mass Customization, Part 1

Part1: The Digital Future

I was challenged to forecast changes in economic systems based on both my knowledge of spatial economic systems and on my experiences with computers, data networking, and automation.  What I have come up with is a four part article on the digital future.

Since I normally tend to build to a thesis like any good engineer designing a product, instead of stating my thesis and then defending it, like lawyers, and journalists normally do, I will take a shot at the thesis of this paper first.

 

We have entered the Digital Age in which Capitalism, which describes the economic system of the Age of Print will be succeeded by Economic Services Oriented Architecture producing Mass Customization.  It will be an age where consortiums are formed small and entrepreneurial organizations to produce products, systems, and services the customer wants.  This architecture will replace the current organizational architecture of a single large organization producing a large quantity of products that “satisfice”, that is, they come somewhat close to satisfying the customer’s requirements—they suffice.

The article is constructed in five parts.  This part, Part 1, discusses the economic history of humankind based on how they have communicated and stored data and information.  I feel it’s important to provide the context for my forecast.

The Second Part is a discussion of the coming of the Digital Age based on my experiences seeing it over the past 50+ years.  I’ve found a structure in pattern in the seeming chaos of change in data and information storage.  This pattern leads me to architectural pattern changes that lead to my forecast.

The Third Part is a more detailed discussion of this new architectural pattern called Services Oriented Architecture (SOA).  I will give a couple of examples to demonstrate how SOA will work economically.

The Fourth Part will consider how SOA and the Digital Age will change an individual’s life by giving three examples.

The Fifth Part will show how converting to SOA will create changes as drastic as the changes from the feudal economic architecture to the Industrial architecture.   In this part, I will forecast the change to a number of industries.  Most of these changes are already starting to occur, though in a very minor way.

An Exceedingly Brief History of European and American Civilization

There have been four ages for humankind. 

The Age of Speech

The age of speech (verbal communication), from circa 300,000 BC to circa 6,000 BC  was when for the first time data, information, and knowledge could be transferred and store within and between generations.  This was the first time when clans and tribes formed.  And, according to archeologists, there was a glacially slow revolution from hunting and gathering and stone to agriculture and metals.  This was the economic architecture of the time.  During this period, the shaman, or priest was holder of the tribal information base.

The Age of Writing

The age of writing (written communication), from circa 6,000 BC to 1455 AD, was when data, information, and knowledge could be more accurately transferred longer distances and stored for much longer time periods (in fact, there are documents and records over this entire period).  Political institutions increased from tribes migrating all over the landscape to settled (or at least apparently) settled city states, and then to regional and national states.  This was the second form of an economic architecture.

During this age the first known libraries and colleges formed; for example, the library and museum (college/research center) at Alexandria.  And again, more than 600 years later, after the various barbarian tribal invasions sent Europe back to the talking age, (during the dark ages) up to 900 AD when Carolus Magnus (or Charlemagne) manage to very slightly reintroduce writing and then colleges were formed in what is now Italy.

The Age of Printing

 The age of printing, (printed communications), from 1455 AD to between circa 1942 to 1992, data, information, and knowledge, became much more readily available to humankind.  Thanks, in large part to Martin Luther insistence that everyone should be able to read the Bible, Northern Europe learned to read and read ideas and concepts that were not part of the Catholic Church Doctrine.

By 1776, Adam Smith had described how wealth was created, together with the growth of engineering knowledge, and the ability of individuals to take risks and fail or succeed, Humans entered the era of Mass Production and Liberty.  This is the basic economic architecture of the Age of Printing.

Included in the mass production was mass production of education, based on a school for all teaching reading, writing, and arithmetic.  This has led to the mass production educational systems of today.

Knowledge and Wealth

You should note that with the speech, humanity grew significantly wealthier than other animal species.  The reason is that they could accumulate more and better data, information, and knowledge through speech.

With writing, humanity accumulated a much more wealth.  This wealth was exceedingly badly distributed. Nonetheless, looking at places like Pompeii, even some of the slaves could accumulate small wealth (while “the bread and circuses” form of socialism led to the eventual destruction of the Roman Empire).

With printing, a much large chunk of humanity created orders of magnitude more wealth.  The accumulation of knowledge of how the Universe works has led to mass production, which meant mass wealth.  For example, if there is a disaster now, people expect the restoration of power, water, fuel, and communications immediately; this was never true for even the “wealthiest in the age of speech, writing, or even for most of the age of print.  This demonstrates how exceedingly rich even the poor are today, when compared with the rest of human history (This is something the liberal entitlement generation has forgotten).

The Digital Age: The next Age

The next Age has begun.  It began, in a real sense with WWII.  It gestated throughout the 1950s to the mid-1960s.  I will discuss this period and beyond in the next part of this article.

Hurricanes and an Architecture for Process

The idea for this post came from a previous post on this blog regarding enterprise architecture and religious organizations.

Definitions or Definitional Taxonomies

There are three definitions that are needed to understand this post; the definition of Architecture, process using the OODA Loop, and the Hierarchy of Knowledge.  These are somewhat personnel definitions and you may interpolate them into your model universe.

Architecture

Architecture—A functional design of a product, system, or service incorporating all the “must-do” and “must-meet” requirements for that product, system, or service.

OODA Loop

All processes supporting the mission, strategies, or infrastructure of any organization fall into the OODA model of Col. John Boyd.  The OODA, as discussed in several previous posts includes four steps: Observe, Orient, Decide, and Act.  I will discuss this shortly (below) using the example of the prediction of a hurricane.

A Taxonomy for the Hierarchy of Knowledge

I defined the following taxonomy of knowledge based on much reading and some thinking.  I will demonstrate what I mean for each definition by using it in the appropriate place in my example of the hurricane prediction.

Datum—some observation about the Universe at a particular point in four dimensions

Data—a set consistent datum

Information—patterns abstracted from the data.

Knowledge—identified or abstracted patterns in the information.

Wisdom—is the understanding of the consequences of the application of knowledge

Process Architecture and Forecasting Hurricanes

Since the process for predicting or forecasting hurricanes is most likely to be familiar to anyone watching weather on TV, it is the best example I can think of to illustrate how the OODA Loop process architecture works with the taxonomy of knowledge.

Observe

Initially, data is gathered by observing some aspect of the current state of the Universe.  This includes data about the results of their previous actions.  In point of fact,

Datum—some observation about the Universe at a particular point in four dimensions

Data—a set consistent datum

Obviously observations of current temperature, pressure, humidity, and so on, are basic to making weather forecasts, like the prediction of a hurricane.  And each observation requires latitude, longitude, height above sea level, and the exact time to be useful. So one datum, or data point would include, temperature, latitude, longitude, height, and time.

When the temperature is measured at many latitudes and longitudes concurrently, it produces a set of temperature data.  If other weather measurement, like pressure and humidity, are taken at the same locations at the same time, then you have several data sets with which to make a forecast (or a composite weather data set for a particular time).  Because this is a recurrent measurement process, the weather folk continue to build a successively larger composite weather data set.  But large data sets don’t make a forecast of a hurricane.

Orient

The Orient step in the process is inserting the new data into an individual’s model of how the world (Universe) works (descriptive) or should work (prescriptive).  These models are sometimes called paradigms.  Rules from Governance enable and support the Orient step, by structuring the data within the individual’s or organization’s model.  Sometimes these model or paradigms stick around far after they have demonstrated to be defective, wrong, or in conflict with most data and other information.  An example would be the model of the Universe of the Flat Earth society.

Information—patterns abstracted from the data.  This is the start of orienting the observations, the data and information.  Pattern analysis, based on the current model converts data into information is derived from the organization’s model of its environment or Universe.  For hurricane forecasting this would mean looking for centers of low pressure within the data.  It would also include identifying areas with high water temperatures in the temperature data, areas of high humidity, and so on.  This and other abstractions from the data provide the information on which to base a forecast.  But it is still not the prediction of a hurricane.

Knowledge—identified, combined, or abstracted patterns in the information.  Using the same paradigm, environmental model, or model Universe, people analyze and abstract patterns within the information.  This is their knowledge within the paradigm.  In weather forecasting the weather personnel uses the current paradigms (or information structures) to combine and abstract the knowledge that a hurricane is developing.

When they can’t fit information into their model, they often discard as aberrant, an outlier, or as an anomaly.  When enough information doesn’t conveniently fit into their model the adherents have a crisis.  In science, at least, this is the point of a paradigm shift.  And this is what has happened to weather forecasting models over the past one hundred and fifty years.  The result is that the forecasting has gotten much better, though people still complain when a storm moves off its track by fifty miles and causes unforecast wind, rain, and snow events.

Decide

Once the organization or individual has the knowledge, he or she uses input their knowledge within their models of the Universe to make decisions.

Wisdom—is the understanding of the consequences of the application of knowledge. 

This is the hard part of the OODA Loop because it’s difficult to understand both the consequences and the unintended consequences of a decision.  If your paradigm, environment, or Universe model is good, or relatively complete, then you’re more likely to make a good decision.  More frequently than not people make decisions that are “Short term smart and long term dumb.” 

Part of the reason is that they are working with a poor, incomplete or just plain wrong paradigm (view of the world or universe).  This is where the Risk/Reward balance comes in.  When choosing a path forward, what are the risks and rewards with each path?  [Sidebar:  A risk is an unknown and it is wise to understand that “you don’t know what you don’t know”.]  For weather forecasters it’s whether or not to issue a forecast for a hurricane.  To ameliorate the risk that the are wrong, weather forecasters have invented a two part type of forecast, a hurricane watch and a hurricane warning. [Sidebar: It’s split more by giving a tropical storm watch and warning.]

The reason for this warning hierarchy is that the weather forecasters and services are wise enough to know the public’s reaction to warnings that don’t pan out and lack of warnings that do; they understand the risks.  So when they give a hurricane warning, they are fairly sure that the area they have given the warning for will be the area affected by the hurricane.

Act

Once the decision is made people act on those decisions by planning a mission, strategies, and so on within their paradigm.

For governments and utilities in the U.S. this means putting preplanned hurricane preparations into effect.  For people this generally means boarding up the house, stocking up on food, water, and other essentials or packing and leaving.  And for the drunk beach comber this means grabbing the beers out of the frig, and either heading to the hills or back to the beach to watch the show.

Programmed and Unprogrammed Processes

In the 1980s I wrote a number of articles on process types.  After doing much research and much observation, I came to the obvious conclusion that there are two types of processes, Programmed, and those that I will call Unprogrammed and which includes design, discovery, and creative processes. [Sidebar: These three sub-types differ in that design processes start from a set of implicit or explicit customer requirements, discovery processes start from inconsistencies in data or in thinking and modeling the data, and creative really have no clear foundation and tend to be intuitive, chaotic, and apparently random.] 

Programmed Processes

Programmed processes are a set of repeatable activities leading to a nearly certain outcome.  Almost all processes are of this type.  The classic example is automobile manufacturing and assembly.  Since they are repeatable how does the process architecture model apply?

The short answer is to keep them repeatable and to increase their cost efficiency (that is making them less expensive in terms of time and money to create the same outcome).

Keeping Programmed Processes Rolling

To keep your truck or car operating requires maintenance.  How much and when is an OODA Loop process.  For example, you really should check your tires, both for wear and pressure regularly; otherwise the process of driving can become far too exciting.  So you are in the “observe” step.

As the tires wear out or lose pressure too often, the data set becomes information triggering the “orient” step in the process.  At this point a driver starts to gather data on which tires are wearing fastest and where on the tire they are wearing.  This last point may indicate that tire rotation is all that’s needed.

Based on this data our driver models the information (using the computer between his or her ears) to determine how much longer it’s safe to drive on the tires.  Based on results of that model, the driver will “decide” to buy tires, rotate tires, or wait a little longer.  [Sidebar: Actually, many drivers will go through the buy/wait decision based on additional data, the cost of new tires, and their budget.]  The driver will then “act” on the decision, either get new tires or wait (which is really an action).

The point here is that like everything else in our Universe, over time, everything breaks down; so repeatable processes require an OODA Loop process to maintain them.

More Cost Efficient

However, the OODA Loop process architecture elucidated here is important for programmed processes for another reason.  For repeatable processes the key OODA process is always how to make the faster, cheaper, or making a higher quality product at the same or lower price, and producing it in the same or less time.

This is famously what Adam Smith described in his pin example in the first chapter of “The Wealth of Nations”. By creating an assembly line process that he called “the division of labour”, he demonstrated that the same number of workers produce hundreds of more pins.  He went on to describe that each worker might now invent (or innovate, that is, refine or enhance) tools for that worker’s job. [Sidebar: Henry Ford went on to prove this in spades or should I say Model-T’s.]  [Sidebar: To me the worker’s inventing or innovating tools to help them in their job is quite interesting and often missed.  Tools are process or productivity multipliers.  In the military they are called force multipliers; you always want your enemy to bring a knife to your gunfight, because a rifle multiplies your force.  Likewise, the tooling in manufacturing can increase the quality of the product, the uniformity of what is produced, while reducing the cost and time to produce…fairly obvious.]

Both the “division of labour” and the creation of tooling require the use of the Architectural OODA Loop, which means that increasing the cost efficiency of manufacturing uses the OODA Loop with the knowledge taxonomy.

Unprogrammed Process

Unprogrammed processes have stochastic (creative and unplanned activities within them).  There are really three sub-types of unprogrammed processes, design, discovery, and creative. The key differences between the design, discovery, and creative processes are the whether the process has customer requirements driven and what the requirements are.

Design

The design process has customer requirements.  [Sidebar: As I’m using it, the design process also includes custom development, implementation, and manufacturing.]  It uses a semi-planned process, that is, program or project planning creates a plan to meet the objectives, but with latitude for alternative activities because there is significant risk.  The actual design activities within the programmed process are stochastic, from a program management perspective.  That is, the creative element of these activities makes less predictable, and therefore with programmatic risk with respect to cost and schedule.  Therefore, program management must use (some form or) the OODA architecture to manage the program or project.

The stochastic activities are themselves OODA Loop processes.  The designers have to identify (observe) detailed data of the functions they are attempting to create (the “must do” requirements), while working to the specifications (the”must meet” requirements) for the product, system, or service.  The designers then have to “orient” these (creating a functional design or architecture for the function), “decide” which of several alternate proposed designs is “the best”. Finally, they “act” on the decision.

Discovery

The requirements for research, risk reduction, and root cause analysis are generally unclear and may be close to non-existent.  One of my favorite research examples, because it’s well known, and because it so clearly proves the point is the Wright Brothers researching and developing the aircraft.  In 1898 the brothers started their research efforts.  Starting with data and information then publicly available they built a series of kites. With each kite, they collected additional data and new types of data.  They then used this to reorient their thinking and their potential design for the next kite.  They found so many problems with the existing data that they created the wind tunnel to create their own data sets on which to base their next set of designs.  By the end of 1902 they had created a glider capable of controllable manned flight and by 1903 they created the powered glider known as the Wright Flyer.  It took them at least two more cycles of the OODA Loop to develop a commercially useful aircraft.

Risk reduction also uses the OODA Loop.  A risk is an unknown.  It requires some type of research to convert the unknown into a known.  There are four alternative.  First, the project/research team must decide whether or not to simply “accept” the risk. Many times the team orients the observed risk in a risk/reward model and accepts the risk.  However, another to orient is to determine if there is knowledge or knowledgeable people to convert the unknown into a known. So “transferring” the risk is one way to reduce the risk.  A second method is to “avoid” the risk.  This means redesigning the product, system, or service, or changing the method for achieving the goal or target.  The final way is to “mitigate” the risk.  This is nothing short of or more than creating a research project (see above) to convert the unknown into a Known.

Likewise, root cause analysis is a research effort.  However, the target of this analysis is to identify why some function or component is not working the way it is supposed to work.  Again, its observing the problem or issue, that is gathering data, orienting it through modeling the functions based on the data, deciding what the cause is or causes are for the problem and how to “fix” the problem, and then acting on the decision.  Sound a whole lot like the OODA Loop.

Creative

Creative processes like theory building (both through extrapolation and interpolation), and those of the “fine arts” that come from emotions also use the Architecture of Process defined in this post.  For some theories and all fine arts, “the requirements” come from emotion, based on an intuitive belief structure [Sidebar: a religious structure, see my post on architecture of religion].  This intuitive structure provides “the data, information, and knowledge” on which the creativity builds.

However, for scientific theories, at least, they are sometimes based on inconsistencies in the results of the current paradigm.  The theorist attempts to define a new structure that will account for these aberrant data.

Why make a Deal about the Obvious?

To many it might seem that I’m making a big deal about what is obvious.  If it is the case, why hasn’t it been inculcated into enterprise architecture and used in the construction and refinement of processes and tooling to support the charter, mission, objective, strategies and tactics of all organizations?  Using formal architectural models like the architecture for process, presented in this post, will enable the enterprise architects to “orient” their data and information more clearly making Enterprise Architecture that much more valuable.

Agility, SOA, Virtual Extended Enterprise, Swarming Tactics, and Architecture

Agility and the Virtual Extended Enterprise

In the 1990s, The Agility Forum of Lehigh University defined Agility as "The ability to successfully respond to unexpected challenges and opportunities." The forum chartered a technical committee that created two documents.  The first defined the Virtual Extended Enterprise.  A second, that I co-authored was entitled The Key Needs of the Virtual Extended Enterprise.  As I describe in my book, Organizational Economics: The Formation of Wealth, a Virtual Extended Enterprise (VEE) is a cross ownership domain organization whose purpose/goal/mission is to create a particular product or service and to support it through its life cycle.  This means that a group of skilled individuals and organizations band together to form an "enterprise" whose mission is to create or innovate and design a product, produce it, support it, and finally dispose of it. 

In my book I predict that more of the VEEs will come into existence within the "high-tech" industries because consortia of small highly innovative organizations, loosely coupled (in a legal sense), will be able to out adapt larger organizations, that is, they will work inside the large organization's OODA loop.  Col. John Boyd, father of the OODA loop, based its concept on taking effective action in a time efficient manner.  In the context of battle, this is either hitting the enemy from an unexpected direction or at an unexpected time, and so on, or counter-punching a strike by the enemy in the same manner--that is, out thinking the enemy. To do this, they will need to use swarming tactics. 

Agility and Swarming Tactics
Swarming is "A method, procedure, or process for meeting unexpected challenges and taking advantage of opportunities"; sounds a lot like agility.  A swarming process has many organizational components "gang-up" on the challenge or opportunity using various swarming tactics; some of which are discussed in the examples in the following section.  To some degree, swarming is already used in "Silicon Valley" where individuals and very small firms can organize, almost over lunch, to produce a new product.  This is very much a Service Oriented Architecture (SOA) model for business, where each part of the organization is a "business service component", and the whole is a "business composite enterprise".  Still, this sounds like nothing more than a more flexible version of the waterfall (straightline) business and product life cycle model.

However, nature presents a more agile way to "out-think" the enemy or meeting a new challenge of any type, swarming tactics.  Again, Swarming is "a large number of animate or inanimate things massed together and usually in motion : throng <swarms of sightseers> <a swarm of locusts> <a swarm of meteors>."  Normally, these "things" are apparently autonomous, but  acting, operating, or moving as if in concert or by some "invisible hand".  The key characteristic of all of these examples is that the individual "things", self-directing components are much smaller than their prey or opponent, yet as a team or organization they overcome with ease.  The agility comes from the different types of (service) components, their assembly, and the tactics of their response.  Here are three well known examples.

Army Ants--Army Ants use at least two forms of swarming in their exploitation of their environment.  In fact their "Missions" are "to provide food for the army" and "protect their bivouac (nest)".  This is true of all life.  However, the strategies and swarming tactics they use are different from the "biggest animal with the most teeth, wins" strategy of most animals.  Instead, the ants search for prey either in somewhat parallel columns or in a fan shape.  When they come upon a prey, the first ants to encounter it signal to others while immediately latching onto it.  Then additional ants join, swarming on the unfortunate prey. [Sidebar: This could include humans that are young enough, silly enough, injured enough to not walk away from the ant army.  According to one article, up to 5 people a year are killed by army ants.]

Antibodies--White blood cells and other antibodies are the second example.  Their mission is to defend the body from invading microorganisms including viruses.  They are on the alert for anything that is not foreign to the body and when discovered swarm in on it.  The antibodies are agile in both that they have ability to sense, methods for swarming, and the organization/tooling enable the methods.  One of the reasons for the effectiveness of flu shots (or vaccines) is that they introduce "life-like" targets for the antibodies to practice on (the actual microorganisms, but dead).  This enables the antibodies to reconfigure or reorganize themselves to sense the microorganisms and plan the swarming attack.  Consequently, when the body is attacked by a microorganism resembling the dead microorganism in the shot, the antibodies execute the plan and, hopefully, the microorganism.

Wolf Packs--Wolf packs are yet another organism (animal) that uses swarming tactics to survive.  Like the army ants, the wolf pack has two missions; "to provide food for the army" and "protect their bivouac (i.e., the den and the young and old)".  A single wolf, however large and ferocious, is not able to "take down" even the weakest bull elk or moose by itself.  However, wolves have senses, intelligence, and communications skills that the ants may not have.  For example, they smell their prey herd at a significant distance; they use a relay method to chase a herd until the weak cannot keep up; and they then swarm the weak member using their combined fangs to overwhelm the larger prey/opponent.  On defense of their den they are equally as voracious.

Examples in the organizational Environment

Strategies are used by organizations to achieve their Mission(s) and Mission(s) are used the organization to achieve its Vision.  Strategies are the plans for achieving a goal (or Mission), tactics are the patterns of attack whether it's another organization or a problem or obstacle in the way of the achieving the Mission and Vision.  This is important for Enterprise Architects to remember.

One set of organizational tactical patterns is the set swarming patterns.  It is one of the most sophisticated, complex, yet elegant patterns when successfully executed.  The risk is that it is agile, which is both its strength and its weakness.  Here are examples of each.

WWII US Army Task Groups and Task Forces

In WWII, both the US Army and Navy used reconfigurable units, and to some degree swarming tactics, to their advantage.  After WWI, that is in the 1920s and early 1930s, the cavalry and the infantry each thought they knew the properly role for tanks, the cavalry for exploiting breakthoughs (a typical mission of the cavalry), the infantry to support the infantry on defense and in creating breakthroughs.  Consequently, the infantry wanted a slow moving tanks (maximum speed of about 5 MPH) that was heavily armored.  In contrast, the cavalry wanted fast moving tanks, which meant that they would be much more lightly armored.

In the two years that followed the appointment of George C. Marshall as Chief of Staff of the US Army, he transformed the upper echelons of that organization.  Among the many reforms, he created the Chief of Armor.  Prior there was a armored "cavalry" regiment at Camp (Fort) Knox and an armored "infantry" regiment at Fort Benning.  These had been organized as "experimental" units in the late 1930s after Chief of Staff MacArthur disbanded the early experimental units.  However, under the direction of General Edan Chaffee, the leading doctrinal theoretican and architect of armor and previously in command at Fort Knox, the initial functional design of the WWII Armored Division was constructed. [Sidebar: It is interesting to note that a couple of the leading proponents of armor in Nazi Germany were sent to Fort Knox to "observe" the experimental regiment.  When Germany attacked Poland, setting off WWII, they used most of General Chaffee's architecture and doctrine in the attack, "the Blitzkrieg".]

General Chaffee's architecture for the armored division, the basic unit at the time, included tanks and infantry regiments, with supporting artillery, engineering, and air units--in other words, all of the elements of the US Army at the time.  Consequently, the Armored Division Commander could reconfigure the elements of the attack depending on the mission and strategies employed.  After maneuvers in 1940, the Divisions were further architecturally refined to make them more agile, by eliminating the brigade structure, and having the regiments report directly to the division Commanding Officer (CO).  Then they added task group COs.  The Division CO would assign units to the task group/force CO depending on the mission and strategies to be employed.

However, it was not until the 3rd Army under General Patton's command that the agility of the armored divisions and the swarming tactics showed its full flower with air/ground teams using units from armored and infantry brigaded together in task forces.  This is the well known and documented race through France in August 1944.  It is shown additionally by the 90 degree swing of the 3rd Army in the Battle of the Bulge.  And finally, by the 1st Army's ability to swarm the Ludendorf railroad bridge across the Rhine at Remagen, creating a crossing where none had been "planned" and exploiting it.

The Film-Making Industry

An example that I use in my book is of the film industry.  One well known example of this type of enterprise are movie companies. They form to make a movie and dissolve once the final product is on the market (for some movie franchises, this may be a fair number of years).  The movie company forms (and swarms) around a movie script or concept. Once the movie is made the organized swarms dissolves with each component off in search of new opportunities.

ABB

One project oriented organization, ABB, one of the largest European corporation with about 125K employees, is built on a Swarming Architectural Pattern.  This organization is known for its management structure, which has only five levels (many organizations in the US with only 5000 employees have at least that many or more).  Yet, ABB makes money.  They do it by having many different engineering disciplines formed into relatively small teams.  According to Tom Peters, these teams are mostly self-directing/managing.  If one team finds an Request For Proposal (RFP) they cannot support or support in total, that team networks with other teams in the corporation to create a proposal, and if they win, to execute against that proposal.  In other words, the ABB team swarms on any RFP through the ABB organizational network.

Tiger Teams

Many organizations use "Tiger Teams" to rescue development and transformation programs and project that are in cost and schedule difficulties (the Programmatic Requirements).  This may be because of unanticipated technical risks, management mishandling of activities, or many other causes.  A Tiger Team is a set of personnel that are the best (most knowledgable) available to the organization.  These team members may be culled from all of the units of the organization or from other organizations.  One way to dispose of a risk is to transfer it.  [Sidebar: While most managers and financial engineers think the mitigation is the only way to disposition a risk, there are three others, including transference.  See my post The Risk Management Process]  With a Tiger Team, the risk item (the unknown) is transferred to someone who may know a solution or have a quick method for determining a solution.  The reason for having a Tiger Team is to swarm the design risk with the most knowledgeable team to increase the probability for turning the unknown into a known in a reasonable time and least expense, since any design risk may have more than one method for turning the unknown into a known.

Inhibitors and Facilitators

From the examples, above, it is clear that good systems engineering, system architecture, and enterprise architecture are the only processes that will minimize the probability and impact of the design risks in complex development based on pushing the edge of technology, and especially for swarming processes and their enabling and supporting systems and products. 

Good enterprise architecture can model the most effective investment opportunities based on the organization's mission(s), its strategies, and its swarming tactics, while good systems engineering can identify problems with requirements in the initial stages of development (when changes are most cost efficient) and system architect can create an architecture with non-redundant functions, grouped and structure in the architecture, that when allocated reduce the risk of requiring non-standard components and minimizes the effort and risk of development of the non-standard components.

Inhibitors

There are many inhibitors to agility, swarming tactics, and good supporting and enabling systems.  Three of these are described below.

Over Specification and Other Requirements Problems

One inhibitor, as amply shown in the example, is over specification of requirements and other problems with the requirements.  Over specifying requirements is one of several types of requirements issues (see other posts in my blog for additional, as well as Dr. Ralph Young's book on requirements); others being missing and "just plain wrong" requirements.  For example, I worked on an enterprise portal for a major organization.  When I joined the program, it was already in serious trouble because the sales team had promise to have the portal "fully functional" with 6 months and at half the cost that was initially estimated.  Unfortunately "fully functional" was not defined nor were the customer requirements clearly and completely defined, so the customer expected a portal of their wildest fantasies in 6 months.  This included corporate security who saw the portal effort as a way to get control of all applications on the network. 

Based on what had been promised, I tried to put together a customer system requirements document only to find that the security requirements (among others) were so over specified that it would be infeasible to meet, and impossible in the timeframe.  When I pointed this out to the customer, and recommended a series of short cycles for development--boiling a kettle at a time, instead of the ocean--my team was replaced with a new team, which failed.  After a series of catastrophic project failure by several teams, the customer finally understood that not all of the requirements could be met, or at least met in the time frame, they finally had a working enterprise portal.  It likely cost the customer 10 times as much as if the customer had followed my short cycle (agile development) plan and took at least 5 times as long.

Two ideas should be taken from this example.  First, insistent over specification and other failures in requirements management will inevitably lead to failure.  Second, attempting to use the waterfall development process and a highly managed process with incomplete and/or over specified requirements greatly increases the risk of project failure.  A better way to do it is to use short cycle development on large complex projects and programs.  Further, in doing so, swarming tactics could be used.  For example, when I led a process innovation project and concurrently managed an advance IT laboratory at a major corporation, I would hold weekly staff meetings with a team of 16 IT experts and up to 32 others.  The meetings were as much about design risk management as about weekly status for my role a program manager.  When a team member would bring up a risk or an issue, we would have a brief discussion followed by a number of team members getting together after the "status" meeting and coming to a solution.  For the team this was a swarming tactic; one that would not be "allowed" in a control freak management culture.

"Transactional Management"

There are two types of management, transactional and transformational.  Transactional management is what most people would define as "Management" or "Supervision".  This type of management creates "The Plan" and follows it regardless of whether "The Plan" still makes sense.  These are the Control Freak Management and Cultures.  They put following the plan ahead or adapting to the changing environment.  In development this type of management is called transactional (waterfall program management) method for development.

Working on a program where innovation is one key goal or high-level requirement for the program, transactional management will inevitably lead to cost and schedule overruns and most likely project failure (unless the budget and schedule drastically increase). The reason is that by definition innovation is the result of incorporating elements of the technical unknown (called risks) into product.

The level of time and budget required to convert and unknown into a known cannot be defined, only forecast; and as I can attest, sometimes if turns out much simpler than forecast, but many times even what appears to be a minor risk (a small can of worms), turns into a research effort the equivalent of the entire effort.  At this point the transactional manager goes into freak mode, creating 16 levels in the Gantt charts and holding program reviews every half day, in a futile effort to keep the project meets the business requirements of cost and schedule.

Transactional management greatly impedes swarming tactics, because swarming tactics require that the small groups are semi-autonomous. That is, they are told what their objective is, but not how to reach their objective.  It is up to them to meet the objective and to call on other teams with other skill sets as needed.  This gives control freak managers apoplexy because they have only a limited degree of control.

Communications

Early in my corporate career I had a boss who always said, "Communications is a Bitch", and it is. Words and phrases have culturally connotative and well as denotative meanings; which is one reason translation is so difficult. On technology projects they can be even more so.  One project on which I worked had a 300 page glossary of acronyms, with one acronym that was used over 50 times with completely different meanings depending on the engineering or manufacturing discipline.  Obviously, in the process of developing, constructing, or implementing a new product, system, or service poor communications will cause a great deal of process friction, which in turn adds to cost and time to a project to get everyone speaking using the same terms [Sidebar: I've termed this "violent agreement". Either having a long vehemently intense discussion around using different terms to mean the same concept or using the same terms to mean different concepts] and misunderstanding customer, functional or component requirements and specifications leading to (potentially catastrophic) product defects.  I know of an internal unpublished corporate study of several major development programs where 25 to 50 percent of the typical engineer's time was spent in meetings to get clear requirements and other sort out other engineering issues, while maybe 25 percent was spent on the actual engineering. [Sidebar: the rest was spent getting ready for program reviews, in the reviews, or getting feedback from the reviews]

Communications issues can be exacerbated when many small groups swarm on an issue for the same reasons that a single team have issues, as stated above, except they may be several orders of magnitude more difficult.

Facilitators

There are some facilitators of agility, swarming tactics, and good supporting and enabling systems.  Four of these are described below.

Standards

As weird as it sounds, having a clear and appropriate sets of standards and policies is a key facilitator for agility and swarming tactics for a project.  But, this is if and only if the set is appropriate in what the standards are for.  Too many will kill off a project in a sea of intermediate artifacts (reports, presentations, and other documents that have no purpose but to meet some policy or standard).  Too few will kill off a project slowly by the need to reconcile interfaces, among components, assemblies, and functions.  This is a great facilitator of swarming tactics

In any software development process, I would suggest that the team start with a minimum set of software interface and documentation standards.  Interface standards reduce issues with connecting modules, components, etc.  There are several such sets of standard for SOA.  Software documentation standards, including configuration management standards, greatly reduce the time for defect analysis, and will greatly help future maintenance and upgrades when the product is put into operation.  In general, the standards for hardware products and interfaces among components are well defined.

Process Design and System Architecture Derivation

A second facilitator is the combination of process design and functional derivation/functional design/system architecture.  Obviously, if the customer's "must do" requirements are poorly identified, no amount of design work will create a product that will meet what the customer really wants. [Sidebar: As defined in the CARRMA(r) product I've developed, must do  or event driven requirements define the methods, procedures, and processes the product must support.]  Consequently, a well defined process is of seminal importance when using using swarming tactics. The conundrum is that well defined methods, procedures, and processes are not always possible, especially went including technical innovations in the product, system, or service.

The derivation of clear, concise System Architecture or specification, or of a functional design is exceedingly important because it enables the program management to communicate the requirements of the functions, assemblies, and components to the swarm of technical personnel.  Again, as the many research articles and documents, the many methodologies and procedures, and the many program failures demonstrate deriving clear and concise system architecture or functional design is nearly impossible; another conundrum.

Short Cycles

One way, perhaps the only way, out of the conundrums, above, is through the use of short cycles of design, development, and construction.  That is, plan for incomplete, fuzzy customer requirements at the start of the project and an evolving system architecture/functional design.  This builds agility directly into the design/development/implementation process of the project.

To do this, first, create a high-level notional design based on the known customer requirements.  With the customer, identify any defects in this design and with the design/development team identify any unknowns (risks).  Second, build a prototype of the core of the application and share it with the customer.  Constructing this first prototype should take about one month.  The core should include the main interface with the user(s) and the access control and/or safety function(s).  I have found that showing the customer this prototype will do two things, elicit wire brushing from the customer for miss interpretation of the fuzzy requirements the customer identified and much clearer and more complete requirements.

At the same time, start to prototype all unknowns in the system in short monthly cycles.  This is where the swarming tactics begin.  The initial build of the prototypes will demonstrate which unknowns will be easy to make known and which won't.

After the first monthly cycle have the customer choose which of the know set of requirement to implement first (prioritization of the customer's requirements), then choose one or more to implement during the next short cycle.  At the same time continue to prototype the unknowns.  Demonstrate to the customer where a particular customer requirement is leading to unknowns (risks) that will shortly create issues (technical, cost, and/or scheduling problems).  If there is a required function with a high risk (very likely to occur) with a high impact on the effort, focus the swarming on that unknown.

At the end of the second monthly cycle demonstrate the product to the customer.  I have found that after the second or third cycle (two to three months) that the customer will deem the product the IOC (initial operating capability) and want it put into production.  In each short cycle you can add functionality to the production system and with the customer, you may (and most likely will) add to the customer's requirements list.  This method will greatly increase customer satisfaction.

Transformational management

To use swarming tactics requires a transformational management style.  A transformational management style means that the management adapts the project to the changing customer requirements and priorities of the requirements, the identification of unknowns (risks) and possibly the introduction of new technology in the process of designing, developing, constructing, and delivering a new (custom) product.  This is in stark contrast to the transactional management that cast the requirements and project plan in concrete and allows for no deviation until the project fails.

Enter EA, SA, and SOA

At first blush, Agility, and System Architecture/Enterprise Architecture seem to be diametrically opposed to one another; agility requires flexibility while one characteristic of architecture is rigidity.  Yet, in fact, architecture is a method for structuring, ordering, and organizing functions (a functional design) to meet the customer's requirements.  So what is an agile architecture that enables swarming tactics?  Enterprise Architecture provides the mission or goal, the strategies, processes, procedures, and governance supporting early, clear, and complete identification of the customer requirements for the project.  System Architecture provides the ordered and structured functions of the product supporting the design to meet the customer's requirements.

VEE, Swarming Tactics, and SOA

As defined and discussed in many of my prior posts, a VEE or Virtual Extended Enterprise is a form of a supply chain.  However, it is a supply chain with differences.  First, its not a parts ordering supply chain.  Instead all members of the VEE have a stake in the success of the product.  Second, its members all have peer-to-peer communications.  This means there is a coordinator of the activities in creating the product rather than a top down command and control organization.

Ford: The Integrated Supply Chain 

A typical supply chain has an Integrated Command and Control Organization to support highly structured projects and use "waterfall" transactional program management.  Each organization in the supply chain has a has a clearly defined position in the supply chain with (hopefully) clear and complete contractual technical and business requirements to meet in order to create the product.  This of supply chain organization performs quite well when technology is changing at a slow rate.  And it works particularly well for the reduction of manufacturing or implementation costs, that is making the product more cost efficient.  For example, Henry Ford introduced the Model T to the American public in 1908.  It was the first cost efficient automobile that met, from the start, the public's requirements for dependability.

The reason oft cited for Ford's and the Model T's success is the assembly line, though it was really Fort's focus on a cost efficient supply chain including the assembly line.  Ford treated the entire supply chain as a integrated command and control organization to the point that he built the Ford River Rouge facility in Dearborn Michigan as nearly a complete supply chain for the manufacturing of cars.  The integrated manufacturing facilities included docks to off load iron ore, a steel mill, a glass factory, a rubber and tire plant, and most of the other facilities needed to turn raw materials into an automobile. Now that's integration of the supply chain with a capital I.  For 19 years the Ford Motor Company manufactured the Model T successfully.  However, by 1927 other manufacturers were building cars in the same price range but with functions the Model T did not possess.  If I remember right that included electric starting (also, the only color the Model T came in was black).  So by 1927, the customer's business requirements, (cost), was superseded by the customer's increased technical requirements (new functions).   So, The Ford Motor Company had to design and manufacture a competitive vehicle or go out of business; enter the Model A.

By the 1950s, technology was changing at a rate such that the auto companies were coming out with a new design every year.  [Sidebar: In many cases it was more of a change in the surface sheet metal than a technology change, but technology was trickling in during the 1950s, and was streaming in by the 1960s.]  However, the customer could still order and pay for a car with just the functions (options) the customer desired well into the 1960s.

By the 1980s, the custom options had metamorphosed into options packages.  The reason was simple, the plethora of options, in an era of paper, custom manufacturing of each vehicle very expensive and ended up many defective vehicles; cars with the wrong options, which cost significantly more to retrofit than if they had been built with the options to meet the customer's requirements in the first place.  Additionally, it reduced the number of vehicle configurations from a very large to a more cost efficient number.  Consequently, the customers had to get used to paying for options they didn't want to get the ones they wanted--a somewhat mixed blessing for the customer.  And this is where automobile manufacturing processes supporting the customer's requirements stands at this point.

The members of the Integrated Supply Chain are hierarchically organized with functional and component requirements communicated down the chain and components and assemblies moving up the chain for final assembly.  This works well within the mass production paradigm, but not within the coming disruptive mass customization paradigm.

VEE: The Swarming Peer-to-Peer Supply Chain

As opposed to the integrated supply chain, the members of the Virtual Extended Enterprise are not hierarchically organized.  Instead, all members of the supply chain are members of the same team and sit at the same table.  This enables swarming.  When I was leading an advanced IT development and integration lab for a major defense contractor, initially I set up a hierarchical organization based on the conceptual design I had for the effort.

The goal of the effort was to virtually locate aircraft engineers, from multiple aircraft companies, rather than actually locate them.  Up to the early 1990s when there were joint efforts among major aerospace companies, engineers from all of these companies would be sent to a single location (generally the location of the contract winner) to be able to work together. The engineers at this location would include only the prime contractor and those in the second tier.  Still, the costs to the program of moving and temporarily housing all of those engineers (and sometimes their families) for years became a significant cost.  The job of our project was to use technology to reduce and/or eliminate most of those costs.

Through the first six months our effort struggled.  Then I started to empower the team of initially 12 to 16 very smart, self-starting IT professionals.  Later on 8 to 20 aircraft engineers were added.  I would provide guidance of what was needed (the requirements and to some degree the architecture (the structure and ordering of the processes) at a weekly team meeting.  The team would then identify the unknowns (risks) in potential designs.  After the meeting I would see groups form (specialty swarm) to reduce the risks and to design and code products to meet the requirements.  These groups would assign themselves tasks to be accomplished during the following week.  At the end of the week the team had generally done an astonishing amount of designing and development work.  If the team or I had identified a risk, that had not yet been reduced, a team member would bring it up at the next meeting, and another group would form after the meeting to reduce that risk.  Again, swarming tactics. [Sidebar: We actually used the virtual collocation environment (now called a collaborative environment) ourselves.  While it was against de jour corporate policy, the products of our lab enabled two or three young ladies with babies and toddlers to work from home; only coming in once a week for the team meeting] Once I empowered the team to swarm, and moved away from transactional management to transformational management, the project exceeded the expectations of most of the corporate management.

While I admit this was on a very small scale, it demonstrates the power of the VEE architecture and with swarming tactics for research and development efforts, that is, turning unknowns into knowns (i.e., risk reduction).  And risk reduction through innovation is the key to new technology insertion and process improvement (which means cost efficiency).

SOA and Swarming Tactics

Enter Services Oriented Architecture.  Services are components that serve a particular function and that have a standardized and published interface, which can be linked to other services.  This combination for software is an analog to children's building blocks, of which the best known are Legos.

Web Services, at the "application as a service" or the "Platform as a layer are one set of building blocks.  [Sidebar: In a typical SOA there are also, Infrastructure as a Service (better known as a component of Cloud Computing), and Data Center Facilities (also a component of Cloud Computing).]  To be a "web service" the code must have one or more "published" interfaces that is XML-based.

The job of the development team for an implementation to enable and support some new organizational process is to assemble these Web Services by structuring and ordering them, while identifying missing services that the team will need to design, code, and implement.  "Structuring and Ordering" sounds a whole like System Architecture; and it is exactly that.

While I've discussed Systems Architecture in other posts the part that's of interest in this post is the identification of missing functions and the design, development, and implementation of the functions, as well as their integration into the complete assemblage of the product. This is where the swarming tactics come in.  Like my experience in the lab, the program manager, in this case using the transformational management style, would present the entire team with the unknown (risk item) and the team members would decide if that have talents or expertise that would contribute to creating a usable service; that is the essence of the swarming tactic applied to development and implementation.  And, I've found that it's the only method that is both effective and cost efficient to creating innovative products, systems, and services.