Showing posts with label SOA. Show all posts
Showing posts with label SOA. Show all posts

Wednesday, January 10, 2018

The Digital Future: Services Oriented Architecture and Mass Customization, Part 4

From Previous Parts

Part 1 discussed the four ages of mankind.  The first was the Age of Speech; for the first time humans could “learn by listening” rather than “learn by doing”; that is, data could be accumulated, communicated, and stored by verbal communications.  It also transformed the hunting and gathering into an economic architecture of small somewhat settled communities over the course of 300,000 years.   Settlement produced first significant increase in economic activity, wealth per capita, and in the academics in the form of the shaman for tribal organization.

The second, the Age of Writing, produced a quantum leap in data and information that could be accumulated, communicated, and stored.  This was over a period of at least 6,500 years.  During this time, academic activity evolved from everyone working to survive to a diversity of jobs and trades and the economic stratification of political organizations.   Again, the total wealth of humanity took a leap of orders of magnitude as the economic architectures of city states, then countries, and then empires evolved.  The academics evolved from the shaman, to priests, clerics, researchers, mathematicians, and universities (e.g. the Museum at Alexandria ~ 370 BC and the University of Bologna, 1088) and libraries.

The third, the Age of Print, started with Gutenberg’s press in 1455, but blossomed with Luther’s radical admonition that everyone should “read” the bible about 1517.  Suddenly, the quantity of information and knowledge to a leap of several orders of magnitude as all types of ideas were accumulated, communicated, and stored.
 
Part 2 dealt with history of Services Oriented Architecture (SOA) as it developed hand in glove with computing architecture—a natural fit.

Part 3 A, deals with how SOA works with mass customization of products, systems, and services.

Part 3 B, discussed the three economic architectures, infrastructure, mass production, and mass customization will be employed in the Digital Age.

This part, Part 4 discusses how the Mass Customization (Services Oriented Architecture) will change the culture within which the individual lives.

Changes for the Individual in the Digital Age

As can already be seen the Digital Age and the Mass Customization (Services Oriented Architecture) will change human culture; and will likely change it as much as the cultural change with the discontinuity caused the move from the Age of Writing to the Age of Print.

Already children, those below the age of 21, spend as much or more time interacting with others through their informational interfaces, (smart phone) as they do in direct contact with their peers and elders.  They have little idea what a dictionary or an encyclopedia are, using a web browser and search engine instead.  Additionally, they no longer learn cursive writing, except perhaps to sign their name; they have little need for this artifact from the Age of Writing.  They will continue to shop online more and go to the malls less; with some exceptions that I will discuss later.

The Economics of the Digital Age

 The economics of the Digital Age will change drastically.  There is a paradigm shift with the discontinuity caused by the revolutionary changes of Mass Customization.  And it is a shift only whispered about in the financial markets and by economic gurus.

During the Age of Print, and especially with Mass Production, there was always a need for an increasing customer base.  This was based on the increase in the economies of scale as production increased.  At the time this increased the need for additional labor.  In short, the total wealth and the general wealth of a culturally/political/geographically cohesive organization increased with increased population.  This is with the caveat that the organization had access to the raw materials.

An additional factor, already discussed in this paper was the skill-level of the population, that is, the education by listening and by doing.

So in the Age of Print, the creation of wealth (the amount of value) of a country was linked to:
·         The population size and its increase
·         The integration of society (not its diversity)
·         The ability to have access to raw materials and  to transport the finished goods to markets
·         The Security of the country

So much for wealth creation in the Age of Print.

In the Digital Age the creation of wealth will have different linkages.  The reason for this is the overlay of Mass Customization (Services Oriented Architecture) Architecture over the current Mass Production Architecture and Infrastructure Architecture.  And it will replace them in much of the economy.  I will give a few examples in the next section of this paper.

One thing that will happen is that economies of knowledge will continue to replace economies of scale as I’ve already described.  What this means is what I’ve discussed in one of the later chapters in my Book, Organizational Economics: the Formation of Wealth, that a great many small firms will replace the Fortune 500 of today.
 
In the future, either confederations of small organizations (using orchestration as a guiding principle) or concierge-based alliances of small organizations (using choreography as a guiding principle) will create both giant and small products, systems, and services. Additionally, I would expect that in some or many instances, alliances of confederations to implement the greatest/largest projects and programs.

Both confederations and alliances will require standardization of contract laws, regulations, and business rules to operate effectively.  Knowing the government and the current power and lobbying power of Wall Street (financial firms), large corporations, and unions, the ride to full acceptance of this new economic architecture will put the most fantastic roller coaster to shame just as moving from the Age of Writing to the Age of Print did.

In the end, these current economic empires will fall, just like the political/economic power of King and Pope did.  And since this is a revolutionary change, (while I hope not), there may be revolutions, counter-revolutions, and strife caused by it.

The New Order

At the end there will be a new order of economic activities.  It will be very much more a meritocracy not caused by policy but by “the invisible hand” (in evolution it is called natural selection).  In totality, there will be much more wealth because the knowledge of individuals will be turned into value.

 Additionally, it will be a version of “Jeffersonian Democracy.”  There will be a great many small independent firms, and so there will be a great many owners of these firms, in the way Jefferson envisioned it for small farmers/land owners.  This means a resurgence of the “middle class” as small business owners.

These small businesses will need to fully understand the requirements of their customers and tailor their product, system, or service to meet those requirements.  Consortium/confederation and alliance program manage process will be wrapped around the customer’s product, system, or service requirements, instead of focusing primarily on cost, schedule, and the finance engineers bottom line.
It also means that the Mass Customizers will focus more attention on meeting the customers’ requirements than advertising to tell the customer what the customer should want, which is their product, system, or service.

Government will provide security and infrastructure (meaning physical, cyber, and basic research).  Everything else will change based on Mass Customization (SOA) Architecture.  This includes many industries that most people think of as infrastructure-based, including education and medicine.  These two among several I will describe in more detail in the next section.

In economic terms, the result could be that the GDP will fall, while the GDP per person and the wealth of people generally, will increase.  The reason is that knowledge of the steps in a process breeds tooling to make the steps simpler, easier, and faster.  Tooling is the economic equivalent of weapons being a “force multiplier” in the military.  In the economic case, tooling is a “process multiplier”, as first discussed by Adam Smith.

As a process multiplier, tooling enables the same number of workers to create orders of magnitude more uniform quality products, faster and therefore cheaper.  But tooling costs stored value (i.e., money).  This is the reason that the Mass Production Architecture and money are linked and called Capitalism. (Aside: It probably should be called something else like Value Creation because it isn’t a form of idealism or utopianism, like socialism or communism, and it isn’t a religion like Liberalism, it’s a proven model of the way things actually work, at least in part.)

Throughout the Age of Print tooling was constantly improved and refined.  In the early and mid-1950s, there was an ongoing controversy as to whether analog (mechanical) or digital computers were the wave of the future—that’s how good the mechanism became. (Aside: And it is a clear demonstration of the Superiority Principle: “The first instance of a superior principle is always inferior to a mature example of an inferior principle.”)

Then, in the 1960s to 1990s, enter the CNC machine, that is, the Computer Numerically Controlled machine.  At this same time robotics was being hyped as the way to continue to reduce costs for Mass Production in the future.

In the 1960s through the 1990s, the Mass Production Architecture (and paradigm) focused on a totally automated factory, with little or no warehousing because everything would be JIT (Just In Time).  I worked on two or three such projects during that time, wrote a number of articles the future of the automated factory based on my experiences, and worked on several national and international standards committees.

In 1985 and 1986 I worked on a paperless shop floor information system that became the information backbone of a state of the art factory.  It not only integrated the shop floor systems, but had communications linkages with the engineering and MRP (Materials Resource Planning) systems. It won the Society of Manufacturing Engineers Lead Award in 1987.

I read about the GM’s fully automated engine plant that built to produce a single engine.  The assembly line was a mile and a half long, but without a single worker.  I thought this might be a poor investment because I could already see the glimmer of the Digital Age on the horizon.

Subsequently, I visited The GM SATURN plant.  This plant was both state of the art and attuned to Mass Customization (SOA) Architecture.  It could produce multiple types of engines off a single assembly line and it could produce (within reason) multiple types of vehicles.  While it wasn’t totally automated, at assembly stations with workers, the stations were set up ergonomically so that petite or large workers could work at the station comfortably.  Unfortunately, the SATURN product either did not meet the customers’ requirements or was not properly marketed.

In the future, all manufacturing will be somewhat the same, but also quite different.  There will be two key differences.  The first is the standardization of contractual, requirements, and design interfaces among the various functions of the confederation and/or alliance. 
   
The second is that there will be standardization of highly business sensitive communications across a physical and logical network separate from the current Internet.  The interfaces for users will be locked down both in terms of communications protocols and data formats.  No documents or interpersonal communications (e.g. e-mail) will be allowed.

These two key differences will enable and support both business confederations and alliances the agility to act quickly to customers’ requirements, the heart of the Mass Customization (SOA) Architecture.

Changes to Habits (Cultures)

In the Digital Age there will be significant cultural change.  I will discuss two of these changes.

Home office/business

Beyond these two key requirements for the successful implementation of Mass Customization using Services Oriented Architecture, a third, only slightly less important is product transportation.  In the future, allied organizations are less likely to be collocated and therefore transportation of their output to final assembly and to the customer enables and supports Mass Customization.

In the Age of Print collocation was of seminal importance.  In a paper I wrote (Industrial Location Behavior and Spatial Evolution, Journal of Industrial Economics, Vol. 5 (1977) pp. 295-312.) I discuss how a region around Detroit MI became the center of the automotive industry, though initially there were auto manufacturers all over the country.  The reason was simple; it was the center of innovation, the way “silicon valley” has been for information technology.

Now there is no need for geographically regions to be the centers of innovation; I know.  By 1990, I had the personnel of the advanced data communication laboratory I managed, working from home most of the time.  By 2003, I was leading nationwide software development teams and producing software that met the customers “product” requirements and their “business” requirements (cost and schedule).  So, I know there is no need to collocate software development personnel in a high cost, high tax state where the environmental restrictions are such that houses can’t be built and people live in RVs.

In the Digital Age, centers of innovation will occur in virtual space.  Personnel can work from home, wherever home happens to be.  For example, this means that someone who wants to sail around the world, but has a high tech design job can do both.  That is he or she will be able to design the latest widget from the middle of the Pacific Ocean.  Finance and work on various financial markets can already be done from anywhere, so there is really no need for the New York Stock Exchange being located in New York. It could be located in eastern Kentucky or in West Virginia, with backup systems in Mississippi.  This would mean a lower cost of living and lower taxes for all.

Additionally, this means that there will be less or no need for office buildings.  Since people can work from their homes in home offices, or from wherever and get the same tasks and activities accomplished, it’s pretty silly to spend money because of an “edifice complex”.

I would see the “great cities” and regions diminish in size and population, while small and medium cities and regions regain population.

Developmental versus operational/maintenance

That doesn’t mean that there won’t be all facilities, just that facilities will be sized for the product that the organizations are implementing; i.e., you can’t build an aircraft carrier in a bathtub.
Consequently, for the near-term future these plants and the infrastructure supporting them, supporting the communication, and the rest of the physical and information infrastructure will require maintenance personnel.

In the Digital Age there will be two general categories of jobs/workers.  Not White Collar and Blue Collar, but Developmental and Operational/Maintenance.

In fact, most jobs will be in the operational/maintenance category.  This will include plumbers, lawyers, electricians, nurses, auto mechanics, medical doctors, landscape maintenance, and so on.  Most of these jobs will be structured after the Mass Customization Architecture.

For example, when the doctor provides the analysis of an illness together with a customer’s DNA, the pharmacist will manufacture a dosage of one or several medicines (or drugs) in exactly the formulation that will best target the disease, heal the customer, with the least side effects.  While some of these “drugs” may be mass produced, most will be formulated at the pharmacy from various chemicals.  This means that every pill will be produced for just one person.  This is an example of the future in the Digital Age.


Relatively few people will be involved in development of new products, systems, or services on Earth.  These will be talented out-of-the-box thinkers and doers.  And they will not be what we think of today as “college educated”.  In all likelihood, there will be no such thing as a college degree in the future.  Education too will be on a Mass Customization basis, as I will discuss in the next section of this paper.

Thursday, December 7, 2017

The Digital Future: Services Oriented Architecture and Mass Customization, Part 2


From Part 1

There have been four ages of mankind.  The first was the Age of Speech; for the first time humans could “learn by listening” rather than “learn by doing”; that is, data could be accumulated, communicated, and stored by verbal communications.  It also transformed the hunting and gathering into an economic architecture of small somewhat settled communities over the course of 300,000 years.   Settlement produced first significant increase in economic activity, wealth per capita, and in the academics in the form of the shaman for tribal organization.

The second, the Age of Writing, produced a quantum leap in data and information that could be accumulated, communicated, and stored.  This was over a period of at least 6,500 years.  During this time, academic activity evolved from everyone working to survive to a diversity of jobs and trades and the economic stratification of political organizations.   Again, the total wealth of humanity took a leap of orders of magnitude as the economic architectures of city states, then countries, and then empires evolved.  The academics evolved from the shaman, to priests, clerics, researchers, mathematicians, and universities (e.g. the Museum at Alexandria ~ 370 BC and the University of Bologna, 1088) and libraries.
 
The third, the Age of Print, started with Gutenberg’s press in 1455, but blossomed with Luther’s radical admonition that everyone should “read” the bible about 1517.  Suddenly, the quantity of information and knowledge to a leap of several orders of magnitude as all types of ideas were accumulated, communicated, and stored.  

This created for many changes in sociopolitical organizations and cultural upheaval.  After major wars (e.g., the 30 years, the 100 years, WWI, and WWII, with continuous warring in between) and “the age of exploration and exploitation (colonization)” which created the footings of globalization, in 1776 economic architecture was formalized into the mass production industrial architecture called Capitalism.  Capitalism, with its risk/reward, and mass production has created far more wealth for humanity, though several new religions have destroyed a significant percentage of this wealth; religions, like Fascism (a throwback to feudalism), Communism, Socialism, and Liberalism (All which replace “personal responsibility” with “social responsibility” as their major article of faith).

Now humanity is on the cusp of the Digital Age.  It too, will create orders of magnitude more data, information, and knowledge.  And, it promises another giant leap in the wealth, but with commensurate risks of barbarianism and wars, from all sides, unless there can be integration of cultures, not “cultural diversity”.  History graphically demonstrates that cultures always clash and that “diversity” cultures implode from the clash.  I will discuss these later, but first Part 2 will discuss the inception and gestation of the Digital Age.

Part 2: The Digital Age: A Personal Perspective of How Services Oriented Architecture Evolved


“Intel giveth, Microsoft taketh away”.
A saying by computer software developers circa 1975

From its start to today, the Digital Age could be called the Age of Assembly.  I know because I was there.

Starting in earnest sometime in the late 1960s, humanity has entered a new age, the Digital Age!  And, as in the past three occasions, humanity has not realized the potential of this change in information technology.

Computing Power

In 1965, Gordon Moore, the founder of Intel, observed that “the number of transistors in a dense integrated circuit doubles approximately every two years.”  Effectively, what this means is that raw computing power doubles approximately every two years.

Coding Cost Efficiency

But, the digital age requires three technologies.  In addition to faster and more powerful hardware, it requires the same abilities of the other ages, the ability to accumulate and analyze the raw datum and to communicate both the data and store the results of the analysis.

The ability of a digital system to accumulate and analyze data is based on its programming.  In 1956, I played with my first computer, or what was referred to at that time as a computer.  This computer was programmed with a wire board—a board with a matrix of connectors; the programming came of wires connecting the “proper” connectors together.  Data was inserted using Hollerith cards (referred to as punch cards).

 By 1960, programming had graduated to computer coding using the punch cards.  I coded my first program in Symbolic Programming System (SPS), a form of Assembler—the first step up from coding in machine language (1s and 0s)—as a member of my high school’s math club.  An Assembler is little more than a useful translator of machine code to make it simpler of the coder to create code and to more easily identify bugs—both, greatly increasing the coders and codes effectiveness and also the cost efficiency of creating code.

By 1964, I started taking the first of three computer courses offered by the Math Department of the university I attended.  This class included programming in machine code (literal ons and offs), SPS, and Fortran 1.  The latter was the first use of the concept of Services Oriented Architecture (SOA). 
Fortran 1 (Formula Translation 1) was among the earliest scientific programming languages.  These languages were made of a set of computer commands (functions), read (getting input), print (provide output), do mathematically calculations (add, subtract, etc.) and perform some logical step (loop, branch, and so on).  Actually it’s much more complex than this, but I’m not quite ready to take a swan dive into the minutia of computer software and hardware design and architecture.

By 1965, a coder could create hundreds of instructions per hour, rather than a couple of dozen like in 1956.  Since the computing power of the hardware was (and is) dramatically increasing, making the coder more cost efficient makes sense.  Additionally, it meant that computers could handle much larger and more complex tasks.

Data Storage and Data Communications

Between 1964 and 1980 two other technological developments occurred that have led to the start of the Digital Age, data storage devices, and data communications. 

In 1964, I first saw, wrote code for, and used a storage device called a disk drive.  Prior to this data was stored either on cards or on tape drives.  Like the CPU, data storage hardware has continued to follow Moore’s Law.  Today, the average smart phone has 100 to  1000 times the storage that the “mainframe” computer had when I was working on my Ph.D. and that storage 300MB costs tens of thousands time more and took up more than a basketball court sized room.

So the abilities to store and analyze data and information have become much more effective and cost efficient.   So has the technology’s ability to communicate data, information, and knowledge.
In 1836 Samuel Morse demonstrated telegraphic communications, the first machine language method for communicating information.  It took until the mid-1970s telegraphic communications to evolve into a wide variety of data communications hardware, software, and communications protocols.  Then it took the next 20 years to coalesce into the hardware, software, and communications protocols we know as the “Internet” and the “web”.

During the same 20 years the final element for the digital age evolved, Services Oriented Architecture. 

Services Oriented Architecture

At the dawn of the digital age programming, all programs were simply an order set of machine instructions, no loops and no logic.  When computer languages, like FORTRAN 1 first evolved, they were created with a good deal of branch logic to allow code to be reused.  Why: because the memory and data storage on the machine was so small.  So, at the time it made sense to reuse sections of code that were already in the program, rather rewrite the same code.

Inevitably this led to what “computer scientists”, people that taught programming rather than writing programs for a living, called officially “unstructured” programming, or in the slang of the day “spaghetti code”.  In production at the time, unstructured programs were much faster in execution on the hardware available.  However, they were also much more difficult to understand especially if they weren’t properly documented. This meant that they were hard to debug and hard to update or upgrade.

According to the computer scientists, the chief culprit of unstructured programming was the unconditional branch, called the “goto” statement.  This statement indicated the location of the next coded statement that the computer should execute, and this, in general, was back to some location earlier in the program.  This meant that in following the program, it jumped around, rather than going from top to bottom, making it easier for the inexperienced to follow.  So, like all liberal bureaucrats they outlawed unconditional branching.

Then they replayed the goto statement with an unconditional branch to another program, initially called a subroutine.  This fragmented the program and intuitively created what could become the services of SOA, but didn’t.  Instead, they became the Dynamic Link Library (DLL) of computer functions; utilities of the operating system that can be used to support all programs on a computing system.
Instead of structured programs, I opted for modular programming.  These were programs that performed a given function for the program.  For the application that I wrote as part of my Ph.D., I used this architecture; and I submit that it’s the basis for the SOA-based applications.
Be that as it may, I’ve described how I see by the concepts for SOA evolving from machine language of the late 1950s, to assembler, then programming languages with DLLs, and then modular programming.

Each time, the number of machine instructions increased by orders of magnitude, meaning that a single instruction to call a “subroutine” or function could generate 50 to 500 or more instructions and each of these instructions could generate 1000 to 10,000 instructions.  Actually, one time I did some research and found that one 20 line program actually generated 63 mega-bytes of code.

There was still a problem.  It really helped the cost efficiency of the coder to be able to create massive amounts of code fast, but many times the customer for whom the program was being created wanted to interlink that program with other programs.  Most often these programs were created in different computer languages (e.g., FORTRAN, COBOL, PL1), and on various brands of computers (e.g., IBM, DEC, HP, Sun, Silicon Graphics).

Obviously, the problem was that all of these various software and hardware suppliers were competing for business and therefore making it hard to interlink their products with competitors’ products.  The reason was simple; to force their customers to buy only their products.  Today you can see the identical strategy, with Apple, Microsoft, and others building their own “ecosystems” to ensure their customers stay their customers.

In the early 1980s, customers began to recognize this, especially with the advent of data networks.  This set off a series of international standards committees for all components, from data and how to store it, to data communications.

For example, I was on a number of Open Systems Interconnect (OSI) data communications committees starting in 1982.  One of the team members of the team I led had been a coauthor of the Standard Generalized Markup Language (SGML).  Both HTML and XML, the languages of the web, were derived from SGML.  I was peripherally involved with STEP (PDES) for engineering data, X.400 for e-mail and X.500 Directory Services from which LDAP was derived.  This was all between 1982 and 2009.

Another step in creating SOA for information and knowledge-based system (the so called “Big Data” systems) is a standardized interface for the Services (i.e., components or functions).  Various organizations including the W3C worked on this issue and came up with Web Services.  Web Services uses XML in a defined and standardized manner to enable the software functions to communicate without having to write an interfacing routine between and among functions.\

The final step in creating SOA was the design and development of a reference architecture or model denoting the components of its supporting infrastructure, assembly process, and functions needed to create a robust application using SOA.  I was a team member on the OASIS Team the created this model.

So from personal experience, I’ve seen the international standards for data formats, and data communications develop.  All of the above are the precursors for Services Oriented Architecture as used for a new economic architecture.

Mass Customization Using Services Oriented Architecture--Part 3, Coming Soon


Part 3 will discuss an economic version of Services Oriented Architecture and how it will reformulate business organizations in the Digital Age.

Thursday, July 13, 2017

An Architecture for Creating an Ultra-secure Network and Datastore

The Problem
According to United States records, from 2006 cyber attacks to 2016, (crimes, intelligence gathering, and warfare) have gone up 1300 percent.  Other reports identified in Forbes Magazine indicate that between 2015 and 2016 there was a 200 to 450 percent increase in attacks.  I suspect that though the numbers vastly underestimate the total number of attacks.  I know that in the late 1980s, one company was averaging 10,000 attacks per day on its website and access points to the internet; of which 4000 originated in Russia (then the USSR), China, North Korea, and the like
.
There are two goals for attacks, to disrupt the entire IT infrastructure or to gather or change protected data for various nefarious purposes.  There is a multiplicity of reasons for these attack, monetary gain, political change, and so on; the “so on” is too long to enumerate.

The cost for preventing and mitigating the effects of these attacks has spawned a new multi-billion dollar industry.  Consequently, the need is for an entirely new system (network and datastore) that completely defeats all attack vectors.  That is what I’m proposing here.

The Solution A Disruptive Architecture: The Once and Future System

The Goal

The goal of the architecture presented here is to define a highly secure system for the transmission and storage of data.

The architecture is for a fundamentally different “new” network and datastore.  I put “new” in quotes because I based the architecture on a number of concepts and standards from the late 1970s to the mid-1990s.  For reasons of economies and business politics these concepts and standards were abandoned.  When I submitted the architecture for a patent and even though the architecture uses old concepts and standard in a new way, I was told that since it was based on well known concepts and standards the architecture it is unpatentable. 

Consequently, I’m presenting it in this post in the hope that someone will take serious look at it and communicate with me so that I can present the details and we can build a secure network and datastore.

The Architecture

My fundamental idea is to create a separate “data only” network and datastore.  While initially, having a worldwide network for the storage and transmission of data separate from the Internet “of everything” may seem as a ludicrous idea for those looking at the “short-term” costs for an organization; what the cost of having data stolen, corrupted, or destroyed would be for an organization?  And remember that there are  initial and recurring costs for data security on a cloud or across the internet.

This new architecture has five components.  One of them has evolved over the past twenty years.  One of them was declared obsolete thirty years ago.  One of them is based on petrified standards of the 1980s.  And one uses a new twist on current hardware and software.  The fifth is a particular form of governance.

New User Interface Security

The base technology of the new user interface has been evolving over the past twenty years at least.  It is a combination of three functional technologies.  The first is biometric recognition.  Any secure system requires some form of authentication; that you are who you say you are.  Various forms of biometric authentication, facial recognition, fingerprint identification, retinal pattern recognition, and so on, are currently the least likely forms of identification to be broken by cyber attacks.

The second security technology is a version of the smartcard.  These are credit-card-like with a data storage computer chip embedded.  Under this new function the card reader would communicate the location, time of day, and date, whereupon the card would generate a pass code based on those parameters.
 
At the same time the reader would generate a pass code also based on those parameters.  The system would accept the identification if and only if they matched.  Since any secure system requires at least to factor authentication, a user would need both the smart card (which additionally could store the biometric data) and their own body.

Finally, authorization and access control are both static for a given user interface to the system.  This means that the user of a given device (be it a terminal, PC, smart phone, etc.) can only gain access to the set of data, records, or summaries to which their entitled. 

So a contract specialist has no access to engineering data for the contract or only a limited set.  If the contract specialist attempted to sign on to another device, to which he was not preapproved, he could not get to the data to which he is entitled. The reason is that an individual must be preapproved of every terminal the individual wants to use. 

Or a doctor may not see a patient’s complete medical history without the patient’s permission. This would be a two step process.  The doctor would have to sign in on his or her device using the two-factor authentication, described above.  Then the patient would have to sign on to the same device using the same two-factor authentication to give the doctor permission to access his or her medical record.

The security meta-data and parameters are stored on the ultra-secure data network (USDN).  Any updates or changes must be made and approved through the system’s security governance function.  No dynamic changes can be made until the changes are approved.  In a political/cultural context, this governance process will be the most difficult to secure since users expect changes to be made “NOW” and the process doesn’t allow “NOW” to happen.

The Bridge

The second architectural component is the bridge from the Internet to the USDN.  This is really the key component securing the USDN from attacks.  And this is the component that was declared obsolete thirty years ago.  In the early 1980s there were many proprietary data networks.  To communicate data from one network to another required a network bridge.

The following diagram is from the patent that I applied for.  It shows an example of how changing the protocol layers or stacks creates a portcullis in the bridge that provides the ultra-security.  On the left side of the bridge are the standard Internet protocols.  Other than the top layer (called the Application Layer in the OSI model) and the bottom layer (the Physical Layer in the OSI model), all layers link and guide the communications between the sender and receiver.

Notice that the functional protocols on each side of the bridge, with the exception of the physical layer are different.  On the left side all protocols are current Internet standards.  However, on the right side the bridge uses protocols from the Open Systems Interconnect (OSI) suite.  These protocols were abandoned in the 1990s in favor of the earlier TCP/IP suite, that at the time were less expensive and much less capable. [Sidebar: “The first example of a superior principle is always less capable than a mature example of an inferior principle”].

What this means is that the entire USDN will use these OSI protocols.  Any cyber attack software developed for Internet protocols would have to be redesigned for the OSI protocols.

Even if the hackers of whatever stripe did develop software capable of exploiting vulnerabilities in the OSI protocol stack they would still need to get it onto the network.  But the design of the bridge includes a portcullis in the middle of the bridge.

This portcullis is designed to allow only data and records in well defined formats to pass.  This means that no documents can move across the bridge.  In this case “documents” includes e-mail, documents, unformatted text, files, or other unformatted data.

This stringent requirement eliminates nearly every attack vector by hackers.  For example, there is no way that a Trojan horse attachment can get into the system because e-mail, let along e-mail with attachments, is allowed access across the bridge.

As shown in the diagram, only data in specific and static XML formats is allowed to move through the portcullis.  The XML data structures are installed in the portcullis only after approval using one of the governance processes.

So, for example, medical data would use an XML version of the international medical standards, engineering data would use an XML version of STEP, and so on.  Only data exactly following those standards to which the user is entitled would get through the portcullis.. This would initially have a very large overburden in meta-security and access control data about all individuals.

The Network

The third architectural component is the network.  The network is based on petrified standards of the 1980s.  Inside the portcullis-bridge data would be free to move among the various nodes of the network using the same OSI protocol stack that is used on the right side of the portcullis-bridge shown in the diagram.

Additionally, it would use improved versions of the Directory Service (X.500) standard.  This would include using static routing meta-data (which many network analysts would say is not an improvement).  However, static routing meta-data means that if an unauthorized node magically appeared on the USDN (because some hacker tapped one of the USDN lines) the node would be recognized as a threat immediately.  Consequently, any attempt to breach the security imposed by the portcullis-bridge by directly attacking the network would fail, as long as good governance is in place.

Datastores

The last technical function is data storage.  This datastore function uses a new twist on current hardware and software design for the storage of data and information.  The twist is that only specific data and records are store, not files from outside the network. 

An organization using an USDN-like system would have its data file structures created by authorized personnel inside the USDN.  These file structures would follow the various authorized XML data structures.  No freeform data like e-mail or documents would be allowed.  [Sidebar: remember its much much simpler to create documents from data than to glean data from documents.] 

The only applications that are authorized to run on the USDN and its datastore computers are those that create, read, update, or delete records or data elements.  Reading data would include reading for transfer, and for summarization. 

For example, suppose the medical profession of a state or of the United States adopts the USDN to protect patients’ medical records.  A medical researcher may be granted access to summaries of certain data elements of patients’ record that have a particular medical problem.  This access would be granted through an approval process—part of governance—prior to obtaining the summaries.

The advantage is that the medical researcher has access to a complete set of data for the population of an area.  The downside for the researcher is that they need to have a well formulated and defensible hypothesis to work from, in order to obtain the data, and that the governance processes take time.

Governance

The Governance processes function of the system’s architecture is most critically important of the five functions because it is the only one where humans are involved—Big Time.  As discussed above there are many security functions that are static and require administrative functions to change the parameters and meta-data.  While I expect that actually changing the meta-data and parameters will be automated, the various decision making processes will not.

One obvious example is in banking.  Some financial data must be secure within a financial institution and only shared with a client.  Other data, in the form of transactions must be shared between and among banks and other financial institutions.

The USDN security meta-data would determine which data could be sent to another financial organization, what data can be sent, and other characteristics of transaction.  It would be within the USDN and not across any portion of the Internet.  It can then be retrieved by the destination organization.

For example, if all defense contractors were on the USDN then when teams formed to respond to a DoD Request For Proposal (RFP), the various teams of contractors and subcontractors could share requirements and other data within their team.  When the DoD chose the winning team, program/project, risk, and design data could be shared and shared with the customer without fear a cyber attack on one of the sub-contractors leading to the capture or corruption of the program or mission critical data.  [Sidebar: frequently a third or fourth tier sub-contractor has more vulnerabilities than the prime contractor.]

Issues

Again,”The first instance of a superior principle is always inferior to a mature example of an inferior principle.”

There are three issues with the creation of such a system. 

The first is cost; creating an entire nationwide or worldwide network is very expensive in the startup phase.  Creating (or really resurrecting in many cases) software to support the functions of the USDN will be very expensive.  There is the cost of implementing software services to interface with existing organizational applications.  Acquiring the physical cabling for the system will be expensive.  

Modifying routers to use the new protocols will be expensive. Designing, constructing, and testing the new portcullis-bridge will be very expensive.  Most of this investment will need to be done before one data element is protected.

The cost is more than a straight financial issue of building the system.  It will threaten much of the multi-billion dollar cyber security industry’s income stream.  This industry will market and lobby against building out the system.

The second issue may be used by that industry as an argument against the USDN.  The issue that the system only protects data and not other types of information like e-mail and documents.  This is true.  However, the core of any organization is its data.  Documents can be easily constructed from data, but not the other was around.

The third issue, at least initially, is the response time of the system.  Currently applications and users have come to expect nanosecond response times to dynamic requests.  Initially, at least, I predict that the response time to requests will be in terms of seconds; maybe many.  I saw this with Microsoft DOS—until version 3.1 it was bad—other products from Microsoft, Apple, and Oracle [Sidebar: I worked with Oracle 4.1] and many other hardware and software products.]  So it will be a rocky start, but ultimately it will cost much less than the recover, rebuild, patch, upgrade, and get hacked again systems of today.

Summary

While the USDN does not protect an organization from cyber attacks, it does make an organization’s mission critical data nearly invulnerable an organization will be able to recover from an attack and will make it nearly impossible for terrorists, cyber criminals, etc. to get a personal data or its mission critical data protected.


For anyone who is interested, please comment on this post.  I have much of knowledge of the processes, technology, and construction process involved than I can put in a post, but would be happy to discuss it.

Thursday, December 29, 2016

Agility, SOA, Virtual Extended Enterprise, Swarming Tactics, and Architecture

Agility and the Virtual Extended Enterprise
In the 1990s, The Agility Forum of Lehigh University defined Agility as "The ability to successfully respond to unexpected challenges and opportunities." The forum chartered a technical committee that created two documents.  The first defined the Virtual Extended Enterprise.  A second, that I co-authored was entitled The Key Needs of the Virtual Extended Enterprise.  As I describe in my book, Organizational Economics: The Formation of Wealth, a Virtual Extended Enterprise (VEE) is a cross ownership domain organization whose purpose/goal/mission is to create a particular product or service and to support it through its life cycle.  This means that a group of skilled individuals and organizations band together to form an "enterprise" whose mission is to create or innovate and design a product, produce it, support it, and finally dispose of it. 

In my book I predict that more of the VEEs will come into existence within the "high-tech" industries because consortia of small highly innovative organizations, loosely coupled (in a legal sense), will be able to out adapt larger organizations, that is, they will work inside the large organization's OODA loop.  Col. John Boyd, father of the OODA loop, based its concept on taking effective action in a time efficient manner.  In the context of battle, this is either hitting the enemy from an unexpected direction or at an unexpected time, and so on, or counter-punching a strike by the enemy in the same manner--that is, out thinking the enemy. To do this, they will need to use swarming tactics. 

Agility and Swarming Tactics
Swarming is "A method, procedure, or process for meeting unexpected challenges and taking advantage of opportunities"; sounds a lot like agility.  A swarming process has many organizational components "gang-up" on the challenge or opportunity using various swarming tactics; some of which are discussed in the examples in the following section.  To some degree, swarming is already used in "Silicon Valley" where individuals and very small firms can organize, almost over lunch, to produce a new product.  This is very much a Service Oriented Architecture (SOA) model for business, where each part of the organization is a "business service component", and the whole is a "business composite enterprise".  Still, this sounds like nothing more than a more flexible version of the waterfall (straightline) business and product life cycle model.

However, nature presents a more agile way to "out-think" the enemy or meeting a new challenge of any type, swarming tactics.  Again, Swarming is "a large number of animate or inanimate things massed together and usually in motion : throng <swarms of sightseers> <a swarm of locusts> <a swarm of meteors>."  Normally, these "things" are apparently autonomous, but  acting, operating, or moving as if in concert or by some "invisible hand".  The key characteristic of all of these examples is that the individual "things", self-directing components are much smaller than their prey or opponent, yet as a team or organization they overcome with ease.  The agility comes from the different types of (service) components, their assembly, and the tactics of their response.  Here are three well known examples.

Army Ants--Army Ants use at least two forms of swarming in their exploitation of their environment.  In fact their "Missions" are "to provide food for the army" and "protect their bivouac (nest)".  This is true of all life.  However, the strategies and swarming tactics they use are different from the "biggest animal with the most teeth, wins" strategy of most animals.  Instead, the ants search for prey either in somewhat parallel columns or in a fan shape.  When they come upon a prey, the first ants to encounter it signal to others while immediately latching onto it.  Then additional ants join, swarming on the unfortunate prey. [Sidebar: This could include humans that are young enough, silly enough, injured enough to not walk away from the ant army.  According to one article, up to 5 people a year are killed by army ants.]

Antibodies--White blood cells and other antibodies are the second example.  Their mission is to defend the body from invading microorganisms including viruses.  They are on the alert for anything that is not foreign to the body and when discovered swarm in on it.  The antibodies are agile in both that they have ability to sense, methods for swarming, and the organization/tooling enable the methods.  One of the reasons for the effectiveness of flu shots (or vaccines) is that they introduce "life-like" targets for the antibodies to practice on (the actual microorganisms, but dead).  This enables the antibodies to reconfigure or reorganize themselves to sense the microorganisms and plan the swarming attack.  Consequently, when the body is attacked by a microorganism resembling the dead microorganism in the shot, the antibodies execute the plan and, hopefully, the microorganism.

Wolf Packs--Wolf packs are yet another organism (animal) that uses swarming tactics to survive.  Like the army ants, the wolf pack has two missions; "to provide food for the army" and "protect their bivouac (i.e., the den and the young and old)".  A single wolf, however large and ferocious, is not able to "take down" even the weakest bull elk or moose by itself.  However, wolves have senses, intelligence, and communications skills that the ants may not have.  For example, they smell their prey herd at a significant distance; they use a relay method to chase a herd until the weak cannot keep up; and they then swarm the weak member using their combined fangs to overwhelm the larger prey/opponent.  On defense of their den they are equally as voracious.

Examples in the organizational Environment
Strategies are used by organizations to achieve their Mission(s) and Mission(s) are used the organization to achieve its Vision.  Strategies are the plans for achieving a goal (or Mission), tactics are the patterns of attack whether it's another organization or a problem or obstacle in the way of the achieving the Mission and Vision.  This is important for Enterprise Architects to remember.

One set of organizational tactical patterns is the set swarming patterns.  It is one of the most sophisticated, complex, yet elegant patterns when successfully executed.  The risk is that it is agile, which is both its strength and its weakness.  Here are examples of each.

WWII US Army Task Groups and Task Forces
In WWII, both the US Army and Navy used reconfigurable units, and to some degree swarming tactics, to their advantage.  After WWI, that is in the 1920s and early 1930s, the cavalry and the infantry each thought they knew the properly role for tanks, the cavalry for exploiting breakthoughs (a typical mission of the cavalry), the infantry to support the infantry on defense and in creating breakthroughs.  Consequently, the infantry wanted a slow moving tanks (maximum speed of about 5 MPH) that was heavily armored.  In contrast, the cavalry wanted fast moving tanks, which meant that they would be much more lightly armored.

In the two years that followed the appointment of George C. Marshall as Chief of Staff of the US Army, he transformed the upper echelons of that organization.  Among the many reforms, he created the Chief of Armor.  Prior there was a armored "cavalry" regiment at Camp (Fort) Knox and an armored "infantry" regiment at Fort Benning.  These had been organized as "experimental" units in the late 1930s after Chief of Staff MacArthur disbanded the early experimental units.  However, under the direction of General Edan Chaffee, the leading doctrinal theoretican and architect of armor and previously in command at Fort Knox, the initial functional design of the WWII Armored Division was constructed. [Sidebar: It is interesting to note that a couple of the leading proponents of armor in Nazi Germany were sent to Fort Knox to "observe" the experimental regiment.  When Germany attacked Poland, setting off WWII, they used most of General Chaffee's architecture and doctrine in the attack, "the Blitzkrieg".]

General Chaffee's architecture for the armored division, the basic unit at the time, included tanks and infantry regiments, with supporting artillery, engineering, and air units--in other words, all of the elements of the US Army at the time.  Consequently, the Armored Division Commander could reconfigure the elements of the attack depending on the mission and strategies employed.  After maneuvers in 1940, the Divisions were further architecturally refined to make them more agile, by eliminating the brigade structure, and having the regiments report directly to the division Commanding Officer (CO).  Then they added task group COs.  The Division CO would assign units to the task group/force CO depending on the mission and strategies to be employed.

However, it was not until the 3rd Army under General Patton's command that the agility of the armored divisions and the swarming tactics showed its full flower with air/ground teams using units from armored and infantry brigaded together in task forces.  This is the well known and documented race through France in August 1944.  It is shown additionally by the 90 degree swing of the 3rd Army in the Battle of the Bulge.  And finally, by the 1st Army's ability to swarm the Ludendorf railroad bridge across the Rhine at Remagen, creating a crossing where none had been "planned" and exploiting it.

The Film-Making Industry
An example that I use in my book is of the film industry.  One well known example of this type of enterprise are movie companies. They form to make a movie and dissolve once the final product is on the market (for some movie franchises, this may be a fair number of years).  The movie company forms (and swarms) around a movie script or concept. Once the movie is made the organized swarms dissolves with each component off in search of new opportunities.

ABB
One project oriented organization, ABB, one of the largest European corporation with about 125K employees, is built on a Swarming Architectural Pattern.  This organization is known for its management structure, which has only five levels (many organizations in the US with only 5000 employees have at least that many or more).  Yet, ABB makes money.  They do it by having many different engineering disciplines formed into relatively small teams.  According to Tom Peters, these teams are mostly self-directing/managing.  If one team finds an Request For Proposal (RFP) they cannot support or support in total, that team networks with other teams in the corporation to create a proposal, and if they win, to execute against that proposal.  In other words, the ABB team swarms on any RFP through the ABB organizational network.

Tiger Teams
Many organizations use "Tiger Teams" to rescue development and transformation programs and project that are in cost and schedule difficulties (the Programmatic Requirements).  This may be because of unanticipated technical risks, management mishandling of activities, or many other causes.  A Tiger Team is a set of personnel that are the best (most knowledgable) available to the organization.  These team members may be culled from all of the units of the organization or from other organizations.  One way to dispose of a risk is to transfer it.  [Sidebar: While most managers and financial engineers think the mitigation is the only way to disposition a risk, there are three others, including transference.  See my post The Risk Management Process]  With a Tiger Team, the risk item (the unknown) is transferred to someone who may know a solution or have a quick method for determining a solution.  The reason for having a Tiger Team is to swarm the design risk with the most knowledgeable team to increase the probability for turning the unknown into a known in a reasonable time and least expense, since any design risk may have more than one method for turning the unknown into a known.

Inhibitors and Facilitators
From the examples, above, it is clear that good systems engineering, system architecture, and enterprise architecture are the only processes that will minimize the probability and impact of the design risks in complex development based on pushing the edge of technology, and especially for swarming processes and their enabling and supporting systems and products. 

Good enterprise architecture can model the most effective investment opportunities based on the organization's mission(s), its strategies, and its swarming tactics, while good systems engineering can identify problems with requirements in the initial stages of development (when changes are most cost efficient) and system architect can create an architecture with non-redundant functions, grouped and structure in the architecture, that when allocated reduce the risk of requiring non-standard components and minimizes the effort and risk of development of the non-standard components.

Inhibitors
There are many inhibitors to agility, swarming tactics, and good supporting and enabling systems.  Three of these are described below.

Over Specification and Other Requirements Problems
One inhibitor, as amply shown in the example, is over specification of requirements and other problems with the requirements.  Over specifying requirements is one of several types of requirements issues (see other posts in my blog for additional, as well as Dr. Ralph Young's book on requirements); others being missing and "just plain wrong" requirements.  For example, I worked on an enterprise portal for a major organization.  When I joined the program, it was already in serious trouble because the sales team had promise to have the portal "fully functional" with 6 months and at half the cost that was initially estimated.  Unfortunately "fully functional" was not defined nor were the customer requirements clearly and completely defined, so the customer expected a portal of their wildest fantasies in 6 months.  This included corporate security who saw the portal effort as a way to get control of all applications on the network. 

Based on what had been promised, I tried to put together a customer system requirements document only to find that the security requirements (among others) were so over specified that it would be infeasible to meet, and impossible in the timeframe.  When I pointed this out to the customer, and recommended a series of short cycles for development--boiling a kettle at a time, instead of the ocean--my team was replaced with a new team, which failed.  After a series of catastrophic project failure by several teams, the customer finally understood that not all of the requirements could be met, or at least met in the time frame, they finally had a working enterprise portal.  It likely cost the customer 10 times as much as if the customer had followed my short cycle (agile development) plan and took at least 5 times as long.

Two ideas should be taken from this example.  First, insistent over specification and other failures in requirements management will inevitably lead to failure.  Second, attempting to use the waterfall development process and a highly managed process with incomplete and/or over specified requirements greatly increases the risk of project failure.  A better way to do it is to use short cycle development on large complex projects and programs.  Further, in doing so, swarming tactics could be used.  For example, when I led a process innovation project and concurrently managed an advance IT laboratory at a major corporation, I would hold weekly staff meetings with a team of 16 IT experts and up to 32 others.  The meetings were as much about design risk management as about weekly status for my role a program manager.  When a team member would bring up a risk or an issue, we would have a brief discussion followed by a number of team members getting together after the "status" meeting and coming to a solution.  For the team this was a swarming tactic; one that would not be "allowed" in a control freak management culture.

"Transactional Management"
There are two types of management, transactional and transformational.  Transactional management is what most people would define as "Management" or "Supervision".  This type of management creates "The Plan" and follows it regardless of whether "The Plan" still makes sense.  These are the Control Freak Management and Cultures.  They put following the plan ahead or adapting to the changing environment.  In development this type of management is called transactional (waterfall program management) method for development.

Working on a program where innovation is one key goal or high-level requirement for the program, transactional management will inevitably lead to cost and schedule overruns and most likely project failure (unless the budget and schedule drastically increase). The reason is that by definition innovation is the result of incorporating elements of the technical unknown (called risks) into product.

The level of time and budget required to convert and unknown into a known cannot be defined, only forecast; and as I can attest, sometimes if turns out much simpler than forecast, but many times even what appears to be a minor risk (a small can of worms), turns into a research effort the equivalent of the entire effort.  At this point the transactional manager goes into freak mode, creating 16 levels in the Gantt charts and holding program reviews every half day, in a futile effort to keep the project meets the business requirements of cost and schedule.

Transactional management greatly impedes swarming tactics, because swarming tactics require that the small groups are semi-autonomous. That is, they are told what their objective is, but not how to reach their objective.  It is up to them to meet the objective and to call on other teams with other skill sets as needed.  This gives control freak managers apoplexy because they have only a limited degree of control.

Communications
Early in my corporate career I had a boss who always said, "Communications is a Bitch", and it is. Words and phrases have culturally connotative and well as denotative meanings; which is one reason translation is so difficult. On technology projects they can be even more so.  One project on which I worked had a 300 page glossary of acronyms, with one acronym that was used over 50 times with completely different meanings depending on the engineering or manufacturing discipline.  Obviously, in the process of developing, constructing, or implementing a new product, system, or service poor communications will cause a great deal of process friction, which in turn adds to cost and time to a project to get everyone speaking using the same terms [Sidebar: I've termed this "violent agreement". Either having a long vehemently intense discussion around using different terms to mean the same concept or using the same terms to mean different concepts] and misunderstanding customer, functional or component requirements and specifications leading to (potentially catastrophic) product defects.  I know of an internal unpublished corporate study of several major development programs where 25 to 50 percent of the typical engineer's time was spent in meetings to get clear requirements and other sort out other engineering issues, while maybe 25 percent was spent on the actual engineering. [Sidebar: the rest was spent getting ready for program reviews, in the reviews, or getting feedback from the reviews]

Communications issues can be exacerbated when many small groups swarm on an issue for the same reasons that a single team have issues, as stated above, except they may be several orders of magnitude more difficult.

Facilitators
There are some facilitators of agility, swarming tactics, and good supporting and enabling systems.  Four of these are described below.

Standards
As weird as it sounds, having a clear and appropriate sets of standards and policies is a key facilitator for agility and swarming tactics for a project.  But, this is if and only if the set is appropriate in what the standards are for.  Too many will kill off a project in a sea of intermediate artifacts (reports, presentations, and other documents that have no purpose but to meet some policy or standard).  Too few will kill off a project slowly by the need to reconcile interfaces, among components, assemblies, and functions.  This is a great facilitator of swarming tactics

In any software development process, I would suggest that the team start with a minimum set of software interface and documentation standards.  Interface standards reduce issues with connecting modules, components, etc.  There are several such sets of standard for SOA.  Software documentation standards, including configuration management standards, greatly reduce the time for defect analysis, and will greatly help future maintenance and upgrades when the product is put into operation.  In general, the standards for hardware products and interfaces among components are well defined.

Process Design and System Architecture Derivation
A second facilitator is the combination of process design and functional derivation/functional design/system architecture.  Obviously, if the customer's "must do" requirements are poorly identified, no amount of design work will create a product that will meet what the customer really wants. [Sidebar: As defined in the CARRMA(r) product I've developed, must do  or event driven requirements define the methods, procedures, and processes the product must support.]  Consequently, a well defined process is of seminal importance when using using swarming tactics. The conundrum is that well defined methods, procedures, and processes are not always possible, especially went including technical innovations in the product, system, or service.

The derivation of clear, concise System Architecture or specification, or of a functional design is exceedingly important because it enables the program management to communicate the requirements of the functions, assemblies, and components to the swarm of technical personnel.  Again, as the many research articles and documents, the many methodologies and procedures, and the many program failures demonstrate deriving clear and concise system architecture or functional design is nearly impossible; another conundrum.

Short Cycles
One way, perhaps the only way, out of the conundrums, above, is through the use of short cycles of design, development, and construction.  That is, plan for incomplete, fuzzy customer requirements at the start of the project and an evolving system architecture/functional design.  This builds agility directly into the design/development/implementation process of the project.

To do this, first, create a high-level notional design based on the known customer requirements.  With the customer, identify any defects in this design and with the design/development team identify any unknowns (risks).  Second, build a prototype of the core of the application and share it with the customer.  Constructing this first prototype should take about one month.  The core should include the main interface with the user(s) and the access control and/or safety function(s).  I have found that showing the customer this prototype will do two things, elicit wire brushing from the customer for miss interpretation of the fuzzy requirements the customer identified and much clearer and more complete requirements.

At the same time, start to prototype all unknowns in the system in short monthly cycles.  This is where the swarming tactics begin.  The initial build of the prototypes will demonstrate which unknowns will be easy to make known and which won't.

After the first monthly cycle have the customer choose which of the know set of requirement to implement first (prioritization of the customer's requirements), then choose one or more to implement during the next short cycle.  At the same time continue to prototype the unknowns.  Demonstrate to the customer where a particular customer requirement is leading to unknowns (risks) that will shortly create issues (technical, cost, and/or scheduling problems).  If there is a required function with a high risk (very likely to occur) with a high impact on the effort, focus the swarming on that unknown.

At the end of the second monthly cycle demonstrate the product to the customer.  I have found that after the second or third cycle (two to three months) that the customer will deem the product the IOC (initial operating capability) and want it put into production.  In each short cycle you can add functionality to the production system and with the customer, you may (and most likely will) add to the customer's requirements list.  This method will greatly increase customer satisfaction.

Transformational management
To use swarming tactics requires a transformational management style.  A transformational management style means that the management adapts the project to the changing customer requirements and priorities of the requirements, the identification of unknowns (risks) and possibly the introduction of new technology in the process of designing, developing, constructing, and delivering a new (custom) product.  This is in stark contrast to the transactional management that cast the requirements and project plan in concrete and allows for no deviation until the project fails.

Enter EA, SA, and SOA
At first blush, Agility, and System Architecture/Enterprise Architecture seem to be diametrically opposed to one another; agility requires flexibility while one characteristic of architecture is rigidity.  Yet, in fact, architecture is a method for structuring, ordering, and organizing functions (a functional design) to meet the customer's requirements.  So what is an agile architecture that enables swarming tactics?  Enterprise Architecture provides the mission or goal, the strategies, processes, procedures, and governance supporting early, clear, and complete identification of the customer requirements for the project.  System Architecture provides the ordered and structured functions of the product supporting the design to meet the customer's requirements.

VEE, Swarming Tactics, and SOA
As defined and discussed in many of my prior posts, a VEE or Virtual Extended Enterprise is a form of a supply chain.  However, it is a supply chain with differences.  First, its not a parts ordering supply chain.  Instead all members of the VEE have a stake in the success of the product.  Second, its members all have peer-to-peer communications.  This means there is a coordinator of the activities in creating the product rather than a top down command and control organization.

Ford: The Integrated Supply Chain 
A typical supply chain has an Integrated Command and Control Organization to support highly structured projects and use "waterfall" transactional program management.  Each organization in the supply chain has a has a clearly defined position in the supply chain with (hopefully) clear and complete contractual technical and business requirements to meet in order to create the product.  This of supply chain organization performs quite well when technology is changing at a slow rate.  And it works particularly well for the reduction of manufacturing or implementation costs, that is making the product more cost efficient.  For example, Henry Ford introduced the Model T to the American public in 1908.  It was the first cost efficient automobile that met, from the start, the public's requirements for dependability.

The reason oft cited for Ford's and the Model T's success is the assembly line, though it was really Fort's focus on a cost efficient supply chain including the assembly line.  Ford treated the entire supply chain as a integrated command and control organization to the point that he built the Ford River Rouge facility in Dearborn Michigan as nearly a complete supply chain for the manufacturing of cars.  The integrated manufacturing facilities included docks to off load iron ore, a steel mill, a glass factory, a rubber and tire plant, and most of the other facilities needed to turn raw materials into an automobile. Now that's integration of the supply chain with a capital I.  For 19 years the Ford Motor Company manufactured the Model T successfully.  However, by 1927 other manufacturers were building cars in the same price range but with functions the Model T did not possess.  If I remember right that included electric starting (also, the only color the Model T came in was black).  So by 1927, the customer's business requirements, (cost), was superseded by the customer's increased technical requirements (new functions).   So, The Ford Motor Company had to design and manufacture a competitive vehicle or go out of business; enter the Model A.

By the 1950s, technology was changing at a rate such that the auto companies were coming out with a new design every year.  [Sidebar: In many cases it was more of a change in the surface sheet metal than a technology change, but technology was trickling in during the 1950s, and was streaming in by the 1960s.]  However, the customer could still order and pay for a car with just the functions (options) the customer desired well into the 1960s.

By the 1980s, the custom options had metamorphosed into options packages.  The reason was simple, the plethora of options, in an era of paper, custom manufacturing of each vehicle very expensive and ended up many defective vehicles; cars with the wrong options, which cost significantly more to retrofit than if they had been built with the options to meet the customer's requirements in the first place.  Additionally, it reduced the number of vehicle configurations from a very large to a more cost efficient number.  Consequently, the customers had to get used to paying for options they didn't want to get the ones they wanted--a somewhat mixed blessing for the customer.  And this is where automobile manufacturing processes supporting the customer's requirements stands at this point.

The members of the Integrated Supply Chain are hierarchically organized with functional and component requirements communicated down the chain and components and assemblies moving up the chain for final assembly.  This works well within the mass production paradigm, but not within the coming disruptive mass customization paradigm.

VEE: The Swarming Peer-to-Peer Supply Chain
As opposed to the integrated supply chain, the members of the Virtual Extended Enterprise are not hierarchically organized.  Instead, all members of the supply chain are members of the same team and sit at the same table.  This enables swarming.  When I was leading an advanced IT development and integration lab for a major defense contractor, initially I set up a hierarchical organization based on the conceptual design I had for the effort.

The goal of the effort was to virtually locate aircraft engineers, from multiple aircraft companies, rather than actually locate them.  Up to the early 1990s when there were joint efforts among major aerospace companies, engineers from all of these companies would be sent to a single location (generally the location of the contract winner) to be able to work together. The engineers at this location would include only the prime contractor and those in the second tier.  Still, the costs to the program of moving and temporarily housing all of those engineers (and sometimes their families) for years became a significant cost.  The job of our project was to use technology to reduce and/or eliminate most of those costs.

Through the first six months our effort struggled.  Then I started to empower the team of initially 12 to 16 very smart, self-starting IT professionals.  Later on 8 to 20 aircraft engineers were added.  I would provide guidance of what was needed (the requirements and to some degree the architecture (the structure and ordering of the processes) at a weekly team meeting.  The team would then identify the unknowns (risks) in potential designs.  After the meeting I would see groups form (specialty swarm) to reduce the risks and to design and code products to meet the requirements.  These groups would assign themselves tasks to be accomplished during the following week.  At the end of the week the team had generally done an astonishing amount of designing and development work.  If the team or I had identified a risk, that had not yet been reduced, a team member would bring it up at the next meeting, and another group would form after the meeting to reduce that risk.  Again, swarming tactics. 
[Sidebar: We actually used the virtual collocation environment (now called a collaborative environment) ourselves.  While it was against de jour corporate policy, the products of our lab enabled two or three young ladies with babies and toddlers to work from home; only coming in once a week for the team meeting] Once I empowered the team to swarm, and moved away from transactional management to transformational management, the project exceeded the expectations of most of the corporate management.

While I admit this was on a very small scale, it demonstrates the power of the VEE architecture and with swarming tactics for research and development efforts, that is, turning unknowns into knowns (i.e., risk reduction).  And risk reduction through innovation is the key to new technology insertion and process improvement (which means cost efficiency).

SOA and Swarming Tactics
Enter Services Oriented Architecture.  Services are components that serve a particular function and that have a standardized and published interface, which can be linked to other services.  This combination for software is an analog to children's building blocks, of which the best known are Legos.

Web Services, at the "application as a service" or the "Platform as a layer are one set of building blocks.  [Sidebar: In a typical SOA there are also, Infrastructure as a Service (better known as a component of Cloud Computing), and Data Center Facilities (also a component of Cloud Computing).]  To be a "web service" the code must have one or more "published" interfaces that is XML-based.

The job of the development team for an implementation to enable and support some new organizational process is to assemble these Web Services by structuring and ordering them, while identifying missing services that the team will need to design, code, and implement.  "Structuring and Ordering" sounds a whole like System Architecture; and it is exactly that.

While I've discussed Systems Architecture in other posts the part that's of interest in this post is the identification of missing functions and the design, development, and implementation of the functions, as well as their integration into the complete assemblage of the product. This is where the swarming tactics come in.  Like my experience in the lab, the program manager, in this case using the transformational management style, would present the entire team with the unknown (risk item) and the team members would decide if that have talents or expertise that would contribute to creating a usable service; that is the essence of the swarming tactic applied to development and implementation.  And, I've found that it's the only method that is both effective and cost efficient to creating innovative products, systems, and services.