Systems Engineering and System Architecture in an Agile and Short-cycle Transformation Process (Second Try)

I haven't updated the blog in a while; but I've been working. Here is a copy of a paper that I will present at an INCOSE Meeting in Detroit early in November.  I tried posting a link to a PDF version of the paper and it doesn't seem to work, so here is a copy.

Bob

Abstract

For a Systems Engineering and Architecture process to tame uncertainty in a continuous change organizational and technology environment requires that the transformation process be agile.  Agility is “the ability to successfully respond to unexpected challenges and opportunities.”  Fundamental to agile systems and engineering and architecture is that assumption that “not all requirements are known upfront”.  This means that the process must allow the customer to add requirements throughout the transformation process.

In this paper, first, I will present a short-cycle CMMI Level 3 software development process, based on Extreme Programming, and anecdotal results from over 100 efforts using the process.  Second, I will generalize this process to include systems engineering and system architecture procedures and functions many types of agile and short-cycle development and transformation efforts.  Third, I will discuss the implications of this process on the roles of the program manager, systems engineer, and system architect.

What and Why Short Agile/Short-cycle Development?

This paper will describe, somewhat anecdotally and somewhat analytically, my experience with agile and short-cycle development and transformational processes. Then it will discuss what I’ve found relative to the change in emphasis in the roles and responsibilities of a project’s team members.

Since the terms “agile” and “short-cycle” are among the many hyped and over used terms, first, let me define what I mean by the terms. 

·         Short-cycle is “One implementation cycle Every 1 to 3 months” depending on the type of development or transformation, as I will describe later.

·         Agility is “The ability to successfully respond to unexpected challenges and opportunities”.[1]

Additionally, the reason I include both “development” and “transformation” is that they are somewhat different from a process perspective.[2]  In development, the team is creating an entirely new product, while in transformation; the team starts with a current product and revises or transforms it.

So why is it important that projects migrate to agile/short-cycle development and transformation processes?  To meet the challenge of “better”, “faster”, ”cheaper” that all customers want.[3]

·         “Better” means producing a higher quality product, whether the “product” is a product, service, or process.  Quality is defined as “Conformance to [customer] requirements” by Phil Crosby, in the seminal work on the topic, entitled, Quality is Free.[4]  Any project that focuses on requirements will produce a highly effective, therefore, high quality product.  In any project, “Better” is the responsibility of the systems engineer and the system architect for that project.

·         “Faster” means creating or implementing a useable product in the short period.  Most customers prefer a usable part of a complex system in operation early, than the complete system and wait.  For example, Henry Ford did not wait until electric start was invented to start manufacturing the Model T; yet he sold a fair number of them.  Though his company and others manufactured a number of models before the Model T, many automotive historians would argue that the Model T Ford was the Initial Operating Capability (IOC) for the automotive industry.[5]

·         Cheaper” means more effective and more cost efficient, not the lowest cost.  There is a big difference that “finance engineering” tends to forget.  A stone from the backyard, a hammer, and a nail gun can all drive a nail, so why do carpenters building a house choose a nail gun, by far, the most expensive option?  The reason is that a nail gun can drive nails much more uniformly (i.e., more effectively) and that one carpenter can drive more nails much faster than using a hammer (i.e., more cost effective).

An Example: The Web Application Development Methodology (WADM)[6]

In late 1999 and early 2000, after the group supporting the organization I worked for announced a software CMM Level 3 process, it almost had a rebellion on its hands among the software developers.  The reason was that it could take longer to do the “necessary” paperwork than it took to do the job for small to medium projects (40 to 700 hours).  In the case of the minor project it nearly always took longer.

True of all large organizations, the internal information systems organization I worked for formed a team to “reduce” the SW CMM intermediate artifacts (the paperwork that is used only during the process, but has no value  beyond the end of the project—things like status reports).  Looking at the requirements (in the following section), I realized that a radical rethinking was required to meet them.  I had been researching the Software Engineering Institute’s “evolutionary spiral process” and other Rapid Application Development (RAD) processes and concluded that the simplest approach would be to add functions and documentation to the simplest short-cycle process of them all (eXtreme Programming—XP).  XP started from the premise that developers should develop and not write documentation, which apparently made it the opposite of SW CMM applied to a waterfall process.  In using this approach, but simplifying the process documentation of XP—which for me proved almost indecipherable—and the adding functions and documentation as needed to meet, the SW CMM, and later CMMI level 3 created a highly usable processes and one that was low cost with high customer satisfaction, as I will show.

The requirements

The requirements (or capabilities) for this new process and system had four high-level requirements; they should customer focused, produce a high-quality product, in a short time span, at low cost.

Customer Focus

What the team meant by the requirement for customer focus was:

·         Customer has ownership of, and a high degree of visibility into, the project.  In order for a customer to feel ownership of the project and its deliverable, the product, the customer needs insight into what is going on in the project and a feeling that the product is being influenced by the customer’s inputs (in the form of requirements).  This requires a:

·         High level of customer commitment and involvement in the development team.  The customer must feel like a team member and be treated as such by the rest of the team.  The customer’s role on the team is as the source of the requirements.  This means that the:

·         Customer can add, delete, modify and/or reprioritize functions as the application is being developed. Giving the customer the ability to add, delete, and modify or reprioritize the requirements during the process enables the customer to identify “new” real requirements that were forgotten in the original RFP or contract, but are of higher utility to the customer than the ones in the original source.  Consequently, this:

·         Meets customer’s real demand due to the customer’s degree of influence and control over the development effort.

Produces High Quality Product

What the team meant by the requirement for producing a high quality product is based on Phil Crosby’s definition of quality cited earlier,

·         Product has minimum number of defects.  There are two types of defects.  First, the product does not meet the customer’s requirements (fails validation (i.e., Customer acceptance tests) or second, there is a basic flaw in a function or component of the product.  These defects are minimized by clear identification of requirements—as required above—and clear and consistent high quality verification and validation procedures.  This means that:

·         Testing and defect prevention are integral to the process.  In fact, it means that regression testing is to used, that is, retesting for each build.

Short time Span

What the team meant by short time span is that:

·         The development cycle is divided into successive, short-duration builds.  A short-duration build is one that produced a build every month to six weeks.  Further,

·         The first build and each successive build are usable.  We did not consider it a build if it was not useable.

Low Cost

·         No unnecessary functions are allowed.  Functions that are not required are not developed.  Many times, especially for software, the designers, developers, and implementers add functions to the product that customer’s requirements do not call for.[7] Most of the time, creating functions that are not called for in the requirements means additional time removing them from the product and retesting.  Another requirement for low cost is to ensure that:

·         Project documentation and reviews are minimal.  This is key to a more effective process.  I did this by eliminating all intermediate artifacts, like status reports, and reviews like all of the XDRs (PDR, CDR, and so on) and all PMRs.  Finally, the last requirement of the team for low cost was that the process should:

·         Work with virtual teams.  This was important in the organization I worked for at the time because it was nation-wide.  It had well trained, but idle team members in West Texas, while at the same time, it had a work overload in Baltimore.  Both idle team members and overloaded team members, increases the cost to the organization.  Enabling the team members to work virtually, greatly reduces both costs.

WADM Structure

Figure 1 shows the structure of the WADM process.  It consists of an initialization phase and three cycles; Development, Design, and Construction.

Figure 1 – The WADM Process

Development Cycle

The objective of the Development Cycle is to ensure that the customer’s requirements for development (in priority order by the customer) are understood and met (that is identify customer requirements and validate that the requirements are met). Its duration is approximately one month to six weeks.  It is made up of:

·         A Planning Game[8] where the customer decides on the priority of all unfulfilled requirements and the team, including the customer, decides on which of the highest priority requirements will be implemented during the next cycle.  In many ways the Planning Game replaces the Program Management Review (PMR), except the Planning Game’s emphasis is on meeting the customer’s system requirements, and not its programmatic requirements of cost and schedule.

·          Three to four Design Cycles

Design Cycle

The objective of the Design Cycle is to design an application to meet a prioritized subset of the customer requirements.  Its duration is approximate one week.  It is made up of:

·         A Design Cycle Planning Meeting where the customer reviews the progress of the design in meeting the current requirements and recommends changes.  Sometimes, the developers can incorporate the changing during the next series of construction cycles and sometimes these are really new requirements.

·         Generally five Construction Cycles

Construction Cycle

The objective of the Construction Cycle is to code and unit test modules of the application to meet the current requirements.  Its duration is daily.  It is made up of:

·         A Standup Meeting that is supposed to be 15 minutes long, but that sometimes runs to a half hour.

·         Construction Activities where the developers actually code.

The WADM process meets all of the requirements for an Agile/Short-cycle development or transformation process.  First, it is a Short-Cycle process because a new usable version of the software product every 4 to 6 weeks.  Second it is Agile because it is based on the principle that “Not all the requirements are known upfront”; therefore: it allows the customer to identify new requirements anytime in the process and allows the customer to reprioritize the requirements every 4 to 6 weeks.

Systems Engineering Functions in the WADM

When compared with his or her role and responsibility in a waterfall-base development process the importance of Systems Engineer is much greater in the WADM.  The emphasis in the WADM is on the customers system requirements.  In 2000, I realized there were two types of requirements, the “must perform” and the “must meet”.[9]  Consequently the WADM requirements management procedures use a combination of: Use Cases[10] “One way one type of actor will use the product”, which are the “must perform” requirements, and Design Constraints, which are the “Must Meet” requirements (e.g., standards, regulations, codes, policies, etc.).

Additionally, the Systems Engineer must perform Risk Management procedures.  As defined in the WADM process this is management of the “unknowns in the design”.  These risks frequently arise at the daily standup meetings and can, in most cases, be disposed of at the same meeting or during the next construction cycle.

Finally, the systems engineer must perform the Verification and Validation procedures.  The procedures, themselves must be base on scenarios of the use cases (a scenario is a data instantiated version of the use case, or the verification procedures documented in the design constraints.  Of particular importance for defect minimization is a regression verification and validation procedure, that is, going through all of the verification and validation procedures from the previous development cycles.

Experience with WADM

My experience with the WADM process falls into three part, the pilots, what happened in production, and why were there failures.

WADM Pilots

Starting in 2001, before the process was rolled into production, we had two pilots.  The first we called the Health Visibility Management System (HVMS).  It is a management dashboard for programs and projects graded against program metrics.  Several large programs had their own management dashboard systems.  This made it cumbersome for executive management.  Further, the metric categories were not in same, in the same order, and were not based on uniform data or reporting methods.  Consequently, our customer representative, a former COO of the corporation led a customer team to resolve these issues.  Initially, this project was to provide a framework within which the executive management could gain access to a more uniform set of data from the other HVMS systems.  According to my best estimates of the initial set of requirements, this should have taken about 3 cycles (3 months).  However, as we developed the first three releases, it became obvious to the customer that a) the system was usable, and b) they had many more requirements.  The net result was the initial 3 month project continued at least 7 years, the last time I checked.  Further, our customer representative stated, “This is the only way I’ll ever build software again.”

A second project was the development of Financial and Asset Management System for Data Networks.  At the time, the corporation had merged with nine major organizations and close to 50 small organizations over a 10 year time frame.  At the same time, the data network grew to 8 times its initial size, yet the number of personnel managing “the phone bills” for the corporation had stayed the same.  The result was they were working 80 hours or more, they were getting further behind, and they were catching fewer billing errors.

After I gathered their first set of use cases (their “must perform” requirements), I estimated that it would take at least 6 development cycles (with the personnel available) to meet them.  Unfortunately, the networks department had only enough money for 3 cycles.  So we pared back the requirements to their highest priority set that would make a useable product.  We developed the initial datastore, security, and inputs displays during the first cycle.  While, we were developing some initial reports, in the second cycle, they were inserting data into the datastore.  In this insertion process, the network personnel found enough errors to more than pay for the additional 3 cycles.  By the end of the first six months, also the end of the financial year, the personnel were back to working 40 hours a week, they were catching most of the charging errors, and they could provide senior and executive management timely and correct reports on the data network costs.  Consequently, the management had more requirements.  The last I checked, the project had continued 5 years.  The customer said “This is the first time I got what I wanted without multiple defects or changes and added cost.”

Production (2003 to 2005)

The WADM process was roll into production in late 2002.  By 2005, well over 500 small and medium sized efforts used the process successfully.  The reason it was so successful in its adoption throughout the internal organization was that it met its requirements, better, faster, cheaper.

However, I was called in on two projects that used it that didn’t meet those requirements.  In both cases, the reason is “they didn’t follow the process”.  Instead the program managers attempted to follow the waterfall process and identify all of the requirements upfront and set up a project schedule based on those requirements and stick to the schedule.  The result was they each cost more than four times as much as estimated and did not roll out even the initial iteration of the product.

Shortcomings

The WADM process has some major shortcomings.  Chief among these is that it is designed and implemented only for software development efforts.  It is not adaptable for hardware development or for hardware/software integration efforts.

Further, it cannot handle the complexity of large systems because it lacks systems architecture procedures (functional design processes), component requirement allocation procedures and tradeoff study procedures.  However, these two major limitations can be addressed.

The Generalize Agile Development and Transformation Process

The Generalize Agile Development and Transformation process is envisioned to eliminate the shortcomings of the WADM process.  It requirements includes the WADM Process Requirements of being customer focused, producing a high quality product, in a short time span, and at low cost; all of which were discussed in the previous section.

Additionally, it must support the development or transformation of both hardware and software products.  This would include:

·         System Architecture procedures to develop functional requirements (or specification) and the ordering of these to create a system architecture (or functional design) and the allocation of the functions to components.

·         A more integrated risk management functions for traceability

·         And a Tradeoff study procedure that is traceable to the requirements.

Figure 2 shows the process.

Figure 2 – Generalized Agile Development and Transformation

The Process

This process is made up of the three cycles of the WADM process, plus a fourth System implementation Cycle.  As shown in Figure 2, this process has a Construction Cycle, a Design Cycle, and a Development Cycle or all of its software.  However, additionally it has the System Implementation Cycle.  This cycle has many functions familiar to systems engineers.

The objective of the system implementation cycle is to ensure that the customer’s requirements for implementation of the system are understood and met (that is identify customer requirements and validate that the requirements are met for hardware and the integration of hardware and software).  The duration of this cycle is 3 to 6 months depending on the feasibility of useable hardware and its integration with software and other hardware.  It is made up of Systems Engineering, Detailed Design, Implementation and Procurement Activities.

As was noted in an earlier section, hardware of all types has frequently been developed using this type of short-cycle process.  The first model might only be a prototype or an X-model to try out a new concept.  That is followed by a number of test articles culminating in a fully functional produce.  Even then the process continues with block point upgrades and so on.  Sometimes the Systems Implementation Cycles are treated by program management as a single cycle.  But a close look at the process frequently reveals a hidden short-cycle process.[11] In taking advantage of this “short-cycle attribute of a large process, the development or transformation process can be much more agile.

Changing Roles and Responsibilities in Short-cycle Processes

To take advantage of this attribute within all programs, the team must revise their thinking about the various roles and responsibilities of the team members.  As mentioned earlier the key change in turning a waterfall-based process into an agile/short-cycle process is to its assumption that, “all requirements are known upfront”.  In and of itself this is a heroic assumption that, for all but the simplest effort, is false.  Further, it minimizes the agility of development or transformation effort, by adding all of the intermediate artifacts and reviews required for a formal waterfall process.

However, a truly agile process that takes advantage of short-cycles assumes that “not all requirements are known upfront”.  Any short-cycle process that does not make this assumption will have greatly increased management cost and schedule problems inherent in the waterfall process.

Program Management

With the assumption that “not all requirements are known upfront” the role and responsibilities of the program manager are radically altered.  For example:

·         Project Planning is minimized.  How can a project be fully scheduled and resources identified for unknown requirements?  Instead, the project plan should:

o   Emphasize Project Goals, Missions, Strategies, etc.

o   Emphasize the Systems Engineering Management Plan (SEMP)

·         Minimizes all intermediate artifacts and reviews.

o   There should be no need for status reports or Program Management Reviews (PMRs) since the customer is involved on the effort on at least a weekly basis.

o   There should be no need for Engineering Change Orders (ECOs), since requirements change is built into the process and procedures.

·         There is no need for scheduling, in the traditional waterfall sense.

o   Only high-level schedule possible over the System Implementation Cycle

o   Detailed scheduling is irrelevant (and a waste of resources) because schedules can change on a weekly or monthly bases.

·         Earned Value Analysis is irrelevant in its present form.  Traditional EVA will not work since there is no way of knowing what percentage of the work is complete when the requirements are continuously changing.  Instead, an alternative to EVA will need to be requirements completion based.

·         Agile/Short-cycle Contract Management cannot use a contract based on meeting the initial set of contracted requirement.  Agile/Short-cycle development and transformation processes require “Level of Effort”(LOE) Contracts, since there is no way to guarantee that all of the contracted requirements will be met—more importantly additional  requirements may be found in the process and some initial requirements may not be met during the contract.  Still, the customer will be more satisfied if the product the customer pays for, actually meets his or her highest priority needs.

·         Procurement is very important in an agile/short cycle process because the timing of the procurement of hardware and software is very important.  If the procurement is made too early the equipment or software will set around for awhile, at least, and as the requirements change the team may find that the equipment is not needed or a different version is needed.  On the other hand, if the equipment is procured too late, the project may come to a halt due to needing that hardware or software (though this is less true of short-cycle efforts and waterfall-based efforts because the functions needing the equipment can be reprioritized and be implemented later in the effort).

Systems Engineering

 With the assumption that “not all requirements are known upfront” the role and responsibilities of the systems engineer changes as well.

·         The Requirements Management procedures become seminal to successful agile/short-cycle development and transformation process.  The Requirements Management procedures must enable the customer to add requirements at any time in any cycle of the process.  It must also enable traceability from the product, through the design to the customer’s system requirements.

·         Risk Management must manage risk; risks are unknowns in the design and are very important to identify early.  With agile/short-cycle development and transformation processes this is particularly true because of the need to estimate the cost of each requirement (in the case of WADM, and the Generalized version in this paper, the use cases and design constraints)

·         Configuration Management is a must since a new version will be rolled out every one to three months.  Since the project team wants to insure that the version and components rolled out are the version and components verified and validated, disciplined configuration management in an imperative.  Configuration Management must include managing the Verification and Validation information and documentation to ensure proper regression testing.

·         Verification and Validation procedures must trace directly to the requirements at all levels, customer (A Spec), functional (B Spec) and Component (C Spec).  And both Verification and Validation procedures must be regressive, that is, they must include all prior verification or validation procedures.

From this you can get the idea that for an agile/short-cycle development and transformation effort there is much less emphasis on Program Management procedures and much more on Systems Engineering.

Summary

Agile/Short-cycle development and transformation can increase quality, decrease cost, and reduce the schedule concurrently, but only if the procedures, functions, roles, and responsibilities are changed and executed in a highly disciplined manner. To quote from a 25+ year veteran software developer who had worked 5 years in a SW CMM Level 3 environment: “In the first iteration of this process (the WADM), I thought that documenting all that requirements cr_p was more busy work.  I was wrong, this documentation is really helpful”.

---

[1] This definition comes from Agility Forum’s Virtual Extended Enterprise committee. The Agility Forum is based at Lehigh University.

[2] Unfortunately, I will have to leave the description of the differences for another presentation.

[3] An aircraft engineer had a saying posted on the wall of his office, “Cost, quality, schedule; choose two”, all customer prefer all three.

[4] Phil Crosby, Quality is Free.  Penguin Group, New York, 1980.

[5] For a discussion of the growth of the automotive industry, see my paper, Industrial Location Behavior and Spatial Evolution, Journal of Industrial Economics, Vol. 5 (1977) pp. 295-312

[6] In 2006 The Northrop Grumman Corporation applied for a patent on this process.  The process has been presented at several open forums since.  It is a CMMI level 3 conformant process

[7] It does also occur in hardware.  One famous example was the WWII M-3 tank where it showed up with a police siren.

[8] For more detail of the Planning Game, looks for papers on XP.

[9] While I will not claim to be the only one that came up with this concept for requirements, I did come up with it independently of others over a period of some 25 years of tortured and frustrating attempts to find a good method to identify requirements.  I had tried use cases without design constraints and found the shortcoming in that and thus backed into the concept of design constraints.

[10] This came directly from XP.

[11] The aircraft Kelly Johnson took advantage of the inherent short-cycle process within large hardware development efforts when he designed the U-2 and the SR-71.