The Cost of Rockets Built by NASA: Waterfall Process vs Short-cycle and Agile Processes

Shortcomings

This post is really not about the shortcomings of NASA, it's more about the inevitability poor, high cost deliverables when a) an organization looses its focus because of a constantly changing Vision and Mission; b) a development process for a complex system is stilted by formality, documentation, and transactional management; c) the development process assumption is that all of the requirements and risks (unknowns in the design) must be known before the next step in the effort starts.  Unfortunately, NASA is a good example.

NASA and the Waterfall Development Process

NASA does not have a clearly defined mission.  Instead, it has three poorly defined missions.  The first is to improve aircraft.  Currently, I and many others would expect that the Mission statement might be, "Research methods to improve aircraft fuel efficiency"; but apparently not.  From the sales to American Airlines, the Eurobus aircraft has much better efficiency than the Boeing aircraft. 

The second mission is to perform space science, in particular astronomy.  The consequence is that the astronomers have taken over NASA as their personal toy-maker.  While they are attempting to answer some big questions, it is not clear that these questions need to be answered by this generation of astronomers.  In actuality, they are racing each other to get into the "history of science" books.  The value of that work is not readily apparent except in the very long term.  Further, much of it could be better accomplished from space stations.  And while it is interesting to explore Mars by robot, it would be much more interesting to have humans explore and colonize the moon. 

The third mission of NASA is to make it possible to colonize space.  NASA has fallen down completely with this mission.  Over the past twenty years, NASA has been the poster-child for the highly formal waterfall process for development of new systems.  It's process demonstrates what happens when transactional and bureaucratic management takes on development efforts; it's programs fail to deliver.  It seems to me, there are two interconnected reasons for this.  First, they've lost their Big Hairy Audacious Goal (their BHAG, see Jim Colins' book Built to Last for a discussion).  Second, they've tied up their innovative thinking, designing, and developing in a web of red-taped processes and procedures. 

BHAG and NASA

Elton Musk, the creator of PayPal and the Tesla Motors, has also created SpaceX.  The reason that Mr. Musk invested in SpaceX is that he

"...believes the high prices of other space-launch services are driven in part by unnecessary bureaucracy. He has stated that one of his goals is to improve the cost and reliability of access to space, ultimately by a factor of ten." [see SpaceX]

First, this quote demonstrates that SpaceX has a BHAG, "...to improve the cost and reliability of access to space...by a factor of ten."  In fact this is a very ambitious mission for a small organization, yet it seems to be one at which they are succeeding.

NASA had such a Mission, "...to put a man on the moon before the end of the decade."  That's a Mission is short, clear, and measurable, the three criteria of a good mission statement.  NASA performed admirably.  NASA performed the Mission using a strategy of short cycles, each building on the previous.

However, once, it had achieved its Mission, the NASA leadership assumed that colonizing the Moon was its next Mission, but LBJ had bigger plans, the Entitlement Programs, which have saddled the United States with unimagined debt--predictably; at least I could see it coming in 1965.  One of the first results of the increased debt was cutting the Moon landing program was shut off completely.  In substitute, because NASA has a political constituency, the Mission was to "build" the Skylab on the cheap.  They partially achieved this mission by using components from previous manned space programs.  Then NASA was charged by the US Government with creating a Space Transportation System (STS) that would significantly lower the cost to boost a ton of materials or people into low earth orbit.  Initially, this was couple with constructing a US space station in low earth orbit, similar to the one built of the USSR.  This was a fairly ambitious effort, but no where near as ambitious and audacious as exploring and colonizing the Moon.  The result was, in 1971, a Mission to construct the space shuttle and, in 1984, a Mission to construct the United States space station, Freedom.  As NASA moved from the mission to the Moon to subsequent missions, they were hobbled with the "politically correct" requirements.  These includes ensuring that states of politically powerful Senators and Representatives got contracts to firms in their districts, ensuring the minority owned, female owned, veteran owned firms got a certain percentage of the contracts--as mandated by Federal Law.  From personal experience on the Freedom effort, I can pretty much guarantee these mandates added a minimum of 10 percent to the cost of the shuttle and space station efforts--probably much more when added to the second reason for NASA's failure.  This was a big part of the reason that President Clinton cancelled the US space station program.  Since, NASA has had several manned Missions, but the US Congress has not been on board, so these have not been adequately funded--using the NASA processes (which are very financially wasteful).

Short Cycle Versus Big Bang in Aerospace Development and Transformation

The second reason NASA is failing is their choice of processes.  Since the Apollo Program, NASA has become continuously more bureaucratic, with transactional management and processes rather than transformational management and processes.  Their formal processes are based more and more on the waterfall process with all of the Program Management overhead, in terms of intermediate artifacts--again, increasing intra- and inter-organizational friction and pushing up the cost and out the schedule of the program.

Part of the problem is that the waterfall process assumes that "all requirements are known at the start of the effort" and that "risks (unknowns) can be turned into known through schedule-based invention and innovation", that is, invention and innovation can be performed on a timeline.  Both of these are heroic assumptions and were proven false time and again in the NASA Programs.  However, transactional management and finance engineering militated against it because those assumptions allow them to "control" the effort.  The consequence has been, since the Shuttle Program, there have been no successful development of systems to deliver materials to space; there have been no lessons learned about how to build better shuttles, or even how create better heat shield for the shuttle.  Instead, to "protect" the personnel, these transactional managers have added more audits and check lists and redundant check lists.  And one administration after another cut off one of the key technical centers for real innovation, which from work in the 1960s, led in the 1990s, to advances in computing, the Internet, medicine and so on; instead, choosing to expend the funds on entitlement handouts. ["Feed a man a fish and he satisfied for a day, teach a man to fish...", do research on fish to ensure their survival and you feed generations.]

The results have been similar in character to what Boeing found.  In selling the 777 to Japan, they guaranteed that the Japanese aerospace industry would have a work share.  However, before they let the contract for a tail surface, Boeing asked the Japanese firm for a "test article".  When they received the tail surface, it met all of the specifications and met them more closely than the Boeing manufactured tail surfaces.  However, they had a mystery, the Japanese tail surface weighed ~150 lbs less than the one Boeing built internally.  On taking the Japanese test article and an "identical" Boeing test article apart, the Boeing engineers solved the mystery, the Boeing test article had ~150 lbs of shims to bring into conformance, while the Japanese built it to specs, rather than shimming it to specs.  Since reducing aircraft weight is one key to reducing fuel costs per passenger mile, the shims all over the aircraft were a big issue.

I've found that many program managers of governmental contracts require program management shims.  When a program gets into trouble the first thing they require is more status, more PMRs, and more detailed schedules.  All of this requires formal replan documents, which takes a significant chunk of the program's budget.  The is what has happened to the NASA man-in-space programs and is happening in most other federal, state, and local programs.  And again, it is exacerbated by the great additional friction of federal "fairness" policies, which, as noted earlier, direct funding to organizations with certain types of ownership, regardless of competence.  These are in fact, blatant attempts at quick cultural change.  They may be somewhat successful in meeting their mission, but they have wrecked programs like those of NASA.  This is noted in the quote from Mr. Musk, "the high prices of other space-launch services are driven in part by unnecessary bureaucracy." 

Since the aerospace industry started just over 100 years ago with a flight of 120 feet and in slightly over sixty years, was flying at supersonic speed (in fact Clarence "Kelly" Johnson was one designer who 1933 helped develop the Lockheed Model 10 Electra and finished his career by designing and developing the SR-71 Blackbird, flying at Mach3+) and had reached the Moon.  How did these inventors, innovators, and designers do it?

The Wright Brothers versus Langley

Invention and innovation in the aerospace industry is rife with examples of both short cycle and Big Bang development and transformation, but especially development, starting with the Wright Brothers.  Starting in 1899, the Wright brothers started active work on developing a "controllable heavier than air craft" (their BHAG).  Prior, they had identified that building a "stable" aircraft, an aircraft that would fly straight in still air, as all previous aircraft pioneers were doing, and then adding control functions would not achieve the goal of heavier that air flight.  Instead, they decided that they needed an unstable aircraft, one that the pilot would actually have to fly.

[Sidebar: This concept is found in sailboats as well.  Cruising sailboats have very long keels and small rudders.  The long keel provides stability for going in a straight line.  In fact, frequently, the crew can leave the helm unattended for 5 to 10 minutes without have the boat change direction by 5 degrees. on the other hand, racing sailboat have to be maneuvered before and during the race.  Therefore, they have short, deep keels and large rudders.  This makes then inherently unstable for holding their course.  In fact, my boat, which is a combination racer/cruiser will wonder 30 degrees or more off course in a matter of seconds; but it will turn around in practically its own length.  This shows the difference between stability in instability.]

Consequently, in 1899 the Wright Brothers started to build a series of controllable and maneuverable kites.  By 1901, they felt they had a flyable design, but it behaved poorly when compared with the predictions of the research current at the time.  They felt that there was a risk that the research was wrong.  Since there was no way to avoid, transfer, or accept the risk and be successful, they instituted a mitigation plan.  Initially this consisted of attaching small model airfoils to a bicycle and pedaling as fast as possible.  While the data from this "exercise" showed that the data, current at the time was not reliable, they had to invent the wind tunnel to get accurate data.  Again, they went through short-cycle experiments on very small wind shapes and developed the first set of highly accurate data on airfoils. 

With this data, in 1902, they were able to build a guilder that was fully controllable.  But after a series of test flights, the 1901 manned kite also showed a need for a vertical tail, and again, after a series of short cycles, they implemented the vertical tail surface and allowed it to move.  This too, was incorporated into the 1902 guilder.  The 1902 guilder proved that they were ready to add an engine to create the first fully controllable aircraft.

In 1903, the Wright Brothers found two more unknowns.  First, they needed a light-weight engine and second, they needed to determine what shape of propeller would produce the most thrust.  With respect to the light-weight engine, since they couldn't buy one, they built one with the aid of an employee/team member.  With respect to the prop, the conventional wisdom of the time said that the shape should be much like a ship's propeller blade.  However, the Wright Brothers found that thinking of a propeller as an airfoil that spins produced much more thrust.  Thus by the end of 1903 they were ready and flew the Flyer 120 feet on their second attempt (their first, the previous day had just gotten of the ground when they crashed it).  As the gained experience and confidence they flew farther, to 852 feet.

In 1904, they built a new Flyer based on their experience.  They flew in Ohio, close to home.  There they gained more piloting experience as well as refinements to the design, again in short-cycles, mindful or new requirements and risks as they came along.  They considered the 1904 craft merely a design step and at the end of the 1904 flying season, salvaged it, and burned the leftovers.  By 1905, they finally designed a Flyer that was truly usable, at least for the time and it was from these aircraft design that all useful controllable heavier than aircraft can be traced.

I left out two parts of the story.  First, they spent ~$1000 to create their aircraft (excluding their own time, as an investment) and Second, they were in a competition with Samuel Langley.  Samuel Langley was a well connected researcher that, having shown was funded by the US Government through the Smithsonian Institution (Museum) to over $50,000.  The reason that Langley was funded was that he had created a model unmanned aircraft that flew 3/4 of a mile under ideal conditions; it was stable in its design. 

[Sidebar: Again, as noted earlier, stability allows an aircraft to continue in a straight line.  It does enable the course of the aircraft (or boat) to turn easily.  This is a good analog to lean versus agile processes and waterfall versus short-cycle processes.  To create a lean process, the process engineer looks for waste.  While there are many forms of waste, one is "unnecessary" activities and procedures; and there are many of those, as well.  reducing these enables to process to flow more quickly to "the solution" or "the deliverable".  Therefore, this is a very stable process.

However, if external conditions effecting the lean process change, the process has no ability to effectively or successfully respond.  The same thing happens with the waterfall process.  The process is based on the assumption that "all of the requirements are known up front".  Then the Program Manager can plan out the effort (sometimes to the bathroom breaks) and manage to the schedule.  Unfortunately, this assumption is entirely false.  Consequently, projects using the waterfall process have to undergo much replanning, Engineering Change Orders (ECOs) and so on (that keep Program Managers employed).  This too means that waterfall process-based programs are stable, but brittle in that changing direction, however, slight, requires major effort.]

Langley scaled up this design, changed the powerplant from steam to gasoline, and built a bigger barge; but apparently not big enough since his craft hit something on the barge and immediately crashed.  As several writers has pointed out, even if the aircraft had flown, Langley did not have a good plan for his pilot to land his craft.  Since the pilot wasn't hurt, this attempt ended much better than it might have even if it were "successful", that is, flown successfully only to have the pilot killed at the end of the flight.  So Langley rebuilt his craft again and again it crashed.  While Langley was trying to assess the results and the damage, the Wright Brothers flew their aircraft.

However, because of the secrecy in which they flew, it wasn't until 1908 that they demonstrated their accomplishments to the world.  They flew both in the US and Europe, literally and figuratively flying circles around their competitors; they could control their agile aircraft while their competitors could fly their stable aircraft in straight lines.  The Wright Brothers succeeded because they understood their Vision as flying (controlling) a heavier than air craft where they wanted.  They spent from 1899 to 1903 getting the basics right in a series of short cycles, then refined the design in a series of short cycles.  On the other hand, Langley and a fair number of European competitors had a Vision of a stable heavier than aircraft, which they then hoped to find means to control.  Consequently they built small stable models, then scaled up the results in a waterfall like process--and failed.

Robert Goddard and Wernher Von Braun and Short Cycle Development

Rocket science too, used short cycle development, with little formal program management.  The father of modern rocket science, Dr. Robert H. Goddard worked to a single Vision (his BHAG), manned space travel virtually his entire life.  To realize this Vision, Dr. Goddard started by creating strategies for getting into space.  This led to two landmark patents in 1914 of 214 that he was granted.

By 1915, Dr. Goddard was working on meeting one of the requirements derived from his strategies, creating an engine with sufficient thrust to get into space.  He found that the rocket engines of the time actually converted only 2 percent of the energy they produced into thrust.  When he applied steam turbine nozzle technology to the nozzle of a powder rocket, he could the conversion rate to above 40 percent.  However, the total thrust produced by powder was not great enough to achieve his Vision.

Therefore, he started to investigate other fuels.  At the time, liquid fuels had the greatest chance of producing the needed thrust to weight ratio (and had an added advantage of being controllable, that is, reducing the flow of fuel to the engine reduced the thrust, while increasing the fuel to the engine increased the thrust).  In 1926, Dr. Goddard is credited with the first liquid fuel rocket to lift off, after working a number of years, with many tests on this engine.  Like the Wright Brothers, the first flight Dr. Goddard's flying machine was very short in both time and distance, but it proved that liquid fuel could be used.  And like the Wright Brothers, he then both continued to refine the engine design, and went to a new challenge.  In the case of the Wright Brothers, they learned to control their aircraft first, then to power it, while Dr. Goddard first learned to power the craft and then spent a significant amount of time learning to control it; even while refining the propulsion system.  Additionally, Dr. Goddard spent a good deal of time on raising funds for his research and development efforts.

By 1937, Dr. Goddard had flown a number of his rockets.  While none of them were particularly successful, at least to the general public, they did draw the attention of the rocket science community around the world,  and while Dr. Goddard, like the Wright Brothers, tended to be secretive, he did share technical information with others in the community.

In Germany, one who paid attention was Dr. Wernher Von Braun.  Like Dr. Goddard, Dr. Von Braun had had a Vision (again, his BHAG) of human space travel since he was a child.  By 1930, Dr. Von Braun had joined the "Space Flight Society" in Germany and started working on liquid fuel rockets; this is what brought Dr. Goddard's work to Dr. Von Braun's attention.  By 1934, the work of Dr. Von Braun came to the attention of the Nazis.  When in 1934 he was ready to publish his doctoral thesis, the Nazis classified it.  They then offered to support his work.  Obviously, the Nazi Vision and the Mission for rocket technology differed wildly from Dr. Von Braun's.  Still, both Dr. Braun and the Nazis wanted to develop rocket technology (and there was an implied, but very real, threat that non-cooperation would have dire consequences).

The net result was that Dr. Von Braun built on Dr. Goddard's work during the 1930s and until 1939, asked technical questions of Dr. Goddard from time to time.  During this time he first created the A-1, then A-2 and A-3 series of rockets in a series of short cycles.  Built on these prototypes, the A-4 series first flew in 1941, but due to reliability problems and interference from the Allies, it was not put into production until 1943--the A-4 was then redesignated as the V-2.  This was the first rocket to demonstrate the potential of space travel as well as the first intermediate ranged rocket.  Both Dr. Goddard and Dr. Von Braun confirmed that the V-2 design was actually a refinement of Dr. Goddard's original work.

While it did not affect the course of WWII as Hitler hoped, it did get the Allies attention. At the end, both the USSR and the US captured some V-2 missiles and while the USSR captured some of the German rocket scientists, the majority followed Dr. Von Braun into surrendering to the US.  In the period of the late 1940s to 1957, the USSR worked in secret on derivatives and upgrades of the V-2, while the US made minimal use of the technology and expertise they had available.  Only when the USSR launched Sputnik in 1957 and several Vanguard rockets failed, spectacularly, was the Von Braun team asked to put a satellite into space.  Basicly, this team rolled out the Redstone rockets they had prepared for launching a satellite three years before (They had not been allowed to launch it because it would have been politically incorrect to do so).  This was Explorer 1.

Dr. Von Braun went to NASA and continued his work.  NASA itself, in the early days created a short cycle plan to get a manned landing on the Moon by 1969.  It started with the Mercury Program, which "simply" got men into space, then graduated to Gemini, which verified that vehicles could meet and mate in space (necessary for the method the US chose to go to the Moon).  And finally, using Dr. Von Braun's Saturn 1 and Saturn 5 rockets, move in relatively short cycles to the moon landing.

Notice that from 1915 to 1969 and actually to 1972, the space/Moon landing program was not a single Big Bang process, but a series of small steps, each building on the previous.  This is the only type of process that would work for a Moon landing.  However, since, NASA has abandoned this process in favor or working the way other government departments work, with much red tape, little flexibility, few successes or deliverables...as described so succinctly by Musk.

Final Thoughts

Short cycles allow for experimentation, the waterfall process doesn't.  As noted by Dr. Goddard,

"It is not a simple matter to differentiate unsuccessful from successful experiments. . . .(Most) work that is finally successful is the result of a series  of unsuccessful [short-cycle] tests in which difficulties [risks and issues] are gradually eliminated." (Written to a correspondent, early 1940s, See Lehman, Milton, This High Man: The Life of Robert H. Goddard [N.Y., N.Y.: Farrar, Strauss, and Co., 1963], p. 274.)

 Experimentation is not economic in the calculations of finance engineering, since most experiments are failures, and cannot be preplanned.  The first generation of NASA personnel and management understood this, but later generations of NASA management have not.  Since the Moon Mission, NASA has lost its way.  First, NASA's Mission keeps changing with the political winds, both within government and within the "scientific" community.  There has been no clarion clear Mission since the Moon landing Mission.  In the last 10 years, NASA was given a Mission to set up a Moon colony, a Mars colony, then a Moon colony (again), then a manned Mars mission.  Each of these had different strategies for achieving their mission.  From what I've read, none of these include short-cycle processes...that's too expensive.  Instead, they have been Program Management and Finance Engineering controlled waterfall-like programs.
[Sidebar: If it were me, I would choose a Mission to Mars by way of the Moon.  The reason being to learn more about space travel and colonization of a planet in an environment where emergency and other short-cycle risk reduction flights would have a chance, rather than one that the "short duration" is seven months.  But, that's only my opinion.]

Unfortunately, as events over the past 10 to 15 years have shown, it's not only NASA, but the entire Federal Government that needs to overhaul its Missions, Strategies, laws, regulations, policies, and standards.  Given the two current diameterically opposed views of the roll of government (see my post on "The Purpose of Government" and linking posts, for my thoughts on the roll of government) the US Federal Government remains uncontrolled for all practical purposes.