A good many of the issues result from a poor understanding by economists and Finance Engineers of the underlying organizational economic model embodied in Adam Smith's work, which is the foundation of Capitalism. The result of this poor understanding is an incomplete model, as I describe in Organizational Economics: The Formation of Wealth.
"Cost avoidance is a cost reduction that results from a spend that is lower then the spend that would have otherwise been required if the cost avoidance exercise had not been undertaken." ]
- Quantifiable--Metrics that organization is currently using to measure its process(es) performance and dependability that will predictably change with the development or transformation; the metrics will demonstrate the benefits (or lack thereof). This type of metric will provide hard, but not financial, evidence that the transformation has benefits. Typically, the organization knows both the minimum and maximum for the metric (e.g., 0% to 100%).
- Measurable--Metrics that organization is not currently using to measure its performance, but that should measurably demonstrate the benefits of the development or transformation. Typically, these metrics have a minimum, like 0, but no obvious maximum. For example, I'm currently tracking the number of pages accessed per day. I know that if no one reads a page the metric will be zero. However, I have no idea of the potential readership for anyone post because most of the ideas presented here are concepts that will be of utility in the future. [Rant 5: I had one VP who was letting me know he was going to lay me off from an organization that claimed it was an advance technology integrator that "he was beginning to understand was I had been talking about two years before"--that's from a VP of an organization claiming to be advanced in their thinking about technology integration--Huh....] Still, I have a good idea of the readership of each post from the data, what the readership is interested in and what falls flat on its face. Measurable metrics will show or demonstrate the benefits, but cannot be used to forecast those benefits. Another example is of a RAD process I created in 2000. This process was the first RAD process that I know of, that the SEI considered as Conformant; that is, found in conformance by an SEI Auditor. At the time, I had no way to measure its success except by project adoption rate (0 being no projects used it). By 2004, within the organization I worked for, that did several hundred small, medium, and large efforts per year, over half of them were using the process. I wanted to move from measurable to quantitative, using metrics like defects per roll out, customer satisfaction, additional customer funding, effort spent per requirement (use case), and so on, but "the management considered collecting this data, analyzing and storing it to be an expense, not an investment and since the organization was only CMMI level 3 and not level 4, this proved infeasible. [Rant 6: It seems to me that weather forecasters and Wall St. Market Analysts are the only ones that can be paid to use measurable metrics to forecast, whether they are right wrong, or indifferent--and the Wall St. analysts are paid a great deal even when they are wrong.]
- Observable--Observable is the least quantitative, which is to say the most qualitative, of the metric types. These are metrics with no definite minimum or maximum. Instead, they are metrics that the participants agree on ahead of time--requirements? (see my post Types of Requirements.) These metrics are really little more than any positive change that occurs after the transformation. At worst they are anecdotal evidence. Unfortunately, because Financial Engineers and Managers (for reasons discussed above) are not willing to invest in procedures and tooling for better metrics like those above, unless they are forced into it by customers, (e.g., requiring CMMI Level 5), Enterprise Architects, System Architects, and Systems Engineer must rely on anecdotal evidence, the weakest kind, to validate the benefits of a transformation.
"Every revolutionary idea evokes three stages of reaction. They can be summed up as:
–It is Impossible—don’t Waste My Time!
–It is Possible, but not Worth Doing!
–I Said it was a Good Idea All Along!"]
Consequently, "proving" that the engineering and implementation of the transformation actually reduced the cost, and not the "manager's superior management abilities" is difficult at best--if it weren't the manager's ability, then why pay him or her the "management bonus" [Rant 8: which is were the Management Protective Association kicks in to protect their own].
The Benefits Measurement Process
As usual, I will submit to the reader that the keys (culturally and in a business sense) to getting the the organization to measure the success (benefits) of its investment decisions and its policy and management decisions is twofold. The first high-level activity is a quick, (and therefore, necessarily incomplete) inventory of its Mission(s), Strategies, Processes and tooling assets. As I describe in Initially implementing an Asset and Enterprise Architecture Process and an AEAR, this might consist of documenting and inserting the data of the final configuration of each new transformation effort as it is rolled out into an AEAR during an initial 3 month period; and additionally inserting current Policies and Standards (with their associate Business Rules) into the AEAR. Second, analyze the requirements of each effort (the metrics associated with the requirements, really) to determine the effort's success metrics. Using the Benefits Context Matrix determine where these metrics are incomplete (in some cases), over defined (in others), obtuse and opaque, or conflicting among themselves. The Enterprise Architect would present the results of these analyses to management, together with recommendations for better metrics and more Process Effective transformation efforts (projects an programs).
The second high-level activity is to implement procedures and tooling to more effectively to efficiently observe and orient the benefits through the metrics (as well as the rest of the Mission Alignment/Mission Implementation Cycles). Both of these activities should have demonstrable results (an Initial Operating Capability, IOC) by the end of the first 3 month Mission Alignment cycle. The IOC need not be much, but it must be implemented, not some notional or conceptual design. This forces the organization to invest resources in measurements of benefits and perhaps, in which component the benefits exist, control, process, or mechanisms.
Initially, expect that the results from the Benefits Metrics to be lousy for at least three reasons. First, the AEAR is skeletal at best. Second, the organization and all the participants, including the Enterprise Architect have a learning curve with respect to the process. Third, the initially set of benefits metrics will not really measure the benefits, or at least not effectively measure the benefits.
For example,I have been told, and believe to be true, that several years ago, the management of a Fortune 500 company chose IBM's MQSeries as middleware, to interlink many of its "standalone" systems in its fragmented architecture. This was a good to excellent decision in the age before SOA, since the average maintenance cost for a business critical custom link was about $100 per link per month and the company had several hundred business critical links. The IBM solution standardized the procedure for inter-linkage in a central communications hub using an IBM standard protocol. Using the MQSeries communications solution required standardized messaging connectors. Each new installation of a connector was a cost to the organization. But, since connectors could be reused, IBM could right claim that the Total Cost of Ownership (TCO) for the inter-linkage would be significantly reduced.
However, since the "benefit" of migrating to the IBM solution was "Cost Reduction", not Increased Process Effectiveness [RANT 9: Cost Avoidance in Finance Engineering parlance], Management and Finance Engineering (Yes, both had to agree), directed that the company would migrate its systems. That was good, until they identified the "Benefit Metric" on which the management would get their bonuses. That benefit metric was "The number of new connector installed". While it sounds reasonable, the result was hundreds of new connectors were installed, but few connectors were reused because the management was not rewarded for reuse, just new connectors. Finance Engineering took a look at the IBM Invoice and had apoplexy! It cost more in a situation where they had a guarantee from the supplier that it would cost less [RANT 10: And an IBM guarantee reduced risk to zero]. The result was that the benefit (increased cost efficiency) metric was changed to "The number of interfaces reusing existing connectors, or where not possible new connectors". Since clear identification and delineation of metrics is difficult even for Increased Cost Efficiency (Cost Reduction), it will be more so for Increased Process Effectiveness (Cost Avoidance).
For example, BAMM used in conjunction with SOA-based Services will enable the Enterprise Architect to determine such prosaic metrics as Process Throughput (in addition to determining bottlenecks) before and after a ttransformation. [RANT 11: Management and Finance Engineering are nearly psychologically incapable of allowing a team to measure a Process, System, or Service after its been put into production, let alone measuring these before the transformation. This is the reason I recommend that the Enterprise Architecture processes, Like Mission Alignment be short cycles instead of straight through one off processes like the waterfall process--each cycle allow the Enterprise Architect to measure the results and correct defects in the transformation and in the metrics. It's also the reason I recommend that the Enterprise Architect be on the CEO staff, rather that a hired consulting firm.] Other BAMM-derived metrics might be the cost and time used per unit produced across the process, the increase in quality (decreased defects), up-time of functions of the process, customer satisfaction, employee satisfaction (employee morale increases with successful processes), and so on. These all help the Enterprise Architect Observe and Orient the changes in the process due to the transformation, as part of the OODA Loop-based Mission Alignment/Mission Implementation process.