24 October 2001

On 19 October 2001 the US General Accounting Office (GAO) published a critical and comprehensive report on the status of the Joint Strike Fighter (JSF) acquisition program.

"Although the Joint Strike Fighter program has made good progress in some technology areas, the program is at risk of not meeting its affordability objective because critical technologies are not projected to be matured to levels that we believe would indicate a low risk program at the planned start of engineering and manufacturing development in October 2001."

"To eliminate one of the major sources of cost and schedule risk, we recommend that the Secretary of Defense delay the start of engineering and manufacturing development until critical technologies are matured to acceptable levels."

The GAO report has been submitted to the Hon Christopher Shays, a leading member of the House of Representatives, and Chairman of the Subcommittee on National Security,Veterans' Affairs, and International Relations Committee on Government Reform. The essence of the GAO's report is reproduced below.

The Joint Strike Fighter Program is the most expensive aircraft program in the Department of Defense (DOD). It is intended to produce affordable, next-generation aircraft to replace ageing aircraft in the military services' inventories. As currently planned, the program will cost about $200 billion to develop and procure about 3,000 aircraft and related support equipment.

Two contractor teams, led by the Boeing Company and by the Lockheed Martin Aeronautics Company, are competing for the engineering and manufacturing development phase. This phase of the program is projected to last about 8 years and cost about $20 billion and typically involves large, fixed investments in human capital, facilities, and materials.

Last year, we testified and reported that a key objective of the program's acquisition strategy is affordability and that a part of that strategy - entering into engineering and manufacturing development with low technical risk - would not be achieved because technologies critical to meeting the program's cost and requirement objectives were projected to be at low levels of technical maturity in April 2001, the date then scheduled for awarding the engineering and manufacturing development contract.

We stated that the program's approach was not consistent with best practices in which technologies are more fully developed before proceeding into product development. Organisations that use best practices recognise that delaying the resolution of technology problems until product development - analogous to the engineering and manufacturing development phase - can result in at least a ten-fold cost increase; delaying the resolution until after the start of production could increase costs by a hundred-fold.

Because of concerns about the adequacy of the Joint Strike Fighter's short take-off and vertical landing flight test program, the maturity of its critical technologies, and other factors, the Fiscal Year 2001 National Defense Authorisation Act directed that the contract for the aircraft's engineering and manufacturing development not be awarded until certain criteria were met.

For example, the act required that the program's short take-off and vertical landing demonstration aircraft accumulate at least 20 hours of flight test time before the program could enter the engineering and manufacturing development phase. The engineering and manufacturing development contract award is now scheduled for October 2001. At your request, we assessed whether the Joint Strike Fighter's critical technologies will have matured to a low technical risk at the start of the upcoming engineering and manufacturing development phase.

We assessed technical maturity based on technology readiness levels - measures pioneered by the National Aeronautics and Space Administration and adopted by the Air Force Research Laboratory to determine the readiness of technologies to be incorporated into a weapon or other type of system.

The research laboratory considers a technology to be low risk for the engineering and manufacturing development stage when a prototype of that technology has been developed that includes all of its critical components in approximately the same size, weight, and configuration of the end product and that prototype has been demonstrated to work in an environment similar to the planned operational system.

Results in Brief

Although the Joint Strike Fighter program has made good progress in some technology areas, the program is at risk of not meeting its affordability objective because critical technologies are not projected to be matured to levels that we believe would indicate a low risk program at the planned start of engineering and manufacturing development in October 2001.

In fact, the critical technologies are projected to be at the same level of maturity in October 2001 as they were projected to be in April 2001, the previous scheduled date for the engineering and manufacturing development contract. Several of these technologies under development are critical to making the long-term ownership costs of the program affordable.

These technologies remain at higher risk levels for engineering and manufacturing development because (1) they have not been developed to approximately the same size, weight, and configuration of the end product and/or (2) they have not been demonstrated to work in an environment similar to the planned operational system.

For example, the contractors have demonstrated certain key component technologies in flight environments, but the tested hardware was not always the same size and weight required for the Joint Strike Fighter aircraft. In other instances, components built to the required size and weight were only demonstrated in ground-tests.

Based on our work on best practices in product development, and the importance of the Joint Strike Fighter to DOD and the industrial base, we believe that DOD needs to ensure that the program's critical technologies are at demonstrated levels of maturity before making engineering and manufacturing investments in the program. Failure to do so could result in increases in both the production and long-term ownership costs, schedule delays, and compromised performance as problems arise in product development.

Moreover, the impact of failing to mature one critical technology could ripple throughout the program. Therefore, we are recommending that DOD ensure that critical technologies are mature before proceeding into engineering and manufacturing to improve the likelihood of meeting program expectations or to take additional actions if it chooses to accept the risk of immature technologies.

In commenting on our report, DOD said that it assessed the technology maturity of the Joint Strike Fighter to be sufficient for the next phase of the program. DOD also said that the Joint Strike Fighter Program Office has implemented a rigorous risk management program that will continue to monitor and address technology risks, as well as other risks, throughout the program's life. We disagree with DOD's assertion that technology is mature enough to move forward.

The technology readiness level assessment conducted as part of our review of the Joint Strike Fighter showed that critical technologies are not projected to be matured to levels that would stem risks at the start of engineering and manufacturing development. Our previous work has shown that when programs proceed in this fashion, they experience delays, rework, and substantial cost increases that could force the Department to divert much-needed funds from other important weapon system programs.

Program Background

The Joint Strike Fighter Program is structured to use a common production line to produce three versions of a single aircraft. These aircraft will be tailored to meet conventional flight requirements for the U.S. Air Force, short take-off and vertical landing characteristics for the U.S. Marine Corps, and carrier operation suitability needs for the U.S. Navy. The program will also provide aircraft to the British Royal Navy and Air Force. Table 1 shows the services' planned use for the Joint Strike Fighter.

Aquisition Strategy

A key objective of the Joint Strike Fighter acquisition strategy is affordability - reducing the development, production, and ownership costs of the program relative to prior fighter aircraft programs.

To achieve its affordability objective, the Joint Strike Fighter program has incorporated various acquisition initiatives into the program's acquisition strategy and various technological advances into the fighter. Among the acquisition initiatives planned was to develop critical technologies to a level where they represent low technical risk before the engineering and manufacturing contract is awarded.

The expectation was that incorporating these initiatives into the acquisition strategy would avoid cost growth, schedule slippage, and performance shortfalls that have been experienced in other weapon acquisition programs.

To date, the Joint Strike Fighter Program has awarded contracts totalling over $2 billion to Boeing and Lockheed Martin for the current concept demonstration phase. During this phase, DOD required each contractor to design and build two aircraft to demonstrate the following:

• Commonality/modularity to validate the contractors' ability to produce three aircraft versions on the same production line

• The aircraft's ability to do a short take-off and vertical landing, hover, and transition to forward flight

• Satisfactory low airspeed, carrier approach flying and handling qualities

Each contractor was required to submit a Preferred Weapon System Concept, which outlines its final design concept for developing a Joint Strike Fighter aircraft that is affordable and meets performance requirements.

The Preferred Weapon System Concept includes results from the flight and ground demonstrations and is being used by DOD to select the winning aircraft design and to award the engineering and manufacturing development contract. During engineering and manufacturing development, the Joint Strike Fighter will be fully developed, engineered, designed, fabricated, tested, and evaluated to demonstrate that the production aircraft will meet stated requirements.

Critical junctures in engineering and manufacturing development are the preliminary and critical design reviews and commitments; testing of aircraft; and commitments to production hardware, including the purchase of long lead production items. It is at the critical design review that decisions are made toward finalising the aircraft design and begin building test aircraft. About two-thirds of engineering and manufacturing development funding will be spent after this review. Figure 1 shows planned Joint Strike Fighter aircraft designs by contractor.

Technology Readiness Assessments

In our previous work on best business practices, commercial firms have told us that a key part of product development is getting the technology into the right size, weight, and configuration needed for the intended product - in this case, the final Joint Strike Fighter design. Once this has been demonstrated, the technology is at an acceptable level for product development.

Technology Readiness Levels (TRL) can be used to assess the maturity of technology and can reveal whether a gap exists between a technology's maturity and the maturity demanded for successful inclusion in the intended product. Defining this gap for the Joint Strike Fighter technologies is important for determining whether they can be expected to demonstrate required capabilities before being integrated into the aircraft design.

Readiness levels are measured along a scale of one to nine, starting with paper studies of the basic concept, proceeding with laboratory demonstrations, and ending with a technology that has proven itself on the intended product.

The Air Force Research Laboratory considers TRL 7 an acceptable risk for starting the engineering and manufacturing development phase. The readiness level definitions state that for a technology to be rated at TRL 7, it must be demonstrated using prototype hardware (such as a complete radar sub-system) that is the same size, weight, and configuration as that called for in the final aircraft design and that prototype has to be demonstrated to work in an environment similar to the planned operational system. We have previously reviewed the impact of incorporating technologies into new product and weapon system designs.

The results showed that programs met product objectives when the technologies were matured to higher levels and conversely showed that cost and schedule problems arose when programs started when technologies were at low readiness levels.

For example, the Joint Direct Attack Munition (JDAM) used modified variants of proven components for guidance and global positioning. It also used mature, existing components from other proven manufacturing processes for its own system for controlling tail fin movements.

The munition was touted for its performance in Kosovo and was purchased for less than half of its expected unit cost. However, the Comanche helicopter program began with critical technologies such as the engine, rotor, and integrated avionics at TRL levels of 5 or below. That program has seen 101% cost growth and 120% schedule slippage as a result of these low maturity levels and other factors.

In commenting on our report concerning better management of technology development, DOD agreed that TRLs are important and necessary in assisting decision makers in deciding on when and where to insert new technologies into weapons system programs and that it is desirable to mature technologies to TRL 7 prior to entering the engineering and manufacturing development phase of a weapon system program.

Since that time, DOD has adopted the technology readiness levels as a means of assessing the technological maturity of new major programs. In a July 5, 2001, memorandum, the Deputy Under Secretary of Defense (Science and Technology) stated that new DOD regulations require that the military services' science and technology executives conduct a technology readiness level assessment for critical technologies identified in major weapon systems programs prior to the start of engineering and manufacturing development and production.

The memorandum notes that technology readiness levels are the preferred approach for all new major programs unless the Deputy Under Secretary approves an equivalent assessment method.

Conclusions

Moving into engineering and manufacturing development creates an expectation that the Joint Strike Fighter can be delivered for a stated time and dollar investment and with a given set of capabilities. The decisions the Department of Defense makes now and over the next 2 years will largely determine whether those expectations can be met.

A key component of the Joint Strike Fighter Program's acquisition strategy is to enter the engineering and manufacturing development phase with low technical risk. The program will not have achieved that point by October 2001 because technologies, which the Joint Strike Fighter Program Office identified as critical to meeting the program's cost and requirements objectives, will not have been matured to an acceptable risk level.

By entering the engineering and manufacturing development phase with immature critical technologies, the program will need to continue to develop those technologies at the same time it will be concentrating on production issues and the integration of subsystems into a Joint Strike Fighter.

This approach would not be consistent with best practices. In fact, it would more closely follow DOD's traditional practices in weapon system programs that have often resulted in cost increases, schedule delays, and compromised performance.

Recommendations for Executive Action

To eliminate one of the major sources of cost and schedule risk, we recommend that the Secretary of Defense delay the start of engineering and manufacturing development until critical technologies are matured to acceptable levels.

Alternatively, if the Secretary of Defense decides to accept these risks and move the program into engineering and manufacturing development as scheduled, we recommend that the Secretary dedicate the resources to ensuring that maturity of the critical technologies is demonstrated by the critical design review or defer the inclusion of immature technologies from the approved design.

REF XQQFS

editor@defence-data.com

Copyright(c) 1996 - 2001 Defence Data Ltd. All rights reserved. Reproduction in whole or in part in any form or medium is prohibited except as provided in the Conditions of Use.The Defence Data logo is the trademark of Defence Data Ltd. Defence Systems Daily, and DSD, are the trademarks of Defence Data Ltd. By using this Web Site, you agree to all of the Conditions of Use.