What is the problem
The $1bn failure of Sea 1411 Phase 1—the Super SeaSprite helicopter—was a low point in Australian Government procurement. It seems incomprehensible that Defence could ever replicate it. Or does it?
The Defence Strategic Update 2020, in detailing a rapid deterioration in regional security, highlights that Australia cannot afford to spend time or money on projects that do not deliver the required war-fighting capability in an efficient and effective manner. The scope and complexity of the capabilities required for Australia’s security are expanding, while the timeframe for procurement is decreasing. Supply chain issues are leading to new pressures to design and manufacture more in Australia. In this environment, it is imperative for Defence to be an effective “smart buyer” as envisaged by the First Principles Review (FPR). For this to occur, it is critical that decision makers can, as stated by the FPR, assess “whether risks and interdependencies have been identified and managed”.
Background
T&E is a key systems engineering tool to identify risk across the capability life cycle (as per Figure 1). Defence has long had dedicated policies[1] outlining why T&E is important and detailing how it should be used in acquisition, sustainment and force generation.
Following multiple reviews[3], there have been periodic “new” pathways to establish (or recover) and sustain an effective T&E capability. As the various Defence Procurement and Capability Manuals have been updated, a consistent theme has been the vital role of T&E to inform risk-based decisions.
The First Principles Review, in recommending a “Smart buyer” approach, makes the assumption that a T&E capability exists such that Defence can assess “whether risks and interdependencies have been identified and managed”[4]. T&E has more recently also been recognised as one of the ten initial sovereign defence industry capability priorities.
Given this consistent emphasis on T&E, it should surprise the Australian taxpayer that in almost every review into Defence procurement there is a negative assessment of how Defence deals with T&E. Consistent themes include: difficulties in defining, creating and sustaining an experienced workforce; the lag and surge of experience on a project-by-project basis which makes it difficult to apply effective T&E early in the capability life cycle; a lack of coordination between the various entities that are stakeholders in Defence T&E (including industry); a lack of coordinated investment in T&E infrastructure; and, a lack of accountability to ensure that projects engage T&E and take meaningful account of any subsequent reports. While there are many competent, diligent individuals working to assess risk in some domains, the organisational ability to ensure that there are competent staff across all domains who are consistently tasked and their assessments heeded has been found lacking.
The First Principles Review highlighted the need for:
- Strengthening and placing at arm’s-length a continuous contestability function that operates throughout the capability development life cycle from concept to disposal; and,
- Transferring accountability for requirements setting and management to the Vice Chief of the Defence Force and the Service Chiefs within a regime of strong, arm’s-length contestability.
Some key principles arise from this:
- Independence. The risk identification function must be independent so that assessment is made without bias or influence (intended or unintended). Independence also ensures that the assessor of risk has a voice (NB not a veto) which is heard at each decision-making level of the capability life cycle.
- Task specific competence. The FPR recognised that Defence must ensure that committed people with the right skills are in appropriate jobs. Competence is a matrix of qualifications and experience which are relevant to the task at hand. An orthopaedic surgeon, for example, is a highly competent medical professional but they would never be tasked to conduct neuro-surgery. Likewise, many individuals in Defence have credibility due to their competent performance as an operational commander or maintenance engineer. That does not mean, however, that they are competent to assess technical performance, integration and certification risk.
- Transparency. Previous inquiries highlight that risk assessors working within Defence face various barriers (individual or organisational) that influence whether their voice is actually heard. Risk must be assessed and the results considered by decision makers. Given the costs and national security implications, the taxpayer deserves to know that this is occurring, despite the commercial and security considerations of full transparency. There is already a good model for this, as the public similarly deserve to know that the National Intelligence Community (NIC), operating under appropriate layers of secrecy, are acting legally. The Inspector General of Intelligence and Security (IGIS) conducts regular audits of the NIC as well as specific investigations and reports to the relevant minister and the Parliamentary Joint Committee on Intelligence and Security (PJCIS).
- Consistency. Measures to identify and manage risks and interdependencies must be professional and appropriate to need across the capability life-cycle. This is also true for “off the shelf” [5] products that may be used by an ally, but due to Australian environment, inter-dependencies or role requirements, may not be capable to perform all the desired functions in ADF service. The capability manager, Government and taxpayer deserve to know that what is being bought may require additional funding or a supplementary capability may be required for some tasks. This knowledge is important for operational planning, forecasting funding, and even reputation management.
The US Congress, facing a similar dilemma chose to legislate for independent T&E that provides mandatory independent annual reporting of all T&E measures to Congress on all major Defence acquisitions. The UK engaged a UK company, Qinetiq, on a 25-year Long-Term Partnering Agreement (LPTA) to provide a range of technical support to the Ministry of Defence (MoD), including T&E. This model provides independence, and highlights that industry can play a leadership role in providing training, establishing requirements for, and managing, test ranges and infrastructure, as well as the conduct of T&E.
The path to assured risk management
The Defence Strategic Review should be the trigger for a different approach that is based on the principles outlined above, and includes:
- A Defence Capability Assurance Agency (DCAA). An independent statutory body that is responsible for assessing risks[6] associated with materiel procurement and sustainment. The governance of the DCAA could be analogous to the Civil Aviation Safety Authority (established under the Civil Aviation Act 1988), led by a Director appointed by a DCAA Board. The DCAA workforce would be qualified and experienced T&E practitioners drawn from both Defence and industry. The DCAA should be funded[7] through a Department (probably Defence or PM&C), inform Defence of outcomes, but report to the Defence Minister and the Parliament.
- Assurance. An audited and enforceable requirement that the DCAA is engaged to evaluate risk at agreed points across the capability life-cycle. The DCAA reports are specifically to be included in the briefs provided to project managers, assurance bodies, Defence IC and NSC. A DCAA representative should be a member of these decision-making bodies for agenda items considering defence acquisition or sustainment decisions. This does not constitute the DCAA being given a veto. It is ensuring that the DCAA is given a voice so that risk-based decisions have a credibly informed basis.
- A leadership role for industry. The DCAA should have a long-term agreement with an Australian industry partner to provide depth of domain expertise and a consistent comprehensive approach to T&E through the entire capability life-cycle across multiple platforms, environments and systems. The industry partner would not necessarily conduct all T&E, but as a minimum would be responsible for regulating the qualifications and professional standards of the workforce that are tasked to conduct T&E.Defence already has a good model for this approach – the technical airworthiness system. The Defence Aviation Safety Authority (DASA) as a regulator provides assurance (through assessment of candidate suitability and ongoing audit) that anyone working in this safety and mission critical area have appropriate qualifications and experience for the particular task they undertake. DASA also provides a range of specialised support services where deep domain expertise is required (eg: aircraft structural integrity).
The industry partner should provide a regulatory function analogous to DASA, in essence, acting as the DCAA Regulator (DCAAR). Similar to DASA, the industry partner would be required to sustain an expert workforce in a range of T&E disciplines, providing the capacity to surge, mentor, support and develop T&E practice in support of Defence capability. The industry partner should also be responsible for the quality and operation of T&E infrastructure and efficacy of training[8].
Scale and flexibility would be achieved by the DCAA being able to draw on any defence personnel or industry provider who meets the qualifications and experience requirements of the DCAAR. This T&E domain expertise would be complemented by the recent operational experience of Defence force operators/engineers who have graduated from the relevant T&E training being embedded within DCAA.
- Transparency . Defence should be audited to ensure appropriate engagement of the DCAA and transparency of subsequent reporting of identified risks. This should be conducted by a small team who hold suitable security clearances and are subject to protections for commercial-in-confidence information, working as part of an independent assurance office. This is analogous to the role of the Inspector General Intelligence and Security (IGIS) across the National Intelligence Community (NIC).
Conclusion
There are individuals and units within defence that are professional and competent technical risk assessors. The organisational inability of Defence, however, to generate, sustain, consistently task, and transparently respond to a credible assessment of risk in procurement and sustainment is well known, and has been the subject of too many reviews. The strategic circumstances of this decade – and beyond – require timely, effective and sustainable change.
The Defence Strategic Review (DSR) should not seek to initiate yet another review of T&E within Defence. The strategic imperative for Defence to be a smart buyer now, should compel the Government to instigate legislative reform to establish a Defence Capability Assurance Agency operating on the basis of principles outlined in this paper.
[1] These existed in single-service instructions eg: ABR 6205, Naval Test, Evaluation and Acceptance Manual or DI(AF)LOG 2-7 Test and Evaluation of Technical Equipment prior to reforms that led to whole of Department documents such as: DI(G)LOG 08-10, DI(G)OPS 43-1, Defence 2007 T&E Policy, Draft Defence Instruction DI(G) ADMIN 05-1 Defence Capability Systems Life Cycle Management 2001, Part Three of the DCDM and various interim policies and instructions internal to DAO/DMO/CASG.
[2] MoD Head of Weapons, Evaluation & Capability Assurance, ITEA Presentation
[3] Eg: Kinnaird 2003, Mortimer 2008, ANAO 2002, ANAO 2015, Senate 2003, Senate 2012
[4] https://www.defence.gov.au/Publications/Reviews/FirstPrinciples/Docs/FirstPrinciplesReviewB.pdf, pg 28
[5] ANAO Test and Evaluation of Major Defence Equipment Acquisitions, Nov 2015
[6] Risk can include: technical risk (eg: compliance with a requirement or standard), systems integration risk, force-integration risk, contractual risk (eg: what form of contract appropriately shares risk between the Commonwealth and tenderer for the provision of the particular capability/product required), process risk (eg: where DIC or central agency concern regarding IIP phasing may undermine good engineering practice), or even reputational risk whereby a product performs as designed but not as expected by the media, public or the parliament.
[7] Offsets are available. Defence already funds a disparate range of people, organisations and infrastructure conducting T&E that could be made more efficient with cohesive leadership. Individual CASG or individual service projects are also expected to provide funding for T&E, noting that previous Defence policy has highlighted that up to 25% of materiel funding should be allocated to T&E for developmental or highly complex projects. The opportunity cost of failed and “remediation” of failing projects should also be considered when debates around funding arise (eg: the wasted A$1bn on Super Sea Sprite Troubled $1bn helicopter project axed (afr.com)).
[8] Subject to an assessment of probity measures, the industry partner may be limited to selecting, auditing and managing contracted training providers or may also provide a range of in-house T&E training for defence and industry stakeholders.