Skip navigation | Accessibility and accesskey details | Sitemap

• home
• news & press
• about us
• publications
• good practice
• vacancies
• contacts
• search
• links
•  bigger text 
•  smaller text 
•  reset text 

National Audit Office Value for Money Report: Executive Summary

Ministry of Defence: The Defence Information Infrastructure

Summary

1. The Ministry of Defence (the Department) requires secure, high-quality information technology both for the success of operations and to conduct its day-to-day business. As in many other organisations, the Department developed numerous systems to meet specific needs over many years. In 2000, the Department began to develop plans to replace these diverse systems with a single information infrastructure to enable better communication, to promote more efficient ways of working, and to obtain better value for money. In March 2005, the Department let a contract with ATLAS, a consortium with EDS as the prime contractor, for the installation and management of a new infrastructure over 10 years. The Defence Information Infrastructure Programme (referred to in this report as DII) will, when delivered in full, incorporate 150,000 terminals for 300,000 users at over 2,000 defence sites, including on ships and deployed operations. The parts of the Programme which the Department has on contract, including payments to ATLAS, are estimated to cost £4.9 billion.
2. In addition to its scale, the DII Programme is highly complex. It must meet challenging security requirements and needs to function in operational theatres and on ships. As well as installing hardware and software at sites, the Programme requires a network of data centres to store Departmental information and two large call centres to provide service management to users of DII and many legacy systems. During implementation, DII has also undertaken to maintain the quality of service provided to users of legacy systems. The Programme is being implemented during a time of major change in the Department, including mergers of various Departmental organisations and the associated movement of personnel between sites. Figure 1 gives a summary of the DII Programme.

Preparation

3. Inevitably after many years of fragmented information technology, the DII Programme has become a key enabler of many other major business change application programmes. To keep risks manageable the Department decided in 2001 that the DII Programme would deliver only infrastructure and core software, while new applications would be delivered through separate programmes. Between 2005 and 2007, the business change programme which most urgently required assistance from DII and which had the biggest impact on how the Programme was structured was the Joint Personnel Administration Programme, which planned to automate and improve Armed Forces personnel pay and benefits processes and introduce self-service capabilities.
4. The Department did considerable work to understand the required scope of the DII Programme and to understand and mitigate inherent and anticipated risks. It sought to learn from other large computer programmes, including the United States Navy Marine Corps Intranet Programme, which has a similar purpose to DII and is also being delivered by EDS. In part on the basis of lessons learned on the United States programme, the DII Programme team decided to take control of existing systems before letting the contract, improving their knowledge of what was to be replaced and allowing them to decide the best order in which to install DII. It also reduced substantially the number of applications that would need to run on DII. However, in a situation where the Department had limited knowledge of the condition of its estate, the Programme did not do enough work to understand the physical environment into which the new infrastructure would be installed, and consequently made too optimistic an evaluation of the physical condition of many defence sites.
5. The Department devised a robust commercial structure for the DII contract. The structure includes contractor shadowing and an incremental approach to awarding work to ATLAS to manage the risks of the Programme and incentivise the contractor. Payment for performance means that, with limited exceptions, the Department only pays the contractor when terminals have been installed and are working. To better support system management during the implementation of DII, legacy systems are transferred to ATLAS with the contract increment in which they will be replaced. The Department also created sound governance and decisionmaking structures which have endured since the start of implementation.

Implementation

6. Following extended negotiations to ensure a better deal, the Department and ATLAS signed the DII contract three months later than intended. To meet the timetable for the Joint Personnel Administration Programme, the Department decided not to change the schedule. Though this delay meant the loss of a three-month startup phase, ATLAS believed that it would be able to find premises and staff and start to deliver in the compressed time period.
7. During 2005 and 2006, the Programme delivered a number of important enhancements to the Department’s information technology. This included the transfer of a number of legacy systems and the staff that managed them to ATLAS, and the establishment of two toplevel data centres to store Departmental information. Most significantly, users of legacy systems which were now being managed by ATLAS benefited from many improvements, which made their systems more reliable. From November 2005, when the Single Point of Contact call centre was opened, users of legacy systems have also been able to call a single telephone number to solve any problem they have with their computer system.
8. Throughout 2005 and early 2006, however, problems emerged with two key elements of the Programme: the rollout of new computer hardware to sites, and the creation of the software. These problems caused major delays to the rollout of the first stage of the DII Programme. The Department contracted to have 62,800 DII terminals in place at permanent defence sites by the end of July 2007. At the end of April 2008, only 29,000 had been delivered. There have been significant delays to the commencement of the installation of terminals on Royal Navy ships and submarines. Currently, the end date for the installation of Increment 1 of DII is running 18 months late against the estimated latest completion date at contract signature.
9. The DII Programme assumed that the rollout of infrastructure and terminals would be more straightforward than transpired and that it would proceed at an identical pace at all sites, irrespective of their size, complexity and the condition of the environment and supporting services. The Department accepted ATLAS’ proposal to use a Fixed Rollout Methodology, understanding that it would minimise the project management required, and thereby prove to be cost-effective and capable of meeting the Programme’s challenging deadlines. This methodology proved to be inappropriate and unresponsive in many cases to the circumstances of the Department. Resulting delays, in particular because of work to improve the condition of many defence sites and the level of organisational change, have meant that legacy systems have had to run for longer, in some cases requiring additional maintenance. The realisation of some benefits, particularly those associated with improved ways of working, will take place later than initially envisaged.
10. The DII Programme took some time to understand and address the root causes of these problems. During this period, however, the Programme sought to ensure that the rollout schedule adopted did not prevent the introduction of the Joint Personnel Administration application. The Programme did so through rescheduling the DII rollout and through expenditure of some £12 million to install additional terminals on legacy systems. This protected the Joint Personnel Administration Programme’s ability to realise gross benefits of £972 million from 2005-06 to 2014-15. The Department intends to re-use much of the additional hardware when it later replaces these systems with DII. The Joint Personnel Administration application is now running in all three Services – the Army, Royal Navy and the Royal Air Force – and is reporting that it is on track to achieve significant financial savings. The Department has monitored closely the impact of changes on the delivery of benefits and, to a significant extent, has protected benefits it enables in other programmes.
11. At the end of 2006, the Department and ATLAS agreed to replace the Fixed Rollout Methodology with a Decision Point Process, which is more responsive to the variable condition of defence sites and requires more active project management. From early 2007, the new methodology led to a considerable increase in the number of terminals delivered each month: on average 3,000 terminals were installed monthly in the last five months of 2007.
12. As well as hardware, DII requires core software, including tools to run, monitor and protect the system, and software to enable office automation, web-browsing and other standard activities needed by all or most users. The Department’s requirement for core software has remained largely unchanged since the contract was let, but the Programme has been unable to deliver this to the schedule anticipated at contract award. It took longer than anticipated for the Programme to translate that contracted requirement into the detailed requirements needed for design and development and it has been unable to deliver the software to the schedule anticipated at contract award. The original plan was for all of the core software to be delivered in two releases by June 2006. Following difficulties, Release 1 was split into a Restricted capability, most of which has been delivered, and a Secret capability, which has not been delivered yet. None of Release 2 has been delivered yet. The Programme’s inefficient processes for software design, issues with the designs themselves and changes in the Department’s detailed requirements for core software have been the main causes of delay.
13. The Department and ATLAS have taken considerable time to understand the underlying problems with the design of core software. After previous attempts to improve the situation had limited impact, the DII Programme believes that changes implemented in early 2008 will be effective. If additional core software is not delivered soon, the rollout of DII terminals to sites that already have access to electronic document and record management services through legacy systems will not be able to proceed as the DII Programme has undertaken to maintain, as a minimum, users’ existing levels of functionality. This includes the Department’s Main Building and the headquarters of the Royal Navy and the Royal Air Force.
14. As a result of problems with preparation at some sites and difficulties with the delivery of network infrastructure and core software, the rate of installation of terminals slowed considerably in early 2008, with only 6,700 out of 21,000 terminals delivered between January and April. Despite the challenges in early 2008, from April month on month roll-out performance has improved. Similar progress also needs to be made with the delivery of software and the Department has this work in hand. If there continue to be problems with the development of core software, they will have a significant impact on rollout schedules.
15. In addressing the specific problems that have affected the implementation of DII, the Department and ATLAS have exploited their partnering approach. Robust governance structures have been strengthened further and key personnel have remained in place for much longer than normal to see the Programme through difficulties. At the highest levels within the Department and ATLAS’ constituent companies, senior management have been well engaged in the DII Programme. The Department’s and EDS’ senior management have done much to instil a partnering ethos throughout their organisations and the relationship between the Department’s Senior Responsible Owner and the ATLAS Senior Responsible Industry Executive is a strength of the Programme.

The Deployed IT system

16. In September 2007, the Department awarded a further increment of the DII Programme to ATLAS, to deliver a computer system that can be used by the Armed Forces to handle Secret material when on operations. By the end of 2010, the system will comprise some 1,500 deployable terminals, supported by a similar number operating in the United Kingdom. It is planned to cost £385 million between 2008 and 2015, and is currently running to schedule, with the first unit due to receive equipment at the beginning of 2009. Part of the original scope of Increment 1, to put DII terminals on the Department’s ships and submarines, is now being managed in one project with the deployed terminals.
17. Before giving ATLAS responsibility for developing the deployed system, the Department and the consortium conducted substantial work to understand and mitigate risks. The Department is now carrying out similar work, using a technical demonstrator, to understand better the nature of its requirement for a Top Secret system. At short notice, ATLAS also helped the Department by developing and installing two interim computer systems, known as OVERTASK, for command and control, and J1J4 Interim Operating System, an administrative and logistics system, to assist in operations in Afghanistan. These have been a success and are already delivering operational benefit to the front line.

Service Management

18. The number of users of the new system has increased rapidly to 82,000 since the middle of 2007, requiring the DII Programme to focus increasingly on service delivery. The Key Performance Indicators through which the performance of the DII system is measured are of a high quality. Crucially, the Department has not tried to be exhaustive, but has focused on a manageable number of indicators, which will give a good overall picture of how ATLAS is performing.
19. To date the system has generally been available when it should be and a recent customer satisfaction survey was encouraging. Where ATLAS measures performance indicators, its performance has been good, although since December 2007 it has found it more difficult to meet some targets as more users gain access to the DII system. Since March 2007, the Department has started to adjust payments to the contractor to reflect under-performance against those Key Performance Indicators currently being measured. These indicators are mostly being met, but the Department has decided not to abate payments made to ATLAS fully for under-performance. ATLAS is not yet measuring the full range of Key Performance Indicators as stated in the contract. Users who require changes to be made to their system or who have complex problems have a more negative experience of the quality of ATLAS’ service, but the Programme continues to take action to address these issues.

Costs

20. The Department estimates that it will cost £4.9 billion to deliver those increments of the DII Programme which are currently on contract. This estimate includes payments to ATLAS, retained costs and contingency for the mitigation of risks, totalling £4.5 billion. The estimate also includes some £300 million for Departmental staff and programme management costs, and around £100 million for some legacy system costs. The total cost to the Department to deliver this work, including programmes on which DII depends which cost £1.2 billion, is £6.1 billion. The direct forecast costs of the Programme have increased by £182 million, some three per cent, since the Department let the contract for Increment 1 in March 2005. Cost changes of a further £179 million have occurred to the programmes on which DII is dependent, but these changes are not due to DII and most have not increased the overall cost to the Department. Payments of £959 million have been made to ATLAS up to 31 March 2008 for the implementation of DII, acquisition of assets and management of legacy systems.
21. Following necessary work to clarify better its requirements for deployed and Top Secret systems and the installation of DII terminals at defence sites not yet on contract, the Department currently estimates that it will have to spend £984 million to deliver the remaining parts of the Programme that are not yet on contract. So, if all planned increments of DII were to go on contract, the current estimated cost of realising DII, including the cost of related programmes, will be some £7.1 billion. This estimate also includes the cost of additional capabilities for the deployed environment which were not in the original scope of DII. In its recent planning round, the Department allocated funding and set efficiency targets, which together will enable it to fund around 140,000 of the planned 150,000 terminals. However, it is yet to place on contract work to complete the rollout of 30,000 to 40,000 terminals to permanent defence sites. The Department is exploring further changes to its approach, which would allow it to close the remaining gap in the number of terminals it can afford.

Overall Value for Money

22. As with other major IT programmes, DII is intrinsically complex and challenging. In this case, the Programme’s size and demanding requirements for security and deployment to theatres of military operations are particularly exacting. It is also challenging to manage in terms of the complex interconnection with other business change programmes and the level of churn in the Department’s business, and has had to be introduced into a diverse, and in places poor quality, estate.
23. The Department had a sound rationale and convincing business case for the Programme in terms of the improved military operational effectiveness, and more effective and efficient running of the business, particularly through the business change programmes DII supports. This continues to be true. The Department calculated that to provide the same improvements without placing a service contract with the private sector would have cost more. The Programme reports that it has already achieved or enabled benefits to date of £916 million, including £640 million of costs it has avoided by placing the contract.
24. The Programme’s implementation difficulties have led to key elements of it running 18 months late. This delay has, in turn, led to postponement of the achievement of some benefits, particularly the savings associated with switching off legacy systems and the longer term benefits from improved ways of working. The Department has, however, to a large extent, protected its financial position. The direct forecast costs of the Programme have increased by £182 million, some three per cent. It has adhered to the principle of payment on delivery, when delays have been due to the contractor. It has taken concerted action to protect benefits, particularly the enabled benefits of other programmes, and to get the Programme back on track. Nonetheless, key elements are running late and the delays have led to continuing expenditure on less capable legacy systems, albeit largely offset by paying for fewer DII terminals to date. There have been efforts to remedy the problems identified, but without improvement in the rate of rollout of terminals and the completion of software development to meet the latest timelines, significant risks remain to the timely delivery of the Programme.

Key Recommendations

Our key conclusions and recommendations are below. More detailed conclusions and recommendations can be found at Appendix 1.

Preparation

1. The Department and ATLAS gained a good understanding of the legacy systems that DII would replace, but did not do enough to understand the physical condition of the environment into which the DII system would be installed. This lack of emphasis along with an inappropriate rollout methodology, was a major cause of delay. In planning any major business change programme, the Department should pay greater attention to any land and buildings aspects. It should secure the necessary support from Defence Estates and a good knowledge of the contractual obligations of third-party contractors, local councils’ planning departments and heritage bodies.
2. The decision that the contractor would receive the majority of the payment for their work only when terminals had been installed was sensible and has protected the Department from paying for services before they have been delivered, when delays have occurred because of contractor error. When delays in installing terminals occurred in the first year, the Department rescheduled payments of some £11 million for other deliverables. A larger proportion of payments is being made against work delivered in Increment 2b also, to reflect the greater capital outlay required to build a deployable version of DII. For the increments not yet on contract, there will be less time over which the consortium can be paid for delivering the service. In negotiating future increments, the Department should, where appropriate, adhere to its existing principle of paying for the DII service only when terminals have been installed and are in use.
3. The Programme did not conduct a formal pilot because it believed that the implementation of DII at permanent defence sites would be relatively straightforward. The Programme’s use of a demonstrator to reduce risks on the deployable part of the DII system has been effective. The Department should run risk mitigation and piloting phases for the remaining increments similar to that on the deployed element in advance of each increment that is still to be let, irrespective of how straightforward the requirement seems initially.

Implementation

4. There have been persistent weaknesses in the design and accreditation of core software functionality, which contributed to earlier delays to the DII Programme and poses the risk of delaying further the remaining implementation. There is a risk that the Programme’s recently implemented recommendations of its review of the delivery of software may not be enough to address performance problems. If large elements of Release 1 and Release 2a software remain undelivered, the Department should oblige ATLAS to bring in additional expertise to deliver the software solution.
5. As the rollout moves to sites with more capable legacy systems, it may become difficult to transfer users to DII, given the Department’s understandable decision not to move such users until DII is at least as capable as their legacy systems. The Department has only been prepared to accept claims from ATLAS for lost revenue caused by problems with the rollout methodology and the physical estate where it was culpable. If the rollout of hardware should be stalled in future because core software is not available, the Department should maintain the same principle in settling claims from ATLAS.
6. During the first two years of the Programme, payments for legacy systems did not reduce as quickly as the Department had hoped, even allowing for the slower rate at which DII was being installed, since rollout schedules have not been designed to optimise legacy closure. The Programme is now seeking to terminate payments for legacy systems more quickly through improved processes and spend-to-save measures. The DII Programme should design any subsequent rollout schedule to achieve closure of legacy systems commensurate with the rollout of DII terminals but without compromising the delivery of other benefits.

Service Management

7. The Key Performance Indicators through which the quality of the DII system is to be measured are of good quality, though ATLAS cannot yet measure all of them. The Department should exercise its right under the contract to abate payments to ATLAS for nonmeasurement of Key Performance Indicators and introduce revised measures where a robust assessment can be made. The Department should also fully abate payments to ATLAS for poor performance against those Key Performance Indicators that are being measured. It should also maintain an accurate list of abatements it has foregone when indicators are not measured and use this information in future commercial negotiations.

Costs

8. The Department’s Programme team has kept detailed records of additional costs it has incurred as a result of delays caused by ATLAS, but not for all costs incurred by other Departmental programmes as a result of delays to DII. The Department should maintain and use detailed records to achieve the best possible settlement in outstanding commercial negotiations including, where appropriate, additional costs caused to other programmes.