Let the news come to you. Get ESN updates sent direct to your inbox.

Join our ESN Community

NAO flags up high risks of ESN, but sees potential benefits

The National Audit Office has delivered its first report on the UK’s Emergency Service Network. Its assessment so far is that ESN is the right direction strategically, but the Home Office has downplayed both the technical and timescale risks, reports James Atkinson

NAO flags up high risks of ESN, but sees potential benefits

The UK is in the early stages of delivering a new communications network for use by the emergency services and other public sector organisations. It will replace the current dedicated Airwave narrowband TETRA network with a broadband 4G LTE network shared with 20 million-plus consumer subscribers.

The Emergency Services Network (ESN) is designed to match Airwave in all respects, as well as: make high-speed data more readily available to the emergency services to improve their performance; provide more flexibility to take advantage of new technologies as they emerge; and, cost less than Airwave.

The three main contracts were awarded by December 2015 with KBR responsible for project management and delivery; Motorola Solutions for user services and applications; and EE for the main 4G network. The emergency services are due to transition to ESN between September 2017 and December 2019 when the Airwave network will be switched off.

ESN - Key Facts
• 70%: percentage of GB landmass, as measured for ESN purposes covered by EE’s 4G Network, July 2016: needs to be increased to 97% to match Airwave’s coverage
• £3.6bn: estimated value of the quantified benefits over 17 years resulting from switching to ESN
• £1.2bn: estimated cost of ESN, April 2015 to March 2020. After March 2020 ESN is expected to save money compared with Airwave
• £500: estimated annual saving per device used by emergency services once transition to ESN is complete; the Government expects to spend £377 million on Airwave in 2016-17 – equivalent to £1,300 per device.

The National Audit Office has just published its first assessment of ESN (15 September 2016) entitled: Upgrading emergency service communications: the Emergency Services Network.

The scope of the report examines ‘the significant upcoming challenges that the programme will need to manage if it is to be successful, how it is managing them and why it has chosen this approach’.

The NAO notes that as the report looks at a live programme early in its delivery phase, it is too soon to assess whether the programme has achieved value for money. The report only looks at the services provided by Airwave that are relevant to the future programme.

Key findings
The NAO’s main assessment is that: ESN is inherently high risk as such an approach has not yet been used, nationwide, anywhere in the world – only South Korea is adopting a similar path and it starts with a more extensive 4G base and is providing dedicated spectrum for exclusive use by its emergency services.

The report identifies three main categories of risk: technical; user take-up; and commercial arrangements. These roughly align to the three main phases of the programme: design, build and test; transition; and, operation.

It also makes the point that there is also ‘an overarching risk due to the ambitious nature of the timeline adopted’.

Technical challenges
The report says that ESN is ‘technically cutting edge’ and that there are ‘some significant technical challenges’ to delivering ESN, the foremost of which are:

• Increasing the percentage of GB’s landmass covered by EE’s network from 70% (as of July 2016) to 97%. The current projection is that sufficient coverage will be available by September 2017.
• Developing handheld and vehicle mounted devices that will work with ESN as no suitable devices currently exist.
• Developing new push-to-talk software to enable ‘radio-like’ communications between emergency services personnel and control rooms.
• Implementing the software and protocols that are needed to give emergency services personnel priority over commercial users of EE’s network.

The NAO observes: ‘Delivery of the programme against these technical challenges is by no means certain and, while total failure seems unlikely, there remains a risk that the programme will not be able to overcome these challenges for the cost or timetable proposal in the full business case, or to the satisfaction of users.’

User take-up
Success depends on the emergency services choosing to take up ESN and making full use of it. Users have been promised they can stay on Airwave until ESN is ‘at least as good as Airwave’. The NAO points out that ‘defining this is complex and leaves room for disagreement’ - particularly over coverage.

The NAO reports that in its talks with end users, ‘there were indication in these discussions that users may be very demanding of ESN; expecting an almost exact match of where is covered compared with Airwave’. However, ESN contracts do not specify which areas of the country are to be covered.

End users also voiced concern that ESN will not replicate all of Airwave’s functionality. The NAO notes: ‘It is unclear, for example, whether the current specifications for ESN will meet the security needs of counter terrorism and covert operatives.’ Device-to-device direct mode voice calling was also a concern.

The NAO warns that if even a small number of emergency services choose to delay transition then this will ‘reduce benefits compared to the full business case’. It also notes that: ‘The full benefits of ESN rely on the emergency services exploiting high-speed data services by changing their operational behaviour, but supporting this is not part of the programme’s scope and the government is not yet clear on what support it may need to put in place.’

Commercial arrangements
The commercial arrangements for ESN have separated the operational responsibilities of the emergency services from the commercial levers, which are held by the programme and therefore the Home Office. With ESN the majority of the cost is paid for centrally.

The emergency services do not have their own contractual arrangement for the full scope of ESN. Instead, they will have a call-off arrangement with one of the ESN suppliers, EE, but the terms of this are more limited than their current contract with Airwave. The NAO says: ‘Their contract with EE will give them very limited direct recourse for poor service.’

End users have no direct contract with Motorola for the user services or with the provider of the control room interface into ESN, despite the importance of these suppliers in the end-to-end service.

The NAO argues that these commercial arrangements create a risk that the emergency services feel they do not have sufficient control over the service they receive and may continue to make use of supplementary services, as they currently do with Airwave, leading to a reduction in the benefits of ESN.

Despite the inherently high level of risk, the programme has adopted a timeline for delivering ESN that is very ambitious, according to the NAO. From its interviews, the NAO concludes that both programme staff and emergency services all saw delivering ESN in line with the timeline of the full business case as ‘very difficult’. In notes that the timeline contains no contingency during the design, build and test phase.

A notable difference between the programme and end users is that programme officials consider that there is contingency to compress the planned timeline during the transition period. Emergency services personnel do not agree, arguing that the September 2017 to December 2019 period ‘already gave them limited opportunity to plan or learn lessons from each other’.

The programme’s approach to managing these risks
A 12-month delay to ESN could cost up to £475 million (the cost of keeping Airwave running) so the programme has put in place commercial and funding mechanisms that are designed to manage this risk.

Technical design risk
What this means is that ESN’s commercial arrangement pass many elements of the technical risk to suppliers because, in the opinion of programme officials, they are best placed to manage these risks.

The NAO’s assessment of this is that: ‘While this is true if the risk materialises on a small scale, we consider that these arrangements could be detrimental to the overall commercial relationship between the programme and its suppliers if there are high cost increases or long delays.’

It added that: ‘A large number of programmes, including the Home Office’s own e-borders programme, have demonstrated that suppliers are often not well placed to bear technical design risk in complex ICT programmes.’

Volume risk
An interesting revelation is that the programme tried to get suppliers to price in at least some of the risk for getting end users onto ESN. Suppliers were not willing to do this, so the responsibility for volume risk of getting users onto ESN is largely borne by the Government.

Commercial exploitation risk
Motorola is developing new software for ESN, while EE has the option of using sites built in remote areas under the separate Extended Areas Services contract. In pricing ESN, both suppliers will have made assumptions about how much they can earn from these opportunities. The taxpayer gets nothing from these, so the exploitation risk lies with the suppliers.

The NAO observes that: ‘This risk allocation means that there is a higher chance of tension in the commercial relationship between the programme and suppliers as there may be significant financial gains or losses by suppliers compared to what was expected.’ Some, but not all, financial gains by suppliers will be shared with the programme.

Delivering to time
Between August and December 2015, the programme slipped by two months due to delays in the procurement process. Since contract award, detailed designs were finalised three months late.

Elements of the functionality contracted to be delivered by August 2016 will now be delivered in a phased manner, with some coming as much as eight months later than planned. Overall the programme is therefore between five and 10 months behind the targets it set itself in the full business case.

Good programme culture
However, on the plus side, the NAO says that, in general, the programme has a positive delivery-focused culture that has helped it retain staff and manage issues as they have emerged. In contrast to other programmes recently observed by the NAO, ESN has benefitted from stability in staffing at both senior and junior levels – these staff also have a strong record of delivering other projects.

Programme management
Despite this, the NAO argues that the programme’s management of its key risks needs to improve if it is to deliver ESN successfully. It identified four main areas of concern:

• The programme’s approach to technical assurance and testing needs to be better – the board lacks independent telecommunications expertise.
• User engagement could be better, particularly with police and non-emergency services users of Airwave - some emergency services representatives told the NAO that programme officials ‘do not always listen to challenges that they raise’.
• The circumstances in which the Airwave contract will be extended need to be more clearly set out: i.e. how will this contingency be invoked and for how long?
• The service management arrangements once ESN is operational need to be more clearly articulated – it is unclear who in the Home Office will be responsible for ensuring ESN delivers its predicted benefits once it is operational.

In two further observations the NAO noted that: ‘The programme is behind schedule compared to the full business case and has responded by squeezing the time available rather than extending the overall time frame.’ And: ‘Overall, the programme, the Home Office and other sponsor bodies appear to be underrating the seriousness of the risks ESN poses.’

The NAO states: ‘Programme officials recognise many of the technical challenges the programme faces. However, in interview, programme officials and key stakeholders consistently appeared to view these challenges as easier to overcome than we would….On the other hand, the emergency services consistently identified technology, and in particular, the coverage and resilience of ESN, as their main concern.’

In summing up the risk factors, the NAO says: ‘The programme considers it has a strong approach to mitigating risk. Our view is that despite the programme’s mitigations, ESN remains an inherently high-risk programme that will require the highest levels of senior oversight throughout its lifetime.’

Why the programme chose to adopt these risks
The NAO agreed with the programme that: ‘ESN is the right direction strategically and the programme’s planned approach to delivery, if successful, will maximise benefits.’

But it went on to say: ‘However, we consider that in seeking to maximise benefits the programme’s planned approach to delivery has also maximised risk.’

In other words, the Home Office’s decision to adopt a big bang approach by moving to a fully combined data and mission critical voice network at the same time, before mission critical 4G standards are in place, rather than taking a slower or more phased approach (e.g. data first, then voice) as other countries are doing, has maximised the risks involved.

The NAO concedes that ‘Airwave does look expensive for the taxpayer’ because it is a dedicated service built from scratch and uses the private finance initiative model, which means the cost of designing, building, maintaining and operating Airwave is spread over the 20-year contract.

Access to broadband data is seen as a key benefit of ESN, but the NAO says: ‘We are unconvinced that the programme needed to adopt ESN to get the data capabilities the emergency services need’, noting that on its field visits the emergency services already access mobile data using existing 4G contracts.

In its observation on value for money, the NAO delivered a warning shot across the bows, observing: ‘The benefits of ESN should be substantial but we consider that the business case may be overly optimistic in its valuation of these.’

It noted that the programme has estimated that the benefits of ESN will be worth £3.6 billion between April 2015 and March 2032. It went on to say: ‘Valuing benefits is always difficult but we consider that a number of the assumptions that the programme made in valuing these benefits may be optimistic.’

One such assumption being that Airwave will continue to cost the same in the future as it does now, when in the NAO’s view ‘some discount should have been assumed’.

Another is that productivity and service improvement benefits from using EE’s 4G network are deemed to be worth £841 million. Of this calculation, the NAO says: ‘The way the productivity benefits have been valued, including the very limited nature of consultation with emergency services about them, mean that we cannot be assured that £841 million is the correct valuation.’

NAO recommendations for the programme
• The programme should improve the independence of the technical assurance arrangements it has in place.
• The programme needs to urgently develop a detailed contingency plan
• The programme needs to improve communications with the emergency services and other users of Airwave
• The programme needs to work with the Home Office, other sponsors and users to develop the service management arrangements for when ESN is fully operational.

Responses to the report
In response to the report the Home Office asked the NAO to include the following: ‘The Home Office has asked us to record that they have adopted their approach to equip the emergency services with the modern data communications capabilities they need and so welcomes the report’s key findings that ESN is the right direction strategically. The Department has also accepted the key recommendations.

‘However, the Home Office does not agree with the NAO’s judgement about the Department’s acknowledgement of the programme’s risk, on incentives on users to transition, or the scale of the benefits in the business case, considering the programme and commercial approach are designed to maximise value for money and comply with procurement law.’

An EE spokesperson said: ‘As the report suggests, Emergency Services Network is a state of the art technology programme that will allow Britain’s police, fire and ambulance services to benefit from a world leading communications network.

‘We’re proud to be a part of this programme, and we’re confident in delivering our commitments so that the lives of Britain’s Emergency Services workers will be improved, not put at risk.

‘Failing to replace the current, outdated systems will prevent Britain’s Emergency Services from becoming safer, more efficient, and more effective, and risks leaving them behind as technology advances around them.’

There is no doubt that ESN is a hugely ambitious programme and the NAO is right to highlight the risks involved, but perhaps the UK should be applauded for pioneering this approach to next generation mission critical communications. Time will tell, but it is not surprising that end users are cautious at this stage of the project.

They are comfortable with what they have, although they are clearly aware of its limitations. But as yet, they haven’t seen the new ESN devices or been able to really engage in the opportunities a 4G network can give them compared with what they are used to.

Many recognise that a mission critical broadband communications network could be a game changer. The ‘big bang’ approach and compressed timeline adopted by the Home Office may well be risky, but as some end users have remarked to Wireless this year, if all ESN delivers in the end is a like for like equivalent of Airwave then a huge opportunity will have been lost.

The real benefits are not in having faster download speeds, but in the opportunities a 4G system can deliver for operational changes among the emergency services. A live video link from ambulance crew at the scene of an accident back to consultants in a hospital has the potential to deliver on-site interventions that could save lives, for example.

Advocacy is hardly the NAO’s job, and keeping ESN on track is not going to be easy, but now the UK is committed to this path, let’s hope it all falls into place and delivers the potential it is capable of.

The full NAO report can be found here

Leave a Comment