Crisis Exercises: Are You Wasting Your Time and Money? A Blueprint for Better Exercises

All incident and crisis management doctrine includes a preparedness cycle (figure 1) emphasising the principles of continuous improvement and learning from lessons. Exercising plays a critical role in the cycle.

Whilst in reality exercises are used to achieve a range of outcomes, in theory at least the core purpose is to “rehearse and evaluate arrangements, give staff practice in the roles they have been assigned and trained for, and confirm that arrangements actually work in practice” (BSI, 2014 p27).

One of the most important functions of exercises is to facilitate the learning and embedding of lessons identified in previous exercises or incidents and identify new lessons that can be addressed before a future incident occurs.

Exercises can take various forms, from very simple to extremely complex. As can be seen from Figure 2, there is a broad but not entirely consistent agreement between guidance bodies on the terminology (and underlying definitions) used to characterise exercise types.

Figure 2 – Various taxonomy of exercise types

Whilst there are many sources of guidance on exercising, there is little quality theoretical or empirical evidence on what makes an effective exercise. An analysis of incidents and resultant lessons clearly demonstrates recurrent and persistent weaknesses, despite exercises.

Example: UK Government Response to the Pandemic

The response to the pandemic is an example of many positives – leadership, innovation under pressure and global collaboration, but it also highlights failures not only in exercising but in the risk-specific approach to preparedness. To use the UK as an example, The UK Government “Coronavirus: lessons learned to date” report, published in October 2021 found: 

The report acknowledged that the exercises were based on contingency plans derived from the National Risk Register and that the shortcomings of the exercises were caused in part by a focus on risk specificities (the characteristics of the disease), limited consequence analysis (cascading and longer-term consequences) capability and capacity assumptions, complacency and “austerity” which informed the Government’s pandemic preparedness.

The report recommended that:

Example: Multi-agency lessons identified in the UK since 2005

The UK’s Emergency Planning College, part of the Cabinet Office, published a summary of multi-agency lessons learned from 24 exercises and incidents that occurred in the UK since 2005.

It identifies a number of common and persistent trends. Perhaps unsurprisingly to experienced practitioners, communication and situational awareness were the most common themes, both recurring repeatedly over 13+ years.

In its conclusion, the report states:

This article presents a Blueprint for Better Exercises. The goal of the Blueprint is to provide a set of considerations and approaches for implementing existing guidance on exercise design and evaluation, based on academic literature and evidence from exercises.

Figure 3 - Blueprint for better exercise, Architects: Dave Rouse (OSRL) and Dr El Parker (Independent Consultant)

Considerations:

• Proactive: part of a long-term programme with QA and accountability mechanisms
• One-off reactive: ad hoc; triggered after change in legislation, new lesson or plan
• Both demand forensic needs assessment – root analysis cause of previous problems/consequences of change
• Outline parameters (budget, timescale)
• Gain genuine support from senior leadership* 

Significance:

Establishing an annual cycle of quality-assured exercising and improvement will allow development and improvement to build year on year.

Reactive exercising as a response to change may reap transient benefits but lessons are unlikely to be learned and retained unless exercise forms part of broader learning processes.

Considerations:

Ensure one purpose:

• Develop and integrate new or revised arrangements
• Test arrangements
• Train and develop people’s competence
• Validate & assure arrangements and competence

One Focus:

• People vs process vs the whole system
• Is knowledge development needed prior to practice 

Significance:

One exercise can not address multiple needs – no matter the resource thrown at it. Define a series of smaller exercises.

Only competent people can test arrangements.

Only responders with high-level expertise can test and assure complex interdependent arrangements.

Trained and experienced assessors are needed to assess competency

Considerations:

• Relate to needs assessment and reason to hold exercise
• More parties = more planning and more politics…
• …BUT potential for more value if executed well
• Evidence of the benefits for key stakeholders*

Significance:

Identify needs – Are people competent? do your arrangements work? Is there interoperability across services and agencies?

Response requires effective teams – not just individuals. Crises are often managed by peripatetic/inexperienced teams.

Don’t design the exercise around those available to play – highlight the risks of limitations in those capabilities and capacity. 

Considerations:

• Represent ALL stakeholders
• Diverse (thinking, expertise, demographic, experience)
• Break silos and avoid groupthink
• Team members :
  • Project management
  • Technical advisors
  • Scenario development researchers
  • Evaluation/QA experts
  • Exercise delivery leads

Significance:

Diversity builds resilience. Maximise insights which build competence and capability – a diverse development team can bring a breadth of experience, and a depth of understanding and challenge each other’s assumptions.

Don’t overlook second-order stakeholders’/potentially impacted communities’ perspectives.

Considerations:

• Collective approach – fidelity & trust
• In/out of scope – manage expectations
• Assign responsibilities
• Milestones and timelines
• Budget/in-kind
• Baseline Workshops
• Understand current capabilities ecosystem – links and interdependencies
• Define the capacity needed (staff, assets etc) to implement the plan AND what is typically available 

Significance:

Planning the exercise is a learning opportunity. The developers of London’s largest exercise (Unified Response 2016) reported the value of shared learning and understanding gained thorough collaborative design (changes were identified during this phase).

The exercise is a project. Adopting project management discipline is critical to success.

Don’t overlook or underestimate the importance of ‘front end loading’ – the robust planning, design and preparation carried out in the early stages of a project’s lifecycle which set the project up for success.

Considerations:

• Clear statement of need – strategic goal
• Measurable objectives or learning outcomes
• Define standards expected for arrangements or competence
• Align the scenario & injects
• Capture in an evaluation framework
• Progress indicators
• Monitoring & Information Management mechanism

Significance:

Align all your elements (the golden thread) – create a “map” that shows how the injects test or evidence competencies or standards and which objectives cover each cluster.

Create measurable performance standards that observers and debrief facilitators can use for evaluation.

Considerations:

• Overcome politics, risk-averse culture and fear of losing face
• Map capability and capacity assessment against credible WCS – be clear on the risk of not stress testing
• Challenge assumptions (scenario, capability & capacity)
• Navigating and influencing takes time
• Many organisations have a preference for exercises that deliver good news…
• Learning organisations strive for improvement… Good news!

Significance:

Too often exercises play safe, the scenarios are within capability and capacity – success is guaranteed! By definition, a crisis is a situation beyond the capacity of those involved.

High-reliability organisations (potential for large-scale risk and harm but balance effectiveness, efficiency and safety) demonstrate deference to expertise, reluctance to simplify, sensitivity to operations, commitment to resilience and preoccupation with failure.

Considerations:

For scenario:

• Don’t make it up – define evidence-based credible worst-case scenarios (WCS)*
• Resist pressure to deviate

For injects:

• How much role-player free-play?
• Realism – technical expert actors

For assessment:

• Align injects to performance standards indicators
• Provide individual/team feedback using competence statements & confidence ratings
• Agree on predefined feedback statements – “working towards” rather than “needs improvement”

Significance:

Avoid playing safe – by researching case studies, applying predictive models, and applying systems thinking (Checkland 1989 & 2011) to define a credible worst-case scenario that details hazard/threat – impacts – consequences.

Look beyond the risk register when developing credible WCS. The risk register is not always imaginative enough. Consider not only what could credibly happen but the paths that could lead there.

The organisational culture may mean teams make safe assumptions – use objective experts to challenge decisions and free-play reactions during exercises.

Use the performance standard response to an inject as the basis of feedback to the team.

Considerations:

• Understand perceptual biases through consultation e.g. assumed effectiveness of risk controls
• Test confirmation, anchoring & availability bias via tailored injects
• Test procedural vs adaptive expertise
• Provide the evaluation framework (standards expected) to participating organisations/ delegates 

Significance:

Don’t play safe - design exercise scenarios that challenge nontechnical skills (Flin et al 2008).

Areas of weakness commonly stem from poor situational (risk) awareness, unwillingness to deviate from planned actions despite evidence to the contrary, and sticking to failing procedures in times of cognitive overload.

Tabletops with breakouts for discussion provide a safe space for analysis and feedback.

Resilient practitioners/responders demonstrate metacognitive expertise – they know when they can't solve a problem and adopt effective (collective) strategies.

Considerations:

• Train a diverse range of evaluators from inside & outside the sector
• Develop a better debrief than ‘three things that went well, three things to do better AND provide debrief training for delegates
• Use the evaluation framework (standards expected)
• Establish a moderation process for consistent evaluation
• Governance process needs to ensure lessons are actually implemented across teams/agencies – track accountability
• Measure, evidence & report impact and ROI
• Publish transparently 

Significance:

Ensure everyone follows the golden thread, and evaluate what the exercise intended to achieve; not what was salient during the debrief.

Empowering people to contribute in a debrief – to identify areas of weakness – requires an understanding of performance standards and trust that contributions will be used ethically, and belief that positive change will result.

Assign ownership to each lesson, agree on appropriate actions and ensure the quality cycle demands evidence of change and retention.

Considerations:

• Less is usually more
• Exercise management software has pros and cons
• Tendency to go big, but several shorter activities may be better for retention, developing dynamic risk decision-making, confidence…
• Resist the use of tools and templates that restrict thinking/decision-making. 

Significance:

Exercises are not a showpiece, no organisation is invincible.

Expertise and resilient arrangements take time and practice (Ford & Schmidt 2001).

Competence has a half-life.

Aide Memoires are useful but ensure they drive creative/adaptive rather than proceduralist thinking.

Considerations:

• Balance between time/cost/quality
• Purpose (teaching vs deliberate practice vs testing vs performance demonstration)
• Focus (people vs process vs the whole system)
• Where it fits in the programme/recency of previous exercises
• Competence of participants vs gaps
• Maturity of arrangements vs gaps.

Significance: 

With a diverse team and a bit of time, it is possible to create focused, challenging and beneficial exercises.

Having an exercise programme (a variety of types/scales) with a quality assurance mechanism will reinforce lessons that need to learn, embed changes, and save money in the longer term.

Too often, exercises play safe. They use scenarios that are within existing capability and capacity, and designed to finish within a neat eight-hour window. 

Success is virtually guaranteed. 

Exercising routine arrangements like this are a start and may have a role in developing muscle memory and mitigating skill fade. However this will not prepare teams for complex incidents and crises, which by definition are unpredictable, do not go to plan, and threaten to exceed the capability of those involved.

Thoughtful application of the approach outlined in this paper will provide strategic benefits to any organisation – improving preparedness and improving the return on exercise investment.

A programme of thoughtfully-sequenced, well-defined small-scale exercises, complemented by guided self-study, will develop and embed the knowledge required by responders. The programme is repeated with new cohorts of responders, evolving where appropriate. This approach is sustainable, efficient and agile. 

This allows large-scale exercises to focus on the most complex areas, delivering maximum benefit from the investment they entail, and to take resilience to the next level. Using expert-informed, credible worst-case scenarios, large-scale exercises can genuinely challenge participants and stress-test interdependencies present in complex organisational

systems. This is only possible when the focus is clear and the fundamentals have been developed progressively beforehand. 

Exercising in this manner, responders develop the mindset necessary to work effectively under pressure, expand shared mental models of system dependencies, and build the confidence needed to perform at their best when the worst happens. 

In addition to improving response readiness, organisations with high crisis competence develop a resilience mindset and improved risk management culture. Staff who are confident, adaptable, quick thinking, agile and collaborative, who trust each other from having worked together under pressure, can apply worst-case-scenario thinking to business

as usual challenges, and are better equipped to anticipate and manage unforeseen consequences.

If the pandemic has shown us anything, it is that we must plan for uncertainty and develop capabilities to deal with it.

Organisations already have the capability to critically assess needs and gaps, develop performance standards, and consistently evaluate competencies and arrangements.

The main shift most organisations need to make to improve the return on investment of exercises and realise the full preparedness and strategic benefits outlined above is cultural.

The shift is from a culture where exercises are ad-hoc, standalone projects towards one where they form part of a much broader quality and improvement cycle.

By integrating exercise planning, delivery and evaluation phases into wider quality governance systems, organisations will benefit from improved readiness, a more effective response, swifter recovery and strengthened resilience.

British Standards Institution, 2014. BS 11200:2014 Crisis management – Guidance and good practice. London.

Checkland, P. B. (1989). Soft systems methodology. Human systems management, 8(4), 273-289.

Checkland, P. (2011). Autobiographical retrospectives: Learning your way to ‘action to improve’ – the development of soft systems thinking and soft systems methodology. International Journal of General Systems, 40(05), 487-512.Gunns, J. and Leigh, M., 2016. Developing and Delivering Exercises. UK Government Cabinet Office Emergency Planning College. York.

Demmar, D. and Leigh, M., 2020. EPC Lessons Digest – Summary of Multi-Agency Lessons Identified in the UK Since 2005. UK Government, Cabinet Office, Emergency Planning College. York.

Flin, R., O’connor, P., & Crichton, M. (2017). Safety at the sharp end: a guide to nontechnical skills. CRC Press.

Ford, J. & Schmidt, Aaron. (2000). Emergency Response Training: Strategies for Enhancing Real-World Performance. Journal of hazardous materials. 75. 195-215. 10.1016/S0304-3894(00)00180-1.

Gunns, J. and Leigh, M., 2016. Developing and Delivering Exercises. UK Government Cabinet Office Emergency Planning College. York.

IOGP-IPIECA, 2014. Oil spill exercises – Good practice guidelines for the development of an effective exercise programme. London.

London Fire Brigade (LFB) 2017. Exercise Unified Response Evaluation Report.

New Zealand Government, Ministry of Civil Defence & Emergency Management, 2019. CDEM Exercises Director's Guideline for Civil Defence Emergency Management Groups [DGL 10/19]. Wellington.

US Government, Department of Homeland Security, Federal Emergency Management Agency, 2020. Homeland Security Exercise and Evaluation Program (HSEEP). Washington DC.

UK Government, House of Commons Health and Social Care and Technology Committees, 2021. Coronovirus: lessons learned to date. London.