Short answer
After an ordinary failure, recovery often means restoring the service you had.
After a cyber incident, that may not be enough. If an attacker changed systems, created accounts, altered configuration or established another way back in, restoring the existing environment may preserve the problem as well as the service.
An alternative is to build a new environment from controlled definitions and trusted sources. Infrastructure is deployed again. Software is rebuilt or installed again. Configuration is generated from reviewed sources. Data is restored from a point considered sufficiently reliable.
The compromised environment can remain isolated for investigation and reference. It does not need to become the foundation of the recovered service.
This does not prove that nothing harmful will return. Repositories, templates, pipelines, images, backups, identities and data can also be compromised. Rebuilding changes the problem: instead of depending entirely on finding every unwanted change in the existing environment, you establish which sources are trustworthy enough to build from.
Recovery normally assumes that something remains trustworthy
A failed disk, server or data centre creates an availability problem. The system may no longer work, but the organization may still trust its application, configuration, identities and stored data.
A cyber incident creates a different question.
You may not know:
- which systems were accessed;
- which files or settings were changed;
- whether new accounts, credentials or access paths were created;
- how long the attacker was present;
- whether backups or administrative tools were reached;
- or whether the visible attack was the complete attack.
Cleaning the existing environment requires you to find and remove everything that should not be there. In a small and well-understood environment, that may be possible. In a larger or poorly documented environment, confidence can be much harder to establish.
Rebuilding offers another route. You create a new trust boundary and decide what is allowed to cross it.
Cloud makes a parallel environment practical
A conventional recovery may require spare hardware, another location and enough capacity to reproduce the affected environment. Maintaining those resources in advance can be expensive.
Cloud infrastructure changes that constraint. Networks, access controls, computing capacity and managed services can be created when they are needed. A complete replacement environment does not necessarily have to remain provisioned while waiting for an incident.
If the environment has been described in Infrastructure as Code, much of its structure can be deployed consistently into another account, subscription, project or region. Application code and configuration can then be deployed through a controlled pipeline, followed by the data required to make the service useful.
That does not mean every cloud environment is ready to be rebuilt. Manually created resources, undocumented dependencies, configuration stored only on running servers and credentials tied to the affected environment can make cloud recovery just as uncertain as recovery elsewhere.
The practical advantage appears when rebuildability was designed and maintained before it was needed.
Build a new trust boundary
A new environment should not simply be another set of servers connected to everything the old environment used.
Depending on the incident, meaningful separation may require:
- a different cloud account, subscription or project;
- newly established administrative access;
- new passwords, keys, tokens and certificates;
- reviewed network connections;
- replacement service accounts and machine identities;
- newly established trust between applications and external systems;
- and restricted communication with the affected environment.
If the attacker’s entry point is known, it must be addressed before the relevant service returns. Recovery should not assume that the known entry point was necessarily the only one.
The same caution applies to the cloud control plane. Rebuilding resources inside an account that remains under an attacker’s control does not create a clean environment. The identities, automation and administrative paths used for the rebuild also need an appropriate level of trust.
Different parts of the service return in different ways
Rebuilding does not mean copying the contents of old servers into new ones. Infrastructure, software, configuration and data need different treatment.
Infrastructure
Networks, access rules, computing resources and managed services can be deployed again from reviewed Infrastructure as Code.
Those definitions need version control, review and protection of their own. Automation can reproduce an intended environment quickly, but it can reproduce a malicious or accidental change just as efficiently.
Software
Application software should normally be rebuilt or installed from controlled source code, packages, artefacts and images.
Copying binaries from a compromised machine may carry changes into the new environment. The repositories, build systems and dependencies used instead must therefore be included when deciding what can be trusted.
Configuration
Infrastructure configuration may already be represented in code. Application configuration may be stored elsewhere: in deployment systems, configuration repositories, managed services or, less desirably, only on the affected machines.
Configuration should be regenerated from reviewed sources where possible. Secrets do not belong in ordinary Infrastructure as Code and should be replaced rather than casually recovered from the compromised environment.
Static content
Static files can be republished from a source repository or another validated copy. A large collection of apparently harmless content still needs consideration if attackers could have inserted scripts, changed downloads or altered material distributed to users.
Dynamic data
Business data usually cannot simply be created again. It must be restored, recovered or reconstructed from legitimate activity.
The most recent backup is not automatically the safest one. If the incident began before that backup was created, it may contain unwanted changes. Recovery may require the last known good point, followed by controlled replay of later legitimate transactions.
That replay also needs validation. Logs and transaction records may include actions performed by the attacker alongside normal business activity.
The objective is not always perfect certainty. It is a documented decision about which recovery point and which subsequent changes provide an acceptable level of confidence.
Restore the smallest useful service first
Trying to reproduce the complete former environment in one operation can make recovery slower and harder to control.
A more manageable sequence begins with the minimum service the organization needs to resume its most important activity. That includes the application and the dependencies required for it to function, but not automatically every system that previously existed.
Additional services and integrations can return in stages. Each step creates an opportunity to validate behaviour, access and data before expanding the new environment further.
This process may reveal that the documented architecture and the actual architecture were different. An application may depend on a server, scheduled task, identity, file share or external connection that was never recorded as part of the service.
That discovery should not be addressed by reconnecting the old environment wholesale. The isolated environment can be consulted as a reference while the missing dependency is understood and recreated appropriately.
Preserve the affected environment
The old environment may contain evidence needed to understand the incident. It can also help identify configurations, data and dependencies that were not captured elsewhere.
Preserving it does not mean leaving it connected or continuing to operate it normally. Appropriate isolation, evidence handling and access control should be agreed with the people responsible for incident response, security, legal obligations and any required investigation.
There is an important distinction:
- the affected environment can remain a source of evidence and information;
- it should not automatically remain a trusted source of software, configuration or identity.
Investigation and rebuilding can therefore proceed alongside each other. What is learned from one may influence the other without making the new environment dependent on the old one.
Rebuilding requires more than Infrastructure as Code
Infrastructure as Code is an important foundation, but a set of templates is not yet a recovery capability.
A usable rebuilding process also depends on:
- protected and version-controlled infrastructure definitions;
- access to trusted source code, packages and images;
- a deployment pipeline that can operate outside the affected environment;
- known methods for replacing credentials and trust relationships;
- backups that are separated from production compromise;
- tested restoration procedures;
- an understanding of application and data dependencies;
- sufficient cloud capacity and account access;
- people who know who can authorize and perform the recovery;
- and a method for validating that the rebuilt service actually works.
These dependencies need their own failure assumptions. A pipeline hosted inside the compromised environment may no longer be usable. An image registry may contain images created after the incident began. A backup account administered through the same compromised identity may not provide meaningful separation.
The recovery path should not depend entirely on the thing from which the organization is trying to recover.
Design for rebuilding before an incident
Rebuildability is an architectural property. It is difficult to add under pressure after trust in the environment has already been lost.
When an environment is designed, ask the architect not only how it will be built, but how it could be built again.
That includes decisions about:
- which resources are represented in Infrastructure as Code;
- where application and infrastructure definitions are stored;
- how configuration is separated from secrets;
- how trusted images and packages are produced;
- how data is backed up and restored;
- which dependencies must return first;
- how a separate recovery environment would receive administrative access;
- and how the complete process will be tested.
A successful first deployment proves that the environment could be created once. A rebuilding exercise shows whether it can be created again when the normal environment and its ordinary administrative paths are unavailable.
Rebuilding does not replace incident response
This article describes one recovery capability, not a complete response to a cyber incident.
Organizations may also need to contain the attacker, preserve evidence, determine reporting obligations, understand affected data, communicate with customers and partners, monitor for renewed activity and coordinate with specialist responders or authorities.
A newly built environment can be compromised again if the attacker remains active, the original access path remains open or an affected identity is reused.
Rebuilding is therefore not a shortcut around investigation and containment. It avoids making the complete restoration of service dependent on proving that every part of the existing infrastructure is clean.
Questions to ask before you need the answer
- Could we deploy our critical infrastructure into a separate environment today?
- Which resources exist only because somebody created them manually?
- Can our deployment pipeline operate if production accounts and credentials are unavailable?
- Which software, images, packages and repositories would we trust?
- Where is application configuration stored?
- Which credentials, certificates and trust relationships would need to be replaced?
- How would we identify the last sufficiently trustworthy data recovery point?
- Which later business transactions could be replayed, and how would we distinguish legitimate activity?
- What is the smallest useful service we could restore first?
- Which dependencies are documented, and which have only ever been observed in production?
- Can backups be altered or deleted using the same identities that administer production?
- Have we rebuilt the service from these sources and validated the result?
- Who can decide that the new environment is ready to serve users?
If several answers depend on inspecting the existing servers during the incident, the organization may have backups but not yet have a reliable rebuilding capability.
What cloud changes
Cloud does not make a compromised environment trustworthy. It makes it possible to create replacement infrastructure without first purchasing and installing another physical environment.
Infrastructure as Code, deployment pipelines, managed services and capacity on demand can make rebuilding faster, more consistent and more testable. They can also spread a harmful change quickly when the definitions, identities or automation have been compromised.
The important capability is therefore not merely that the infrastructure is in the cloud. It is that the service can be recreated from sources whose integrity can be assessed, within a boundary that does not inherit trust from the affected environment.
Recovery brings a service back.
Rebuilding determines what is allowed to return.
Sources and further reading
- CISA: #StopRansomware Guide
- NIST SP 800-184: Guide for Cybersecurity Event Recovery
- NIST SP 1800-11: Data Integrity — Recovering from Ransomware and Other Destructive Events
- AWS Well-Architected Framework: Use defined recovery strategies to meet recovery objectives
- AWS Well-Architected Framework: Automate recovery