In enterprise IT, the worst outcome of a bad incident response decision is a longer outage. In OT, the worst outcome is a process condition that kills someone. These are not the same problem, and they must not be treated as the same problem.
The playbook that does not transfer
Enterprise IT incident response has matured considerably over the past decade. The frameworks are well-established, NIST SP 800-61, SANS incident response methodology, the MITRE ATT&CK-aligned detection-and-response models that underpin most commercial SIEM deployments. The tooling is mature: EDR platforms with automated containment capabilities, SOAR platforms that execute response playbooks at machine speed, threat intelligence feeds that enrich alerts with adversary context in real time. For enterprise IT environments, this ecosystem represents genuine progress, faster detection, faster containment, faster recovery.
The problem arises when this ecosystem is extended, without adaptation, into operational technology environments. And it is being extended, by IT security teams that have been given responsibility for OT security without adequate OT context, by vendors whose IR platforms were designed for enterprise IT and repackaged for ICS markets, and by organisations that have treated “unified security operations” as a goal without examining whether unity is actually achievable across environments with incompatible operational requirements.
The failure modes of IT-centric incident response applied to OT environments are not hypothetical. They are documented, they are recurring, and in the most serious cases they are dangerous. This post is an attempt to describe precisely why the transfer fails, what an OT-appropriate incident response capability looks like, and what the organisational and architectural requirements are for building one that works when you actually need it.
I want to be honest about the title. The claim that IT incident response playbooks applied to OT environments will get people killed is not hyperbole for effect. It is a statement about what happens when automated containment actions are taken against systems whose continued operation is a safety requirement, when network isolation decisions are made without understanding the process consequences, and when recovery actions are executed without the engineering validation that safe process restart requires. These are specific, foreseeable failure modes. The purpose of this post is to make them concrete enough that organisations can design around them.
Part 1: The operational constraints that make OT IR fundamentally different
To understand why enterprise IR methodology does not transfer to OT environments, you need to understand the operational constraints that govern what actions are permissible in a control system environment. These constraints are not bureaucratic obstacles or organisational conservatism. They reflect the physical reality of operating infrastructure where digital systems are coupled to physical processes with safety and availability consequences.
Constraint 1: Availability is a safety requirement, not a business preference.
In enterprise IT, availability is a business requirement. Downtime costs money. Extended downtime costs more money. The incident response decision to take a system offline, to isolate a compromised host, to shut down a service, to sever a network connection, is fundamentally a cost-benefit calculation: the cost of continued operation with a known compromise versus the cost of taking the system offline to contain and remediate.
In OT, availability of certain systems is a safety requirement. A DCS controlling a chemical process must continue to control that process because the alternative, an uncontrolled process without active control, may be more dangerous than the controlled process with a compromised DCS. A SCADA system monitoring a gas pipeline must continue to provide operator visibility because operators making decisions without process visibility are more dangerous than operators making decisions with compromised visibility. Taking these systems offline is not a cost-benefit calculation. It is an engineering decision with process safety implications that must be made by people with process engineering competence, not by incident responders working from an enterprise IR playbook.
This does not mean that OT systems can never be taken offline during an incident. It means that the decision to take them offline must be made through an engineering decision process, one that explicitly evaluates the process safety implications, identifies the safe state that the process will be driven to before the system is isolated, and ensures that the isolation can be executed safely. The timeframe for that decision process may be hours or days, not minutes.
Constraint 2: The safe state must be explicitly defined and reachable.
In enterprise IT, a system that is isolated or shut down typically fails to a state that is acceptable by default: services become unavailable, users cannot access data, but no physical consequence follows. In OT, the failure mode of a system depends entirely on the process it controls and the engineering decisions made when the system was designed.
Some OT systems fail safe by design: a safety instrumented system that loses power or communication drives the process to a defined safe state, closing valves, stopping pumps, isolating sections of plant, using hardwired logic that does not depend on the compromised system. Other OT systems have no inherent safe failure mode: a DCS that loses communication may leave control outputs in their last-known state, which may or may not be safe depending on what the process was doing at the moment of failure. Understanding the failure mode of every control system in scope for an incident response action is a prerequisite for taking any containment action in an OT environment.
Constraint 3: Forensic evidence collection conflicts with operational requirements.
In enterprise IT, incident response forensics typically involves imaging the memory and disk of affected systems, collecting volatile data before it is lost, and preserving evidence in a forensically sound manner. The affected system is typically taken offline for this process.
In OT, the affected system may be a SCADA server or HMI that must remain operational because it is the operator’s primary interface to the physical process. Imaging the disk of a live SCADA server, even if technically possible without taking it offline, may introduce performance degradation that affects control response times. Installing forensic agents on OT systems may conflict with vendor support agreements or introduce software not validated for the OT environment. The forensic collection process must be adapted to the OT context, prioritising evidence that can be collected without operational impact and deferring intrusive collection until a planned maintenance window.
Constraint 4: Recovery requires engineering validation, not just IT verification.
In enterprise IT, recovering a system from an incident typically involves reimaging the system, restoring from backup, and verifying that services are functioning correctly. The validation is primarily functional: does the application start? Do users have access? Are services responding to requests?
In OT, recovering a system from an incident requires engineering validation: does the control system’s configuration accurately reflect the intended control logic? Are the I/O mappings correct? Are the setpoints, alarm limits, and interlock configurations consistent with the process engineering design? Is the system’s behaviour consistent with expectations when connected to the live process? These are questions that cannot be answered by an IT system administrator and cannot be answered quickly. They require process engineers who understand the intended behaviour of the system and can validate the recovered system against that understanding.
An OT system that has been recovered from an incident, even one that has been reimaged and restored from a clean backup, must be validated by process engineering before it is reconnected to the live process. This validation takes time, and the time it takes should be explicitly planned for in the OT incident response framework rather than treated as an unexpected delay.
Part 2: Process Hazard Analysis integration: the safety case for IR decisions
The most important architectural integration in OT incident response is the one that is most consistently absent: integration with the process safety function through the organisation’s Process Hazard Analysis (PHA) documentation and process safety management (PSM) framework.
PHA/HAZOP studies, which responsible energy and process industry operators maintain for every process unit above a trivial risk threshold, document, in exhaustive detail, every process deviation that could result in a safety event: the cause, the consequence, the safeguards in place, and the residual risk after safeguards are applied. This documentation is directly relevant to incident response decisions because the most dangerous incident response actions in OT environments are those that inadvertently remove or compromise the safeguards documented in the PHA.
Consider the following scenario: during an active OT incident, an incident responder identifies that the SIS network has been compromised and recommends isolating the SIS server to prevent further lateral movement. The SIS server is the software component that monitors safety function status and provides operator visibility into safety system state. The SIS hardwired logic, the actual safety functions, operates independently of the SIS server. From an IT security perspective, isolating the compromised SIS server is correct containment. From a process safety perspective, removing operator visibility into safety system state during an ongoing process operation is a significant safety risk, because it removes the operator’s ability to confirm that the safety functions are armed and available.
A PHA-integrated incident response framework would catch this conflict before the action is taken, because the relevant PHA safeguard, operator visibility of safety system status, would be explicitly identified as a constraint on any incident response action involving the SIS network. The IR decision authority would know, before making the isolation decision, that it removes a documented safeguard, and the process safety function would be consulted on the risk of proceeding.
Building PHA integration into the IR framework requires several things:
A structured mapping between the control systems in scope for OT incident response and the PHA safeguards that depend on their availability. For each control system, the mapping should identify which PHA safeguards would be impaired if that system were isolated, shut down, or degraded, and what the consequence of that impairment would be.
An escalation path from the incident response function to the process safety function that can operate at the timescales required by an active incident. PHA consultations that take days are not useful in an incident context. The process safety team must have pre-designated members who can provide rapid assessment of safety implications for incident response decisions, using the pre-developed mapping between control systems and PHA safeguards.
Pre-approved containment actions that have already been validated as safe by the process safety function. For each category of OT incident response action, network isolation, system shutdown, configuration rollback, remote access termination, a set of pre-approved variants should be developed and safety-validated before an incident occurs. Incident responders working from pre-approved actions are less likely to improvise actions with unvalidated safety implications than those working from generic IR playbooks without OT context.
Part 3: The OT-specific incident classification framework
Enterprise IR frameworks typically classify incidents by severity, P1, P2, P3, or equivalent, based on the scope of impact, the sensitivity of data affected, and the operational disruption caused. This classification model is insufficient for OT incidents because it does not account for the process safety dimension of impact.
An OT incident classification framework must include a safety dimension that governs the response posture independently of the IT-centric severity assessment. The following classification model reflects the approach I consider most appropriate for energy operator OT environments:
Class S, Safety system involvement. Any incident where the compromise or suspected compromise extends to safety instrumented systems, emergency shutdown systems, fire and gas detection systems, or any other system whose function is to prevent or mitigate process safety events. Class S incidents are automatically the highest priority regardless of other severity indicators, require immediate notification of the process safety function and site management, and may require process shutdown to a defined safe state before any further incident response actions are taken. The decision to maintain process operation with a potentially compromised safety system is a process safety decision, not an incident response decision.
Class O, Operational technology affected. Incidents where compromise is confirmed or suspected in Level 1 or Level 2 OT systems, PLCs, DCS, SCADA, that have direct control of physical processes. Class O incidents require joint response from the security incident response team and the operations function, with operational continuity decisions made by the operations function with security input, not by the security function independently.
Class B, OT boundary systems affected. Incidents where compromise is confined to OT DMZ systems, historian servers, jump hosts, remote access infrastructure, without confirmed propagation to Level 2 OT systems. Class B incidents can be managed primarily by the security function but require OT network isolation assessment before any containment actions are taken that could affect data flows to or from the OT supervisory zone.
Class I, IT systems with OT adjacency. Incidents in enterprise IT systems that have defined interfaces to the OT environment, ERP systems with interfaces to production data, engineering workstations that connect to both IT and OT networks, vendor systems with remote access to the OT DMZ. Class I incidents require assessment of whether the compromise could propagate to OT systems through the defined interfaces, and pre-emptive monitoring enhancement of the OT DMZ boundary while the IT incident is being resolved.
This four-class framework provides incident responders with a clear decision tree for escalation and response posture that accounts for the OT-specific dimensions of safety system involvement and operational impact. It should be documented in the OT IR plan, trained with the response teams, and exercised in tabletop scenarios before it is needed in an actual incident.
Part 4: Containment in OT, what you can do and what you cannot
The containment phase of incident response is where the incompatibility between enterprise IR methodology and OT requirements is most acute. Enterprise containment actions, isolating a host, blocking network traffic, disabling a user account, quarantining a device, are all actions that assume the affected system can be taken offline or degraded without catastrophic consequence. In OT environments, this assumption frequently does not hold.
The OT containment toolkit must be built around what is operationally permissible rather than what would be optimal from a pure security perspective. The following distinguishes between actions that are generally permissible, actions that require engineering assessment before execution, and actions that are generally not permissible without a planned process shutdown.
Generally permissible containment actions:
Blocking inbound remote access connections from external sources to the OT DMZ, provided that no active vendor support session is in progress and that the operational team is notified. This action does not affect control system function but removes a potential adversary access path.
Disabling specific user accounts in the OT identity infrastructure that are suspected to be compromised, provided that the accounts are not service accounts required for control system function and that the operational team confirms no active sessions are in progress using those accounts.
Increasing monitoring fidelity on the OT DMZ boundary, enabling additional logging, increasing packet capture scope, adjusting SIEM alert thresholds, without changing traffic flows or connectivity.
Isolating OT DMZ systems from the IT-side network, blocking the IT/DMZ firewall interface, provided that this does not interrupt data flows required for operational safety, specifically, that historian data required for safety case monitoring is either not affected or that alternative monitoring is available.
Actions requiring engineering assessment before execution:
Isolating specific OT supervisory zone systems from the OT DMZ. The impact depends entirely on which system and what functions it serves. An engineering workstation used only for infrequent configuration changes can generally be isolated with minimal operational impact. A SCADA server that provides primary operator visibility into a live process cannot.
Changing firewall rules on the OT-side firewall interface. Any change to traffic flows within the OT environment requires assessment of operational impact, because OT traffic flows are tightly coupled to control and monitoring functions.
Deploying any new software or agent to OT systems, including forensic collection tools, security sensors, or monitoring agents. All software deployed to OT systems must be validated for compatibility with the OT environment before deployment, the same validation required for any OT software change.
Generally not permissible without a planned process shutdown:
Isolating Level 2 OT systems en masse. Mass network isolation of SCADA and DCS systems is not a containment action in an OT context. It is a process shutdown, with all the safety and operational implications that a process shutdown entails. It must be planned, communicated, and executed through the operational shutdown procedure, not through a security containment playbook.
Reimaging OT systems while the process is running. Reimaging a SCADA server or DCS engineering station requires that the system be taken offline, which requires that its control functions be transferred to alternative systems or that the process be shut down to a safe state.
Revoking OT system certificates or rotating cryptographic credentials while the process is running. Certificate or credential revocation that affects authentication for control system components can cause control system failures that manifest as process upsets.
Part 5: The safe state procedure:engineering’s input to incident response
The concept of the safe state is fundamental to process safety engineering and largely absent from incident response planning in most organisations with OT environments. Every process unit operated by an energy company has a defined safe state, a configuration of the physical process in which it can remain indefinitely without risk of escalation to a dangerous condition. The safe state is typically achieved by driving the process to a low-energy, controlled configuration: closing isolation valves, reducing temperatures and pressures to safe levels, stopping rotating equipment, engaging safety interlocks.
The relevance of the safe state to incident response is this: if a control system incident is severe enough that the affected systems cannot be trusted to continue controlling the process, the appropriate response is to drive the process to its safe state and then take the time required to properly assess and remediate the incident, without the time pressure of an ongoing process operation. Attempting to continue operating a process while simultaneously responding to a severe control system incident is the worst of both worlds, the incident response is constrained by operational requirements, and the operational management is compromised by the incident.
Integrating safe state procedures into the OT IR framework requires:
Documenting the safe state procedure for each process unit, including the sequence of operations required to reach the safe state, the estimated time required, the engineering personnel required to execute the procedure, and the operational notifications required before initiating the procedure.
Defining the trigger criteria for initiating a safe state procedure during an incident. These criteria should be defined in advance, in consultation with process safety and operations leadership, rather than determined ad hoc during an active incident when decision-making is most difficult. Typical trigger criteria include: confirmed compromise of safety instrumented systems, confirmed capability of an adversary to issue control commands to process equipment, or loss of sufficient control system visibility that safe process management is no longer possible.
Assigning clear authority for the safe state decision. The decision to initiate a safe state procedure for a process unit is an operational decision with significant safety, operational, and commercial consequences. It must be assigned to a named role, typically site director or operations manager, with clear criteria for when that authority should be exercised. This authority must not rest with the incident response team, which has the security perspective but not the operational authority or the process knowledge to make the decision safely.
Part 6: The OT-specific IR team : roles and capabilities
An effective OT incident response capability requires personnel with capabilities that are distinct from enterprise IT IR competencies. Building an OT IR team from enterprise IT security professionals, even highly competent ones, without OT-specific training and integration with the operational functions produces a team that is well-equipped for the wrong environment.
The core capabilities required in an OT IR team are:
OT network and protocol knowledge. The ability to analyse traffic captures from OT networks, including the ability to parse and interpret industrial protocol transactions, Modbus, DNP3, OPC-UA, PROFINET, IEC 104, in the same way that enterprise IR responders parse HTTP, SMB, and DNS traffic. Without this capability, the IR team cannot distinguish between normal OT traffic and adversarial traffic, cannot assess the significance of protocol anomalies, and cannot interpret the operational significance of traffic patterns identified in OT network captures.
Control system platform knowledge. Familiarity with the major DCS and SCADA platforms, Honeywell Experion, Emerson DeltaV, Siemens PCS 7, Schneider EcoStruxure, GE iFIX/CIMPLICITY, OSIsoft/AVEVA PI, sufficient to perform forensic analysis on those platforms, identify configuration changes, and assess the integrity of control logic. This is specialist knowledge that takes time to develop and that typically requires hands-on training with the specific platforms in the operator’s environment.
Process safety awareness. Not process safety engineering expertise, that is the domain of the process safety function, but sufficient awareness of process safety principles, PHA methodology, and safety instrumented system design to work effectively with the process safety team and to understand the operational constraints that govern IR decisions. OT IR responders who have no process safety awareness will make decisions that the process safety team cannot support.
Forensic capability adapted for OT. The ability to collect forensic evidence from OT systems in ways that do not interfere with operational function, passive network captures, filesystem imaging of non-production systems, memory acquisition techniques compatible with real-time operating systems used in embedded OT devices. Standard enterprise forensic tools and techniques are frequently incompatible with OT environments and may cause control system malfunctions if applied without adaptation.
Established relationships with the operational function. OT incident response is a cross-functional activity. The security team brings the incident response methodology and the adversary knowledge. The operations team brings the process knowledge, the operational authority, and the relationship with the engineers who understand the specific systems being protected. These relationships must exist before an incident, they cannot be built during one. Regular engagement between the security IR function and the operations team, through joint exercises, through security reviews of OT systems, through regular security briefings to operational leadership, is the mechanism for building the trust that effective joint response requires.
Part 7: Exercises and simulation : preparing for what you cannot predict
OT incident response capability is not built through documentation. It is built through exercise. The gap between a documented IR plan and an effective IR capability is bridged only by repeated practice under conditions that approximate the stress and uncertainty of an actual incident.
OT IR exercises present specific challenges that enterprise IR exercises do not. You cannot realistically simulate a compromised SCADA system on the production control network. The exercise environment must be separate from the operational environment, which typically means either a dedicated test environment that mirrors the production control system or a tabletop format that simulates the operational decisions without executing them on real systems.
Tabletop exercises are the most accessible format and the most valuable for testing the decision-making processes, particularly the cross-functional decision processes between security, operations, and process safety, that are the most critical and the most likely to fail during an actual incident. A well-designed OT IR tabletop exercise:
Presents a realistic scenario drawn from the threat actor activity documented in ATT&CK for ICS and current threat intelligence. The scenario should reflect the actual adversary techniques that are relevant to the operator’s threat profile, not a generic “ransomware” scenario, but a scenario that reflects the specific techniques used by threat actors known to target energy infrastructure.
Involves all relevant functions: security, operations, process safety, site management, legal and regulatory, and external communications. OT incidents have consequences that extend well beyond the technical response, and the tabletop should exercise the full organisational response including regulatory notification, stakeholder communications, and management decision-making under uncertainty.
Includes decision points that expose the cross-functional conflicts described in this post: a decision point where the technically optimal containment action conflicts with operational safety constraints, a decision point where the safe state trigger criteria are ambiguous, a decision point where forensic evidence collection conflicts with operational requirements. These are the decisions that will be hardest under real incident conditions, and they must be worked through in exercises before they arise in practice.
Documents the lessons identified from each exercise in a structured format, and tracks those lessons through to closure. An exercise that produces lessons that are never implemented is a compliance activity, not a capability improvement programme.
Conclusion: The IR capability that matches the threat
The energy sector is operating infrastructure that sophisticated adversaries, including nation-state actors with the capability, intent, and patience to conduct multi-year intrusion campaigns, are actively targeting. The incident response capability that this threat demands is not the enterprise IT IR capability that most organisations have built and that most IR consultants know how to provide. It is a specialised capability that integrates security methodology with process safety engineering, that is calibrated to the operational constraints of control system environments, and that has been tested through exercises that expose the cross-functional decision failures that will occur if it has not.
Building this capability requires investment that is difficult to justify in advance of an incident. The ROI of incident response capability is, by definition, measured in incidents that were contained rather than catastrophic, counterfactuals that are invisible when things go well. This is the persistent challenge of security investment justification in any domain.
In OT environments, the additional argument is available: the consequence of inadequate IR capability in a control system environment is not just operational disruption or data loss. It is the possibility of physical harm to people and the environment, harm that is foreseeable, preventable, and that an organisation with inadequate OT IR capability has consciously chosen not to prevent.
That argument should be sufficient. In my experience, it is not always heard until after the incident that makes it concrete.
Build the capability before you need it. Test it until it works. And ensure that the people who will make the critical decisions during an OT incident understand both the security context and the process safety consequences of what they are being asked to decide.
Key references and further reading
- IEC 62443-2-1, Establishing an IACS Security Management System. Section 4.3.6 covers incident response requirements for industrial automation and control systems.
- NIST SP 800-82 Rev. 3, Guide to Operational Technology Security. Section 6.3 covers OT incident response with specific attention to the operational constraints described in this post.
- CISA, ICS-CERT Incident Response, cisa.gov/ics, CISA’s ICS incident response guidance and the reporting mechanisms for CNI operators experiencing OT incidents.
- NCSC, Cyber Incident Response: Regulated Industries, NCSC guidance on incident response for operators of critical national infrastructure, including notification obligations under the NIS Regulations.
- IEC 61511-1:2016, Functional Safety: Safety Instrumented Systems. The process safety standard whose safe state and SIS independence requirements are directly relevant to the incident response constraints described in Parts 1 and 5.
- Dragos, OT Incident Response Playbooks, Technical documentation on OT-specific incident response procedures, including containment approaches validated for control system environments.
- MITRE ATT&CK for ICS, Inhibit Response Function, attack.mitre.org/matrices/ics/, The adversary techniques specifically targeting incident response and safety system capabilities described in Part 3.
- Robert M. Lee, The Sliding Scale of Cybersecurity, The SANS paper that provides the conceptual framework for understanding where IR capability fits in the broader OT security architecture.
- ISA-TR84.00.09, Cybersecurity Related to the Functional Safety Lifecycle. The technical report addressing the intersection of process safety and cybersecurity directly relevant to the PHA integration in Part 2.
This post is part of an ongoing series on security architecture for critical infrastructure and energy systems. The next post examines the security architecture of distributed energy resources and the smart grid edge, the challenge of securing millions of internet-connected devices that are increasingly integral to grid management.
About the author: Infrastructure Architect with 14 years of experience spanning enterprise architecture, operational technology environments, and regulated digital infrastructure. Currently leading infrastructure architecture for a major energy operator. Holds CISSP certification. Previously Technical Director at a digital health startup. Views expressed are personal.