After AWIA: Planning and Implementing Effective Cybersecurity
Bob George, director of cybersecurity and network infrastructure for Tetra Tech who serves water sector clients throughout the United States and Canada, discusses cyber-related outcomes and strategies for water utilities working toward America’s Water Infrastructure Act (AWIA) compliance.
Water utilities are preparing risk and resilience assessments (RRA) and emergency response plans (ERP) to meet the deadlines of the AWIA. These efforts address both physical assets, such as pipelines and storage tanks, and cyber assets, such as networks and control systems. Having worked with utilities throughout the United States to conduct these assessments, Tetra Tech has identified several cyber-related outcomes that are common to utilities of all sizes:
- Estimates of the consequences associated with failure of a critical cyber asset often fail to account for the additional staff and overtime required to operate manually during prolonged outages, water production impacts, potential safety and supply impacts, and compliance-related penalties. Information technology (IT) and operational technology (OT) consequences can exceed system replacement costs by more than 10 times
- IT departments often are unfamiliar with the unique cyberphysical considerations associated with industrial control systems (ICS), such as health and safety risks, regulatory data compliance, and environmental consequences that can have impacts beyond the organization
- Threats to physical assets tend to have high consequence but very low probability, while threats to cyber assets have both high consequence and high probability. Cyber threats are far more prevalent and surprisingly devastating
- Municipalities and utilities, which often lack cybersecurity on ICSs running 10 years or more with minimal updating, are increasingly becoming the targets of sophisticated advanced persistent threat (APT) attacks
Water utility owners may struggle to address such issues cohesively, effectively, and without disrupting critical production systems. Tetra Tech has identified strategies to help utilities design, implement, and maintain IT and OT cybersecurity improvements in a phased approach. It is essential to prioritize improvements and look for opportunities to implement projects in parallel that provide the most benefits.
A first step toward effective implementation of needed cyber improvements is to categorize the improvements in groupings such as:
- Organization—Cybersecurity awareness and skills training are essential to staff development. Training can be planned and implemented independently of technology deployments
- Methods—Policies and procedures used in day-to-day operations are typically undocumented. In times of emergency, documentation can significantly improve system recovery times by allowing additional personnel to assist and verify that essential steps are not missed
- Technology—Cybersecurity improvements for network infrastructure takes time and can impact communications infrastructure. They require a phased approach for design and implementation
- Security—Cybersecurity deployments, such as anomaly detection and monitoring, can often be incorporated at the edge of IT or OT networks and not require significant changes to the internal network
Developing policies and procedures can be laborious and time-consuming and may not appear to have immediate results. However, policies and procedures are the only protection for the most vulnerable part of any system—people. Clearly defined policies should:
- Define what is and isn’t permissible when accessing a system
- Provide clear instructions for responding to suspicious events or system anomalies
- Enable new personnel to quickly identify appropriate actions
- Identify essential cybersecurity capabilities to be included in future improvements
New cybersecurity policies typically require organizational changes and a shift in priorities from user comfort to security. Cybersecurity awareness training for all employees accessing critical cyber assets is particularly important in our age of ransomware and APT attacks.
Short-Term Improvements: The Cybersecurity Overlay
Enhancing cybersecurity in the short term relies on cybersecurity overlays and controls that can be implemented quickly, inexpensively, and without disrupting critical production systems.
This involves identifying and consolidating a secure interconnect between critical systems and other ICS networks to enhance defense-in-depth between networks. This requires a clear perimeter between systems to secure user access and data exchange.
Specific strategies include:
- Consolidating connections between networks (e.g., enterprise and ICS)
- Eliminating dual-homed hosts or other devices with multiple network interfaces
- Providing logical, and ideally physical, separation between networks, typically in the form of dedicated internet protocol (IP) subnets and virtual LANs (VLAN)
- Installing a firewall or other dedicated security appliance between networks to restrict access
- Eliminating the ability to connect directly between IT and ICS networks
- Configuring a demilitarized zone (DMZ) network with firewall rules controlling and monitoring DMZ access
- Installing one or more DMZ-based hosts to secure user access or data exchange between systems
- Requiring a protocol break between systems so there is no direct network connection between internal and external systems specifically for user access and data exchange. This approach provides multiple layers of security that are overlapping, complementary, and independent
Depending on the size and complexity of the system, multiple secure interconnects may be appropriate. At a minimum, utilities must secure and isolate critical traffic crossing shared network infrastructure.
Long-Term Improvements: Secure Network Architecture
A secure network architecture needs to incorporate cybersecurity throughout the infrastructure, with both logical and physical separation between IT and OT systems. Utilities must identify controls that require substantial budgetary outlay, staff training, and modifications to existing production networks.
This architecture can incorporate and expand on a previously implemented existing cybersecurity overlay effort, leveraging the costs and effort involved in the initial improvements.
Of particular concern during network design and implementation are rapid technology changes and new technologies that can significantly reduce cost and complexity. To incorporate flexibility in the design to avoid technology lock-in, it is important to do the following:
- Focus on a logical design regardless of the underlying communication technologies
- Avoid specifying proprietary solutions and ensure that technology is interoperable with other parts of the system based on established standards
- Reduce spot application of technology, securing one part of a network while leaving other parts exposed provides little net gain
- Ensure that communications infrastructure can support additional features such as enhanced logging and encryption
It’s not uncommon to find critical systems that are 10 to 15 years old and can’t be replaced or upgraded for some time. A common mistake is to leave these systems in place unmodified until fully replaced, with security vulnerabilities unaddressed.
A better strategy is to designate these as legacy networks—essentially separate security zones with dedicated firewall ports and IP subnets—and incorporate them into the overall architecture. These systems can remain as is, and yet still be integrated into a secure architecture.
The impending advent of 5G technologies will significantly change the wide area networking landscape. Affordable LAN-speed connections are becoming a reality. Therefore, utilities should provide sufficient flexibility in their architecture to support easy upgrade and exchange of communications infrastructure and avoid vendor lock-in.
OT vendors are rapidly expanding their industrial internet-of-things offerings, making a compelling case for inexpensive, low-power, and low-cost communications even to extremely remote locations. Other technologies are striving to replace copper wiring within facilities with low-power Bluetooth or other wireless technology. Close to ground computing pushes decision-making onto the plant floor, building wiring closet, or remote site.
Virtualization and hyper-convergence have completely reshaped the data center concept. Racks of chunky servers have been replaced by small rack-mounted modules, which can support dozens of virtualized servers. In addition, most firewall manufacturers now offer virtual appliances that can extend full security architecture into the virtual world.
AWIA assessments have elevated the awareness of threats posed to utility systems from physical and cyber threats. By developing strategies that address immediate needs while allowing for system growth and technological change, we can bring the security of SCADA and ICS up to modern security standards.