For the past 25 years, I have been working with Operational Technology (OT), and for the last 10 I have been solely focused on the security side of OT.  Until recently, cyber-attacks remained almost solely within the “IT” world, however in 2010 this all changed with Stuxnet, a malicious computer worm that targeted the centrifuges of Iran’s uranium enrichment facilities by attacking the PLCs (programmable logic controllers) that run the the refinement process by compromising the workstations used to configure those PLCs.  Stuxnet used the compromised workstations to reprogram the centrifuge PLCs to make them spin up to the point where they would tear themselves apart.  Unfortunately, Stuxnet’s design could be tailored as a platform for attacking modern SCADA and PLC systems in the United States.  Targeting industrial control systems (ICS), the worm infected over 200,000 computers and caused 1,000 machines to physically degrade.  Since then, there has been a constant increase in cybersecurity attacks on industrial organizations. Furthermore Stuxnet had security researchers and adversaries start thinking about the possibilities in attacking Industrial Control System environments. Due to this and other threats, securing your OT environment is as necessary as securing your IT environment, which was previously done through two separate and different solutions.  But these days with modern ICS residing on both the OT as well as the IT side of your organization, your security program should follow suit and cover both IT and OT as well.

Well, ICS security is a challenge for many organizations as many companies have implemented digital solutions in their networks bringing together both IT and OT, and although this opens up many new ways of improving the production process and the management of that process, if not properly designed and secured, this can leave a multitude of cybersecurity threats.  An attack can not only jeopardize your high priced equipment and network, but it can also cause devastation to communities and economies all over.  Fortinet found in their 2022 State of Operational Technology and Cybersecurity report that 93% of all organizations with OT environments experienced attacks in the past twelve months, and 93% of organizations had 1 or more intrusions in the past year.  

Before we go on, let’s take a trip back in time, back to the early to mid-90s, when automation and control systems had evolved into complex mazes of interconnected systems, devices, and other equipment, running on proprietary communication media, talking proprietary but inherently insecure protocols. Life was good, at least from an ICS security perspective. Even though the traffic between all these devices was wide open, plain-text and ripe for manipulation and attack, there were very few attackers that could make it onto those communication networks. It had to be someone inside the ICS facility with the equipment and the knowledge to be able to start messing with stuff. And all along regular networking technology and information technology (IT) had evolved into where the business side of the manufacturing facility was now completely converged onto Ethernet, some sites had even tossed their hubs and installed switches. So, by the end of the 90’s a typical manufacturing plant would look like this, from a business network versus ICS environment perspective:

What is Operational Technology (OT)

Operational Technology, or OT for short, is the practice of using hardware, software and other controls and automation equipment to monitor, control and regulate a production process. OT equipment and systems primarily interact with the physical world (open valves, move actuators, read temperatures, etc.). OT includes control systems and equipment like programmable logic controllers (PLCs), distributed control systems (DCSs), and supervisory control and data acquisition (SCADA) systems, among other things.

OT equipment and systems can be found in many industries including manufacturing, energy, medicine, building management, and as ecosystems within other industries.  Traditionally Operational Technology consisted of proprietary equipment and software but let’s explore how this evolved over time.

The Evolution of Operational Technology

Operational technology has been around longer than modern computer technology.  Initially a typical OT setup consisted of racks of wired relays, timers, switches, dials, readouts, indicators, sensors, actuators, etc. but over time these types of setups got replaced with installations built around Programmable Logic Controllers (PLCs) and Human Machine Interfaces (HMIs). 

When these PLCs first started replacing the racks of relays that used to run the automation process, they weren’t much more than that, a replacement for the racks of relays that were much more compact and sophisticated.  This allowed changes to be made without having to rewire half your plant and only took up a fraction of the control room space. The first PLC implementations were standalone islands of automation where you needed specialized “programming terminals” to connect to the PLC and make changes.

The popularity of the PLC quickly grew, and vendors started adding convenience in terms of remote IO, universal programming stations, HMIs and inter PLC communications. To achieve these new features automation companies started developing communication protocols and media, mostly based on some form of serial communications and all of them proprietary and different from one another. Some early industrial protocols are Modbus, ICCP, DNP and CIP.

Inherently Insecure

As these proprietary communication protocols were initially designed for short distance point to point communications, they didn’t incorporate security measures such as authentication, authorization, or encryption. Why would you need that on a single wire between two devices where only those two devices can talk? The inherent ICS security problems started evolving when these early protocols started to be used for communications between multiple devices, over longer distances. Even though the media the protocols ran on changed in some cases, to allow for the longer distances, the protocol itself, the way commands and instructions are communicated remained the same (for backwards compatibility).  

Convergence of IT and OT

In the early to mid-90s, automation and control systems had evolved into complex mazes of interconnected systems, devices, and other equipment, running on proprietary communication media, talking proprietary but inherently insecure protocols. And life was good, at least from an ICS security perspective. Even though the traffic between all these devices was wide open, plain-text and ripe for manipulation and attack, there were very few attackers that could make it onto those communication networks. It had to be someone inside the ICS facility with the equipment and the knowledge to be able to start messing with stuff. All along regular networking technology and information technology (IT) had evolved into where the business side of the manufacturing facility was now completely converged onto Ethernet, some sites had even tossed their hubs and installed switches. So, by the end of the 90’s a typical manufacturing plant would look like the graphic below, from a business network versus ICS environment perspective:

From a cybersecurity perspective this architecture is very desirable. You would need the maintenance laptop to be infected with a remote access trojan (RAT) or in some other way remotely accessible, while connected to the ICS equipment before this setup can be remotely compromised. So pretty ideal right? Well as with all good things, this didn’t last. The biggest drawback of the setup shown here is the variety of technologies, with every vendor bringing their own type of communication media and modules.   The number and variety of spare parts needed to keep your facility safely stocked, exponentially grows with the amount of ICS equipment you own, as well as the technical knowledge a person needs to have to support everything.  Not only does every vendor implement their own proprietary solutions, but within their offerings, vendors can change things up as well. This mere fact along with the increased demand of communications bandwidth as well as ease of use and manageability drove the demand for a common denominator in the communications field. As the ICS vendors were not likely to agree on a single protocol, the next best thing was chosen, a common way to transmit commands over a shared media network, and ethernet was chosen.  Vendors were able to easily adopt their proprietary protocols to the Ethernet networking standard by simply encapsulating the protocol in an Ethernet frame. They were even able to add routing and session, by incorporating IP and TCP. Things started to look up for the overwhelmed support engineer, who now only needed to know how to deal with IP networking. So, everything slowly moved onto the Ethernet bandwagon, and when I say everything, I mean everything. PLCs, HMIs, Sensors, Actuators, Valve blocks, signal lights, … By the mid-2000s a typical plant would look like this from an ICS and business network perspective:

Fantastic! Everything was now connected; anything could talk to anything else and all of it was accessible from the maintenance office or wherever you could hop on the network. The controls engineer in me found this the golden age of automation networking. I remember commissioning a new production line from a rooftop in Los Angeles while connected to the Wireless network we had just installed, what a freedom! But here comes the caveat, that wasn’t enough. Engineers wanted to have business email on their engineering workstation, the plant manager wanted on-the-fly and up-to-date production stats on his PDA while sitting at the beach in Tahiti and plant controllers, quality personnel and six sigma blackbelts wanted to get uptime numbers, production statistics, counts, and downtime tracking and other “Overall Equipment Effectiveness (OEE)” data. So, what was the most natural thing to do? Why not tie the two separate networks together? They both are Ethernet and run IP and TCP and once connected we can access all the data we need. So, this happened:

Or this architecture in some cases:

Most of us control engineers made the mistake of using one of these two “fixes” to the problem. I know I did. But after coming to our senses and forming a bond with our IT department we settled on the following:

This is a pretty decent compromise and seems to cover the functionality and security requirements we are after, or at least we control engineers think so. The reality is that over time that firewall in the middle there becomes Swiss cheese. We find more and more protocols and features we want to allow through the firewall so before long that firewall will look like:

With all these firewall exceptions in place, is it still a firewall? We now have unfettered access from the enterprise network to the industrial network, and we have industrial protocols with no regard for security leaking from the ICS network to the business side. Now we don’t need an attacker to be on premises to attack the ICS system. Any attacker with a foothold into the business network can now do damage. This setup also allows for the pivot attacks we have been seeing more and more often, where any individual in a company who receives phishing emails with malware attached that once opened, allows a remote connection to that individual’s computer from anywhere in the world. This access allows the attacker to pivot their attacks and find a way into the ICS network, which with a Swiss cheese firewall in place is a matter of scanning for the right protocols.

This is how things evolved over time. I am not trying to generalize every ICS owner but in my experience the bulk of them will have some form or shape of the Swiss cheese architecture in place. I have seen companies implement a way more secure setup and I have also seen some implement way less but typically this is what you find out there.

OT is Now Exposed to IT Threats 

The convergence of OT and traditional IT infrastructure into ICS environments has led to easier operational oversight, but it also introduces new avenues for attackers to exploit. The convergence has opened paths for attackers to wander into or target production environments. Furthermore, tools developed for traditional IT environments work on OT networks that have been converted to operate over TCP/IP protocols. Not only can OT devices now be interrogated by attack tools, but their inherent insecurity also opens them up for attacks like Man in the Middle (MitM) and Denial of Service (DoS) that have plagued IT space for decades now. 

ICS has Become a Juicy Target for Attackers

To pile onto the issues is the fact that criminals are starting to catch on to the fact that industrial environments make juicy targets. Where traditional IT is starting to defend against common attacks like MitM, DoS, and ransomware, the OT space is still wide open and susceptible to these kinds of attacks. Compounding this problem is the fact that an OT/ICS operator is more likely to pay the requested ransomware to be able to quickly restart production or prevent a safety or hazardous situation.

The ICS Kill Chain

The ICS kill chain was described by SANS as a way to depict how a typical attack on a production environment is undertaken by the adversary. The full writeup for the ICS kill chain can be found here: https://www.sans.org/white-papers/36297/ , and we will give you a high-level perspective upon further reading. 

Shown in the figure below, a typical (targeted) attack on a production facility starts with an attack on the enterprise. This can be the exploitation of an exposed service (RDP, Web, File share, etc.), a (spear) phishing campaign that lures a victim in opening a malicious attachment or navigating to a compromised website, or it could be some other way for the attacker to gain a foothold into the corporate (Enterprise) network.

Once the attacker has established a foothold into the enterprise network, they will start pivoting around, looking for a way to pivot onto the industrial network. 

Once a pivot into the industrial network has been discovered, the true objective of the attack can start, attacking the production environment. 

The ultimate objective of the attack can be anything from stealing proprietary information (secret recipe, competitive information, etc.) or something more devious like causing equipment failure, with all the dire consequences that go with it.

Attacks on ICS cover IT and OT and so Should Your Security 

As we can see from the previous section, a typical attack on an ICS environment starts on the traditional IT side of the organization (the enterprise network) and works its way down into the industrial environment. With your adversaries attacking both IT and OT in your organization, it is only logical that your security program should cover both IT and OT. Just looking at the security of your OT network can cause you to miss certain aspects of the security posture and leave out testing for and securing of all potential vectors/avenues of attack on your ICS environment.

GuidePoint Offers Holistic ICS Cybersecurity Services

All of this to say, I’m delighted to let you know that GuidePoint Security’s ICS Security Services is launching our ICS Penetration Testing and Security Program Review.  Our ICS Security Services combine our expertise in ICS Penetration Testing and Governance, Risk, and Compliance to deliver a holistic view of the security posture of your environment. We make it possible to secure IT/OT networks without disrupting operations or risking non-compliance.  Our solution allows complete visibility of network control traffic and helps establish the right security policies that will protect your processes, people and profit and significantly reduce security vulnerabilities and incidents. With GuidePoint Security, bridge the IT and OT gap with ICS Penetration Testing, Security Program Review, and Security Architecture Review that will bring together a holistic view of your ICS security posture.