Podcast: Play in new window | Download (Duration: 35:42 — 32.7MB) | Embed
In this Spotlight Podcast,
In this Spotlight episode of the Security Ledger podcast, I interview Chris Walcutt of DirectDefense about the rising cyber threats facing operational technology (OT). Chris and I talk about how organizations that manage OT – including critical infrastructure owners – are being targeted by sophisticated cyber actors and the strategies best suited to manage increased cyber risks to OT environments.
[Video Podcast] | [MP3] | [Transcript]
Cyber attacks on critical infrastructure have gone, in the past two decades from the hypothetical, to the actual, to the epidemic. Today, malicious actors from cybercriminal ransomware gangs to nation-state affiliated hacking groups are teeing up vulnerable operational technology (OT) environments. As CISA noted in a February Advisory about Chinese infiltration of critical infrastructure providers, the goal of many of these groups is long term persistence and – eventually – disruption of critical functions such as power distribution at a time of their choosing.
How should companies respond to the increasing risks to OT systems and environments? In our latest Spotlight episode of the Security Ledger podcast, I sat down with Christopher Walcutt, Chief Security Officer at DirectDefense, to talk about the changing cybersecurity landscape for critical infrastructure and the challenges (as well as the solutions) that organizations face today.
Chris’s Cybersecurity Journey
Starting his career on a help desk for a Fortune 200 energy firm, Christopher’s path to infosec is a testament to the many unexpected routes leading to cybersecurity expertise. Starting out on a help desk, Chris worked his way up to roles as a system administrator and network engineer, eventually taking the IT helm at a power provider with a portfolio of over 30 North American plants, including three nuclear facilities.
Chris’s time in the industry saw the inception of NERC CIP regulations – the first cybersecurity rules directed at critical infrastructure (with the exception of nuclear facilities). Since then, the dialogue about cybersecurity has evolved from a focus on checking compliance checkboxes to addressing cybersecurity as an existential organizational risk amid mounting threats and attacks. Chris and I dig deep on this paradigm shift, and the growing focus within critical infrastructure sectors on resilience vs. simple compliance.
Addressing the Human Factor in OT Cybersecurity
While OT environments present a number of challenges, many of the most significant risks facing OT environments stem from “layer 8” in other words: “the human factor.” As Chris and I discuss, social engineering attacks are the first step in many sophisticated attacks. Accordingly, Chris stresses the importance of security training for employees that is focused on creating memorable learning experiences. For example: by sharing real-world examples as a part of awareness education, organizations can discuss practical measures they use to bolster defenses against sophisticated cyberattacks, underscoring the nuanced nature of cybersecurity threats which defy mere technical solutions.
Tackling OT Supply Chain Risks
We also talk about the growing specter of supply chain risks in OT and critical infrastructure. Chris talks about the challenges and strategies for managing these risks, highlighting the necessity of thorough vetting, compliance, and due diligence in the selection of technology partners and solutions.
In the end, Chris makes a strong case that securing OT environments requires a multi-faceted approach which encompasses compliance with stricter regulations, as well as more expansive user education, and increased financial support for cybersecurity protections for OT environments.
This is a conversation you don’t want to miss! You can check out the podcast using the player (above). Or, check out a video of our conversation below or on Security Ledger’s YouTube channel.
Video Podcast and Transcript
Video Podcast
Transcript
[00:00:00]
Paul: Okay. Welcome back to another episode of Security Ledger podcast. I’m your host, Paul Roberts. I’m the Editor in Chief at Security Ledger. And we are very pleased today to have with us Chris Walcutt, who is the Chief Security Officer at DirectDefense. Chris, welcome.
Christopher: Oh, thank you for having me, Paul.
Paul: I think it’s our first time having you on the show and I’m thrilled to have you. For listeners, viewers who aren’t familiar with DirectDefense, could you just give us an idea of what DirectDefense does?
Christopher: Absolutely. we are a full service cyber security organization. we have pillars that support penetration and application security testing. We are an MSSP, we do some compliance work. And we have a specialty practice called Connected Systems, which is focused around the [00:01:00] OT SCADA IoT world and hardware and device testing.
Paul: Again, you’re chief security officer. You’re kind of, You know, there’s so many interesting journeys and paths to cyber security. What is your background and how did you find your, your path to the cyber security industry?
Christopher: So, interestingly enough, and probably like some of our compatriots, I started on a helpdesk. So, I started on the Energy Commodities Trade and Helpdesk for a Fortune 200 energy firm. Very fast paced. No room for downtime. Everything was focused on uptime, you know, trading losses could total into the millions of dollars per minute.
Christopher: but that was a great place to learn because I went from there.
Paul: High stakes there. High
Christopher: yeah, but I learned, you know, I learned sysadmin. I learned application packaging. I learned network engineering. And then I was interested in network. And so I took that path, network engineering, network architecture, and then somewhere along the way through a variety of projects, I ended up as the head of it for power generation.
Christopher: So we had, more than 30 plants around North America, including three nuclear [00:02:00] plants. And was in that role when the NERC CIP regulations came out, which was the first real foray into cybersecurity regulation for critical infrastructure in the U. S. So that was the 2006 era.
Christopher: In that role, I was helping to put together policy and procedure for cybersecurity for these, plants and helping to try to put a security wrapper around old infrastructure. I went from there to Sundart, spent some time there helping run threat and vulnerability management, you know, understanding more in the vulnerability space and then some breach work. And then spent seven years at Black & Veatch, ~Veatch, which is a hundred year old engineering firm that builds critical infrastructure, uh, and the security risk and compliance practice. Uh, and helping out in a WDC, so role,~ and then I’ve been with DirectDefense for almost seven years,
Paul: I mean, a really interesting journey. And you were sort of there when, like you said, when, when NERC CIP came through prior to that, there wasn’t much of a conversation in critical infrastructure around cybersecurity. It was mostly kind of companies were left to their own devices, more or less. Yeah,
Christopher: Except the nuclear world. But yes, yes,
Paul: Right. How did you see the conversation evolve around cyber risk, from, hey, you know, [00:03:00] NERC CIP, we just, we got to kind of check our boxes and be compliant to, oh, you know, actually, no, this is an existential risk for us as an organization.
Christopher: You know, I think that migration really kind of flowed from a need for resilience. And so the focus on resilience originally was critical infrastructure systems. And this is also true of manufacturing, because they use the same type of. products for control under the under the hood, really was focused around uptime.
Christopher: So a programmable logic controller that might start or stop a breaker in the power grid or might turn her on or off a combustion turbine in a power plant is the same one that might start or stop a conveyor or an oven or a metal press and manufacturing, and they’re built to withstand heat and vibration and last 20 years.
Christopher: There is no component in the I. T. world that’s really built to design that way. They all sort of have this 3 to 5, maybe 7 year at a stretch life cycle. And so the focus went from we have this need for resilience and uptime. How do [00:04:00] we move into being able to protect these things? With an understanding that we can’t do it the way we would normally, you can’t put antivirus on a PLC, you can’t use normal monitoring tools because the devices in those environments don’t use the same communications protocols.
Christopher: Those monitoring platforms, even the best of them, that have made an effort to push into that space, That’s not really something natively that they’re going to understand. So in the very early days, there were a few companies that made a push into the space from the vendor side. I’m thinking industrial defender, which then got sold and resold a couple of times around.
Christopher: And originally they were product and then they were service and. but what has happened over the years is there’s been more of a spotlight. And so it started with an experiment that was done at Idaho National Labs in 2008 called the Aurora experiment. If you haven’t seen it, there’s a fun video on YouTube.
Christopher: I’m sure you have. Yeah, famous, right?
Paul: Shows this generator kind of basically shaking itself apart. Yeah. It’s
Christopher: Self destructive, right? Because some commands that were normal commands, but sort of [00:05:00] out of sequence were sent through to it remotely, uh, by some people at I N L. that started the awareness. Is that the nurse ship? And frankly, you even go back a little bit further in the resilience discussions to the blackout in 2003. And that got everybody thinking because that impacted such a large percentage of the population, right? Northeast. That everybody started to go, Oh, well, what if somebody did this on purpose? You know, what would that look like?
Paul: ~I had just started the cyber security beat in September of 2002. So I actually actually went home early that day because I was working in downtown Boston and we were affected by that blackout. And I do remember my editors being like, is there a cyber angle here? Is this a cyber attack? And, you know, there were some stories in the wake of that of like, Oh, you know, maybe, you know, I.~
Paul: ~I can’t remember the exact details, but there were these, you know, substations and stuff that were having issues around availability with some of their systems. And there was this question of like, was this, you know, but yes, that there was enough going on in the cyber realm that people started to kind of connect the dots and be like, Hmm, you know, down the road, this could be an issue and the Aurora experiment really made real for people this notion of cyber physical risk, which at that point was really not something people talked a lot about, right?~
Paul: ~That some software based attack could have physical consequences on machinery or yeah,~
Christopher: The very first SANS SCADA security conference was in Orlando in 2006. that’s where a lot of these discussions started. NERC CIP had just been put in place. And we had been talking about it for a couple of years. I had, like 12 or 14 power plants that were in the western portion of the US that fell under our regulatory audit body called WECC.
Christopher: And we had volunteered to be sort of the guinea pig for them and go early and pre audit. And the head of that group at that time was a gentleman named Patrick Miller who’s, you know, still a friend and actually spoke to him the other day. So, we still talk about these things, right?~ ~
Christopher: ~And that’s kind of how my journey went. I went straight from IT to moving into the OT sort of industrial control space, and then security sort of naturally followed, and I was interested in it, so I took that path.~
Paul: So one of the things that DirectDefense does, you [00:06:00] mentioned you’re an MSSP, but you also do security assessments of your customers or clients, pen testing, red teaming, that type of thing. so give me a sense, without naming names, but what are you seeing, both in terms of security posture of some of these, infrastructure owners, and then on the MSSP side, what are we seeing in terms of threats, attacks, you know, activity? Is it more of the same or are we seeing real changes just in the, you know, quantity and quality of, of threats out there facing these types of organizations?
Christopher: The assessment work probably opens a bit of a Pandora’s box, depending on the industry. Right. So the energy sector has had regulation in place for a long time. Some of the other sectors have made pushes in that direction. Water with the America’s Water Infrastructure Act and the EPA regulation. Homeland Security through CISA and TSA has made pushes into transportation and pipeline.
Christopher: And I know we’re going to talk about some of that stuff. But, for the most part, a lot of these organizations are still [00:07:00] focused around this kind of Hard exterior, hard and crunchy on the outside, soft and chewy in the middle, right? They’re focused on uptime resilience and allowing data flow where it needs to Within that contained space and the contained space is you know It could be a substation or a power plant or a manufacturing facility and that assessment work a lot of times is driven around trying to help them push towards visibility Because there are now visibility tools that function in the space, that allow the system owners to do more with understanding what’s going on.
Christopher: Establish that baseline so that they know when something’s different, they know when something’s potentially wrong. The newer tools that have come out understand the protocols and allow for just a deeper level of intelligence. And then add to that, the part of the cross sell for the, for the software vendors is that those tools are also allowing for inventory and maintenance improvements and things that have always been a challenge, right?
Christopher: Those have always been stuck in the world of, well, we’ve got an Excel spreadsheet. So the clients are really [00:08:00] focused a lot of times on network segmentation because the networks are not set up properly to allow for the visibility because we still need basically to be able to have inspection choke points for the traffic.
Christopher: We’re still in a world of span and tap where we can’t go down the path of I’m going to put an agent on this thing. You’re not going to do that with a PLC. There’s not a lot of Windows and Linux. Operating system computers in those environments. It’s a lot of real time operating system stuff.
Paul: And that’s been an issue for, for a long time. a couple decades, right? Which
Paul: is a sort of the, the IT, the IT security community wants to come in and talk to you about monitoring and detection and so on. But OT folks are like, you don’t really speak our language. Like you’re not, you can’t manage any of these protocols that Our equipment uses and your stuff was designed to work on totally different type of device, and it’s going to completely brick this device.
Paul: So,
Paul: and
Paul: the end of the conversation really, basically, although, you know, there are obviously companies now that are, that do specialize around and we know [00:09:00] who they are, but you
Christopher: yeah, there are some and we’ve partnered with some of them. we try to remain vendor agnostic, but the partnerships have allowed us to have access to training and better understanding of how the tools work. So when we have clients that have one in place or they’re considering one of a couple, we can help them make informed decisions.
Christopher: And in some cases, maybe even consider changes in, in architecture that would allow them to work better. And that’s also why, The connected systems team at DirectDefense has built the way it is. So, you know, a lot of organizations would take an information security Practitioner, you know, white hat penetration tester and give them training in the OT space, or maybe it’s a fun side project.
Christopher: And they, they go and buy some IOT project products for home and break them and kind of see how that all works. Everyone in that practice was a practitioner. So we had people who have worked for energy and water utilities, people who’ve been responsible directly for manufacturing people who’ve worked for, software companies that make IOT devices work.
Christopher: So when they come into that [00:10:00] environment. There’s an easier conversation and a better level of trust off the bat that that and not necessarily the corporate side of the customer might engage with us directly, but the people that are running the power plants or the substation, the power grid, the water and wastewater, the gas pipeline, the manufacturing plant.
Christopher: When you speak their language and they understand you’ve done their job They know that you get how important the uptime is to them And it focuses that discussion They also trust a little bit more when you tell them that there are solutions that might
Paul: yeah, so getting things, I mean, do you think it impacts actually security outcomes to have folks who, who, again, speak the language, have, have history, like, you know, do you feel
Christopher: It absolutely does because I think there’s always a fear that when a third party comes in that they’re there for their own agenda to make money, to find more business, to continue to keep you on the hook to do things. if you have been the person that gets the call at 2 o’clock in the morning because the conveyor system is down, or because this power plant tripped offline, or because [00:11:00] the water SCADA system is not responding.
Christopher: It’s a shared pain, right? It’s just like anybody who’s worked in a NOC or a SOC who’s had a night shift. Anybody who’s had a help desk who’s gotten the, you know, the page or the call in the middle of the night. When you’ve been in their shoes, there’s that understanding of, okay, you’ve been where I am.
Christopher: You really understand. What I’m concerned about and theoretically, you’re not just gonna advise me to do something that doesn’t make sense for the system. We’re trying to protect ~A lot of MSPs are traditional infrastructure. Put endpoint agents on servers and workstations, monitor the firewalls, monitor the other tools that are out there in the environment, and focus on, okay, this went from green light to red light, or something’s out of normal baseline for traffic.~
Christopher: ~Is this a threat actor? Is this unplanned maintenance? In the OT space, it’s not that simple because a lot of MSSPs are empowered to take direct action. So let’s say a virus pops up on a workstation in an IT space. The MSSP can quarantine that system and take it offline and then maybe do some investigation.~
Christopher: ~And by the time they’re having a conversation with the client, the client understands, Okay, this person clicked on the link in an email they shouldn’t have. And now we’re in a position where we have to deal with cleaning the machine. In the OT space, you get a PLC, let’s say with an unplanned firmware change in the middle of the night. You have to understand what that means. So you have to know that if that is something nefarious going on, that that could impact, it could take that section of the power grid down or shut that power plant down. If it is what I like to call change management gone wild, you know, when you just don’t have something planned properly, or maybe it was planned, but not in documentation, then that points to a potential resilience gap that not everyone in the organization who should have known about it in advance. the reason that’s important for the OT system owner for that, that critical infrastructure, and I’ll use that term even with manufacturing, because it’s their business critical infrastructure, if that critical infrastructure system. is being maintained in a way that isn’t aligned with all the security monitoring going on, then that makes it harder for the organization monitoring and the software to understand when there’s an actual breach.~
Paul: ~mm ~
Christopher: ~Or potential signs of someone trying to get to that point. So that’s a really important message to communicate.~
Paul: So we’ve had some really scary, dire sounding warnings in, in recent years. Months, from CISA and others around cyber threats, critical infrastructure. I know CISA in February put out an alert specific to PRC, China’s government, getting its hooks into U. S. critical infrastructure with an eye to potentially, you know, very disruptive attacks, presumably as part of some.
Paul: Future conflict, whether it’s around Taiwan or whatever else. so a couple [00:12:00] questions. what are your thoughts on that? Have, has DirectDefense seen evidence of those types of sophisticated nation state campaigns, amongst your critical infrastructure customers and, What is the practical impact of an alert like that for just in your experience working within this sector? So as soon as it puts out an alert like that, does it have an impact? Does it actually change kind of behaviors and, uh, on the ground or is it more just, you know, we’re just covering our butt here.
Paul: You can’t say we didn’t warn you. Um, and, and, and that’s basically the point.
Christopher: You know, I have a lot of respect for what CISO is trying to do. I think that part of the challenge is that they are putting the message out into the world and the people in the organizations. that are really focused on what they’re saying and taking the best information they can from all their data sources, including what’s put out in those threat briefs and intelligence and things like that.
Christopher: Those are the people in the organizations that are already refocused that way. Or the ones that got burned, the ones that have been breached, [00:13:00] getting the message out to the, let’s say the other 80, 85 percent who either don’t feel they have the budget to do anything, don’t feel they’ve got the, the leadership on board to support what they’re trying to accomplish, or maybe just feel that they can fly under the radar.
Christopher: That’s the real challenge. How do you get the message to the organizations and to the people that
Paul: right.
Christopher: You know, they’re still focused on their day job. They’re focused on their resilience. you know, we talked about the water space a lot. There was a, press release that came out, uh, and a CISA alert that came out earlier in the year about targeting of specific PLCs.
Christopher: there was a water utility in Pennsylvania, and that one was pseudo state sponsored, but they were targeting specific companies that had ties to a government because of some political turmoil.
Paul: I remember it. Yes, the, the hardware maker was linked to a non US government entity that, that is,
Christopher: And so this was this, this was a state sponsored attack
Paul: we’ll let you guess who that country was, [00:14:00] but, uh, open your, open your news browser and you’ll figure it out. Anyway, go But, but, but the focus there, is that. The water sector in particular is one that has talked about cybersecurity for a long time. I sat in a meeting a few years ago with some engineering firms and a bunch of water utilities and the EPA administrators, not long after OWEA was signed, the America’s Water Infrastructure Act, and, the goal of the water utilities, the larger ones were focused on protecting themselves.
Christopher: The smaller ones were somewhat focused, not entirely, but at least somewhat focused on, protecting themselves. Resilience, but their answer is always we can run the system manually. We can just disconnect. The only things that they’re required to provide anybody is they have to put water quality reports up, usually to a state level entity.
Christopher: And sometimes that’s multiple times a day, but that can be done manually.
Christopher: And then if they are owned by municipal government, then they have to put some sort of operational, you know, we’re not wasting money type data out on a website, maybe once a month.
Christopher: [00:15:00] And so they were comfortable to say, we could just disconnect and operate manually.
Christopher: the other major difference is that if we compare energy
Paul: We think
Christopher: we think if we compare energy and water sectors in the U. S. right. The energy sector is mostly owned by, merchant energy companies that are for profit organizations, a lot of them are publicly traded. Because of that, there’s a much larger pool of money for them to do things in the cybersecurity area.
Christopher: Most of the water utilities are small, municipally owned, and their funding comes directly out of the taxpayer’s pocket, and really no place else except maybe for some grants. And so they don’t have the same pool of resources. So that’s been part of the challenge, especially for the medium and smaller ones in that space.
Christopher: Thanks.
Paul: Yeah. I’m on my community’s local IT advisory committee. We’ve got a locally owned electric utility, and so I have seen this up close, which is, it all comes down to budget and staffing and, uh, you know, literally the job of the water utility is to keep the water flowing. Job of the electric utility is to keep the lights on and [00:16:00] some of these kind of fuzzy.
Paul: You know, cyber threat conversations, you know, just, it’s, it’s hard to, get them fired up about them because, you know, again, budget and staffing are limited and, and that, that talent is, I mean, this is kind of where companies like DirectDefense come in, right? That cyber talent is very expensive and hard to come by.
Paul: So, unless you’re going with, uh,
Christopher: It’s an unusual skill set for sure, uh, especially in this OTS data space. and you know, so then part of my responsibility that I take personally is, is evangelism in this space. So I’ve been involved with the NERC CIP committees for years and years. I still participate with the CIP 13 supply chain working group.
Christopher: I participate with the American Water Works Association, which is drinking water only, but they’re the main sort of lobbying group around some of this. And they have a cybersecurity director, Dr. Kevin Morley, Who’s constantly trying to work with the EPA to further the cause of cybersecurity resilience in the water space. And so, one of my [00:17:00] colleagues and I were involved in their, the rewrite of their J 100 standard, which is a best practice guide because there is no regulation to enforce it, but it is a best practice guide for cybersecurity that roughly follows the NIST standards. And so there have been pushes in little pockets in the private sector to kind of move in this direction, but that’s why we take part in organizations like that.
Christopher: I do webinars for the American Council of Engineering Companies and the Engineering Change Lab because the engineering firms really play a critical role. If I wanted to attack critical infrastructure and utility critical infrastructure in this country, I wouldn’t go after any one utility because no one can take down any utility.
Christopher: More than a small region of anything. But if you go after some of the engineering firms that have done the integration work that have maybe remote access into some of these systems. And so I also make a point of
Paul: it’s that supply chain, that supply
Christopher: It’s absolutely,
Paul: Yeah, really interesting. so, DirectDefense has written up, you, you’ve done blogging and stuff [00:18:00] about some of the work that you do with your customers, and especially with some of your, you know, pen testing and red teaming engagements talked about, the risk posed by, you know, You know, just social engineering, attacks as a, 1st stage in, in what might be a more sophisticated operation.
Paul: 60 minutes just did a story looking at the, MGM casino hack, out in Vegas. actually, it wasn’t limited to Vegas that started with a pretty targeted social engineering, uh, attack on a, on a, uh, Privilege, you know, MGM, admin, this is just such a, such a fuzzy problem. But do you see as the fix for that for a critical infrastructure owner, given that, you know, every, every organization has a layer 8 problem, you know, humans are humans, it’s hard to.
Paul: Get them to be fully defended against a clever, sophisticated social engineering attack. How do you deal with a squishy problem like that? Cause it’s not solely a technology problem.
Christopher: I think part of that is personalizing it for them. ~So you can send the phishing emails and you can have the conversations and you can give them a little video to watch and a test to take, ~if [00:19:00] you get them some content like yours, or if you get them an opportunity to sit down and listen to, Content like mine, right?
Christopher: Where we can really relate to you. You know, we came in one day that night, we physically broke into this small utility. we found a work order for some networking work for the next day from a vendor. We made a photocopy of it. We went to the front desk, we stole some visitor badges. We came back the next morning with hard hats on and those badges and our paperwork and a box of donuts.
Christopher: And they let us write in. And so, and this true story, right?
Christopher: The donuts were key there.
Christopher: Yeah, they were. We had another one of our testers that, found a job listing and applied for the job, was offered an interview, went in person to the interview, and they left him alone to fill out the paperwork. He plugged the device into the network as part of the penetration test, right?
Christopher: So, the human element, when you can personalize it, it allows people to say, You know, that’s a funny story. They’re going to remember it a lot better than, than your video that you got from a vendor. They’re going to remember it [00:20:00] better than the email that they didn’t really read that they clicked through.
Christopher: and then when they see something weird, like somebody trying to tailgate into the building that they don’t know, Let me actually say something.
Paul: ~Or setting up in an office and kind of looking busy and like, what are you doing here? Yeah. ~
Christopher: ~we occasionally have somebody who actually catches that stuff and then comes back and says, Hey, you know, what are you actually doing here? Let me see your badge or credentials or whatever. Sometimes someone pays attention and it’s a lot of time. It’s just a random employee. But if you can get them to personalize it in their brain, then it sticks with them.~
Christopher: ~And ~that method of education I think is one of the best ones, is that personalization. I wanted to go back, you asked the last question, had a couple of parts to it. We didn’t get to one or two of them. Sophisticated threat actors on the state sponsored side. We have recently worked breaches.
Christopher: We do breach work both on the I T and the O T side, and we worked to breach a few months ago for a larger client it absolutely appeared to be state sponsored threat actor, and they were very sophisticated. We were working alongside a couple of other larger firms together, and the mechanisms they used were frankly impressive.
Christopher: they sent an actual person. With a fake ID and enough information into a cellular phone store in a big U. S. city. And Sim swapped an IT admin for that organization. And the IT admin [00:21:00] figured it out pretty quick and swapped it back. So, the same person went into a second store location in the same city that same day and swapped it again.
Christopher: They are getting pretty brazen. more sophisticated threat actors are willing to go to quite a length to get what they want to get.
Paul: They’ve got operatives on the ground, which we generally don’t assume. I mean, we know the Chinese and Russians and so on have spies in this country, but we don’t think about them in the context of that type of attack.
Christopher: And in this particular case, they then contacted the help desk. And they got the help desk to break one of their own protocols through some social engineering. But then there also was something of a broken process in the midst there, right? So, in my personal opinion, and we also do virtual CISO services.
Christopher: I’m virtual CISO for a couple of our clients. Your help desk level one staff should not be able to reset admin credentials. you want to reset admin credentials, get someone who knows the person, get them on a video call with that person and then reset their credentials.
Christopher: Just some basic protocols, some basic [00:22:00] cyber hygiene stuff.
Christopher: Those are the kinds of things that don’t cost any money. But if you make the mistake, then they cost you your, your
Paul: Right. Making sure all your employees or your valued or highly privileged employees have, pins set up on their, mobile phones. Right. So you can’t just SIM swap, right.
Christopher: Yeah. And exactly. And so one of the, one of the things that we helped them with in the aftermath was send a message to all your employees. This should be part of your annual training. There is swap prevention offered by all the major cell phone carriers. Go down that path. Take that extra step.
Paul: So, one of the things that I think you know, we’re, really starting to wrestle with is, of course, supply chain risk. you know, every organization is addressing. We saw this most recently with, uh, you know, pulse secure Avanti, disclosure where you had, you know, You know, security hardware running an 11 year old open source operating system.
Paul: That was four years end of life, but had encrypted software updates. [00:23:00] So nobody could, wasn’t, it was kind of a black box. It wasn’t easy to, reverse engineer those binaries and figure that out. and ended up getting, you know, having, uh, exploitable vulnerabilities that were taken advantage of by a nation state actor.
Paul: that just encapsulates really nicely, the sort of. What’s in the sausage question that so many organizations are wrestling with? I would assume this is true in the OT space and critical infrastructure space as well, that, that everybody’s sort of, yeah, looking at their, yeah. And saying, gosh, what are we running here?
Paul: Especially some of these systems are quite old. do you see any like initiatives or actions to address that supply chain risk? or is that just kind of. the elephant in the living room, and folks are trying to figure out how to, how to even get around it.
Christopher: My marketing people are going to kill me for this, but it’s one of my favorite tag lines. I coined a phrase for a speaking engagement in 2017, I start out with how well do you think you should know your vendor before you let them into your environment? Do you just let them take you to lunch?
Christopher: Do you maybe [00:24:00] have a couple of dinners? Do you find out anything about them? Because if you’re not careful, you’re going to contract a vendor transmitted disease.
Paul: That’s right,
Christopher: And so we really want to avoid those VTDs. And this, this guides some of the work that I’ve done with
Paul: I’m seeing connections to other areas of our lives. Yeah, right, it’s the same concept,
Paul: right?
Christopher: right?
Christopher: So this, this is, this is the, the concept of supply chain risk really in a nutshell.
Christopher: You do want to know that they have done some security testing. If you’re buying hardware, ask them. There’s a few, there aren’t a lot, but there’s now a few cybersecurity hardware certifications,
Christopher: right? So IEC 62443, which is also ISA 99 is out there.
Christopher: UL 2941 is sort of in the process of being wrapped up. and these are things that we’ve helped some of our clients with, right? We’ve helped, we got the first battery storage system. I think a shipping container full of full of batteries that’s utilized to store energy from like a solar field or a wind farm lights, you know, when [00:25:00] the sun’s up and the wind’s blowing and then it can be discharged onto the grid when those aren’t the case, those certifications can be used to go through and literally say, Okay, you’ve got all these secure components.
Christopher: You’ve done your testing. Now I’ve got this piece of paper. And if I change any component, I have to recertify. So that functions as your S bond, that secure building materials to say, okay, The things that are here should be okay the way they sit as of this date. Now the secure update process, that’s a whole separate thing.
Christopher: And, you know, SolarWinds is probably the most famous of those, right? From that standpoint, But, the risk here is in not understanding, if you’re working with an integrator or you’re working with a vendor that’s sourcing components, make sure you know what components they’re choosing. you know, famously, the U.
Christopher: S. Government went through and said, okay, certain vendors aren’t allowed, or you’re not allowed to buy surveillance equipment, video cameras, right? High vision was one of those or certain vendors. You’re not allowed to purchase and install within U. S. Government facilities. And a lot of the private sector kind of followed suit on
Paul: Mm hmm. Yeah.[00:26:00]
Christopher: that, getting down that path and figuring out, okay, these are vendors that not necessarily are bad, but just haven’t done their due diligence.
Christopher: Okay. That potentially should be something that takes them off your list. I was doing a penetration test for an energy utility in the northeast, years ago, and they had deployed smart meters. And the smart metering network, when it’s deployed, this is 700, 000 smart meters, more than half a state. When they deploy those networks, they get this huge mesh wireless network.
Christopher: Well, there’s extra bandwidth in it. And so they said, we’d like to use this for recloser automation, which is basically turning on and shutting off parts of the power grid. You know, a tree line falls, you need to shut the power off to that section. They used to send a truck out to the substation and shut it off.
Christopher: But with recloser automation, they can do that all remotely. They said, Hey, we’ve got this network here. We just need a secure way to get the reclosers onto the network. We’re going to buy these little security gateways and let them connect. Well, and this is a big piece of advice here. They could have bought two or three of them.
Christopher: This is the vendor that didn’t do any security testing. They could have bought two or three of them and done their [00:27:00] own security testing either internally or paid somebody to vet them, and they didn’t. And so when I found the vulnerability in them that allowed me to get back into the power grid, they then had to go to the vendor.
Christopher: And the vendor said, you know, we don’t do a lot of work in the space. We’re not really concerned about patching that. They had to pick a new device and buy thousands of them and replace them all.
Christopher: Potentially multi million dollar mistake because it wasn’t part of their programmatic approach to say, either we’re going to buy a few and test them, or we’re going to stand it up in a lab when it’s all ready to go and do a complete end to end test before we turn it loose in the wild.
Paul: I’m going to go out on a limb and say maybe that device they picked had the lowest unit cost of the, of the, you know, you know, And in fact, that vendor name stuck in my head and it came up in a conversation with a client the other day and I said, don’t do that.
Christopher: And I told him the story,
Paul: And, and there’s nothing to compel them regulatory, regulation wise to say, you have to fix that. You can’t. And it’s like, well, we don’t want to. Under CIP
Christopher: 13 [00:28:00] organizations working in the energy sector have to, but nothing would force a large manufacturer to do it. We’re helping a large manufacturer, you know, 37 plants globally go through and do their segmentation and put some security visibility tools in place. And we’ve now had that conversation with them.
Christopher: They’re sticking with mostly primary vendor products from a couple of platforms that are, that are known and tested and known good,
Christopher: but that due diligence is important. And one of the messages also is be friendly with your procurement people, because if they understand some of the things you need, you might have to educate them.
Christopher: They’re not, you know, in your space technically, but even if you give them a checklist or just give them a 20 minute, Hey, this is what we do. This is how these things work. This is why this is important. Okay. That, once again, you’re personalizing that little bit of education so that in the back of their heads when they’re running your RFP, they know to add five or six more questions specifically to help get the information you need.
Paul: So, I mean, we’re hearing a lot from the Biden administration, cyber executive order and stuff around things like software bills of [00:29:00] material and just trying to quantify that ingredients list for hardware and software. and critical infrastructure is a big part of that executive order.
Paul: Um, are you seeing that having any impact in terms of acquisition, you know, technology, purchase, acquisition and so on, critical infrastructure owners. or their suppliers sort of pushing back and saying, we want to see the S bomb, you know, we want to be able to verify what’s in it, that type of thing.
Christopher: There is some, knowledge and acceptance of that. And there’s been a push. There’s been a couple of very, very large companies that have said, not only are we going to do this in our critical infrastructure purchases, but we’re going to do it it wide. And when you’re at 10, 15, 20, 30, 000 person company, that has a bigger impact.
Christopher: And so now you’re putting on notice. Some of the big names, right? Some of the biggest software vendors, some of the biggest hardware vendors, these are things that you’re going to require, that you’re going to make sure that the TPM chip within these laptops that I’m going to buy 5,000 of, has been tested, and you can show me a [00:30:00] certification, or you can guarantee through some other mechanism, that And the S bomb reflects that it’s a secure module and it hasn’t been tampered with and all that fun stuff.
Paul: And sometimes it takes out those, those very large vendors of very large customers to kind of set the standard and, and, and the downstream effects for smaller organizations is good. Right? I mean, yeah,
Christopher: Yeah, if they can ride those coattails, it absolutely is beneficial. Yeah.
Paul: Okay. Final question.
Paul: so if you were you know, the OT czar, the critical infrastructure cyber czar, what would your approach be to addressing some of the risks that we’ve talked about, You know, these sophisticated actors, the kind of, you know, legacy, investments and, and, uh, those types of risks.
Paul: what do you think is, uh, the best approach to, to actually raise the bar on, uh, on the security of, of OT and critical infrastructure?
Christopher: So, I see that as a three pronged approach, really. the first is that most of the regulations have no teeth. So, when organizations [00:31:00] get,
Paul: It’s the truth, man.
Christopher: uh, are found in violation of NERC CIP, they can face fines of a million dollars per day per instance of noncompliance. Basically, none of the other regulations that have come out have that kind of a backbone to them for enforcement.
Christopher: So, you know, WEA has been this sort of, it’s in, it’s out, we’re going to enforce it, we’re not. It’s still, you know, EPA around that right now, and I know AWWA and the work we’ve been doing with them is really sort of still focused on that. there have been some things that have come out in the National Security Memorandum that were part of the National Cyber Security Strategy.
Christopher: the Sector 5 entities really being, Pushed down that path. So the implementation of it is the important part where the rubber meets the road. You can put anything you want out on paper. The H s has had guidelines for cyber security for 18 sectors for probably 15 or 20 years, but they were always sort of best effort.
Christopher: You do the best you can. And so the regulatory side, having some actual [00:32:00] real financial burden for the organizations or like what Sarbanes Oxley did, where if you found a To not be upholding your fiduciary responsibility, the CEO or the CFO gets in trouble, you know, that’s part of it. And we’ve seen the SEC reporting guidelines for the publicly traded entities.
Christopher: The second one is that education piece that I’ve talked about a couple times. And what gets to this point is there are not a lot of people that do this OT SCADA security work. And we’re a smaller group and we kind of, a lot of us know each other. Offer free education or low cost education for these system owners and operators that are not in the energy sector, not in the water sector.
Christopher: You know, oil and gas after the pipe, the Colonial Pipeline attack, TSA came out with some regulation. offer the education so they better understand what they should be doing, and how to get there. And then the third thing is most of these organizations don’t have enough money to do this. And so if you want to protect critical infrastructure here, the government may have to pony up some money.
Christopher: There were a bunch, there’s a bunch of grant money offered [00:33:00] through, ARRA in 2008.
Paul: Yeah.
Christopher: It might be time for another era of some of that, because some of this infrastructure is so old that it’s almost impossible to protect. So, and the organizations that own it, if they’re not big for profit entities, they just may not have the funds, the resources.
Paul: I don’t know. 16 years. You think?
Christopher: Yeah, exactly. You know, the fourth, the fourth piece, and it’s kind of part of the education is how do we, how do we grow the cyber workforce? I’ve been part of the Cyber Patriot organization for about 10 years. so that’s the national middle school and high school cyber defense competition. And then the college version is C.
Christopher: C. D. C. And this is one of our best opportunities. for practitioners to get involved as mentors and really help grow the workforce. So, I coached the national championship team in 2016 and two members of that team worked for us, right? They had followed along and they’ve gotten internships and mentorship and the ability to help grow.
Christopher: And then that team still exists and went back to national finals this year. So this is our capability. That’s the other
Christopher: way to plug the [00:34:00] gap.
Paul: Such a huge opportunity. And I’m always surprised that we’re not, you know, working it into technical education programs or really trying to get it down to the grassroots, you know, because it’s sort of like, oh,
Christopher: Stay tuned on that. I may be working on some things in the wing to see what we can do. You know, there’s a thought that the European model of apprenticeship really would be a good fit in certain parts of American culture. And this may be part of, this may
Paul: Absolutely. I totally agree with you. Yeah. And when you’re looking at like SOC operator and so, you know, the kind of entry level, like those are things that you absolutely could have as part of a, you know, nine through
Christopher: I can see it looking over the shoulder of somebody that knows what they’re doing, you know, gaining responsibility over time and particularly when we’re talking about so our MSSP also functions in the OT SCADA, that sort of SCADA SOC space. You have to know that, okay, that’s not a normal function in that space.
Christopher: This is not like it, or the systems are just going to tell you, and then you have to know, okay, I need to call the plant manager or I need to escalate. And in our case, somebody from the connected systems team who understands this [00:35:00] stuff to help make the determination on whether or not it’s really a problem.
Paul: This is a great conversation.
Christopher: Yeah, I appreciate it. I, uh, I enjoy these conversations. This is, I like talking about this stuff.
Paul: You’re, you’re good at it and you have interesting things to say. So I’m, I’m glad you enjoy it as well. Chris, Christopher Walcott, uh, chief security officer, DirectDefense. It’s been great having you on the security ledger podcast. We will definitely have you on again look forward to seeing you in a few weeks out at RSA.
Christopher: very much for your time. I enjoy this. Love to do it again.
Paul: Great. Absolutely. Take Take care, guys.
(*) Disclosure: This Spotlight Podcast was sponsored by DirectDefense. For more information on Security Ledger sponsored content and the various ways in which we work with sponsor organizations, check out our About Security Ledger page on sponsorships and sponsor relations.
