The new math of cybersecurity value

Jenai Marinkovic doesn’t put much stock into figures that show how many attacks she and her security team have stopped.

Those numbers, she says, really don’t provide any insights.

“Saying we blocked a million doesn’t tell us anything. It doesn’t communicate enough to other executives,” says Marinkovic, who provides virtual CISO services through Tiro Security and serves on the Emerging Trends Working Group with the IT governance association ISACA.

Marinkovic says CISOs instead need to find metrics that provide actionable information that they and the other enterprise leaders can then use to make decisions.

“They should be figures that help the business,” she says, adding that CISOs need to calculate how much they’re impacting the business, how much they’re getting for returns on their investments, and whether and by what degree they’re improving their security posture.

Finding ways to do all that, however, has been a longstanding challenge for security chiefs.

For Marinkovic, that means calculating average time to notification of a breach and average time to containment—both operational metrics that are good for sharing with the board of directors, too.

But, she says, they still don’t give anywhere near a complete picture. They don’t indicate the maturity of the security function nor how well security controls are aligned to strategic objectives. To do that, “you’re probably not going to give a quantitative metric, but a qualitative one.”

In search of something better

Security leaders offer varying opinions on how to quantify how well they’re doing. Yet they all seem to agree that no one metric can capture the full value of the cybersecurity program. They stress that there’s no mathematical equation that can truly measure its effectiveness.

At the same time, however, they acknowledge that there’s pressure to do better.

“There are CISOs who don’t have anything—no metrics, no way to quantify—and they’re aware it’s a problem. Or they have metrics, but they’re awful and they want something better,” says Jeff Pollard, vice president and principal analyst with Forrester, a research firm. “They want a way to know about the effectiveness of their cybersecurity program, to know whether they’re better off than they were.”

Security leaders are tackling this problem, Pollard and others say, by collecting more data that they can turn into a collection of measurements that give them more insights into how well they’re doing, whether they’re improving, and where risks remain. And while it’s not a singular score of success, these metrics are proving useful for CISOs, their executive colleagues, and board members as they assess their security postures and what to do next.

“It’s about creating metrics that let you make decisions,” Pollard says. “You should have a set of metrics that you can share with the boards and that the board and other stakeholders can use to make decisions.”

The challenge of assigning value

CISOs have had a lot to overcome in devising quantifications for their efforts.

To start, they’ve had to gather much of the information they need to evaluate their programs by manually compiling data from multiple disparate sources. (It’s a challenge that remains in play today, too.)

And they’re trying to measure the results of many complex processes and tools, using data that makes no sense out of context. For example, business executives everywhere understand $1 million in revenue, but they’re hard pressed to say whether thwarting 1 million attempted hacks says something positive or negative—or anything at all.

“Think about financial statements: You can bring everyone together and they can take data from different sources and aggregate it and everyone gets it. But there’s no one statement that can be presented to the board around cyber and cyber risks; there’s no single risk scoring engine that can aggregate it into a view everyone understands,” says John Gelinne, a director in Cyber Risk Services for Deloitte Advisory.

He adds: “That’s a challenge. How do you aggregate and bring that information together so everyone gets it?”

Even if they could do that, CISOs have traditionally struggled with assigning value to nonevents. “Coming up with a metric for [the fact] that nothing bad happened and then saying to the board, ‘You should give me more money to make sure nothing bad happens, that’s a hard sell,” adds Tim Rawlins, who as senior advisor and director of security for NCC Group provides risk management, resilience, and strategic advice to the firm’s board.

That, though, is changing.

“They’re coming up with metrics to tie to numbers to show how something made them safer from more harm or risks, how something has saved [the organization] from having to explain to clients and customers that there was a data breach,” Rawlins says. “It’s hard, but leading CISOs are looking at ‘cyber as a science’ and finding methods and metrics with valuations that are repeatable and reproduceable to show trends and for benchmarking.”

Focus on enabling decision-making

Developing the right mix of metrics matters for a few reasons, as is the case with all business metrics. It gives CISOs and others a measure of effectiveness, and it offers insight into whether improvements are happening. But, perhaps even more critically, it enables good decision-making.

As Pollard explains, the only metrics that should be used are those that lead to decisions.

“We’re always looking at information and making decisions, that’s why security leaders need great metrics,” he adds. “If your metrics don’t allow you to do that, then you know they’re not worthwhile. So then you need to create metrics that let you make decisions.”

Borrowing from the principles of conventional business measures, Pollard says those metrics could be lagging indicators, coincident indicators, or leading indicators.

In fact, Pollard notes that he has seen some metrics used as lagging indicators by some CISOs but as leading indicators by others; that’s fine, he says, as long as they work for each individual organization.

Case in point: metrics around insider threat risks. Some CISOs use employee churn and retention rates as a leading indicator of insider threat risks, as departing employees often try to take company information as they leave despite policies forbidding such actions; but tracking insider threat risks could be a lagging indicator if CISOs have been working to tighten employee access as part of an ongoing security initiative.

Either way, Pollard says, the such a metric can help CISOs make decisions about what actions to take—for example, reclassifying roles and reducing permissions to limit access to sensitive data or adding user behavior analytics (UBA) software.

Furthermore, Pollard says, CISOs should develop a set of metrics that they share with the C-suite and the board and another set of operational/tactical metrics that the security function can use internally.

Developing cost-based metrics

CISOs, working with other enterprise executives, need to identify the organization’s assets and likely risks—work many have already done—but next determine the costs associated with security events, Gelinne says.

Deloitte talks about those as being either “above the surface” (and thus better known) incident costs and “beneath the surface” (or hidden or less visible) costs.

In its 2020 report, Beneath the surface of a cyberattack: A deeper look at business impacts, Deloitte lists costs associated with technical investigations, citizen or customer breach notifications, and attorney fees as some-above-the surface costs. It lists insurance premium increases, increased cost to raise debt, devaluation of trade name, and loss of intellectual property as some of the less visible costs.

“Those are harder to quantify if something bad happens, but if you can quantify all that, then you can understand the value of the controls in place and where you need to focus and invest. You can identify what controls you should be investing in and then you can extrapolate returns. That’s the business case side of that [quantification work],” Gelinne says.

Consider the value of this approach for companies considering a merger or acquisition, he says. The CISO can act as a full partner in the company’s strategy by applying metrics to the proposed M&A activities and delivering figures on what risks the deal presents to the company and what it will cost to mitigate those risks. Such security metrics can inform and shape both negotiations and the deal as a whole.

Gelinne adds: “It’s about getting to the point of knowing where to put the investments where they matter most. That’s the greatest value we see in quantifying risk, because it’s grounded in real metrics and it’s defendable, and even where you don’t have data, you can have [reasonable] assumptions.

Tailoring data decisions

Because metrics must meet individual enterprise needs, experts say CISOs have to consider what they need to measure, what data they’ll need for calculations, and how they’ll use that information to make decisions.

Those metrics should be transparent and thus understandable by all stakeholders, Gelinne notes.

He said they also should address what the organization cares about: The board wants to know if the company is investing in the right security capabilities to protect the organization, the CFO wants to know if there’s adequate cyber insurance coverage, and the chief risk officer wants to see risk reductions over time.

Some CISOs successfully devise metrics for areas such as ransomware risk reduction and operational resiliency, drawing on the NIST framework to determine what data to collect and use for calculations and how those figures can show improvements—or declines, if that’s the case—over time, experts say.

Some security leaders are also using the NIST or MITRE frameworks as a way to measure maturity and to set goals for becoming more mature; they’re also using those assessments to benchmark against others.

“We see those as being very effective, because they give CISOs a baseline and a roadmap for where they have to go,” says Dave Cronin, Vice President and Head of Cyber Strategy at Capgemini Americas. “It’s proving your controls are working, and it shows effectiveness to the board.”