Measuring cybersecurity: The what, why, and how

A core pillar of a mature cyber risk program is the ability to measure, analyze, and report cybersecurity threats and performance. That said, measuring cybersecurity is not easy. On one hand business leaders struggle to understand information risk (because they usually are from a non-cyber background), while on the other, security practitioners get caught up in too much technical detail which ends up confusing, misinforming, or misleading stakeholders.

In an ideal scenario, security practitioners must measure and report cybersecurity in a way that senior executives understand, find useful, satisfy curiosity, and lead to actionable outcomes.

What can be measured in cybersecurity?

Most stakeholders usually have questions around risk, compliance, or assurance. Unfortunately, such questions usually cannot be answered using a single data point. Fortunately, there are a wide range of things that security practitioners can measure in order to address stakeholder questions and concerns. These can be broadly categorized under:

• Controls: Measures that are put in place to counter threats and reduce information risk
• Assets: Any item that is of value or is owned by the organization
• Vulnerabilities: Weaknesses in the system that can be exploited by a threat
• Threat events: Actions initiated by a threat capable of causing harm to assets
• Security incidents: Events that successfully impacted the business in terms of disruption, downtime, system shutdown, data breach, phishing, ransomware etc.

Above categories can further be broken down in terms of numbers, time, or cost. For example, numbers can measure totals and percentages of unpatched servers, ratio of unpatched servers in comparison to the required baseline and capacity, or the number of servers possible to patch. Time can measure the amount of time it took to identify an incident, or the frequency of a particular threat over time. Cost can help measure the impact of an incident in financial terms, the cost of recovery, and the cost of lost business due to downtime.

Why focus on KPIs and not metrics?

Security practitioners must select the most relevant measurements when reporting to business teams. Most security teams focus on metrics, which provide low-level measurements related to assets, vulnerabilities, and threat events. Executive teams, on the other hand, care about key performance indicators (KPIs) and key risk indicators (KRIs) because these can help answer specific questions related to information security risk, health, preparedness, and business priorities:

• Are we secure?
• Are security investments delivering value to the business?
• Are we meeting regulatory obligations from a security perspective?
• What is our preparedness for ransomware attacks or supply chain attacks?

These are the types of questions that KPIs and KRIs help answer and this is why practitioners must be laser-focused on KPIs and KRIs to benchmark their security performance, preparedness, and effectiveness.

How can security teams measure cybersecurity?

Building the right measurement framework is a gradual, iterative process. Let’s explore the five main steps involved in building a security measurement cycle:

1. Define requirements

Engage in a two-way conversation with relevant stakeholders to define and understand their needs. When starting small, stakeholders may not always have a good understanding of information risk or their own requirements at this point, so a more bottom-up approach, where security practitioners measure what they think is important and report upwards, is necessary. Security practitioners can use these conversations to ask probing questions themselves, helping to educate and set the agenda if necessary.

2. Select key indicators  

Once stakeholder requirements have been defined, security practitioners should identify and select the key indicators that would help to support those requirements, all stakeholders must be consulted and informed on the measurements that will be presented at a later stage.

Having sight of key indicators should enable stakeholders to take action or make decisions. These key indicators should be at a high level and few. The goal is to help with decision making, not to overwhelm or confuse people with data.

3. Identify metrics

Having identified high-level goals and indicators, security teams must now focus on identifying lower-level metrics that help report on those indicators. Depending on the exact nature of the indicator, this could involve dozens of metrics being required, from across the various categories of measurement outlined above.

4. Collect and analyze metrics to calculate key indicators

Since requirements are now agreed upon, key indicators are selected and metrics are identified, practitioners can now begin collecting and analyzing data based on those key indicators. Metrics must only be derived using data that is accurate, timely, relevant, and trustworthy. Otherwise, the business can make the wrong decisions with serious consequences on the organization’s security posture. Security teams must find ways to collect this data on a continuous basis (most measurements will require a view of trends over time) and preferably make the process as automated as possible (manual process can be tiring and time-consuming).

5. Report key indicators to stakeholders 

Key indicators must be reported to decision makers in a timely manner. Security practitioners and stakeholders should agree on a cadence: How regularly does reporting need to happen? Reporting style must also be agreed upon as different methods suit different stakeholders: Are dashboards required, or would slide presentations do the job? Key indicators should be clearly visible and easily understandable. In the end, reporting should lead to decisions or action.

Finally, after each reporting cycle, it is important to review key indicators and revalidate them with stakeholders. Security teams and stakeholders must ask, do the reported indicators still provide value or does something need to change? If business requirements have indeed changed, then practitioners must again go back to defining requirements and analyzing a different set of indicators and metrics.

Don’t forget, the threat landscape is always evolving and therefore security must also evolve in lock step. Organizations, stakeholders, and security practitioners should not be afraid of going backwards or forwards. The ability to fail fast, move on and improvise or repurpose is critical to achieving success in measuring cybersecurity.