SBN

How to Measure Threat Detection Quality for an Organization?

Sometimes I write blog posts with answers. In other cases, I write blog posts with questions. This particular blog post covers a topic where I feel I am in the “discovering questions” phase. In other words, don’t expect answers — but also don’t expect questions…

So, in recent weeks, I had a few simultaneous conversations with various people that focused on the quality of threat detection. Here I’m talking about the quality of the entire detection capability of an organization. A macro-level detection quality, if you would. Some others framed it as the strength of the detection team, but I think that the focus on the team alone is not sufficient.

How do we discover the right questions? Perhaps we can try this:

  • Who has good detection capabilities?
  • What does it mean to have “good detection capability”?
  • How good detection quality manifests in their security posture?
  • How do you know that they have it?
  • How do they themselves know that they do?

Ok, so I promised no questions, but I just typed some questions. To me, these are questions about what questions we need to be asking …

Let’s explore some of the likely dead ends. Why explore dead ends? Simply because these reflect some of my initial thinking about this that I explored to their logical end and then discarded …

First, I think it is a little easier — yet still difficult — to measure detection quality at a micro-level. Is this a good detection rule? Do I have good ATT&CK coverage in this area? Do my SIEM rules work well here? Does my EDR cover the holes of my SIEM detection posture? These are very useful, yet to me they don’t aggregate to an organizational detection quality. At least, I don’t think they do for now.

Obviously, you can claim that good detection means you are not breached. This is obviously silly — you may not even know that you are breached (because your detection is bad!). You may not be breached because you are lucky; or there may be a million other explanations that have nothing to do with detection strength.

You can say that good detection indicates that you always catch the attacker before they accomplish their goals. To me, this is a variant of the loss reduction argument (“it is not about not being breached, it is about preventing the loss”). Your preventative controls have failed, the attacker got in, but then you detected them before they accomplished their goal. In the old days (eh … 2010?) we called it interrupting their kill chain. There may be something at the end of this line of thinking, just not sure what yet.

Now, please don’t say MTTD. There is nothing substantially wrong with it, however, to me this is very much a micro-level metric, not a measure of the entire capability of an organization. You can have low MTTD for some things, yet miss others altogether.

Also, some of you may correctly point out that some recent discussions about SOC metrics relate to this as well. To me, the quality of thread detection is strongly correlated with SOC quality, there is perhaps a Venn diagram here.

There is probably a detection coverage angle here as well. This line of thinking will produce effective answers, but IMHO they would also be a component of what we’re measuring, not the whole thing. After all, you may have good data collection coverage, good detection tool coverage, but also low signal quality or low triage quality. You may have got good coverage and bad detection.

There is also some kind of scaling question in there. It’s almost as if I want to measure detection quality for each human hired into the detection team or, even more broadly, for each dollar spent. Ultimately, I suspect an organization can have robust detection capability that also costs a lot.

What’s next? Well, we need to get the right questions together so that we know what GOOD detection is and HOW to describe it. I do plan to continue this research so expect more insights to come. Please share yours!

P.S. So, yes, a very incomplete thought post. Have fun with it anyhow!

Related blog posts:


How to Measure Threat Detection Quality for an Organization? was originally published in Anton on Security on Medium, where people are continuing the conversation by highlighting and responding to this story.

*** This is a Security Bloggers Network syndicated blog from Stories by Anton Chuvakin on Medium authored by Anton Chuvakin. Read the original post at: https://medium.com/anton-on-security/how-to-measure-threat-detection-quality-for-an-organization-4cd377ff5dde?source=rss-11065c9e943e------2