Detection as Code? No, Detection as COOKING!

Anton Chuvakin
Anton on Security
Published in
6 min readJun 3, 2022

--

One of the well-advertised reasons for being in the office is about those “magical hallway conversations” (Google it). One happened to me a few days ago and led to a somewhat heated debate on the nature of modern threat detection.

It also resulted in this half-shallow / half-profound blog that relates detection to cooking and farming! Well, the magical discussion was over lunch, so this is perhaps logical.

Part 1 Key Questions

Here on the blog I touched the subject of detection engineering quite a few times, yet I realize that many organizations prefer to rely on their security tool vendor content for threat detection. In essence, they want “detection consumption”, not detection engineering.

To me, this detection engineering-or-consumption conundrum boils down to two key dimensions: detection transparency and detection customizability. Naturally, these are not the dimensions that define good detection (for example, velocity and coverage play a lot too), but we focus on transparency and customizability here.

Specifically:

Transparency

  • Do you want to see the explicit detection logic of your detection content, rule, code, patterns, etc? [yes, I am well aware that for ML-based detections the exact logic may never be available to humans, but let’s table this for now]

Customizability

  • Do you want to be able to modify detection logic?
  • Do you want to be able to create your own detection logic?

BTW, to me, transparency is what leads to trust in detection content, yet I agree with others who say that there are other ways to gain trust other than seeing the explicit detection code.

Part 2 Cooking Analogy

One analogy that came to us in a discussion was cooking. Naturally, this will be a big table (sorry, Anna)

Now, let’s discuss the examples from various detection tools.

Part 3 Brief Review of Detection Tool Categories

For a good number of years, I’ve looked at SIEM customers using — or not — the detection content (rules, models, etc) provided with their product. Quite a few people told me that upon installing an SIEM product, they immediately discard all out of the box content. On the other hand, others told me that they did use the vendor-provided content after some modifications (some light, some heavy). Anyhow, all SIEMs have open and modifiable detection rules, and rely on ML (not open and modifiable) for some things.

Now, if we travel to the world of EDR, we see a lot fewer people writing custom detections. Some successful products don’t even offer an opportunity to write their own detections. We also see EDRs relying heavily on ML, “secret” threat intelligence and other opaque detection mechanisms. Overall, EDRs often have opaque and unchangeable directions (and use ML too) and clients are OK with that.

Now, if we get a time machine and travel to the world of NIDS, this same battle affected intrusion detection systems back in the day. Some products did not expose the logic of the signatures to clients while others did. Some products came without any detection logic, and assumed “all customers will cook” (these products mostly just died). My impression is that in the long run, the vendors with open signatures — like Snort — won the battle, but also that most customers ended up not creating their own signatures (and did light tuning at best). This 2006 (!) Gartner piece pretty much assumes most clients of NIDS/NIPS use vendor signatures with minimal tuning, some only focusing on higher fidelity signatures sets shipped by the vendor (“The signature set most enterprises enable is the vendor-recommended high-fidelity subset of total capability.”) The surviving NIDS had open detections, but clients often chose not to modify them or create their own.

For the remaining “classic” detection technology, NDR, I see a mixed bag of open and modifiable detections (usually zeek-based) and opaque (usually those that are ML-heavy). Thus, some NDRs have opaque and unchangeable directions while others offer open and modifiable ones. This does not really teach us anything, so whatevs.

As a side note, my impression is that many security professionals would undoubtedly answer yes to both questions (transparent and modifiable). For example:

However, surveys and reality don’t always match. Changes in threat landscape and changes in how organizations do IT may affect the operational reality, perhaps disrupting some accumulated detection wisdom.

For example, while signature creation and customization are valuable, no organization will create detection logic to detect threats against 800+ SaaS business applications. It is very likely that there are scenarios where cloud scale will drive people to opaque, but vendor-managed signatures and detections.

Finally, what about ML? The growth of ML and other non-deterministic detection techniques perhaps indicate that lack of transparency and modifiability is not a “red line”, as long — I think — the new methods are significantly better than the old ones (today, this is debatable).

Part 4 The Question

The real question that I am driving at in this post and that caused our discussion to become heated is — what is the model that is best for the majority of organizations?

Frozen food? Meal kits? Gourmet cooking from scratch (probably not)? Or some hybrid approach? Do people want to just add salt to the ready-made recipe? Or replace truffles with Portobello mushrooms? Or meat with soy?

What do you think?

Part 5 The Answer

First, perhaps all this searching for “the” right answer is deluded? Perhaps, as often in security, the answer is “all of the above”?

This post comes with a hidden assumption, which is that a single meal is going to be eaten or all meals would be the same all the time.

In practice, an enterprise, particularly beyond a certain size, looks much more like a working farm than a single family having a meal. On a farm, we have a variety of needs: sometimes the raw strength of oxen to pull a plow, at times the speed and agility of a horse to wrangle wayward cattle, or the wise counsel of an elder farmer. Similarly, we feed these characters differently: our oxen eat silage, our horses hay, and grandma gets the best slice of lovingly prepared pie. A working farm has needs for all kinds of food, just as an enterprise can and should take advantage of each kind of detection!

For most companies it would be bananas (pun intended) to try to feed endpoint detection needs only with lovingly crafted hand tuned rules, just as our oxen don’t solely eat apple pie. At the same time, our business critical crown jewels don’t have merely the out of the box silage NIDS content, we carefully curate our alerts around our key management tooling, because we know grandma won’t eat hay.

So while we’ve spent a lot of time arguing what is best, I think what is truly best is finding the right kind of rule and detection content for the problem at hand, and finding the way for your rules and detection to work in harmony when it comes to prioritizing triggered alerts for triage, investigation, and response.

Thanks to Tim Peacock and to one unnamed, but very intelligent, security product manager for a profound discussion. Separate thanks to Tim Peacock for writing an inspired conclusion to this post!

Blogs about detection:

--

--