Log Centralization: The End Is Nigh?
So I woke up the other day [A.C. — well, the other year as this blog has lingered] with the scary thought: what if we will run out of the opportunities to centralize logs for security (and compliance) purposes at some point in the future.
Or, as I pithily put it on Twitter:
So I wrote some of this and kinda forgot about it for a few months. And then, while at the Gartner Security Summit in June 2023, I saw this:
This was my motivation to finish this as an incomplete thought blog. Also, I had a chance to coolly reflect on this and review my past passionate calls to centralize logs (the earliest of my presentations I found was from 2003), let’s see what do we have now?
So, for many years, security professionals advocated the approach of collecting logs from places where they are generated and centralizing them (1983) into one or few places. First in flat files, then in databases, later in all sorts of fancy distributed systems.
Note that for the sake of this argument, using a distributed system such as Hadoop or its modern peers (located at a single data center or at one cloud provider) is ultimately centralizing logs. This logic applies here because we are collecting multiple types of logs from many places and then storing them in one or a small number of places under our control.
In recent weeks [A.C. — this applies to my original tweet from Aug 2022], I started to see situations where my 20+-year-old call to centralize logs is really not feasible, not practically workable, or the pain of doing so starts to look larger than the value for a typical organization.
Let’s go through a few basic examples. The very example that inspired that line of thinking involved multi-cloud. If you are present in multiple public cloud providers, and present there at scale, it is very likely that you are NOT collecting logs into one place in one cloud. Various complexities, egress costs, storage costs all play into this becoming a questionable decision for most organizations. So you perhaps centralize per cloud, but what if we include SaaS services into this? Then it becomes an even bigger mess, as most large organizations use 100s of those.
Another trivial example refers to the log types that are useful for investigations or in bulk, but where each individual record is unlikely to be used for detection. For example, I’ve noticed that many organizations don’t collect and retain DHCP logs (of course, Chronicle customers do!). They fail to do it not because these logs are not useful (they are very useful as context), but because they don’t use them for any direct detections, and thus see them as “too costly to centralize” (especially if their SIEM vendor charges per EPS…).
Now, while distributed, decentralized and federated sounds really cool, there are many, many challenges with using this approach for log collection, retention and analysis. And no, I don’t just mean the PCI DSS assessors will hate you with all their unregulated passion…
So, yes, many compliance mandates such as PCI DSS directly mandate that you collect and centralize the logs (PCI DSS v4 Requirement 10.3.3 “Audit log files, including those for external facing technologies, are promptly backed up to a secure, central, internal log server(s) or other media that is difficult to modify”). Well, many security professionals nowadays make fun of regulations as outdated for today’s realities (e.g. cloud and DevOps and containers), in this case they represent a stabilizing force for IT security practices.
Before log management and SIEM tools were born, in many cases, logs were left wherever they were produced and no retention or even availability of logs was enforced or guaranteed (or expected, frankly). In that sense, we do not want to move back to that world. While I do love all these “ha-ha, your IT is in the ‘90s” putdowns, in this case we are going to the ’80s perhaps? You know that time before syslog existed? Sorry, that would be the 1970s.
So if you simply hope that the logs will be there when your magical decentralized query tool reaches for them, you are going to be disappointed a lot. Perhaps, you will even cry a bit when your IR consultant finishes the project and then tells you “sorry, not sure what happened here as there were no logs, but here is your $100K bill for all the things we tried.” Even if you rely on some distributed log storage, different for each system and application, the complexities of this approach are staggering indeed. However, are they more staggering than centralizing the petabytes?
Next, those that are centralizing and then normalizing in order to perform analytics on them would be disappointed too (at least until things like OCSF spread, if they ever do). After that happens, distributed analytics need to be created and perfected. However, if your algorithms rely on normalized logs, you may have to wait for a very long time until all logs are normalized naturally. In fact, it is much easier for me to imagine decentralized / federated search rather than to imagine effective decentralized log analytics. After all, we’ve barely managed to make centralized log analytics work well.
You probably sensed where I am going, so here it is: the problem isn’t that the distributed approach is easy, the problem is that the centralized approach is becoming much harder as volumes and log source counts, types and distributed nature go up.
To state an unshakeable truth, while everybody hates log management and SIEM vendors that charge per gigabyte, ultimately collecting and retaining logs has a cost. Somebody has to bear the cost of all those hard drives. If you centralize, you pay. But once you pay, you reliably own the logs (Thank you, Captain Obvious!)
What’s the alternative, really? The more I think about it, the more I’m realizing that a decentralized approach cannot be merely about hoping that the logs are there wherever they are produced whenever you need them.
By the way, some vendors deploy indexers in multiple locations (or multiple clouds) and achieve some form of federated log search. Ultimately, this is not what we are talking about, because in this case you are just creating several big islands of centralization that eventually grow to have the same problems we are trying to solve.
However, some form of centrally defined (and managed?) but distributed log storage is likely to be part of the future (Is Cribl Search a current example of this? Is query.ai? Is Tenzir? Perhaps).
By the way, Gartner BTW optimistically [very!] states that “Federated security log management (SLM) is emerging as an alternative to centrally collecting logs. These different approaches can support the same use cases and outcomes for security operations.” (source)
To conclude, I think the centralized approach to logs would work as long as it can and in as many places as it can. But we may have to augment this with some form of centrally defined, but lightly managed and highly distributed log collection and analysis. So what? Well, nothing yet, let’s revisit this topic in a few years…
Tenuously related blogs:
