Migrate Off That Old SIEM Already!
This is cross-posted from Google Cloud Community site, and written jointly with Dave Herrald.
If you are like us, you may be surprised that, in 2024, traditional security information and event management (SIEM) systems are still the backbone of most security operations centers (SOC). SIEMs are used for collecting and analyzing security data from across your organization to help you identify and respond to threats quickly and effectively. But if you’re still using an outdated SIEM, you’re putting your organization at risk [A.C. — are we a bit harsh here? Frankly no! If your SIEM takes a lot of efforts to maintain, whether on-prem on or via poorly engineering cloud model, you are not spending that time and effort on countering the bad guys].
Legacy SIEMs [A.C. — many people frown at the term “legacy”, but there are clear technology generations in SIEM that a discerning eye can easily see, legacy here means “not the current gen”] are often slow, cumbersome, and difficult to use. They may be unable to keep up with the latest threats or support the latest features and capabilities. They may not offer the flexibility to support your organization’s specific needs. They may not be suited to the multi-cloud strategy that is the reality for most organizations today, or they may be poorly positioned to take advantage of developments in artificial intelligence (AI) [A.C. — here we sound like marketing people, but it is also true; old product with a chat bot is still an old product].
But before we jump straight to “replace the sucker!”, we need to recognize that not all [A.C. — to put it mildly…] SIEM deployment failures are due to the product itself, and simply replacing a product may not make the organization better. Process (and occasionally people) breakdowns also play a critical role in success. [A.C. — this sounds banal and repetitive, yet what else can I say? This still trips a lot of orgs who buy a SIEM in bad faith]
Identifying shortcomings in your current SIEM is much easier than selecting the best replacement and executing a successful migration. That’s where this blog comes in. The authors have seen hundreds of SIEM migrations as practitioners, analysts, and vendors over multiple decades. So, let’s take stock of the top SIEM migration tips for 2024. We’ll divide this list into categories and sprinkle in lessons we’ve learned from the trenches throughout.
Selecting a New SIEM
How do you know when you need a new SIEM? Start by asking yourself and your team some key questions to help uncover each offering’s strengths and weaknesses. We recommend quickly identifying each SIEM’s “superpowers” and planning how your organization can take maximum advantage of those benefits. [A.C. — “shift or persist” really is THE question at this stage, and a lot of energy needs to go into solving this well]
For example:
Is the SIEM offered by a primary cloud service provider (CSP) that can provide world-scale infrastructure at wholesale prices? Tip: Our experience shows that SIEM providers who operate in clouds they don’t own have difficulty overcoming the inescapable “margin stacking” that comes with such models. This question is inextricably linked to cost.
Does the SIEM vendor have a continuous stream of frontline threat intelligence to drive out-of-the-box detection of new and emerging threats? Tip: These golden sources typically arise from top-tier incident response practices, the operation of massive consumer IaaS or SaaS cloud offerings, or massive install bases of security software products or operating systems. [A.C. — again, this is very self serving, yet this is backed up by strong capabilities here]
Does the SIEM offer an extensive library of supported parsers and detection content? Tip: Some SIEM vendors rely almost exclusively on their user community or technical alliance partners to create parsers for popular data feeds. While the existence of a thriving user community is essential, over-reliance on the user community to provide fundamental capabilities like parsing is a problem. Parsers for common data sources should be created, maintained, and supported directly by the SIEM vendor. Take the same approach when looking at detection content. Community rules are essential, but you should expect your vendor to create and maintain a solid library of core detections that are supported and improved regularly.
Does the SIEM incorporate AI, and is it positioned to continue innovating? Tip: The whole story about AI’s role in SIEM has yet to be discovered by any vendor or customer. [A.C. — this is the truth, sorry guys! You can read 1000 articles on how AI will transform the “land of cybers” and barely 1 that describes solid use cases such as for GenAI!] However, leading SIEMs already have tangible AI-driven features shipping today. These features include natural language processing for expressing searches, automated case summarization, and recommended response actions. [A.C. — not magic, to be sure — but very useful! There is no “Bard, detect all threats that matter for me” command, but there are hours of analyst time saved.] Most customers and industry observers consider features like threat detection and predictive adversary analysis to be the “holy grail” of AI-driven SIEM capabilities. No SIEM reliably offers these features today, but as you choose a new SIEM in 2024, consider what kind of vendors are investing the resources necessary to make progress on these big problems.
SIEM Migration
So you’ve decided to make the move. Your approach to migration is critical in ensuring you maintain required capabilities and start extracting value from the new platform as soon as possible. It comes down to prioritization.
A typical trade off is recognizing that, on the one hand, a SIEM migration represents an opportunity to modernize your entire investigation, detection, and response approach. However, many SIEM migrations fail because organizations try to “boil the ocean.” So here are our best tips for planning and executing your successful SIEM migration.
Here are our favorite SIEM migration tips:
Define your migration goals. This sounds obvious, but your SIEM migration is a lengthy process, so the need to define your desired outcomes (e.g., faster threat detection, easier compliance reporting, improved visibility, reduced analyst toil, while also lowering cost) is strongly correlated with success. [A.C. — are you saving money on log collection? Improving detection quality? Improving analyst UX? Switching to a tool that your MDR partner uses?]
Use the migration as an opportunity to clean house. This is a good time to clean up your detection rules and log sources and only migrate the ones that you actually use. It’s also a good time to review your alert triage and tuning processes and make sure that they’re up to date.
Don’t migrate every log source. Moving to a new SIEM is a great opportunity to decide what logs you need, be it for compliance or security reasons. Many organizations accumulate a vast amount of log data over time, and not all of it is necessarily valuable or relevant. By taking the time to evaluate your log sources before you migrate them, you can streamline your SIEM and focus on the data that is most important to your security and compliance needs. [A.C. — yes, this takes work, sorry “Easy Button“ crowd, but this work will absolutely pay for itself for years to come!]
Don’t migrate all content. Migrating all of your existing detection content, rules, alerts, dashboards, and visualizations to a new SIEM is a lot of work and it’s not always necessary. Take the time to evaluate your current detection coverage and only migrate the rules that you need for your new environment. [A.C. — somebody from a high maturity SOC will point out that you have a D&R content lifecycle and old content. unneeded is removed routinely, but admittedly many teams do not have this and do not practice this]
Detection content is recreated, not migrated. There is a process of rebuilding the detection content (rules, alerts, dashboards, models, etc.)from scratch and using your old content as inspiration. In some happy future where Sigma rules, this may change, but today, there is no way to “run a converter” on rules from an arbitrary vendor 1 to convert to rules for any vendor 2 (with some exceptions).
Prepare to spend more time on this than you think is likely. SIEM migrations are complex and they can take a long time to complete. Be prepared to allocate plenty of time and resources for the migration project. [A.C. — definitely ask your new vendor for advice, but perhaps take some of the timelines offered with a grain of salt]
Develop a realistic migration timeline. This includes accounting for data transfer, testing, tuning, training and potential overlaps where you may need to run both systems in parallel. A well-defined migration plan will help you to identify and mitigate risks, and ensure that the migration is completed successfully. The plan should include a detailed timeline, a list of tasks, resources, and a budget. Recognize that major projects like a SIEM migration must be broken into phases.
Testing. We recommend adopting the practice of testing your SIEM and detection content by regularly injecting data that will test your detections, check parsing, and validate data flow from detection to case to response playbook. A SIEM migration is the perfect time to adopt a rigorous detection engineering program that includes testing like this.
Prepare to run both old and new tools for some time. Avoid a disruptive “rip and replace” approach. A phased migration, where you move log sources or use cases gradually, helps control the process and reduces risk. Also, think twice about re-ingesting data from your old SIEM into the new. [A.C. — think twice, and then don’t do this :-)] This approach sounds good in the abstract but rarely pays off. [A.C. — there are definitely cases where ‘run two tools’ period was measured in years]
Enable your teams. Your SIEM migration will fail if your analysts can’t use the new system. A good migration plan will include deep enablement for your teams. Think about training engineers on data onboarding and parsing, training analysts on case management/investigation/triage, threat hunters on anomaly detection/search, and detection engineers on rule-writing.
Get help! If you are lucky (or maybe unlucky?) as a practitioner or leader, you will have perhaps gone through one or two SIEM migrations in your career. Why not seek help from specialists who have done it dozens or hundreds of times? Professional services teams from the vendor and/or consulting teams from qualified services partners are a great choice. SIEM migrations are largely human-centered efforts.
Additional reading:
- “How Chronicle SIEM Can Help Augment Your SOC Stack” paper
- “Future of the SOC: Evolution or Optimization — Choose Your Path” paper
- Detection Engineering Weekly Newsletter
- detect.fyi — Practitioner-centric tips on detection engineering
