Build a mature approach for better cybersecurity vendor evaluation

Seasoned CISO Mike Manrod knows the value of a good cybersecurity vendor evaluation. He recalls that in a past job he inherited some very expensive vaporware under a long-term services agreement. His predecessor had purchased an “innovative” beta identity and access management platform but hadn’t done any analysis on the product, simply accepting the vendor’s claims of its efficacy. It was a dud.

Inversely, as CISO at his current company Grand Canyon Education, Manrod set his team up to evaluate an allegedly “brilliant” web application security product only to discover through testing that its client-side validation was easy to bypass and thus subvert the product. That basic test saved them from making an expensive mistake. “Startups are trysforming, and sometimes they go back to the drawing board. Nothing wrong there, but if we as security leaders purchase something that’s not ready yet, that’s on us,” he says.

A mature evaluation program begins with a clear understanding of a problem and the solutions intended to solve it, Manrod says. An immature team, by contrast, might chase products for no better reason than an executive saw something they liked at a conference without first defining a clear business need for it.

Furthermore, a mature evaluation team understands and reviews a wide range of solutions before narrowing down to those that meet the business-defined needs, Manrod says. Mature organizations also put greater reliance on testing, rather than believing a vendor’s marketing pitches without question. For example, he says, an advanced testing team would replicate the exact tactics, techniques, and procedures (TTP) of an attacker to try and bypass the tool altogether.

No universal framework for vetting cybersecurity vendors

The most important hallmarks of a mature program – standard assessment formats and repeatable processes – are pretty much nonexistent at an industry level, leaving buyers to develop their own checklists and spreadsheets for each new type of solution they’re considering. This causes difficulties on the buyer and seller sides alike, says Chenxi Wang, longtime cybersecurity expert and managing general partner at cybersecurity VC firm Rain Capital, which currently funds 13 cybersecurity startups, such as Living Security and JupiterOne.

“There is no universal framework that people use to evaluate their security vendors, which is a pity because people replicate work across many different vendors, products, and services. And I am on both sides of this issue – I evaluate companies for investment purposes. Then once we invest, we’re on the sell side,” she says. “I would love it if there were universal frameworks and standards to use. That would make the sell-side job a lot easier – knowing what to prioritize and how to communicate to customers.” 

Part of the problem with standardizing evaluations for security services comes from pushback on the part of vendors and testers, suggests Simon Edwards, founder of the UK-based security product assessment provider, SE Labs. Vendors, he says, don’t want criticism because it costs them buyers, while testers tell him that they don’t generally like standards because they’re limiting, although Edwards feels that standards are not restrictive based on his experience. “Another point to make is that testing security products is really hard,” he adds, because of how many types of solutions there are to solve a single problem and how many ways products can be configured and deployed.

Mature vendor evaluation is aligned with business needs

A mature evaluation process begins with business-to-product alignment. For example, Edwards says, antivirus requirements for banking ATM endpoints are more focused on compliance, whereas antivirus on an employee’s laptop must comprehensively protect against continuous and evolving threats such as phishing and “dodgy downloads” coming from almost anywhere on the internet. These must be clearly spelled out in the requirements stage of an evaluation.

This level of analysis also helps form the questions to determine vendor types, which in turn can reveal how well the vendor serves the needs of the business, says Adrian Sanabria, a well-known product tester and director of product management at supply chain risk management vendor Tenchi Security. “Will the solution solve the actual problems you’ve identified? In doing so, will it reduce the load on the security team, or will it require new skills and produce more work for them?”

At a high level, Sanabria sets evaluation requirements and resulting performance indicators around least expensive effort, interoperability, and integration and overhead, with success metrics such as time-to-value, labor ratio, true cost of product, and reliability.

Align vendor evaluation with the standards that do exist

Although there are no standardized methodologies and processes for evaluating cybersecurity vendors, mature programs utilize industry-recognized standards and frameworks to set requirements and conduct tests against them. But these frameworks are not uniformly prescriptive. For example, SE Labs follows the AMTSO Testing Protocol standard for anti-malware solutions and uses what Edwards calls full-attack chain testing, which goes beyond the MITRE ATT&CK framework.

Choosing a framework early is important to align the business and technical goals, adds Manrod. “The process doesn’t start at selecting a vendor. It begins with a strategy anchored to something like the MITRE ATT&CK framework,” he says. “If you’re trying to prevent attacks on email, identities, or websites, then map the need across your exact environment. That leads to your product category. Your functional requirements are defined by your framework of attack and defense, and then aligned to the exact job the product will do.”

Mature organizations base evaluations on frameworks suited to their businesses, including NIST CSF, ISO, SOC 2, and others. Deloitte, for example, is often asked to collect SOC 2 and ISO attestations for critical security partners that their clients are considering. These requests usually come from risk-averse clients in industries such as energy and finance and many of those are international, says Sharon Chand, secure supply chain leader for Deloitte’s cyber risk group.  

Frameworks approach different security domains, such as protect, detect, identify, respond/recover, she continues. So, mapping evaluations to these domains will help the organization focus on what type of outcomes are needed from those security services. For example, she adds: “Am I buying a firewall or zero trust product? Or am I buying an outcome around increased security of my protected control environment?”

Evaluate the cybersecurity vendor, not just the product

Mature programs also have the means to assess vendor stability. If the provider is based in a political hotspot, there could be service interruptions, says Richard Steinnon, who researched 2,800 security vendors for Security Yearbook 2022. A vendor should also show growth, so he advises buyers to look up investor reports, follow the founder or CEO’s leadership background, and check with peers.

Vendor attestation is a part of a mature evaluation process. But if it is the only evaluation process in place, then accepting the vendor’s word for it without review and testing is an indicator of an immature evaluation process, experts say.

Ian Poynter, an established CISO who conducts numerous cybersecurity product evaluations for corporate clients, says that attestation boils down to questions and response. “With a recent client, we started with a list of thirty vendors in the space. We narrowed it down to three, then sent a long list of questions to those vendors, which they answered during our demo.”

The way the vendors respond to the questions is also telling about the vendor, he says. For example, if the vendor doesn’t answer half of them, it indicates that either the product doesn’t meet the standards or the vendor won’t be responsive as a service provider. If this happens with a lot of vendors, however, it could indicate that the buyer’s perceptions are out of alignment with what the products in that space are capable of, Poynter adds.

Test efficacy and security of the vendor

Mature programs have processes in place to test the efficacy of products, according to Sanabria. The depth of the evaluation program depends on criticality of the security service being acquired. For example, if a product is truly critical, testing would go beyond functional requirements to include red team testing against the service itself.

Mature organizations have these skills in-house or have easy access to these testing skills through outside third parties, Manrod suggests. The key to a mature test process, he adds, is to “Test under your terms with your actual use cases to prove whether or not the product properly fulfills your strategy. The test is worth the cost. If your EDR is a big spend, how much more is it to use in-house skills or hire a consultant for a day to make sure your spend is appropriate?”