SolarWinds and Market Incentives

In early 2021, IEEE Security and Privacy asked a number of board members for brief perspectives on the SolarWinds incident while it was still breaking news. This was my response.

The penetration of government and corporate networks worldwide is the result of inadequate cyberdefenses across the board. The lessons are many, but I want to focus on one important one we’ve learned: the software that’s managing our critical networks isn’t secure, and that’s because the market doesn’t reward that security.

SolarWinds is a perfect example. The company was the initial infection vector for much of the operation. Its trusted position inside so many critical networks made it a perfect target for a supply-chain attack, and its shoddy security practices made it an easy target.

Why did SolarWinds have such bad security? The answer is because it was more profitable. The company is owned by Thoma Bravo partners, a private-equity firm known for radical cost-cutting in the name of short-term profit. Under CEO Kevin Thompson, the company underspent on security even as it outsourced software development. The New York Times reports that the company’s cybersecurity advisor quit after his “basic recommendations were ignored.” In a very real sense, SolarWinds profited because it secretly shifted a whole bunch of risk to its customers: the US government, IT companies, and others.

This problem isn’t new, and, while it’s exacerbated by the private-equity funding model, it’s not unique to it. In general, the market doesn’t reward safety and security—especially when the effects of ignoring those things are long term and diffuse. The market rewards short-term profits at the expense of safety and security. (Watch and see whether SolarWinds suffers any long-term effects from this hack, or whether Thoma Bravo’s bet that it could profit by selling an insecure product was a good one.)

The solution here is twofold. The first is to improve government software procurement. Software is now critical to national security. Any system of procuring that software needs to evaluate the security of the software and the security practices of the company, in detail, to ensure that they are sufficient to meet the security needs of the network they’re being installed in. If these evaluations are made public, along with the list of companies that meet them, all network buyers can benefit from them. It’s a win for everybody.

But that isn’t enough; we need a second part. The only way to force companies to provide safety and security features for customers is through regulation. This is true whether we want seat belts in our cars, basic food safety at our restaurants, pajamas that don’t catch on fire, or home routers that aren’t vulnerable to cyberattack. The government needs to set minimum security standards for software that’s used in critical network applications, just as it sets software standards for avionics.

Without these two measures, it’s just too easy for companies to act like SolarWinds: save money by skimping on safety and security and hope for the best in the long term. That’s the rational thing for companies to do in an unregulated market, and the only way to change that is to change the economic incentives.

This essay originally appeared in the March/April 2021 issue of IEEE Security & Privacy.” I forgot to publish it here.

Posted on February 8, 2023 at 6:46 AM20 Comments

Comments

Harald Hanche-Olsen February 8, 2023 7:36 AM

Watch and see whether SolarWinds suffers any long-term effects from this hack, or whether Thoma Bravo’s bet that it could profit by selling an insecure product was a good one.

Since this was nearly two years ago, perhaps now is a good time to ask how it has turned out in the end? Or is the case perhaps still rattling around in the courts?

John Tillotson February 8, 2023 9:18 AM

If the consequences of a data breach aren’t serious enough to make the entire board and C*O suite concerned about their wealth and freedom, then they will just hire a well-paid scapegoat to take the fall. Data breach happens, scapegoat uses golden parachute then gets hired by the next company. Rinse, repeat.

So make the entire board and C*O suite personally responsible for the data breach, no exceptions. Make the penalties REAL. Otherwise it’s just security theater.

Linc February 8, 2023 9:58 AM

“In general, the market doesn’t reward safety and security”

Nonsense. Poorly managed companies go out of business, via competition, low/no profits, customer lawsuits, and shunning by liability insurance companies.
This is how markets routinely and successfully purge bad actors.

“The only way to force companies to provide safety and security features for customers is through regulation.”

… more Nonsense. Government employees (REGULATORS) have no more expertise & honesty & noble service to the public than people in the private sector… and usually less.
A government title and guaranteed paycheck does not eliminate normal human shortcomings.
Plus, big hacks of government computer systems are just as common as in the private sector.

modem phonemes February 8, 2023 10:14 AM

History amply shows that it takes a government to really devastate an economy or country, just as it takes a government to implement certain kinds of economic and political good. As Aristotle said, the same thing is the cause of good and bad, by its presence or absence, illustrating by the boat pilot.

The discussion should center primarily on justice, regulation is just a tool which can promote justice or destroy it.

toni February 8, 2023 11:30 AM

@ Linc,

Nonsense. […] This is how markets routinely and successfully purge bad actors.

Equifax is still around. A family member just had an identification card borrowed for a bit while checking into Marriott. Home Depot doesn’t seem to be having any real trouble. Off the top of my head, I can’t think of any company that went out of business due primarily to a computer security breach (excluding cryptocurrency companies, and I don’t count those because I suspect the whole idea was invented as a covert bug bounty program anyway).

A company that’s badly managed all around will tend to go out of business. But if it’s mostly bad in terms of computer security, that seems to have little effect.

By the way, I hope nobody’s assuming that regulation means we’re wearing “pajamas that don’t catch on fire”. The USA’s Flammable Fabrics Act is pretty strict about children’s sleepwear and all mattresses, but not so concerned about adult clothing unless it burns unusually quickly. There was a push in 1987 to get similar regulation—or a voluntary standard—for adult sleepwear, but I don’t think it ever happened.

Frank Wilhoit February 8, 2023 11:52 AM

Auditors give companies short lists of software to choose from. You have homebrew asset management that actually works, on which you pay no licensing and that your people know how to use? Tough: auditor says scrap it and buy the name brand package. You won’t train your people on it because that’s OpEx, and it will take two months of data fixes to close the books every year, but the auditors will sign and that’s all that matters.

mark February 8, 2023 2:06 PM

It’s long past time to stop pretending that outsourcing will “save money”. For government, esp., it’s time to hire people. Outsourcing was supposed to be for short term issues, and security is a permanent one. You don’t want contracts ending, and people with in-depth knowledge going away, because the contract winner didn’t want to pick up the cost of those people.

JonKnowsNothing February 8, 2023 3:28 PM

@All

re: Modern Good Management and Outsourcing

Both are merely forms enhanced asset transfer.

Companies that have what we term Good Management Models today, evolved from massive asset stripping of other companies. It’s also called Bait and Switch; the stock market calls it Pump and Dump. It has nothing to do with what the company actually does.

Outsourcing, has never been about “saving money”. Employment costs are spread into different parts of the SEC/IRS Books and Annual Review. There are 2 sets of books, one for SEC which, in theory, deals with the business and operations values; the other is the IRS taxation and revenue adjustments. Taxes vary by jurisdiction and country, so there may be many versions of Tax Adjustments. Outsourcing allows the redistribution of work costs into other categories where their P/L ratios have less impact on the numbers used by SEC and Private Equity. Private Equity and the SEC are more than able to re-parse the shifted numbers, others not so much.

Policies that restrict Asset Stripping and Asset Transfer have lower chances of being enacted. Such proposals rarely are effective even if enacted, due to alterations in their structure during the committee re-write phases.

Good Modern Management is: Take Everything, Leave Nothing.

Outsourcing uses any means possible, legal, semi-legal, quasi-legal, sometimes-legal to: Take More, Invest Nothing.

Ted February 8, 2023 6:05 PM

So how big is the software supply chain problem? According to one estimate:

Enterprises and agencies use an average of more than 40,000 open-source software packages downloaded by developers, and each of those can bring in another 77 dependencies.

And that’s just the open source part.

El Reg is reporting on some developments in this space. CISA’s got a new cyber supply chain risk management office (C-SCRM).

It appears there are also some new frameworks coming down the pike: the OSC&R (Open Software Supply Chain Attack Reference) and the OpenVEX spec.

Ted February 8, 2023 6:15 PM

So how big is the software supply chain problem? According to one estimate:

Enterprises and agencies use an average of more than 40,000 open-source software packages downloaded by developers, and each of those can bring in another 77 dependencies.

And that’s just the open source part.

El Reg is also reporting on some developments in this space. CISA’s got a new cyber supply chain risk management office (C-SCRM).

It appears there are also some new frameworks coming down the pike: the OSC&R (Open Software Supply Chain Attack Reference) and the OpenVEX spec.

MarkH February 8, 2023 10:47 PM

@toni:

I can’t think of any company that went out of business due primarily to a computer security breach (excluding cryptocurrency companies …)

I don’t know of such a case either, but readers might be interested in the 1996 crisis at Omega Engineering Corp, a U.S. manufacturer for industrial markets.

A departing employee wiped critical computer-aided manufacturing data, reportedly costing the firm about USD 12 million, which I estimate was comparable to a year of operating profits.

The attack was set up from inside the building, but much of it might easily have been done via network connections.

Clive Robinson February 9, 2023 3:09 AM

@ MarkH,

Re : Insiders playing outsiders

“The attack was set up from inside the building, but much of it might easily have been done via network connections.”

That takes me back… In 1996 I had to give talks to an MSc in Information Systems Design course (mainly non technical managment stream types and a few software types looking to climb the managment pole). I chose to talk about the basic stratagies inside attackers could take to gain their advantage.

I could tell that most of them did not realy get why a sensible inside attacker would do just about every thing they could to make it look like an outside attacker.

A quater of a century later whilst technical security types “get it” the managment stream types still don’t want to go there. They treat the idea like staying in a five star hotel in purgatory… Very costly, and of only minimal conveniance in a place you most definitely do not want to go…

Dean bushmiller February 11, 2023 11:11 AM

No to regulation on practices, it has never worked. Certified public information, security officer, just like we have certified public accountant. Each company that trades publicly or deals with large organizations Must have one of those people in their organization validating their process because internal people can understand the internal processes, business requirements and information security as a collection of processes that is unique to that organization. Professionalization of cyber security, as position is happening slowly through the NICCS. If your cyber professional quits in protest and registers a complaint against cyber practices, it becomes a matter of public record, and is outside the control of the organization, trying to hide those practices for profit reasons Will go away. One of the components of being a professional is having an ethics body just like attorneys.

trouble February 13, 2023 8:38 AM

@Linc

It looks like you think markets are efficient. I encourage you to research terms like “externality” and “market failure.” There are some industries (health care, real estate) where demand is highly inelastic. In others (mining, energy production) many costs are borne by society at large while profits are internalized.

Regulators aren’t perfect, but their incentive structure can encourage outcomes that are good for society. The FDIC makes banks more resilient than they would otherwise choose to be.

Chris Drake February 15, 2023 8:32 PM

Your blame on the corporate sector is horribly unfair.

In my relatively vast experience, getting anyone in Government to take security seriously is near-impossible. They don’t care, they don’t test anything shown to them, they don’t believe anything told to them, and they never buy secure products. You can’t expect commercial operators to BE secure, when they customers do not care and refuse to pay for security!

The reason is simple: there are no laws that force government to be secure (or if anything like a law does exist, it’s ignored), and there are no consequences for anyone in government when they ignore security. Nobody gets sacked, nobody even gets a slap on a wrist. There’s also no working mechanism to report insecure government systems, and if you try, all reports go nowhere.

To make everything worse, mandatory breach reporting (if any) never applies to government, no commercial behavior standards apply to them, they rarely measure anything (so have no idea how broken their stuff is), they practically never report on any of their breaches (despite being more numerous than the entire commercial sector combined), and they’re completely immune to transparency.

Clive Robinson February 15, 2023 9:35 PM

@ Chris Drake,

“The reason is simple: there are no laws that force government to be secure”

Actually thanks to the Republicans the opposite is true.

Some years ago now they forced through legislation handy for their corporate friends.

US Gove Depts are required to buy “Consumer Off The Shelf”(COTS) equipment, ie the cheapest commercial PC equipment and software.

Worse the Deptd have to buy the equipment through a “rigged market place” run by friends of those very same legislators banging on about “saving tax dollars” whilst actually “filling their boots” from “the kick back” they get from those who “milk the public purse”.

Not once has any of those COTS solutions proved to be “cost saving” as you would expect from a faux market designed to bilk the public purse. And the level of security of the consumer grade OS’s and Applications is when properly configured at best pitiful. But due to that rigged market, they are never ever configured properly, only conveniently for those bilking the public pure, oh and Chinese, Russian, and god alone knows who else that want to steal private information.

As such those legislators on the take have done more than any other people in history so far to destroy US National Security… Call it the “fourty pieces of silver” effect.

ResearcherZero February 16, 2023 9:31 PM

@Clive Robinson

How can they possibly have time to produce simple and effective policy, when they they are busy working themselves up into a tizz over a balloon? 😉

“Our national military secrets – including tactics, techniques and procedures – are not for sale.”

https://thenewdaily.com.au/news/politics/australian-politics/2023/02/15/defence-tighten-ship-china-pilot-poach-plot/

State actors using cyber operations as a geostrategic tool to steal national secrets.
https://www.cyber.gov.au/sites/default/files/2022-11/ACSC-Annual-Cyber-Threat-Report-2022.pdf

“outdated and ineffective … allow unrelated staff to use the same inherently weak passwords—meaning there was not a rule in place to prevent this practice.”
https://www.doioig.gov/sites/default/files/2021-migration/Final Inspection Report_DOI Password_Public.pdf

The Committee questioned Defence on who was responsible for conducting the audits and how they are verified, but was unable to ascertain a clear answer from Defence.
https://www.aph.gov.au/Parliamentary_Business/Committees/Joint/Public_Accounts_and_Audit/PersonnelSecurity

“Further, none of the entities had processes for verifying the reliability of cyber security related performance information provided by contracted providers.”
https://www.anao.gov.au/sites/default/files/2022-12/Auditor-General_Report_2022-23_9.pdf

Defence’s systems for managing DISP memberships are not considered to be fit for purpose. Internal review activity has led Defence to conclude that it has had a systemic problem with maintaining accurate records in its systems and data remediation work has been required.

…initial DISO ‘audit’ results indicate DISP participants are not managing Defence’s security risk in accordance with the DSPF or to the standard required
https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2021-22_4.pdf

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.