What it means — CitrixBleed ransomware group woes grow as over 60 credit unions, hospitals, financial services and more breached in US.
How CitrixBleed vulnerablity in Netscale has become the cybersecurity challenge of 2023.
Credit union technology firm Trellance own Ongoing Operations LLC, and provide a platform called Fedcomp — used by double digit number of other credit unions across the United States. This Fedcomp platform was not patched for CitrixBleed, as no Netscaler patches had been applied since May 2023:
A ransomware group gained entry to Trellance via Ongoing Operations. You can read about some of the fallout here. Ongoing Operations’ two Netscaler devices remain offline. This is disrupting operations in a way which impacts millions of Americans.
HTC Global Services, aka HTC Inc, aka Caretech — a large MSP for the US healthcare sector with remote access to hospitals across the US, did not patch Netscaler since July 2023:
HTC Global Services are currently being held to extortion by AlphV ransomware group, who display stolen documents on their ransomware portal which are branded Caretech, a division of HTC.
Earlier in the week, the BBC reported that a ‘cyber incident’ (ransomware) at CTS, a legal tech firm (cloud MSP) in the UK, is leaving UK home sales in limbo:
CTS own Sprout IT as a core brand:
Sprout IT ran Netscaler and hadn’t patched it until late.
ABC report homebuying in the US has stalled due to a ransomware incident at Fidelity National Financial. AlphV ransomware group also claimed them:
Fidelity National Financial also patched CitrixBleed late. You may be spotting a pattern at the victims.
The United States continues to be disproportionately impacted, so I thought it would be interesting to look at what is making this situation so difficult to address. Sadly, CitrixBleed isn’t an isolated situation — it’s just the perfect storm of the style of vulnerability combined with ransomware groups.
First, let’s do a bit of background. The security patch for this issue became available almost two months ago. Back in October, I wrote a Mastodon toot saying CitrixBleed would have more legs than people realise:
The reason I wrote this toot is as follows: in a world where there are thousands of new security vulnerabilities every few months, not all vulnerabilities are equal. This one was, to me, clearly going to be a major issue as it allowed the bypass of multi-factor authentication controls, it didn’t log exploitation and it was easy to exploit. I knew many organisations were not going to be equipped to know to patch the vulnerability, or even know how to identify if they ran the Netscaler product due to the silo’d nature of cybersecurity departments within businesses. Hence why I published information on how organisations could identify their own assets.
I then wrote a blog saying mass exploitation was happening, a companion piece to say ransomware groups are using the vulnerability to backdoor systems for later and continued to track threat actor activity.
I have also established, for example, that one of the ransomware groups, and a state aligned group, both obtained an exploit for the vulnerability on October 23rd, and were using it in the wild:
In my earlier blog post I broke the news, using publicly available information, that the attack on ICBC, the world’s largest bank, was via CitrixBleed. This has since been confirmed by the US Treasury. It also provided evidence of CitrixBleed being used against Allen & Overy and many others.
The blog also broke the news that exploitation leaves no logs for the initial exploit request, since confirmed by Mandiant, due to product deficiencies (which still haven’t been addressed by the vendor). This lead to briefs from the Australian government, the US government and others. Healthcare services issued briefs:
While all this was happening, I was watching people glee-ly posting things on LinkedIn about AI generated malware and such. The cybersecurity industry has amazing problems with understanding threat priority, and really likes to chase after whatever is being sold next.
Now, let’s bring ourselves up to date with what has changed in the three weeks since I last wrote about this. First, many of the victim organisations either never appeared on ransomware group portals, or disappeared from ransomware group portals, because the organisations made the choice to pay the criminals. This is despite mass data theft. Fidelity National Financial and Allen & Overy have both disappeared from portals and refused to confirm what happened.
My research, which has allowed me to track who has been successfully targeted, shows most victim organisations have opted to cover things up and their names are not known to the public. Organisations instrumental to this cover up include leading cybersecurity incident response and insurance firms.
I’m concerned.
Anybody who knows me knows I am a pragmatist. I’ve spent 24 years non-stop working for medium to large size enterprises, from oil companies, telcos, ICS manufacturing, a security vendor etc, in cybersecurity. I’ve seen… a lot. I often get laughed at in professional settings for underselling situations — e.g. I’ll say “that’s not ideal” when things have gone very wrong.
I say this because I want people to understand the tone and weight behind what I’m about to say: I’m really concerned about ransomware groups. So much money is being quietly passed to these guys — often teenagers — that I think there is a very real probability they are going to cause a series of major, global incidents that impact civil society and governments themselves. Not by choice of these groups, but by accident — they’re obtaining the level of access that only nation states should have, where consequences, counterweights and experience normally exist. What’s happening with ransomware isn’t normal, it has just become normalised. We’re allowing teenagers to obtain serious arms and infrastructure at an alarming pace.
I do not think the situation is any longer sustainable. Whilst it is absolutely true that ransomware and extortion groups are just a symptom of poor security — trust me, I’ve seen it — the reality is poor security isn’t fixable any time soon, but the threat uncontrolled groups who’ve monetised said poor security poses is.. legitimately an (inter)national security risk that is going to keep escalating until something goes very wrong, I fear. It isn’t just the criminals who have monetised poor security here — there’s an industry which has sprung up monetising the victims and the fear of being a victim, too.
What needs to change?
- Product vendors need to secure the software and services they sell. For example, Netscaler runs on FreeBSD. FreeBSD supports security jails, to stop processes going rogue. Buffer overflow vulnerabilities like this are very 1990s, and there are multiple mitigations available. In short, I think security vendors like Citrix need to up their game, as things are real now — or be regulated. It cannot be the position that every customer is expected to install 4239 security patches every year for 3294 different vendors largely for variations of the same classes of security bugs that the vendor has failed to address for decades, nor should it be the position that every customer has to apply 398 different security ‘best practices’ each year to make a product they just purchased secure. It simply doesn’t scale and is leading to a security poverty line which risks sinking small to medium sized businesses — which as Satya Nadell has said before, are the life blood of the global economy. Okta just paused new product development for 3 months due to the level of threat their customers are facing from ransomwaresque gang Scattered Spider. That is smart.
- Payments to ransomware and extortion groups need to be outlawed. I know, I know, it will be hard and there’s a million reasons to argue against it and lots of vested interests who don’t want this. But this is the first time I’ve ever said it. I’ve always been pragmatic until now. I mean it — ransomware payments to these groups need to be outlawed, internationally. We have to push through the short-term pain because it is the safer option. Start planning for this, signal it loudly, and do it. This one needs firm leadership from the very top, as the lobbying against will be real. Civil society needs protection via firm leadership, not leadership by a small number of firms profiting from the status quo. This is a chance for world leaders to lead when others haven’t.
- Companies need to examine if they can securely manage the technology they have deployed, and if it is a good fit for them. I know a lot of companies are outsourcing their IT systems to MSPs — they have been for some time — but I think they need to realise that some solutions, particularly ones at the border to the internet, are incredibly risky and need thought put into if they’re the right solution. This comes back to the first question — does Netscaler have a good track record with security vulnerabilities? No. It has been plagued by them. Is it a good idea for SMBs to outsource management around these products? Is that a good match for your business? These are all questions we all need to ask ourselves, along with the vendor owners of these products — security should be the priority, it is the mark of safety of the modern age.
- We need to change how we all deal with ransomware. LockBit encrypted part of Boeing, the international defence firm. Their response? They contained the incident and gave CISA all their important technical information about the incident for them to release. CISA released this publicly, and it helped everybody protect themselves. We all know that Boeing is the outlier here, as currently every CISO is playing a Western game on TLP:RED. I think it’s very clearly a race to the bottom, and history shows those who try to cover up ransomware incidents often end up directly in the spotlight as a result. We’ve got to do better at being more open about this — they’re criminal groups, and we defend better together and as transparently as possible. Boeing and CISA’s response have been the gold standard we should aim for. Historically, firms have minimised breaches from nation states and APTs with the aid of legal departments, often for good reason. These play books should not have been used for ransomware. We’re all poorer. If armed thugs started smashing in the windows of schools and hospitals across the world and talking computers and threatening patient safety, we’d be collectively up in arms and sharing photos and videos and demanding action. We shouldn’t have normalised ransomware like we have, especially given the escalating nature of the problem.
I’m career cybersecurity. I’m numb to pain. I genuinely fear we have a problem where we’re pretending that virtual cyber things aren’t real and there’s nothing we can do about it. Bitcoin payments mean exploit development and recruitment. I think it’s going to get very real, and while I’m super optimistic there are more things we can do to pump the brakes.. we shouldn’t do them too late.
You can follow me on Mastodon for the latest cybersecurity news about emerging cyber threats if you’re really bored.
Update: HTC are acknowledging. Their statement: “HTC has experienced a cybersecurity incident. Our team has been actively investigating and addressing the situation to ensure the security and integrity of user data. We’ve enlisted cybersecurity experts and are working to resolve it. Your trust is our priority.”
