Rackspace: Ongoing Exchange outage caused by security incident

American cloud computing services provider Rackspace says an ongoing outage affecting its hosted Microsoft Exchange environments and likely thousands of customers was caused by a security incident.

The list of impacted services includes MAPI/RPC, POP, IMAP, SMTP, ActiveSync, and the Outlook Web Access (OWA) interface used to access the Hosted Exchange instance to manage email online.

"We are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available," Rackspace said on Friday night, at 02:49 AM EST, when it acknowledged the outage.

More than 15 hours later and multiple updates without any info on what is causing what it describes as a "system disruption," the company said it's "aware of an issue impacting" Hosted Exchange environments and that its engineering teams continue to work "to come to a resolution."

Affected customers were advised to check the status page for the latest updates, even though those are also lacking details about the outage's root cause. 

In reply, Rackspace's irked customers asked the company on social media to provide an ETA for when the issue behind this outage will be addressed and shared plans to switch to another, more transparent, managed service provider (MSP).

Almost twenty-four hours later, at 01:57 AM EST, Rackspace revealed the true cause of the outage, a security incident "isolated to a portion of our Hosted Exchange platform" that forced the company to disconnect the Hosted Exchange environment.

"On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact," the company said.

"After further analysis, we have determined that this is a security incident. The known impact is isolated to a portion of our Hosted Exchange platform."

This confirms some of its customers' concerns who, due to the limited information, said that they feared the outage might be the result of a malware or ransomware attack.

Rackspace's Head of Global Public Relations Natalie Silva told BleepingComputer in an email sent Friday evening that the MSP is now providing affected customers with Microsoft Exchange Plan 1 licenses and instructions on how to migrate their email to Microsoft 365 until the outage is addressed.

"As we continue to work through the root cause of the issue, we have provided an alternate solution that will re-activate our customers’ ability to send and receive emails by providing access to an alternative email solution at no cost to them," Silva said.

"This solution will allow our impacted customers to resume regular business as soon as possible."

Detailed instructions on how to activate the free licenses and how to migrate users' mailboxes to Microsoft 365 are available in Rackspace's incident report.  

The ProxyNotShell vulnerability

While Rackspace has shared very little information about the attack, cybersecurity expert Kevin Beaumont has shared a possible explanation.

Beaumont told BleepingComputer that Rackspace appears to have been running a Microsoft Exchange server vulnerable to the ProxyNotShell vulnerability.

ProxyNotShell was a zero-day vulnerability discovered to be actively exploited in September 2022 to install web shells on Microsoft Exchange servers.

Microsoft fixed the vulnerability in November as part of their Patch Tuesday updates.

However, Beaumont discovered through Shodan that one of Rackspace's servers, 'mex06.emailsrvr.com,' was running Microsoft Exchange build 15.0.1497.40, associated with the August patch level. 

"This Exchange build number is from August 2022, before the ProxyNotShell patches became available," explained Beaumont in a post about the security incident.

Beaumont says that while long build numbers are not always reliable, it could be how Rackspace suffered the security incident.

BleepingComputer has reached out to Rackspace with questions about the security incident but has yet to hear back.

Update December 03, 08:31 EST: Revised the article and the title after Rackspace linked its ongoing outage to a security incident.
Update December 03, 12:38 EST: Added information from Kevin Beaumont.