The scourge of software supply chain attacks—an increasingly common hacking technique that hides malicious code in a widely used legitimate program—can take many forms. Hackers can penetrate an update server to seed out their malware, or even break into the network where the software was developed to corrupt it at the source. Or, in the case of one particularly insidious software supply chain attacker known as Jia Tan, they can spend two years politely and enthusiastically volunteering to help.
Over the weekend, the cybersecurity and open source software community was shocked by the news that a relatively new, experimental version of XZ Utils—a compression utility integrated into many popular distributions of Linux—contained a backdoor that would have allowed hackers in possession of a specific private key to connect to the backdoored system and run their own commands as an administrator. Only some chance detective work carried out by a lone Microsoft engineer, Andres Freund—who’d detected a strange delay in how the remote connection protocol SSH was running in a version of the Linux variant Debian—caught the spy trick before it ended up in many millions of systems worldwide.
That XZ Utils backdoor, it’s now clear, was inserted by none other than the lead open source steward of XZ Utils, a developer who went by the name Jia Tan. In the wake of the backdoor's discovery, a mystery percolating through the tech world remains: Who is Jia Tan, and who did he, or she—or very likely they—truly work for?
Jia Tan exploited open source software’s crowdsourced approach to coding whereby anyone can suggest changes to a program on code repositories like GitHub, where the changes are reviewed by other coders before they’re integrated into the software. Peeling back Jia Tan’s documented history in the open source programming world reveals that they first appeared in November 2021 with the GitHub username JiaT75, then made contributions to other open source projects using the name Jia Tan, or sometimes Jia Cheong Tan, for more than a year before beginning to submit changes to XZ Utils.
By January 2023, Jia Tan’s code was being integrated into XZ Utils. Over the next year, they would largely take control of the project from its original maintainer, Lasse Collin, a change driven in part by nagging emails sent to Collin by a handful users complaining about slow updates. (Whether those users were unwitting accomplices, or actually working with Jia Tan to persuade Collin to relinquish control, remains unclear. None of the users replied to requests for comment from WIRED.) Finally, Jia Tan added their stealthy backdoor to a version of XZ Utils in February of this year.
That inhumanly patient approach, along with the technical features and sophistication of the backdoor itself, has led many in the cybersecurity world to believe that Jia Tan must, in fact, be a handle operated by state-sponsored hackers—and very good ones. “This multiyear operation was very cunning, and the implanted backdoor is incredibly deceptive,” says Costin Raiu, who until last year served as the most senior researcher and head of the global research and analysis team at Russian cybersecurity firm Kaspersky. “I’d say this is a nation-state-backed group, one with long-term goals in mind that affords to invest into multiyear infiltration of open source projects.”
As for which nation, Raiu names the usual suspects: China, Russia, and North Korea. He says it’s still too early to know the true culprit. “One thing is for sure clear,” he adds. “This was more cunning than all previous software supply chain attacks I’ve seen.”
A Very Private, Very Busy Programmer
As scrutiny around Jia Tan has mounted since the revelation of the XZ Utils backdoor last Friday, researchers have noted that the persona has remarkably good operational security. Independent security reporter Brian Krebs writes that he could find “zero trace” of Jia Tan’s email address outside of the messages they sent to fellow open source contributors, even after scouring breached databases. Jia Tan also appears to have routed all their communications through a VPN with a Singaporean IP address.
The lack of any other online presence linked to Jia Tan points toward the account being a “single-purpose invented persona” and indicates how much sophistication, patience, and thought was put into developing the backdoor, says Will Thomas, an instructor at the SANS Institute, a cybersecurity training firm. The Jia Tan persona has vanished since the backdoor was discovered, and emails sent by WIRED to a Gmail address linked to it have gone unanswered. Jia Tan’s GitHub account has been suspended, a company spokesperson tells WIRED.
In fact, the only real footprints Jia Tan appears to have left behind were their contributions to the open source development community, where they were a prolific contributor: Disturbingly, Jia Tan’s first code change was to the “libarchive” compression library, another very widely used open source component. That first change swapped one function with a less secure alternative, potentially attempting another malicious code change, notes developer Evan Boehs in his detailed Jia Tan timeline—though the problem has since been fixed.
In total, Jia Tan made 6,000 code changes to at least seven projects between 2021 and February 2024, according to Michael Scott, the cofounder of the cybersecurity firm NetRise who previously worked in the Marine Corps cyberwarfare group under US Cyber Command. Determining all the branching effects of those changes is nearly impossible, Scott says. Because those changes, known as “commits,” are often batched into collections in a process known as “squashing commits,” it’s not always apparent which exact changes were made by Jia Tan. And the difficulty of tracing which of the many versions of a library like libarchive ended up in which software adds yet another layer of obfuscation. “It’s going to be a bit of a mess pulling on this thread and trying to figure out where all these things ended up,” Scott says.
Scott notes that, throughout this time, Jia Tan was also emailing with other contributors, writing in a “very concise, very dry,” but not unfriendly tone that Scott compares to the output of ChatGPT. “Nice job to both of you for getting this feature as far as it is already,” Jia Tan wrote at one point. Or, at another: “Let me know your thoughts on these patches when you have a chance :)” Jordi Mas, a developer who contributed to XZ Utils and had emailed “feedback” from Jia Tan, says in retrospect that the account went to extra levels to build trust in the persona.
Ultimately, Scott argues that those three years of code changes and polite emails were likely not spent sabotaging multiple software projects, but rather building up a history of credibility in preparation for the sabotage of XZ Utils specifically—and potentially other projects in the future. “He just never got to that step because we got lucky and found his stuff,” says Scott. “So that’s burned now, and he’s gonna have to go back to square one.”
Technical Ticks and Time Zones
Despite Jia Tan’s persona as a single individual, their yearslong preparation is a hallmark of a well-organized state-sponsored hacker group, argues Raiu, the former Kaspersky lead researcher. So too are the technical hallmarks of the XZ Utils malicious code that Jia Tan added. Raiu notes that, at a glance, the code truly looks like a compression tool. “It’s written in a very subversive manner,” he says. It’s also a “passive” backdoor, Raiu says, so it wouldn’t reach out to a command-and-control server that might help identify the backdoor’s operator. Instead, it waits for the operator to connect to the target machine via SSH and authenticate with a private key—one generated with a particularly strong cryptographic function known as ED448.
The backdoor’s careful design could be the work of US hackers, Raiu notes, but he suggests that’s unlikely, since the US wouldn’t typically sabotage open source projects—and if it did, the National Security Agency would probably use a quantum-resistant cryptographic function, which ED448 is not. That leaves non-US groups with a history of supply chain attacks, Raiu suggests, like China’s APT41, North Korea’s Lazarus Group, and Russia’s APT29.
At a glance, Jia Tan certainly looks East Asian—or is meant to. The time zone of Jia Tan’s commits are UTC+8: That’s China’s time zone, and only an hour off from North Korea’s. However, an analysis by two researchers, Rhea Karty and Simon Henniger, suggests that Jia Tan may have simply changed the time zone of their computer to UTC+8 before every commit. In fact, several commits were made with a computer set to an Eastern European or Middle Eastern time zone instead, perhaps when Jia Tan forgot to make the change.
“Another indication that they are not from China is the fact that they worked on notable Chinese holidays,” say Karty and Henniger, students at Dartmouth College and the Technical University of Munich, respectively. They note that Jia Tan also didn't submit new code on Christmas or New Year's. Boehs, the developer, adds that much of the work starts at 9 am and ends at 5 pm for Eastern European or Middle Eastern time zones. “The time range of commits suggests this was not some project that they did outside of work,” Boehs says.
Though that leaves countries like Iran and Israel as possibilities, the majority of clues lead back to Russia, and specifically Russia’s APT29 hacking group, argues Dave Aitel, a former NSA hacker and founder of the cybersecurity firm Immunity. Aitel points out that APT29—widely believed to work for Russia’s foreign intelligence agency, known as the SVR—has a reputation for technical care of a kind that few other hacker groups show. APT29 also carried out the Solar Winds compromise, perhaps the most deftly coordinated and effective software supply chain attack in history. That operation matches the style of the XZ Utils backdoor far more than the cruder supply chain attacks of APT41 or Lazarus, by comparison.
“It could very well be someone else,” says Aitel. “But I mean, if you’re looking for the most sophisticated supply chain attacks on the planet, that’s going to be our dear friends at the SVR.”
Security researchers agree, at least, that it’s unlikely that Jia Tan is a real person, or even one person working alone. Instead, it seems clear that the persona was the online embodiment of a new tactic from a new, well-organized organization—a tactic that nearly worked. That means we should expect to see Jia Tan return by other names: seemingly polite and enthusiastic contributors to open source projects, hiding a government’s secret intentions in their code commits.
Updated 4/3/2024 at 12:30 pm ET to note the possibility of Israeli or Iranian involvement.
