AppSec and Software Community Respond to Log4j
The application security and the open source software communities rose to the challenge of the Java Log4j vulnerability, patching software, sharing information and providing mitigations and tools. We aren’t out of the woods yet, but their actions so far have been inspiring.
What Happened?
The new Log4j vulnerability, dubbed Log4Shell, has put the world on red alert. The vulnerability was first reported to the Apache Software Foundation on Nov 24, 2021 by an employee of the Alibaba Cloud security team and an exploit was anonymously posted to Twitter on Dec 9, 2021.
Shortly after the exploit was posted, active attacks increased dramatically. Evidence soon followed that exploits using this vulnerability had been active in the wild for up to two weeks before it became widely known. Given a public exploit for this vulnerability, the situation became an arms race over the weekend of December 10, 2021 to December 12, 2021 with many in the security community, including our teams, spending their weekends and subsequent late nights rushing to get a better understanding of how the vulnerability works and how to best detect and protect against it.
How Serious is It?
This Log4j vulnerability is as serious as it gets. In a briefing, the top U.S. cybersecurity defense official, Jen Easterly, stated that the flaw is “one of the most serious I’ve seen in my entire career, if not the most serious.”
Because of how easy it is to exploit this vulnerability, the possible impact of a successful exploit and the widespread use of Log4j (the vulnerable Java library), the vulnerability was given a maximum criticality rating of 10 in the common vulnerability scoring system (CVSS).
It is likely that this vulnerability will impact hundreds of thousands of organizations and hundreds of millions of devices worldwide. Similar to other high-impact vulnerabilities like Heartbleed and Shellshock, the number of known vulnerable products will continue to rise in the coming weeks and months, and the impact from this vulnerability will last for years.
Summary of the Vulnerability
This vulnerability is designated CVE-2021-44228 by MITRE and has been nicknamed “Log4Shell” by the application security community.
At a high level, the vulnerability combines:
- The “lookup” functionality of the Apache Log4j 2.x package
- The Java Naming and Directory Interface (JNDI) API to run a Java object
- LDAP and RMI, used to retrieve remote Java objects
- DNS query + variable resolution to sneak out sensitive data.
These are combined into a malicious string which, when parsed by a vulnerable Log4j library—in the simplest case—can be used to read environment variables of the attacked system to grab sensitive information and, in the worst case, can be used for remote code execution (RCE) leading to a full system takeover.
There are already plenty of great posts like this one discussing the technical details of what is currently known about this vulnerability, so we won’t cover that again here.
What’s at Risk?
This vulnerability can quite easily give an attacker the ability to execute arbitrary code on the compromised system (known as remote code execution, or RCE). The ability to perform RCE on a target system is an attacker’s goal. Even though the remote command will execute with whatever restricted privileges that Log4j is running with, once an attacker has a foothold in a system, they are much more likely to be able to escalate their privileges and do more damage.
But it’s not just the directly attacked systems that are at risk.
Not Just Edge Systems
The vulnerability also can give the attacker access to any of the systems in the call chain on the way to the target Log4j system (provided they are also logging using the vulnerable Log4j logging library). And that system can be a backend system behind a firewall, an intermediary third-party service, a client application, an IoT device, a mobile application, etc. Basically, anywhere the Log4j library is being used to capture logs.
And that’s part of what makes this vulnerability so dangerous. An attacker doesn’t need to be specific in their targeting. They can send their exploit payload out randomly and wait for an affected system, even ones behind perimeter protections like WAFs, API gateways and firewalls, to come back to their malicious server to get malicious code to execute. Then, they’ll know they have a successful victim they can control remotely.
Data is at Risk, Too
So, certainly, if your software is running a vulnerable Log4j library then you are at risk, but what if you aren’t? Are you safe from being exploited?
Sadly, the answer is no, you still aren’t safe. Your communications and data are still at risk. For example, let’s say that you have a Go service that is sharing data to a Python service that is sharing data to a Java service that uses Log4j. An attacker sends an attack string to your Go service which passes it to the Python service which passes it to the Java service.
While your Go and Python services will not be vulnerable, the Java service which is using Log4j could. Now, if that Java service was handling any of your sensitive data, that sensitive data might now be compromised.
Mitigations
There are generally three mitigations that everyone should consider implementing as soon as possible to protect their apps from this Log4j vulnerability. These are not mutually exclusive, and we recommend practicing defense-in-depth by implementing more than one of them.
Shield
The first mitigation is to shield all of your applications to buy some time to get the other mitigations in place. The other mitigations require knowing where you are vulnerable and then implementing the changes. To make room for that discovery to happen, throw up a shield that can detect and block the attacks by identifying their signature at runtime and stopping the traffic.
Although this is the easiest, quickest and least intrusive mitigation (no changing the app code or configuration), the drawback is that it might not catch all related exploits or might produce high false positives. This is, of course, dependent on how good your runtime defense solution is.
Patch
The best and most effective solution—which everybody should implement as soon as possible—is to patch their Log4j libraries to the latest non-vulnerable version (Log4j 2.17.0 at the time of this writing). This is the only true resolution for this vulnerability that will persist.
However, this is the most challenging mitigation to do thoroughly, because the vulnerable Log4j library can be hidden deep in your application in many different ways and finding it can be very difficult. A good software composition analysis (SCA) tool, such as Snyk, can help speed up and simplify this process in the development phase.
However, even once the vulnerable libraries are found, it could take time to test the changes everywhere before they are put into production. This is why the “shield” step is important.
Disable
The last, but certainly not least, mitigation is to implement a set of temporary countermeasures that basically turn off the vulnerable capabilities either by disabling lookups (by setting a configuration in environment variables or in Java invocation properties) or by removing JNDI libraries from the Log4j package altogether. Either way, these changes can be breaking for some apps and can be accidentally and unknowingly undone, as they are not the default configurations.
Cross-Functional Communities Come Together
The combined software and application security communities collectively answered the call to remediate this vulnerability and continue to do so. I am extremely proud of how quickly these two communities have come together and sacrificed time and sleep to research, inform, create and share tools, update packages and help protect applications within days of the first exploit becoming public. Their efforts, collaboration and information sharing are inspiring.
And we can’t let up. We need to keep pushing to patch every Log4j package everywhere. Now, more than ever, we have to keep working together for secure software.
In an increasingly interconnected world, we must work as one community (development, security and operations) to secure not just ourselves but also our neighbors and their neighbors who all make up our greater software ecosystem.
Next Steps
As we emerge from this crisis—and we will emerge from this crisis—we will have an opportunity to apply the lessons learned and reduce our exposure to future vulnerabilities. I believe there are five things we—and by ‘we’ I mean business, government, researchers and vendors, equally—need to do as industry leaders:
- First, we need to realize that this is really a form of an API-level attack, and we need to collaborate and share lessons learned from this crisis.
- Second, we need to collectively create a framework of educational resources to help educate the industry about how to mitigate and block API vulnerabilities like this.
- Third, we must educate engineers about API security design and coding patterns to effectively shift API security left.
- Fourth, we should increase our investment and commitment to bug bounty programs to uncover other hidden vulnerabilities early so we can proactively patch and mitigate.
- Fifth, vendors need to develop tools that unify development, security and operations as one team in measuring, managing and improving their API security exposure.
I discuss more details of what we are doing to help solve our current crisis in a separate blog.
