The Heartbleed bug: How a flaw in OpenSSL caused a security crisis

What is Heartbleed?

Heartbleed is a vulnerability in OpenSSL that came to light in April of 2014; it was present on thousands of web servers, including those running major sites like Yahoo.

OpenSSL is an open source code library that implements the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. The vulnerability meant that a malicious user could easily trick a vulnerable web server into sending sensitive information, including usernames and passwords.

The TLS/SSL standards are crucial for modern web encryption, and while the flaw was in the OpenSSL implementation rather than the standards themselves, OpenSSL is so widely used—when the bug was made public, it affected 17% of all SSL servers—that it precipitated a security crisis.

Why is Heartbleed called Heartbleed?

The name Heartbleed comes from heartbeat, which is the name for an important component of the TLS/SSL protocol. The heartbeat is how two computers communicating with one another let each other know that they’re still connected even if the user isn’t downloading or uploading anything at the moment. Occasionally, one of those computers will send an encrypted piece of data, called a heartbeat request, to the other. The second computer will reply back with the exact same encrypted piece of data, proving that the connection is still in place.

The Heartbleed vulnerability gets its name because attackers can use heartbeat requests to extract information from a target server—metaphorically, the victim bleeds out sensitive data through its heartbeat requests.

How does Heartbleed work?

Heartbleed works by taking advantage of a crucial fact: a heartbeat request includes information about its own length, but the vulnerable version of the OpenSSL library doesn’t check to make sure that information is accurate, and an attacker can use this to trick the target server into allowing the attacker access to parts of its memory that should remain private. To understand the mechanism behind this, let’s walk through a typical example of OpenSSL in action.

Imagine you’re reading your Yahoo mail but haven’t done anything in a while to load more information. Your web browser wants to make sure Yahoo’s server is still up and listening, so it will send a message saying, in essence, “This is a 40 KB message you’re about to get. Repeat it all back to me.” This is the heartbeat request we discussed earlier. Heartbeat requests can be of variable sizes (up to 64 KB), and each request needs to include information about its specific length.

When Yahoo’s server receives that message, it allocates a memory buffer—a region of physical memory where it can store information—equal in size to the reported length of the heartbeat request. In our example, that’s 40 KB. Next, the server stores the encrypted data from the request into that memory buffer, then immediately reads the data back out of it and sends it back to your web browser. When your browser gets back the same information it sent out, it can be sure it still has a connection to the server it’s been talking to up to this point.

That’s how it’s supposed to work. The Heartbleed vulnerability arose because OpenSSL’s implementation of the heartbeat functionality was missing a crucial safeguard: the computer that received the heartbeat request never checked to make sure the request was actually as long as it claimed to be. So if a request said it was 40 KB long but was actually only 20 KB, the receiving computer would set aside 40 KB of memory buffer, then store the 20 KB it actually received, then send back that 20 KB plus whatever happened to be in the next 20 KB of memory. There could be all kinds of things in that 20 KB, because even when a computer is done with information, that data persists in memory buffers until something else comes along to overwrite it. That extra 20 KB of data is information that the attacker has now extracted from the web server.

Why is Heartbleed dangerous?

Heartbleed is dangerous because it lets an attacker see the contents of that memory buffer, which could include sensitive information. Admittedly, if you’re the attacker, you have no way to know in advance what might be lurking in that 20 KB you just grabbed off the server, but there are a number of possibilities. It could be gibberish or useless cruft. If you’re really lucky, you could get SSL private keys, which would allow for the decryption of secure communication to that server; this is unlikely, but would be the holy grail for an attacker. More commonly, you could get back usernames and passwords that had been submitted to applications and services running on the server, which would allow you to log in to those apps and gain access to user accounts.

Randall Munroe’s web comic xkcd is known for making difficult scientific concepts accessible, especially in computer science, Munroe’s specialty. This xkcd comic from 2014 does a great job of summarizing how the Heartbleed vulnerability works in a concise way.

How was Heartbleed discovered?

Heartbleed was actually discovered by two different groups, working independently, in very different ways: once in the course of a review of OpenSSL’s open source codebase, and once during a series of simulated attacks against servers running OpenSSL. The two independent discoveries happened within weeks of one another, which is somewhat ironic considering that the vulnerability had lurked undetected for two years.

The first to discover Heartbleed was Neel Mehta, an engineer working at Google, in March of 2014. Mehta had decided to do a line-by-line audit of the OpenSSL code because two earlier SSL flaws that had been uncovered earlier that year, goto fail and GnuTLS, made him suspect that other dangers might be lurking elsewhere in the SSL/TLS ecosystem. After he spotted the bug and realized its implications, Google began to privately warn some infrastructure companies like CloudFlare about it, although they did not make it public or say anything to the U.S. government.

The second discovery happened at Codenomicon, a Finnish cybersecurity firm, just a few weeks later. The company was working on a product called Safeguard, designed for penetration testing on encryption and authentication tooling. In the grand tech industry tradition of eating your own dog food, Codenomicon decided to test Safeguard on their own infrastructure—and discovered that they could gain access to a shocking amount of data.

Codenomicon proceeded quite differently from Google: not only did they publicize their discovery, but they branded it: they were the ones who came up with the Heartbleed name, and they even designed a logo for it. It was one of the first (though by no means the last) examples of a security company turning the discovery of a vulnerability into a marketing opportunity.

Heartbleed CVE

The identifier for Heartbleed in the common vulnerabilities and exposures (CVE) system is CVE-2014-0160; you can follow that link for a wealth of information about the bug. “Heartbleed” is obviously a lot catchier, so you can understand why Codenomicon’s name stuck.

Heartbleed code

A single line of code contains the mistake that gave rise to the Heartbleed vulnerability:

memcpy(bp, pl, payload);

memcpy() is the command that copies data. bp is the place it’s copying it to, pl is where it’s being copied from, and payload is the length of the data being copied. As we’ve seen, the problem is that there’s never any attempt to check if the amount of data in pl is equal to the value given of payload.

The most ironic thing here is that OpenSSL is open source software. Anyone could look at the code, and presumably hundreds did, but until Mehta and the Codenomicon team stumbled upon it, nobody noticed this fairly elementary coding error. In fact, because open source projects like OpenSSL meticulously keep track of contributors, we know whose error it was: Robin Seggelman, a German software developer, who had made numerous contributions to the OpenSSL project.

Heartbleed exploits: Who is affected by Heartbleed?

There have been real-world exploits of the Heartbleed vulnerability, though it’s not clear if any took place before the bug was widely publicized. It’s possible that some attempted attacks detected by security companies as early as 2013 were probing for the vulnerability—and some think the attackers were government security agencies.

After April of 2014, when Codenomicon made the vulnerability public, there was a flurry of activity and a certain amount of chaos as companies scrambled to update their systems; for instance, Yahoo and OKCupid users were briefly advised not to log into their accounts until those services managed patch their installs of OpenSSL, and to change their passwords once they did regain access.

While those big companies managed to get their ducks in a row before anything bad befell them, hackers were able to exploit the vulnerability in several cases. An attack on Community Health Systems that stole patient data was blamed on Heartbleed, as was the theft of hundreds of social ID numbers from the Canadian Revenue Agency.

Heartbleed cost

Heartbleed had costs that went beyond the damages caused by these successful attacks; Security Magazine estimated that just the cost of thousands of organizations needing to revoke and replace their SSL certificates could run as high as $500 million. Add in the worker-hours required to check and update systems and you have a big spike in spending that can be directly tied to this vulnerability.

The Heartbleed fix

The Heartbleed fix was rolled out in version 1.0.1g of the OpenSSL library, released on April 8, 2014, and was also included in all subsequent versions of the software. You can fix the Heartbleed vulnerability by upgrading to the latest version of OpenSSL, and can find links to all the latest code on the OpenSSL website.

If you’re curious about the code that implements the fix, you can look at it—after all, OpenSSL is open source:

/* Read type and payload length first */

if (1 + 2 + 16 > s->s3->relent)

return 0;

/* silently discard */

hbtype = *p++;

n2s(p, payload);

if (1 + 2 + payload + 16 > s->s3->rrec.length)

return 0;

/* silently discard per RFC 6520 sec. 4 */

pl = p;

The first part of this code makes sure that the heartbeat request isn’t 0 KB, which can cause problems. The second part makes sure the request is actually as long as it says it is.

Is Heartbleed still a problem?

Given that Heartbleed was discovered and patched more than eight years ago, you may be surprised to learn that many servers are still harboring the Heartbleed vulnerability—in fact, there were over 200,000 online in November 2020, according a researcher at the SANS Internet Storm Center. While that number has probably come down a bit since then, there are almost certainly a number of vulnerable servers still waiting to be hacked. Seasoned security pros probably won’t be that surprised to learn this—it’s all too common for companies to neglect patching to avoid downtime on mission-critical systems without backups or simply out of neglect—but the sheer number of unpatched machines should be a wakeup call on the importance of rolling out a robust patch management program in your own shop.

Heartbleed vulnerability test: How to detect Heartbleed

You can easily test your servers to detect the Heartbleed vulnerability using free online tools. For instance, pentest-tools.com has a free web-based test that lets you input a URL to discover if a server has been properly patched for Heartbleed and a number of other vulnerabilities.

If you discover that a server under your control has been left vulnerable for some time, there’s more to do than just update the OpenSSL code. For instance, you should change the SSL certificates used by the servers, since they may have been compromised without leaving a trace. More pedestrian but still important: users who have accounts on the system should change their passwords.