Securing CI/CD pipelines: 6 best practices

Recent cyberattacks leveraging weaknesses in continuous integration/continuous delivery (CI/CD) pipelines and developer tooling warrant a need for increased security of the developer infrastructure. Prominently, the Codecov supply-chain attack has alerted everyone against storing secrets in CI/CD environment variables, no matter how safe the environment might be.

By breaching a Bash uploader used by thousands of developers, Codecov attackers managed to siphon credentials, keys, and API tokens from customer environments, remained undetected for two months, and further, reportedly breached hundreds of restricted customer networks. Likewise, attacks on automation tools like Jenkins, GitHub Actions and cloud-native containerized environments have further prompted companies to explore and deploy effective defenses for these tools.

Below are some best practices to ensure your CI/CD pipelines remain secure.

1. Stop storing secrets in CI/CD environments

The reason behind the large success of Codecov supply-chain attack remains that the environment variables exfiltrated by the attackers contained hardcoded secrets including passwords, tokens, and keys. Because some of these credentials gave attackers access to the companies’ private GitHub repositories, further data exfiltration could occur from these private repos that contained data that should have remained confidential. 

Although multiple Codecov customers including HashiCorp, Twilio, Rapid7 and Monday.com disclosed the impact from the supply-chain attack, the most significant data breach to come to light thus far occurred at Japanese e-commerce giant Mercari. More than 27,000 total records pertaining to Mercari customers’ finances, merchants, business partners, company employees, contractors, and various entities were exposed to unauthorized external actors following the Codecov attack.

Granted, each of these attacks may have started from the Codecov breach, some have questioned why personally identifiable information (PII) such as customer financial records were stored in private GitHub repositories.

Similar concerns have been raised with HashiCorp’s GPG private key stored in the CI/CD environment. This is the secret key that was used to sign and verify software releases published by HashiCorp. Before the key was revoked, an attacker could have abused the key to forge HashiCorp’s signature on a malicious software release. “Why is no one talking about the fact that HashiCorp, the maker of Vault, had their signing key stored as an ENV? Dear me… This makes me feel better about my life,” tweeted a developer.

Organizations need to rethink what secrets can be stored in CI/CD tooling, environment variables, and private GitHub repositories. If an application requires a credential or token to be stored in these places, it’s best to store credentials to an account or resource with the lowest privileges, just what’s necessary to accomplish the task—often referred to as the principle of least privilege. That way, even if the secret does get exposed in an unprecedented attack, the damage is contained.

2. Scrutinize automated pull requests and scheduled tasks

CI/CD automation tooling like GitHub Actions allows developers to set up scheduled tasks for their code repositories, such as automatically vetting and processing incoming pull requests. But what happens if the contributor making the pull request to an open-source project has malicious intent? In April 2021, GitHub Actions was abused by attackers who made automated pull requests to hundreds of repositories with the intention of mining cryptocurrency using GitHub’s infrastructure. This large-scale attack occurred after the “flaw” with GitHub Actions had been reported earlier in February.

At a minimum, these pull requests can abuse GitHub’s servers to mine cryptocurrency or execute the attacker’s malicious code. If the project owners are negligent and merge these pull requests, they have now introduced malicious code into their repository and the wider software supply chain. In May, GitLab reported tackling similar cryptomining attacks on its platform from attackers abusing “free minutes” (quota) allotted to new accounts. Because the very nature of CI/CD automation tools like GitHub Actions and GitLab is to provide ease of automating key tasks, gatekeeping becomes a challenge. What may have been an intentional feature soon turns into a security flaw, following abuse by threat actors.

GitHub recently announced new features to combat abuse of its Actions platform by cryptomining attackers: “Pull requests from first-time contributors will require manual approval from a repository collaborator with write access before any Actions workflows run. When a first-time contributor opens a pull request, they’ll see a message that a maintainer must approve their Actions workflow before it will run,” stated GitHub product manager Chris Patterson in a blog post. Leading CI/CD solutions and DevOps platforms can follow GitHub’s lead in adding some security checks in place to deter large scale abuse of their infrastructure by malicious actors.

3. Harden and periodically audit your cloud-native containers

Nothing beats following standard best practices such as ensuring your production containers are properly configured and hardened against common attack vectors. This includes securing your pipeline configuration.

However, simple misconfiguration errors sometimes become hard to spot by humans. Then comes the question, are your Docker-based environments free from vulnerabilities? This is why regularly performing a security audit of your containers for weaknesses, scanning container images and manifest files to spot common security issues remains helpful.

Investing in dependable cloud-native container security solutions that can automate much of this is advisable. The huge volume of security vulnerabilities that are reported every year makes it virtually impossible for a human to keep a track of them.

Additionally, as companies adopt Kubernetes frameworks and Docker containers to deploy their applications, container security solutions with built-in web application firewalls can detect and block suspicious network traffic early on. This can help prevent a bigger compromise, even if the attackers are able to penetrate a container and gain initial access.

4. Integrate deep code scanning to automate code quality checks

Having tooling in place to automatically spot code quality issues, security vulnerabilities, and bugs like memory leaks or race conditions before code commits reach production is an effective strategy in securing your CI/CD pipeline from the start. Although the focus seems to be primarily on preventing cyberattacks, innocuous bugs are just as likely to have large-scale impact. We recently saw this with the global Fastly outage that knocked major sites around the world offline.

Solutions like GitHub code scanner or Sonatype’s Lift [full disclosure: Sonatype is my employer] integrate seamlessly within your existing coding workflow and provide basic safeguards at no cost to developers. Ultimately, an organization’s goal should be to support its developers to do their best work, while preventing the introduction of bugs or security vulnerabilities in the applications as much as practically possible. This needs to happen while causing minimal friction between the development and security teams. This is where real-time notifications alerting the developer of a potential oversight as they code can save everyone time and secure the overall CI/CD workflow from the beginning. 

5. Patch early against latest CI/CD tooling vulnerabilities

In March 2021, attackers employed a cryptomining botnet called z0Miner to mine Monero (XMR) cryptocurrency on vulnerable Jenkins and ElasticSearch servers. By exploiting remote code execution (RCE) vulnerabilities in internet-facing servers, attackers attempted to infect and take over the automation infrastructure to conduct their nefarious activities.

Likewise, as reported last year, Jenkins servers could be exploited by attackers to rapidly cause distributed denial of service (DDoS). This resulted from a UDP amplification reflection DoS vulnerability, tracked as CVE-2020-2100, which affects versions below v2.219 of Jenkins, and below 2.204.1 of Jenkins LTS.

Patching automation tooling and pipelines against these severe vulnerabilities as soon as these are discovered remains vital to ensuring the security of your CI/CD infrastructure. 

6. Verify integrity of updates before applying them

Applying latest updates and patches is sound advice, but are you sure the update you are receiving has not been tampered with? The advice to just relentlessly “update to the latest version” which has been the mantra of security professionals for decades has been challenged following the SolarWinds supply-chain attack.

In the SolarWinds incident, malicious updates made to the Orion IT product enabled attackers to distribute malicious code downstream to over 18,000 customers. Yet again, Passwordstate password manager’s “In-Place Upgrade functionality” had been compromised to distribute malicious updates to Passwordstate users. Therefore, blindly applying product updates is a bad idea.

In Codecov’s case, a simple integrity check led to the discovery of the two-month long breach. A customer noticed a discrepancy between the checksum (hash) of the Bash Uploader that was hosted on the server and the legitimate checksum listed on Codecov’s GitHub repository immediately contacted Codecov which remedied the problem. 

As such, a defense in depth approach warrants that the integrity of any updates, patches, and downloads be verified so that risks from a sophisticated supply-chain attack can be ruled out.