Many companies have rushed to implement continuous integration and continuous delivery (CI/CD) pipelines to streamline their software development workflows. Far fewer have taken the additional step to automate continuous deployment, a practice of using CI/CD pipelines to push changes into production continuously. Understandably so.
The thought of pushing code to production as frequently as daily or hourly gives me the chills. In fact, several years ago, I wrote an article about the downsides of continuous deployment. Another article, "When should responsible devops teams increase deployment frequency," challenges the assumption that more frequent deployments are better.
Much has changed over the last few years, however, and many more devops teams are embracing the skills, practices, and tools to automate high quality and reliable deployments. This article explains the differences between continuous delivery and continuous deployment, then proposes five things devops teams should do before automating continuous deployment in their CI/CD pipelines.
Continuous delivery vs. continuous deployment
Kulbir Raina, agile and devops leader at Capgemini, shares a definition that helps us differentiate continuous delivery from continuous deployment. He says, “Continuous delivery is the end-to-end automated flow of software releases until production, while continuous deployment is the automated process that pushes the software package in that flow into production post-continuous integration via a pre-tested process.”
Automating production deployments has more risks because the results impact the business, customers, and end users. If a devops team decides to automate deployments, the deployment process must include continuous testing and robust error handling. Otherwise, a deployment could create performance issues, unreliable systems, security holes, and defects found in production.
Mike Saccotelli, director of software engineering at SPR, adds, “The primary difference between an organization operating a continuous delivery model versus a continuous deployment model is the level of maturity and sophistication of their build and deployment processes.”
Devops teams can use the following checklist to prepare for upgrading CI/CD pipelines for continuous deployment.
1. Evaluate the business benefits
Continuous deployment, as a principle, can be applied to many applications and even in the most regulated industries. Tim Lucas, co-founder and co-CEO of Buildkite, says, “Continuous deployment can be adopted per project, and the best orgs set goals for moving as many projects as possible to this model. Even in finance and regulated industries, the majority of projects can adopt this model. We even see self-driving car companies doing continuous deployment.”
While devops teams can implement continuous deployment in many projects, the question is, where does it offer a strong business case and significant technical advantages? Projects deploying features and fixes frequently, and where a modernized architecture simplifies the automations, are the more promising to transition to continuous deployment.
2. Prepare the development team
Lucas shares some of the prerequisites that should be part of the software development process before moving to a continuous deployment model. He says, “Continuous deployment is true agility, the fastest way from code change to production. It requires always keeping the main branch in a shippable state, automating tests, and high-quality tooling you can trust and have confidence in.”
Some of the development responsibilities and disciplines for developers looking to automate continuous deployment include:
- Continuous testing with sufficient code coverage to ensure changes don’t create new defects.
- Static and dynamic code analysis tools for testing security, performance, and other code quality issues.
- A commitment to shift-left security practices, including best practices around API security.
- Observability standards around applications and microservices.
- Feature flagging, so that new features can be turned on and off, or controlled to a subset of users.
Saccotelli adds, “Continuous deployment is predicated on the development team having a more mature understanding of quality code so that this process can be successful. If the code is poor or untested, that will create an unreliable system and quickly push out bugs and vulnerabilities into production.”
3. Prepare the operations team
So the CI/CD pipeline runs and deploys new code to production. Does this mean that devops teams are in the clear, their work done, and everyone can move on to the next release?
Not so fast. Despite all the work developers do to ensure builds don’t break, automate code testing, and control what code is enabled in production, there is still some risk that a deployment could cause production issues. Monitoring business services, applications, and systems is an operations responsibility within devops, and their competencies to support continuous deployments start well before enabling deployment automation. Best practices include the following:
- Use infrastructure as code and containers to ensure consistent infrastructure configurations between development, testing, production, and other environments.
- Employ application- and system-level monitoring tools that also load and analyze the observability data created by applications and microservices.
- Choose an AIOps platform that integrates monitoring, alerts, and observability data across the full stack from applications to microservices and datastores and correlates events into manageable incidents.
- Set up canary releases to control cutover and traffic to new deployments and reduce the risks of harder cutover strategies.
- Have a robust set of security tools, including API gateways, web application firewalls, container security, threat monitoring, and sensitive data monitoring that mitigate risks in highly dynamic application environments.
4. Integrate ITSM and workflows across teams and tools
We’re not done yet, even with all the development and operations competencies instituted. Let’s say the devops team commits code, continuous deployment moves the change into production, and application performance monitoring tools are running. How quickly will the right devops team members be alerted so that they can triage incidents, investigate the root cause, and quickly address any issues?
Lucas shares that "flaky tests are the #1 risk" when moving to continuous deployment. He cites unreliable CI/CD tools, poor production monitoring and on-call practices, and the lack of a true partnership between engineering and IT as further risks.
Without workflow and integrations between monitoring, AIOps, IT service management (ITSM), agile, and communication tools, a devops team's time to respond and resolve issues may lag behind its deployment velocities. This gap can create stressful moments and erode the partnership between development and operations. A best practice is to ensure IT tools and workflow are integrated, so devops teams can keep up with any issues that continuous deployments create.
5. Define risk-based decision gates and KPIs
Kristin Baskett, director of product marketing at Copado, provides one critical element needed in continuous deployments. She says, “While reckless automation can hinder a system, the right automation helps organizations achieve the flexibility and consistency they need to truly benefit from devops. When developers can integrate code automatically, collaboration improves. And by investing in test automation and quality gates, organizations can innovate faster, with less risk.”
In addition to test automation, Baskett mentions implementing quality gates that should be extended to other risk assessments. When a build triggers a continuous deployment, quality gates implement business rules for determining which deployments can go to production and which might require a compliance review or management signoff.
Other best management practices that support continuous deployments include developing devops key performance indicators (KPIs), formalizing feedback loops, and developing communication strategies.
Continuous deployments can yield many business and technology benefits, but disciplined devops teams and IT organizations should ensure that best practices and tools are in place before using automation to increase deployment frequencies.