Continuous Delivery with Docker and Jenkins, 3rd Edition E-Book

Rafal Leszko

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 374

Veröffentlichungsjahr: 2022

Beschreibung

This updated third edition of Continuous Delivery with Docker and Jenkins will explain the advantages of combining Jenkins and Docker to improve the continuous integration and delivery process of app development.
You’ll start by setting up a Docker server and configuring Jenkins on it. Next, you’ll discover steps for building applications and microservices on Dockerfiles and integrating them with Jenkins using continuous delivery processes such as continuous integration, automated acceptance testing, configuration management, and Infrastructure as Code. Moving ahead, you'll learn how to ensure quick application deployment with Docker containers, along with scaling Jenkins using Kubernetes. Later, you’ll explore how to deploy applications using Docker images and test them with Jenkins. Toward the concluding chapters, the book will focus on missing parts of the CD pipeline, such as the environments and infrastructure, application versioning, and non-functional testing.
By the end of this continuous integration and continuous delivery book, you’ll have gained the skills you need to enhance the DevOps workflow by integrating the functionalities of Docker and Jenkins.

Leseprobe

Continuous Delivery with Docker and Jenkins

Third Edition

Create secure applications by building complete CI/CD pipelines

Rafał Leszko

BIRMINGHAM—MUMBAI

Continuous Delivery with Docker and Jenkins
Third Edition

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Associate Group Product Manager: Rahul Nair

Publishing Product Manager: Niranjan Naikwadi

Senior Editor: Athikho Sapuni Rishana

Content Development Editor: Sayali Pingale

Technical Editor: Shruthi Shetty

Copy Editor: Safis Editing

Associate Project Manager: Neil Dmello

Proofreader: Safis Editing

Indexer: Tejal Daruwale Soni

Production Designer: Alishon Mendonca

Senior Marketing Coordinator: Sanjana Gupta

Marketing Coordinator: Nimisha Dua

First published: August 2017

Second edition: May 2019

Third edition: April 2022

Production reference: 1110422

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

978-1-80323-748-0

www.packt.com

– To my wonderful wife, Maria, for all her love, wisdom, and smile.

Contributors

About the author

Rafał Leszko is a passionate software developer, trainer, and conference speaker living in Krakow, Poland. He has spent his career writing code, designing architecture, and being responsible for tech in a number of companies and organizations, including Hazelcast, Google, and CERN. Always open to new challenges, he has given talks and conducted workshops at numerous international conferences, including Devoxx and Voxxed Days.

About the reviewer

Werner Dijkerman is a freelance cloud (certified), Kubernetes, and DevOps engineer, currently focused on, and working with, cloud-native solutions and tools including AWS, Ansible, Kubernetes, and Terraform. He is also focused on Infrastructure as Code and monitoring the correct “thing” with tools such as Zabbix, Prometheus, and the ELK Stack, with a passion for automating everything and avoiding doing anything that resembles manual work.

– Big thanks, hugs, and shoutouts to Anca Borodi, Theo Punter, and everyone else at COERA!

Table of Contents

Preface

Section 1 – Setting Up the Environment

Chapter 1: Introducing Continuous Delivery

Understanding CD

The traditional delivery process

The benefits of CD

Success stories

The automated deployment pipeline

Continuous integration

Automated acceptance testing

Configuration management

Prerequisites to CD

Organizational prerequisites

Technical and development prerequisites

Combining CD and microservices

Building the CD process

Introducing tools

Creating a complete CD system

Summary

Questions

Further reading

Chapter 2: Introducing Docker

Technical requirements

What is Docker?

Containerization versus virtualization

The need for Docker

Kittens and cattle

Alternative containerization technologies

Installing Docker

Prerequisites for Docker

Installing on a local machine

Installing on a server

Running Docker hello-world

Docker components

Docker client and server

Docker images and containers

Docker applications

Building Docker images

docker commit

Dockerfile

Complete Docker application

Environment variables

Docker container states

Docker networking

Running services

Container networks

Exposing container ports

Automatic port assignment

Using Docker volumes

Using names in Docker

Naming containers

Tagging images

Docker cleanup

Cleaning up containers

Cleaning up images

Docker commands overview

Summary

Exercises

Questions

Further reading

Chapter 3: Configuring Jenkins

Technical requirements

What is Jenkins?

Installing Jenkins

Installing Jenkins with Docker

Installing Jenkins with dedicated packages

Initial configuration

Installing Jenkins in Kubernetes

Jenkins in the cloud

Jenkins – Hello World

Jenkins architecture

Master and agents

Scalability

Test and production instances

Sample architecture

Configuring agents

Communication protocols

Setting agents

Testing agents

Comparing Docker pipeline builds and Docker agents

Custom Jenkins images

Building the Jenkins agent

Building the Jenkins master

Configuration and management

Plugins

Security

Backup

Jenkins Blue Ocean UI

Summary

Exercises

Questions

Further reading

Section 2 – Architecting and Testing an Application

Chapter 4: Continuous Integration Pipeline

Technical requirements

Introducing pipelines

The pipeline structure

A multi-stage Hello World

The pipeline syntax

The commit pipeline

Checkout

Compile

Unit tests

Jenkinsfile

Code-quality stages

Code coverage

Static code analysis

SonarQube

Triggers and notifications

Triggers

Notifications

Team development strategies

Development workflows

Adopting continuous integration

Jenkins multi-branch

Non-technical requirements

Summary

Exercises

Questions

Further reading

Chapter 5: Automated Acceptance Testing

Technical requirements

Introducing acceptance testing

Installing and using the Docker Registry

The artifact repository

Installing a Docker Registry

Using the Docker Registry

Acceptance tests in the Jenkins pipeline

The Docker build stage

The Docker push stage

The acceptance testing stage

Writing acceptance tests

Writing user-facing tests

Using the acceptance testing framework

Acceptance test-driven development

Summary

Exercises

Questions

Further reading

Chapter 6: Clustering with Kubernetes

Technical requirements

Server clustering

Introducing server clustering

Introducing Kubernetes

Kubernetes features overview

Kubernetes installation

Kubernetes client

Kubernetes server

Verifying the Kubernetes setup

Using Kubernetes

Deploying an application

Deploying a Kubernetes Service

Exposing an application

Advanced Kubernetes

Scaling an application

Updating an application

Rolling updates

Kubernetes objects and workloads

Application dependencies

The Kubernetes DNS resolution

Multiapplication system overview

Multiapplication system implementation

Multiapplication system testing

Alternative cluster management systems

Docker Swarm

Apache Mesos

Comparing features

Summary

Exercises

Questions

Further reading

Section 3 – Deploying an Application

Chapter 7: Configuration Management with Ansible

Technical requirements

Introducing configuration management

Traits of good configuration management

Overview of configuration management tools

Installing Ansible

Ansible server requirements

Ansible installation

Using Ansible

Creating an inventory

Ad hoc commands

Playbooks

Roles

Deployment with Ansible

Installing Hazelcast

Deploying a web service

Running the deployment

Ansible with Docker and Kubernetes

Benefits of Ansible

The Ansible Docker playbook

The Ansible Kubernetes playbook

Introducing IaC

Benefits of IaC

Tools for IaC

Introduction to Terraform

Understanding Terraform

Installing Terraform

Using Terraform

Terraform and Kubernetes

Summary

Exercises

Questions

Further reading

Chapter 8: Continuous Delivery Pipeline

Technical requirements

Environments and infrastructure

Types of environments

Environments in continuous delivery

Securing environments

Non-functional testing

Types of non-functional test

Non-functional challenges

Application versioning

Versioning strategies

Versioning in the Jenkins pipeline

Completing the continuous delivery pipeline

Inventory

Versioning

The remote staging environment

The acceptance testing environment

Release

Smoke testing

Complete Jenkinsfile

Summary

Exercises

Questions

Further reading

Chapter 9: Advanced Continuous Delivery

Technical requirements

Managing database changes

Understanding schema updates

Changing the database in continuous delivery

Avoiding a shared database

Preparing test data

Pipeline patterns

Parallelizing pipelines

Reusing pipeline components

Rolling back deployments

Adding manual steps

Release patterns

Blue-green deployment

Canary release

Working with legacy systems

Automating build and deployment

Automating tests

Refactoring and introducing new features

Understanding the human element

Summary

Exercises

Questions

Further reading

Best Practices

Practice 1 – Own the process within the team!

Practice 2 – Automate everything!

Practice 3 – Version everything!

Practice 4 – Use business language for acceptance tests

Practice 5 – Be ready to roll back

Practice 6 – Don't underestimate the impact of people

Practice 7 – Incorporate traceability

Practice 8 – Integrate often

Practice 9 – Only build binaries once

Practice 10 – Release often

Assessments

Chapter 1: Introducing Continuous Delivery

Chapter 2: Introducing Docker

Chapter 3: Configuring Jenkins

Chapter 4: Continuous Integration Pipeline

Chapter 5: Automated Acceptance Testing

Chapter 6: Clustering with Kubernetes

Chapter 7: Configuration Management with Ansible

Chapter 8: Continuous Delivery Pipeline

Chapter 9: Advanced Continuous Delivery

Other Books You May Enjoy

Preface

Continuous Delivery with Docker and Jenkins – Third Edition explains the advantages of combining Jenkins and Docker to improve the continuous integration and delivery process of app development. It starts with setting up a Docker server and configuring Jenkins on it. It then outlines the steps to build applications on Docker files and integrate them with Jenkins using continuous delivery processes such as continuous integration, automated acceptance testing, and configuration management.

Moving on, you will learn how to ensure quick application deployment with Docker containers, along with scaling Jenkins and using Kubernetes. After that, you will get to know how to deploy applications using Docker images and test them with Jenkins. Toward the end, the book will touch base with missing parts of the CD pipeline, which are the environments and infrastructure, application versioning, and non-functional testing.

By the end of the book, you will know how to enhance the DevOps workflow by integrating the functionalities of Docker and Jenkins.

Who this book is for

The book targets DevOps engineers, system administrators, Docker professionals, or any stakeholders who would like to explore the power of working with Docker and Jenkins together.

What this book covers

Chapter 1, Introducing Continuous Delivery, demonstrates the pitfalls of the traditional delivery process and describes success stories, including Amazon and Yahoo.

Chapter 2, Introducing Docker, provides a brief introduction to Docker and the concept of containerization and looks at the benefits in terms of running applications and services using this platform. In addition, we will also describe, step by step, how to set up Docker Community Edition on a local machine or a server running Linux and check to see whether Docker is running properly.

Chapter 3, Configuring Jenkins, introduces the Jenkins tool, their architecture, and procedures to install master/agent instances on a Docker server, without Docker and using Kubernetes. Then, we'll see how to scale agents. Finally, you will get a working Jenkins instance ready to build applications integrated with your source code repository service.

Chapter 4, Continuous Integration Pipeline, describes how the classic continuous integration pipeline entails three steps: checkout, building, and unit tests. In this chapter, you will learn how to build it using Jenkins and what other steps should be considered (such as code coverage and static code analysis).

Chapter 5, Automated Acceptance Testing, explains how, before releasing an application, you need to make sure that the whole system works as expected by running automated acceptance tests. Ordinarily, applications connect with databases, cache, messaging, and other tools that require other servers to run these services. This is why the whole environment has to be set up and kept ready before the test suite is started. In this chapter, you will learn Docker Registry concepts and how to build a system made of different components running as Docker containers.

Chapter 6, Clustering with Kubernetes, explains how to scale to multiple teams and projects using Docker tools. In this chapter, you will be introduced to Kubernetes and learn how to use it in the continuous delivery process.

Chapter 7, Configuration Management with Ansible, describes how, once you have scaled your servers, to deploy your application in production. In this chapter, you will learn how to release an application on a Docker production server using configuration management tools such as Chef and Ansible. Additionally, you will learn about the infrastructure as code approach and the Terraform tool.

Chapter 8, Continuous Delivery Pipeline, focuses on the missing parts of the final pipeline, which are the environments and infrastructure, application versioning, and non-functional testing. Once this chapter has been concluded, the complete continuous delivery pipeline will be ready.

Chapter 9, Advanced Continuous Delivery, explains how, after building a complete pipeline, you can address more difficult real-life scenarios. Beginning with parallelizing the pipeline tasks, we will then show how to roll back to the previous version, how to run performance tests, what to do with database changes, and how to proceed with legacy systems and manual tests.

Best Practices, this includes best practices to be followed throughout the book.

To get the most out of this book

Docker requires a 64-bit Linux operating system. All examples in this book have been developed using Ubuntu 20.04, but any other Linux system with kernel version 3.10 or above is sufficient.

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781803237480_ColorImages.pdf.

Code in Action

Code in Action videos for this book can be viewed at https://bit.ly/3NSEPNA .

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Continuous-Delivery-With-Docker-and-Jenkins-3rd-Edition. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "We can create a new pipeline called calculator and, as pipeline script, put the code in a stage called Checkout."

A block of code is set as follows:

pipeline {

     agent any

     stages {

          stage("Checkout") {

               steps {

                    git url: 'https://github.com/leszko/calculator.git', branch: 'main'

               }

          }

     }

}

Any command-line input or output is written as follows:

$ sudo apt-get update

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select Gradle Project instead of Maven Project (you can choose Maven if you prefer it to Gradle)."

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Continuous Delivery with Docker and Jenkins, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Section 1 – Setting Up the Environment

In this section, we will be introduced to Docker, and we will cover concepts such as continuous delivery and its benefits, as well as containerization. Furthermore, we will also be introduced to the Jenkins tool, and the architecture and procedures required to install master/slave instances on a Docker server, without Docker, and using cloud environments.

The following chapters are covered in this section:

Chapter 1, Introducing Continuous DeliveryChapter 2, Introducing DockerChapter 3, Configuring Jenkins

Chapter 1: Introducing Continuous Delivery

A common problem that's faced by most developers is how to release the implemented code quickly and safely. The delivery process that's traditionally used is a source of pitfalls and usually leads to the disappointment of both developers and clients. This chapter will present the idea of the continuous delivery (CD) approach and provide the context for the rest of this book.

In this chapter, we will cover the following topics:

Understanding CDThe automated deployment pipelinePrerequisites to CDCombining CD and microservicesBuilding the CD process

Understanding CD

The most accurate definition of CD is stated by Jez Humble and reads as follows:

"Continuous delivery is the ability to get changes of all types – including new features, configuration changes, bug fixes, and experiments – into production, or into the hands of users, safely and quickly, in a sustainable way."

This definition covers the key points.

To understand this better, let's imagine a scenario. You are responsible for a product – let's say, an email client application. Users come to you with a new requirement: they want to sort emails by size. You decide that the development will take around 1 week. When can the user expect to use the feature? Usually, after the development is done, you hand over the completed feature to the Quality Assurance (QA) team and then to the operations team, which takes additional time, ranging from days to months.

Therefore, even though the development only took 1 week, the user receives it in a couple of months! The CD approach addresses this issue by automating manual tasks so that the user can receive a new feature as soon as it's implemented.

To help you understand what to automate and how, we'll start by describing the delivery process that is currently used for most software systems.

The traditional delivery process

The traditional delivery process, as its name suggests, has been in place for many years and is implemented in most IT companies. Let's define how it works and comment on its shortcomings.

Introducing the traditional delivery process

Every delivery process begins with the requirements that have been defined by a customer and ends with the product being released to production. There are differences between these two stages. Traditionally, this process looks as follows:

Figure 1.1 – Release cycle diagram

The release cycle starts with the requirements provided by the Product Owner, who represents the Customer (stakeholders). Then, there are three phases, during which the work is passed between different teams:

Development: The developers (sometimes together with business analysts) work on the product. They often use agile techniques (Scrum or Kanban) to increase the development velocity and improve communication with the client. Demo sessions are organized to obtain a customer's quick feedback. All good development techniques (such as test-driven development (TDD) or extreme programming practices) are welcome. Once the implementation is complete, the code is passed to the QA team.Quality Assurance: This phase is usually called User Acceptance Testing (UAT) and it requires the code to be frozen on the trunk code base so that no new development will break the tests. The QA team performs a suite of integration testing, acceptance testing, and non-functional analysis (performance, recovery, security, and so on). Any bug that is detected goes back to the development team, so the developers usually have their hands full. After the UAT phase is completed, the QA team approves the features that have been planned for the next release.Operations: The final phase, and usually the shortest one, involves passing the code to the operations team so that they can perform the release and monitor the production environment. If anything goes wrong, they contact the developers so that they can help with the production system.

The length of the release cycle depends on the system and the organization, but it usually ranges from 1 week to a few months. The longest I've heard about was 1 year. The longest I worked on one was quarterly-based, and each part was as follows:

Development: 1.5 monthsUAT: 1 month and 3 weeksRelease (and strict production monitoring): 1 week

The traditional delivery process is widely used in the IT industry, so this is probably not the first time you've read about such an approach. Nevertheless, it has several drawbacks. Let's look at them explicitly to understand why we need to strive for something better.

Shortcomings of the traditional delivery process

The most significant shortcomings of the traditional delivery process are as follows:

Slow delivery: The customer receives the product long after the requirements were specified. This results in unsatisfactory time to market and delays customer feedback.Long feedback cycle: The feedback cycle is not only related to customers but developers. Imagine that you accidentally created a bug, and you learn about it during the UAT phase. How long does it take to fix something you worked on 2 months ago? Even dealing with minor bugs can take weeks.Lack of automation: Rare releases do not encourage automation, which leads to unpredictable releases.Risky hotfixes: Hotfixes cannot usually wait for the full UAT phase, so they tend to be tested differently (the UAT phase is shortened) or not tested at all.Stress: Unpredictable releases are stressful for the operations team. What's more, the release cycle is usually tightly scheduled, which imposes additional stress on developers and testers.Poor communication: Work that's passed from one team to another represents the waterfall approach, in which people start to care only about their part, rather than the complete product. If anything goes wrong, that usually leads to the blame game instead of cooperation.Shared responsibility: No team takes responsibility for the product from A to Z:For developers: Done means that the requirements have been implemented.For testers: Done means that the code has been tested.For operations: Done means that the code has been released.Lower job satisfaction: Each phase is interesting for a different team, but other teams need to support the process. For example, the development phase is interesting for developers but, during the other two phases, they still need to fix bugs and support the release, which is usually not interesting for them at all.

These drawbacks represent just the tip of the iceberg of the challenges related to the traditional delivery process. You may already feel that there must be a better way to develop software and this better way is, obviously, the CD approach.

The benefits of CD

How long would it take your organization to deploy a change that involves just a single line of code? Do you do this on a repeatable, reliable basis? These are the famous questions from Mary and Tom Poppendieck (authors of Implementing Lean Software Development), which have been quoted many times by Jez Humble and others. The answers to these questions are the only valid measurement of the health of your delivery process.

To be able to deliver continuously, and not spend a fortune on the army of operations teams working 24/7, we need automation. That is why, in short, CD is all about changing each phase of the traditional delivery process into a sequence of scripts called the automated deployment pipeline, or the CD pipeline. Then, if no manual steps are required, we can run the process after every code change and deliver the product continuously to users.

CD lets us get rid of the tedious release cycle and brings the following benefits:

Fast delivery: Time to market is significantly reduced as customers can use the product as soon as development is completed. Remember that the software delivers no revenue until it is in the hands of its users.Fast feedback cycle: Imagine that you created a bug in the code, which goes into production the same day. How much time does it take to fix something you worked on the same day? Probably not much. This, together with the quick rollback strategy, is the best way to keep production stable.Low-risk releases: If you release daily, the process becomes repeatable and much safer. As the saying goes, if it hurts, do it more often.Flexible release options: If you need to release immediately, everything is already prepared, so there is no additional time/cost associated with the release decision.

Needless to say, we could achieve all these benefits simply by eliminating all the delivery phases and proceeding with development directly from production. However, this would result in a reduction in quality. The whole difficulty of introducing CD is the concern that the quality would decrease alongside eliminating any manual steps. In this book, we will show you how to approach CD safely and explain why, contrary to common beliefs, products that are delivered continuously contain fewer bugs and are better adjusted to the customer's needs.

Success stories

My favorite story on CD was told by Rolf Russell at one of his talks. It goes as follows. In 2005, Yahoo! acquired Flickr, and it was a clash of two cultures in the developer's world. Flickr, by that time, was a company with the start-up approach in mind. Yahoo!, on the other hand, was a huge corporation with strict rules and a safety-first attitude. Their release processes differed a lot. While Yahoo used the traditional delivery process, Flickr released many times a day. Every change that was implemented by developers went into production the same day. They even had a footer at the bottom of their page showing the time of the last release and the avatars of the developers who made the changes.

Yahoo! deployed rarely, and each release brought a lot of changes that were well-tested and prepared. Flickr worked in very small chunks; each feature was divided into small incremental parts, and each part was deployed to production quickly. The difference is presented in the following diagram:

Figure 1.2 – Comparison of the release cycles of Yahoo! and Flickr

You can imagine what happened when the developers from the two companies met. Yahoo! treated Flickr's colleagues as irresponsible junior developers, a bunch of software cowboys who didn't know what they were doing. So, the first thing they wanted to do was add a QA team and the UAT phase to Flickr's delivery process. Before they applied the change, however, Flickr's developers had only one wish. They asked to evaluate the most reliable products throughout Yahoo! as a whole. It came as a surprise when they saw that even with all the software in Yahoo!, Flickr had the lowest downtime. The Yahoo! team didn't understand it at first, but they let Flickr stay with their current process anyway. After all, they were engineers, so the evaluation result was conclusive. Only after some time had passed did the Yahoo! developers realize that the CD process could be beneficial for all the products in Yahoo! and they started to gradually introduce it everywhere.

The most important question of the story remains: how come Flickr was the most reliable system? The reason behind this was what we already mentioned in the previous sections. A release is less risky if the following is true:

The delta of code changes is smallThe process is repeatable

That is why, even though the release itself is a difficult activity, it is much safer when it's done frequently.

The story of Yahoo! and Flickr is only one example of many successful companies where the CD process proved to be the correct choice. Nowadays, it's common for even small organizations to release frequently and market leaders such as Amazon, Facebook, Google, and Netflix perform thousands of releases per day.

Information

You can read more about the research on the CD process and individual case studies at https://continuousdelivery.com/evidence-case-studies/.

Keep in mind that the statistics get better every day. However, even without any numbers, just imagine a world in which every line of code you implement goes safely into production. Clients can react quickly and adjust their requirements, developers are happy as they don't have to solve that many bugs, and managers are satisfied because they always know the current state of work. After all, remember that the only true measure of progress is the software that is released.

The automated deployment pipeline

We already know what the CD process is and why we use it. In this section, we'll describe how to implement it.

Let's start by emphasizing that each phase in the traditional delivery process is important. Otherwise, it would never have been created in the first place. No one wants to deliver software without testing it! The role of the UAT phase is to detect bugs and ensure that what the developers have created is what the customer wanted. The same applies to the operations team – the software must be configured, deployed to production, and monitored. That's out of the question. So, how do we automate the process so that we preserve all the phases? That is the role of the automated deployment pipeline, which consists of three stages, as shown in the following diagram:

Figure 1.3 – Automated deployment pipeline

The automated deployment pipeline is a sequence of scripts that is executed after every code change is committed to the repository. If the process is successful, it ends up being deployed to the production environment.

Each step corresponds to a phase in the traditional delivery process, as follows:

Continuous integration: This checks to make sure that the code that's been written by different developers is integrated.Automated acceptance testing: This checks if the client's requirements have been met by the developers implementing the features. This testing also replaces the manual QA phase.Configuration management: This replaces the manual operations phase; it configures the environment and deploys the software.

Let's take a deeper look at each phase to understand its responsibility and what steps it includes.

Continuous integration

The continuous integration (CI) phase provides the first set of feedback to developers. It checks out the code from the repository, compiles it, runs unit tests, and verifies the code's quality. If any step fails, the pipeline's execution is stopped and the first thing the developers should do is fix the CI build. The essential aspect of this phase is time; it must be executed promptly. For example, if this phase took 1 hour to complete, the developers would commit the code faster, which would result in a constantly failing pipeline.

The CI pipeline is usually the starting point. Setting it up is simple because everything is done within the development team, and no agreement with the QA and operations teams is necessary.

Automated acceptance testing

The automated acceptance testing phase is a suite of tests written together with the client (and QAs) that is supposed to replace the manual UAT stage. It acts as a quality gate to decide whether a product is ready to be released. If any of the acceptance tests fail, pipeline execution is stopped and no further steps are run. It prevents movement to the configuration management phase and, hence, the release.

The whole idea of automating the acceptance phase is to build quality into the product instead of verifying it later. In other words, when a developer completes the implementation, the software is delivered together with the acceptance tests, which verify that the software is what the client wanted. That is a large shift in thinking concerning testing software. There is no longer a single person (or team) who approves the release, but everything depends on passing the acceptance test suite. That is why creating this phase is usually the most difficult part of the CD process. It requires close cooperation with the client and creating tests at the beginning (not at the end) of the process.

Note

Introducing automated acceptance tests is especially challenging in the case of legacy systems. We will discuss this topic in more detail in Chapter 9, Advanced Continuous Delivery.

There is usually a lot of confusion about the types of tests and their place in the CD process. It's also often unclear as to how to automate each type, what the coverage should be, and what the role of the QA team should be in the development process. Let's clarify this using the Agile testing matrix and the testing pyramid.

The Agile testing matrix

Brian Marick, in a series of his blog posts, classified software tests in the form of the agile testing matrix. It places tests in two dimensions – business - or technology-facing – and supports programmers or a critique of the product. Let's have a look at this classification:

Figure 1.4 – Brian Marick's testing matrix

Let's look at each type of test:

Acceptance Testing (automated): These are tests that represent the functional requirements that are seen from the business perspective. They are written in the form of stories or examples by clients and developers so that they can agree on how the software should work.Unit Testing (automated): These are tests that help developers provide high-quality software and minimize the number of bugs.Exploratory Testing (manual): This is the manual black-box testing phase, which tries to break or improve the system.Non-Functional Testing (automated): These are tests that represent system properties related to performance, scalability, security, and so on.

This classification answers one of the most important questions about the CD process: what is the role of a QA in the process?

Manual QAs perform exploratory testing, which means they play with the system, try to break it, ask questions, and think about improvements. Automation QAs help with non-functional and acceptance testing; for example, they write code to support load testing. In general, QAs don't have a special place in the delivery process, but rather have a role in the development team.

Note

In the automated CD process, there is no longer a place for manual QAs who perform repetitive tasks.

You may look at the classification and wonder why you see no integration tests there. Where are they according to Brian Marick, and where can we put them in the CD pipeline?

To explain this well, we need to mention that the meaning of an integration test differs based on the context. For (micro) service architectures, they usually mean the same as acceptance testing, as services are small and need nothing more than unit and acceptance tests. If you build a modular application, then integration tests usually mean component tests that bind multiple modules (but not the whole application) and test them together. In that case, integration tests place themselves somewhere between acceptance and unit tests. They are written in a similar way to acceptance tests, but they are usually more technical and require mocking not only external services but also internal modules. Integration tests, similar to unit tests, represent the code's point of view, while acceptance tests represent the user's point of view. In regards to the CD pipeline, integration tests are simply implemented as a separate phase in the process.

The testing pyramid

The previous section explained what each test type represents in the process, but mentioned nothing about how many tests we should develop. So, what should the code coverage be in the case of unit testing? What about acceptance testing?

To answer these questions, Mike Cohn, in his book Succeeding with Agile, created a so-called testing pyramid. The following diagram should help you develop a better understanding of this:

Figure 1.5 – Mike Cohn's testing pyramid

When we move up the pyramid, the tests become slower and more expensive to create. They often require user interfaces to be touched and a separate test automation team to be hired. That is why acceptance tests should not target 100% coverage. On the contrary, they should be feature-oriented and only verify selected test scenarios. Otherwise, we would spend a fortune on test development and maintenance, and our CD pipeline build would take ages to execute.

The case is different at the bottom of the pyramid. Unit tests are cheap and fast, so we should strive for 100% code coverage. They are written by developers, and providing them should be a standard procedure for any mature team.

I hope that the agile testing matrix and the testing pyramid clarified the role and the importance of acceptance testing.

Now, let's look at the last phase of the CD process: configuration management.

Configuration management

The configuration management phase is responsible for tracking and controlling changes in the software and its environment. It involves taking care of preparing and installing the necessary tools, scaling the number of service instances and their distribution, infrastructure inventory, and all the tasks related to application deployment.

Configuration management is a solution to the problems that are posed by manually deploying and configuring applications in production. This common practice results in an issue whereby we no longer know where each service is running and with what properties. Configuration management tools (such as Ansible, Chef, and Puppet) enable us to store configuration files in the version control system and track every change that was made to the production servers.

Additional effort to replace the operations team's manual tasks involves taking care of application monitoring. This is usually done by streaming the logs and metrics of the running systems to a common dashboard, which is monitored by developers (or the DevOps team, as explained in the next section).

One other term related to configuration management that has recently gained a lot of traction is Infrastructure as Code (IaC). If you use the cloud instead of bare-metal servers, then tools such as Terraform or AWS CloudFormation let you store the description of your infrastructure, not only your software, in the version control system. We will discuss both configuration management and IaC inChapter 7, Configuration Management with Ansible.

Prerequisites to CD

The rest of this book is dedicated to technical details on how to implement a successful CD pipeline. The success of this process, however, depends not only on the tools we present throughout this book. In this section, we will take a holistic look at the whole process and define the CD requirements in three areas:

Your organization's structure and its impact on the development processYour products and their technical detailsYour development team and the practices you adopt

Let's start with the organizational prerequisites.

Organizational prerequisites

The way your organization works has a high impact on the success of introducing the CD process. It's a bit similar to introducing Scrum. Many organizations would like to use the Agile process, but they don't change their culture. You can't use Scrum in your development team unless the organization's structure has been adjusted for that. For example, you need a product owner, stakeholders, and a management team that understands that no requirement changes are possible during the sprint. Otherwise, even with good intentions, you won't make it. The same applies to the CD process; it requires you to adjust how the organization is structured. Let's have a look at three aspects: the DevOps culture, a client in the process, and business decisions.

DevOps culture

A long time ago, when software was written by individuals or micro teams, there was no clear separation between development, quality assurance, and operations. A person developed the code, tested it, and then put it into production. If anything went wrong, the same person investigated the issue, fixed it, and redeployed it to production. The way the development process is organized changed gradually; systems became larger and development teams grew. Then, engineers started to become specialized in one area. This made perfect sense as specialization caused a boost in productivity. However, the side effect was the communication overhead. This is especially visible if developers, QAs, and operations are in separate departments in the organization, sit in different buildings, or are outsourced to different countries. This organizational structure is not good for the CD process. We need something better; we need to adopt the DevOps culture.

DevOps culture means, in a sense, going back to the roots. A single person or a team is responsible for all three areas, which are shown in the following diagram:

Figure 1.6 – DevOps culture

The reason it's possible to move to the DevOps model without losing productivity is automation. Most of the tasks that are related to QA and operations are moved to the automated delivery pipeline, so they can be managed by the development team.

Information

A DevOps team doesn't necessarily need to consist of only developers. A very common scenario in many organizations that are transforming is to create teams with four developers, one QA, and one person from operations. However, they need to work closely together (sit in one area, have stand-ups together, and work on the same product).

The culture of small DevOps teams affects the software architecture. Functional requirements have to be separated into (micro) services or modules so that each team can take care of an independent part.

Information

The impact of the organization's structure on the software architecture was observed in 1967 and formulated as Conway's law: "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."

A client in the process

The role of a client (or a product owner) changes slightly during CD adoption. Traditionally, clients are involved in defining requirements, answering questions from developers, attending demos, and taking part in the UAT phase to determine whether what was built is what they had in mind.

In CD, there is no UAT, and a client is essential in the process of writing acceptance tests. For some clients, who have already written their requirements in a testable manner, this is not a big shift. For others, it means changing their way of thinking to make the requirements more technically oriented.

Information

In the agile environment, some teams don't even accept user stories (requirements) without acceptance tests attached. These techniques, even though they may sound too strict, often lead to better development productivity.

Business decisions

In most companies, the business has an impact on the release schedule. After all, the decision of what features are delivered, and when, is related to different departments within the company (for example, marketing) and can be strategic for the enterprise. That is why the release schedule has to be re-approached and discussed between the business and the development teams.

There are techniques, such as feature toggles or manual pipeline steps, that help with releasing features at the specified time. We will describe them later in this book. To be precise, the term continuous delivery is not the same as continuous deployment. The latter means that each commit to the repository is automatically released to production. Continuous delivery is less strict and means that each commit ends up with a release candidate, so it allows the last step (from release to production) to be manual.

Note

Throughout the remainder of this book, we will use the terms continuous delivery and continuous deployment interchangeably.

Technical and development prerequisites

From the technical side, there are a few requirements to keep in mind. We will discuss them throughout this book, so let's only mention them here without going into detail:

Automated build, test, package, and deploy operations: All operations need to be able to be automated. If we deal with a system that is non-automatable, for example, due to security reasons or its complexity, it is impossible to create a fully automated delivery pipeline.Quick pipeline execution: The pipeline must be executed promptly, preferably in 5-15 minutes. If our pipeline execution takes hours or days, it will not be possible to run it after every commit to the repository.Quick failure recovery: The possibility of a quick rollback or system recovery is necessary. Otherwise, we risk production health due to frequent releases.Zero-downtime deployment: The deployment cannot have any downtime since we release it many times a day.Trunk-based development: Developers must check into one main branch regularly. Otherwise, if everyone develops in their branches, integration is rare, which means that releases are rare, which is exactly the opposite of what we want to achieve.

We will learn more about these prerequisites and how to address them throughout this book. With this in mind, let's move to the last section of this chapter and introduce what system we plan to build in this book and what tools we will use for that purpose.

Combining CD and microservices

We live in the world of microservices. Nowadays, every system is either microservice-based or in the process of becoming microservice-based. After the first publication of the bestseller book by Sam Newman, Building Microservices, the software world has shifted into the fine-grained modular systems in which all communication happens over the network. Some companies have gone one step further and realized that they need to consolidate some of the microservices as they created too many of them. Some other companies even take a step back and consolidate microservices into a monolith.

While the topic of microservices is broad on its own and outside the scope of this book, it is important to understand how the microservice architecture affects the CD pipeline. Should we create a separate pipeline for each service? If yes, then how do we test the interaction between the services and the system as a whole?

Before answering these questions, let's look at the following diagram, which represents a small microservice-based system:

Figure 1.7 – Sample microservice system

There are three services in our system, each with a database. A user only interacts with Service 1. As a more concrete example, this system could represent an online store, where Service 1 could represent the checkout service, Service 2 could represent the product catalog service, and Service 3 could represent customer service.

We could either implement one CD pipeline for the entire system or a separate CD pipeline for each microservice. Which approach is the right one? Let's consider both options. If we create one CD pipeline, this means that the automated acceptance testing phase runs against the entire system from the end user's perspective, which seems correct. However, one CD pipeline also means that we deploy all the services at the same time, which is completely against the microservice principles. Remember that in every microservice-based system, services are loosely coupled and should always be independently deployable.

So, we need to take the second approach and create a separate CD pipeline for each service. However, in such a case, the automated acceptance testing phase never runs against the entire system. So, how can we be sure that everything works correctly from the end user's perspective? To answer this question, we need a little more context about the microservice architecture.