Automating DevOps with GitLab CI/CD Pipelines E-Book

Christopher Cowell

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 552

Veröffentlichungsjahr: 2023

Beschreibung

Developers and release engineers understand the high stakes involved in building, packaging, and deploying code correctly. Ensuring that your code is functionally correct, fast, and secure is a time-consuming and complex task. Code implementation, development, and deployment can be conducted efficiently using GitLab CI/CD pipelines.
Automating DevOps with GitLab CI/CD Pipelines begins with the basics of Git and GitLab, showing how to commit and review code. You’ll learn to set up GitLab Runners for executing and autoscaling CI/CD pipelines and creating and configuring pipelines for many software development lifecycle steps. You'll also discover where to find pipeline results in GitLab, and how to interpret those results. Through the course of the book, you’ll become well-equipped with deploying code to different environments, advancing CI/CD pipeline features such as connecting GitLab to a Kubernetes cluster and using GitLab with Terraform, triggering pipelines and improving pipeline performance and using best practices and troubleshooting tips for uncooperative pipelines. In-text examples, use cases, and self-assessments will reinforce the important CI/CD, GitLab, and Git concepts, and help you prepare for interviews and certification exams related to GitLab.
By the end of this book, you'll be able to use GitLab to build CI/CD pipelines that automate all the DevOps steps needed to build and deploy high-quality, secure code.

Leseprobe

Automating DevOps with GitLab CI/CD Pipelines

Build efficient CI/CD pipelines to verify, secure, and deploy your code using real-life examples

Christopher Cowell

Nicholas Lotz

Chris Timberlake

BIRMINGHAM—MUMBAI

Automating DevOps with GitLab CI/CD Pipelines

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Mohd Riyan

Publishing Product Manager: Surbhi Suman

Senior Editors: Divya Vijayan and Athikho Sapuni Rishana

Technical Editor: Nithik Cheruvakodan

Copy Editor: Safis Editing

Project Coordinator: Ashwin Kharwa

Proofreader: Safis Editing

Indexer: Rekha Nair

Production Designer: Shankar Kalbhor

Marketing Coordinator: Ankita Bhonsle

Senior Marketing Coordinator: Marylou De Mello

First published: February 2023

Production reference: 2200223

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80323-300-0

www.packtpub.com

To Marya DeVoto, for reminding me that words actually mean things.

– Christopher Cowell

To my parents, Tom and Deborah, for encouraging me to never stop asking “why?”

– Nicholas Lotz

Contributors

About the authors

Christopher Cowell was formerly a technical trainer at GitLab, and now builds educational content at Instabase. He also worked for 2 decades as a research scientist, consultant, and QA engineer at big and small software companies, including Accenture, Oracle, and Puppet. He thinks the software industry undervalues code quality and thoughtful design and overvalues delivering mediocre code quickly. Slow down, simplify, and get it right!

He dreams of being able to play every Beatles song on guitar, but would settle for knowing just one really well. He holds a Ph.D. in philosophy from Berkeley and a B.A. in computer science from Harvard. He lives in Portland, Oregon, with his wife, his four kids, and the two best cats.

Thank you to my family, for their unending patience whenever I disappeared into the basement to work on this book, which I did a lot. I especially hope I can repay my wife, Marya DeVoto, by being just as accommodating when she launches into writing her Space Rabbi series of novels. Soren, I owe you a ski trip. Kirin, consider yourself to have a lifetime pass for playing Ticket to Ride whenever you want. Thanks also to my wonderful parents, Nick and Priscilla, for giving me whatever minor skills or talent I may have at explaining things.

Nicholas Lotz is a technical trainer at GitLab, where he teaches organizations how to use GitLab to build and ship better software. He previously worked as a systems engineer, trainer, and consultant in the software infrastructure space. He is passionate about open source and its capacity to help teams innovate. Nicholas holds a B.S. in chemical engineering from the University of Pittsburgh. He lives in Nashville, Tennessee, with his Labrador Retriever.

Thank you to my parents, Tom and Deborah, for encouraging me to ask questions and find my voice at an early age. Thank you to my grandmother, Linda, for patiently spending hundreds of hours with me in libraries and bookstores, and to my grandfather, Steve, for instilling in me a love of technology and tinkering. Thank you Jeff, for helping me find humor and joy in the monotonous. And a sincere thank you to Marty, Alex, Kristina, Cameron, and Mary for challenging me to live as I am called.

Chris Timberlake is a senior solutions architect at GitLab, where he works closely with the product, services, and sales teams. Previously, he worked at Red Hat as a senior consultant and owned and managed a digital marketing firm, and he has a background in security and law enforcement. Chris loves technical engineering problems and does whatever possible to ensure successful customer outcomes. Chris is passionate about open source software, collaborative development, and education. He lives in Chattanooga, Tennessee, with his family.

About the reviewers

Jean-Philippe Baconnais has been a developer for more than 15 years. Initially focused on Java, he later learned new languages such as Go, as well as about frontend frameworks such as Vue.js and Angular. Curious by nature, he enjoys learning about and trying out new technologies, applying them, and sharing them, whether in his own company, Zenika, in communities as a speaker at conferences, or at local events in his home city of Nantes, France. As an open source developer, he has had the pleasure of participating in two open source communities and has been given the recognition of GitLab Hero and GitPod Community Hero.

Philippe Charrière is a customer success engineer at GitLab (having worked there for the last 5 years). He started working in computer science in 1995 and has held positions as a developer, architect, project manager, manager, bid manager, customer success manager, solution engineer, strategic account leader, and customer success engineer. Today, his favorite subject is WebAssembly, particularly with Go, TinyGo, and Rust.

Table of Contents

Preface

Part 1 Getting Started with DevOps, Git, and GitLab

1

Understanding Life Before DevOps

Introducing the Hats for Cats web app

Building and verifying code manually

Building code manually

Verifying code manually

More challenges of verifying code

Security-testing code manually

Static code analysis

Secret detection

Dynamic analysis

Dependency scanning

Container scanning

Manual security testing summary

Packaging and deploying code manually

License compliance scanning

Deploying software

Problems with manual software development life cycle practices

Solving problems with DevOps

How GitLab implements DevOps

Summary

2

Practicing Basic Git Commands

Technical requirements

Why use Git?

What is a version control system?

What problems does a VCS solve?

Why Git is popular

Drawbacks of Git

Committing code to keep it safe

Excluding files from a repository

Tagging commits to identify versions of code

Branching code for developing in an isolated space

Git commands for managing branches

Handling merge conflicts

Syncing local and remote copies of repositories

The “golden” repository

Configuring remote repositories

Pushing

Fetching

Pulling

Additional resources for learning Git

Summary

3

Understanding GitLab Components

Technical requirements

Emphasizing the “why” over the “how”

Introducing the GitLab platform

What is GitLab?

What problem does GitLab solve?

The verify, secure, and release stages

Organizing work into projects and groups

Example – organizing your Hats for Cats work

Tracking work with issues

The structure of a GitLab issue

The kinds of tasks that issues can represent

Labels

Issue workflows

Editing files safely with commits, branches, and merge requests

Commit history

Merging one Git branch into another

The three amigos – issues, branches, and merge requests

When two amigos are enough

How are issues and merge requests different?

Enabling DevOps practices with GitLab flow

Summary

4

Understanding GitLab’s CI/CD Pipeline Structure

Technical requirements

Defining the terms pipeline, CI, and CD

Understanding what a pipeline is

Defining one pipeline per project

Understanding different uses of the term “pipeline”

Viewing a list of pipelines

CI – finding out if your code is good

CD – finding out where your code should go (and putting it there)

GitLab Runners

Parts of a pipeline – stages, jobs, and commands

Stages

Jobs

Commands

Fitting the pipeline pieces together

Running GitLab CI/CD pipelines

Branch pipelines

Git tag pipelines

Other types of pipelines

Skipping pipelines

Reading GitLab CI/CD pipeline statuses

Configuring GitLab CI/CD pipelines

Summary

Part 2 Automating DevOps Stages with GitLab CI/CD Pipelines

5

Installing and Configuring GitLab Runners

Technical requirements

Defining GitLab runners and their relationship to CI/CD

GitLab Runner is an open source application written in Go

GitLab Runner runs CI/CD jobs specified in .gitlab-ci.yml

The runner architecture and supported platforms

GitLab Runner is supported on most platforms and architectures

Runners can be specific, group, or shared

Each runner has a defined executor

Runner tags restrict which runners can pick up which jobs

Installing the Runner agent

Installing GitLab Runner

Registering a runner with GitLab

Considerations regarding the various runner types and executors

Performance considerations

Security considerations

Monitoring considerations

Summary

6

Verifying Your Code

Technical requirements

Building code in a CI/CD pipeline

Compiling Java with javac

Compiling Java with Maven

Compiling C with Gnu Compiler Collection (GCC)

Storing built code as artifacts

Checking code quality in a CI/CD pipeline

Enabling Code Quality

Viewing Code Quality results

Running automated functional tests in a CI/CD pipeline

Enabling automated functional tests

Viewing automated functional test results

Fuzz testing in a CI/CD pipeline

The architecture and workflow of fuzz testing

A fuzz testing workflow

Viewing the results of fuzz testing

Extra considerations when fuzz testing

Fuzz testing with a corpus

Checking accessibility in a CI/CD pipeline

Enabling accessibility testing

Viewing accessibility testing results

Additional ways to verify your code

Code coverage

Browser performance testing

Load performance testing

Summary

7

Securing Your Code

Technical requirements

Understanding GitLab’s security scanning strategy

GitLab uses open-source scanners

Scanners are packaged as Docker images

Some scanners use different analyzers for different languages

Vulnerabilities don’t stop the pipeline

Findings appear in three different reports

Pipelines can use non-GitLab-provided scanners

Using SAST to scan your source code for vulnerabilities

Understanding SAST

Enabling SAST

Configuring SAST

Viewing SAST’s findings

Using Secret Detection to find private information in your repository

Understanding Secret Detection

Enabling and configuring Secret Detection

Viewing Secret Detection’s findings

Using DAST to find vulnerabilities in web applications

Understanding DAST

Enabling and configuring DAST

Viewing DAST’s findings

Using Dependency Scanning to find vulnerabilities in dependencies

Understanding Dependency Scanning

Enabling and configuring Dependency Scanning

Viewing Dependency Scanning’s findings

Using Container Scanning to find vulnerabilities in Docker images

Understanding Container Scanning

Enabling and configuring Container Scanning

Viewing Container Scanning’s findings

Using License Compliance to manage licenses of dependencies

Understanding License Compliance

Enabling and configuring License Compliance

Viewing License Compliance’s findings

Using IaC Scanning to find problems in infrastructure configuration files

Understanding IaC Scanning

Enabling and configuring IaC scanning

Viewing IaC Scanning’s findings

Understanding the different types of security reports

Managing security vulnerabilities

Integrating outside security scanners

Summary

8

Packaging and Deploying Code

Technical requirements

Storing code in GitLab’s package registry for later re-use

Locating GitLab’s container and package registries

Getting started with the package registry

Supported package formats

Authenticating to the registry

Building and pushing packages to the package registry

Building and pushing packages to the container registry

Storing code in GitLab’s container and package registries for later deployment

Using images from the container registry

Using packages from the package registry

Deploying to different environments using GitLab Flow

Deploying to a review app for testing

Deploying to real-world production environments

Deploying to a Kubernetes cluster

The CI/CD workflow

A GitOps workflow

Summary

Part 3 Next Steps for Improving Your Applications with GitLab

9

Enhancing the Speed and Maintainability of CI/CD Pipelines

Accelerating pipelines with directed acyclic graphs and parent-child architecture

How to create a DAG in GitLab CI

Building code for multiple architectures

When and how to leverage caching or artifacts

Caching characteristics

Artifact characteristics

Using caching

Using artifacts

Leveraging artifacts as job dependencies

Reducing repeated configuration code with anchors and extensions

Anchors

The extends: keyword

Reference tags

Improving maintainability by combining multiple pipelines and leveraging parent-child pipelines

Leveraging includes for maintainability

Leveraging includes for reusability

Includes from remote areas

Leveraging parent-child pipelines

Securing and accelerating jobs with purpose-built containers

A purpose-built container example

Summary

10

Extending the Reach of CI/CD Pipelines

Using CI/CD pipelines to spot performance problems

How to integrate browser performance testing

How to integrate load performance testing with k6

Using feature flags to allow business-driven release decisions

How to configure your application for feature flags

Integrating third-party tools into your CI/CD pipelines

Creating our tool container’s Dockerfile

Automating our container’s build

Container scanning

Invoking the third-party tool

Using CI/CD pipelines for developing mobile apps

Requirements

Fastlane

Fastlane – deployment

Fastlane – automated testing

Summary

11

End-to-End Example

Technical requirements

Setting up your environment

Making a GitLab project

Planning work with GitLab issues

Setting up a local Git repository

Writing code

Creating a Git branch to work on

Creating an MR

Committing and pushing code

Establishing the pipeline infrastructure

Creating a pipeline

Creating a runner

Verifying your code

Adding functional tests to the pipeline

Adding Code Quality scanning to the pipeline

Adding a fuzz test to the pipeline

Securing your code

Adding SAST to the pipeline

Adding Secret Detection to the pipeline

Adding Dependency Scanning to the pipeline

Adding License Compliance to the pipeline

Integrating a third-party security scanner into the pipeline

Improving your pipeline

Using a DAG to speed up the pipeline

Breaking the pipeline into several files

Delivering your code to the right environment

Deploying the code

Summary

12

Troubleshooting and the Road Ahead with GitLab

Technical requirements

Troubleshooting and best practices for common pipeline problems

Troubleshooting CI/CD syntax and logic

Troubleshooting pipeline operation and runner assignment

Managing your operational infrastructure using GitOps

Using Terraform to deploy and update infrastructure state

Using Ansible to manage resource configurations

Future industry trends

Automation will create more software at a larger scale

Abstraction will lead to everything-as-code business models

Reduced cycle time will help teams release better software faster

Conclusion and next steps

Index

Other Books You May Enjoy

Part 1 Getting Started with DevOps, Git, and GitLab

In this part of the book, you will learn why the software development life cycle was slow and error-prone before GitLab came along, which will help you understand the problems GitLab solves. You will also learn about the basics of the Git version control system and be introduced to the fundamental concepts and components of GitLab. Finally, you will get your first look at GitLab CI/CD pipelines, which will form the focus of most of the rest of the book.

This section comprises the following chapters:

Chapter 1, Understanding Life Before DevOpsChapter 2, Practicing Basic Git CommandsChapter 3, Understanding GitLab ComponentsChapter 4, Understanding the GitLab CI/CD Pipeline Structure