HashiCorp Packer in Production E-Book

John Boero

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 275

Veröffentlichungsjahr: 2023

Beschreibung

Creating machine images can be time-consuming and error-prone when done manually. HashiCorp Packer enables you to automate this process by defining the configuration in a simple, declarative syntax. This configuration is then used to create machine images for multiple environments and cloud providers.

The book begins by showing you how to create your first manifest while helping you understand the available components. You’ll then configure the most common built-in builder options for Packer and use runtime provisioners to reconfigure a source image for desired tasks. You’ll also learn how to control logging for troubleshooting errors in complex builds and explore monitoring options for multiple logs at once. As you advance, you’ll build on your initial manifest for a local application that’ll easily migrate to another builder or cloud. The chapters also help you get to grips with basic container image options in different formats while scaling large builds in production. Finally, you’ll develop a life cycle and retention policy for images, automate packer builds, and protect your production environment from nefarious plugins.

By the end of this book, you’ll be equipped to smoothen collaboration and reduce the risk of errors by creating machine images consistently and automatically based on your defined configuration

Leseprobe

HashiCorp Packer in Production

Efficiently manage sets of images for your digital transformation or cloud adoption journey

John Boero

BIRMINGHAM—MUMBAI

HashiCorp Packer in Production

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Preet Ahuja

Publishing Product Manager: Preet Ahuja

Senior Editor: Arun Nadar

Technical Editor: Rajat Sharma

Copy Editor: Safis Editing

Project Coordinator: Ashwin Kharwa

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Shankar Kalbhor

Marketing Coordinator: Rohan Dobhal

First published: June 2023

Production reference: 1140623

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80324-685-7

www.packtpub.com

To my parents, Joe and Rose Boero, who have tolerated my shenanigans for all these years.

Foreword

Today, HashiCorp is known for building a broad portfolio of products used for infrastructure automation. What most don’t know is Packer was the first tool we created after starting the company in 2012. The timing at which Packer was released made it rather controversial, as that was the heyday of configuration management and it was common practice for organizations to have long-running infrastructure that was patched and upgraded over time.

Early in the history of HashiCorp, we published the Tao of HashiCorp, which was our set of strongly held principles for managing the infrastructure around which our products were built. One of those key principles is immutability, which we felt was a critical way to manage complexity and risk in large-scale infrastructure. One of the key challenges with mutable approaches to infrastructure is that you have no discrete versions and can’t reason about what portion of the fleet is running version 1 or version 2. Instead, you have a continuous spectrum of versions, with partial failures and drift, forcing operators to reason about version 1.5 or version 1.87.

Immutability provides a path out of this madness. By moving our configuration management, patch management, and machine hardening into pipelines that are preproduction, we get golden images that we can version. That allows us to have a sensible inventory of our production environments while eliminating an entire class of complexity and risk that comes with partial failures. Of course, that creates an entirely new set of challenges. Now, we need to be good at building and rebuilding images frequently and across many different platforms. This was the problem Packer was built to solve, by giving us a common workflow to define and build machine images across many different platforms.

Packer has since become an enormously successful product that is used by thousands of organizations as a key part of how they build and manage production infrastructure. Beyond direct users, Packer has influenced cloud providers as well, and services such as Azure Image Builder are built using Packer.

Over the years, Packer has evolved to solve new challenges. Docker and OCI images came to be several years after Packer was released, and it has evolved to support those formats as well. The irony is that Docker helped bring immutable infrastructure into popular practice more broadly. Today, HashiCorp continues to invest in Packer both with the tool itself and through HashiCorp Cloud Platform, or HCP, with the HCP Packer service. HCP Packer provides an image registry with metadata about images, versions, and artifacts that can be queried by downstream automation tools such as Terraform and Ansible. This enables an end-to-end automation pipeline from the image building to the tools that deploy and manage the production infrastructure.

As an industry, we continue to evolve our approach and best practices around infrastructure management. I feel strongly that immutability has a key role to play in managing at scale, and that Packer plays a critical role in the toolchain. I hope this book is a useful resource for readers to learn more about Packer and how it can be used.

Armon Dadgar

Co-Founder and CTO of HashiCorp

Contributors

About the author

John Boero is a solutions engineer with HashiCorp in London covering partners across the EMEA region. He has worked for Red Hat, Puppet, and others. He has held roles as a developer, DBA, consultant, and architect all around the globe, whether San Francisco, Canary Wharf, or Saudi Arabia. He is an infrastructure minimalist and loves to push and bend projects beyond their standard use cases. He is active in the Fedora project and the wider open source community. Of all of the HashiCorp products, he chose Packer to write a book about because of its simple yet powerful functionality and its recent rewrite to make use of HCL2.

I want to thank the people who have supported me during this long writing process, including the Packt team and my friends and family who put up with the long nights and weekends I spent writing sample code.

About the reviewers

Clara McKenzie is currently a team lead for infrastructure at Rebellion Defense with a long career in software, specializing in networking and embedded platforms. Born in San Francisco, she received a BA in mathematics from Reed College in Portland, Oregon. Her work adventures include the NFS/RPC group at Sun Microsystems, the NetWareForMac team at Novell, Core Software at Ascend Communications, the Gracenote SDK team, Planet Labs Dove Satellites, and supporting Terraform at HashiCorp. She goes by the handle @cemckenzieon GitHub.

Rickard von Essen works as a continuous delivery and cloud consultant at Diabol. There, he helps companies deliver faster, improve continuously, and worry less. He was one of the early maintainers of Packer and contributed to the Parallels plugin. In his spare time, he flies his FPV drones, repairs retro computers, and contributes to numerous FOSS projects. He has been tinkering with Linux and BSD since the late 1990s and has been hacking since the Amiga era. When not fiddling with computers, he spends his time with his wife and three children in Stockholm, Sweden. Rickard received an MSc in computer science and engineering from Linköping University.

Table of Contents

Preface

Part 1: Packer’s Beginnings

1

Packer Fundamentals

Technical requirements

Packer architecture

History of Packer

Who uses Packer?

Alternatives to Packer

Installing Packer

HCL versus JSON

Example legacy JSON

Example PKR.JSON

Example HCL

Summary

2

Creating Your First Template

Technical requirements

Hello World template for a local VM

Breakdown of components, variables, and artifacts

Using an IDE to guide templates

Applying the VirtualBox builder

Summary

3

Configuring Builders and Sources

Technical requirements

Simplifying your template with variables

Utilizing local system builders

Using VirtualBox builders – ISO, OVF, and VM

Using VMware builders – ISO, VMX, and vSphere clone

Using the QEMU builder

Adding cloud builders

Providing your cloud credentials

Using the AWS builders

Using the Azure builders

Using the GCP builders

Exploring other cloud builders

Building containers

Using the null builder

Summary

4

The Power of Provisioners

Technical requirements

Configuring communicators

Injecting your config or artifacts

Seeding a file or directory into your image once booted

Using templates to populate configuration resources with variables

Running a script or job across all builders

Developing provisioners with the null builder

Summary

5

Logging and Troubleshooting

Technical requirements

Managing stderr and stdout

Using environment variables for logging and debugging

Controlling flow and using breakpoints

Summary

Part 2: Managing Large Environments

6

Working with Builders

Technical requirements

Adding applications deployable from vSphere

Adding an AWS EC2 AMI build

Using HashiCorp Vault for credentials

Adding an Azure build

Adding a Google GCP build

Parallel builds

CI testing against multiple OS releases

Pitfalls and things to avoid

Summary

7

Building an Image Hierarchy

Technical requirements

Starting with LXC/LXD images

Docker container image format

Podman plugin for the OCI container image format

Base image strategy

Aggregation and branching out multiple pipelines

Summary

8

Scaling Large Builds

Technical requirements

Speeding up your builds with parallel processes

Preventing parallel processes from causing DoS

Troubleshooting logs in a parallel world

Using image compression

Selecting a compression algorithm for Packer images

Selecting the right storage type for the image lifecycle

Building delta and patch strategies

Summary

Part 3: Advanced Customized Packer

9

Managing the Image Lifecycle

Technical requirements

Tracking image lifecycle

Using the manifest post-processor

Creating a retention policy

Summary

10

Using HCP Packer

Technical requirements

Creating an HCP organization

Configuring HCP Packer in your templates

Using HCP ancestry

Consuming HCP Packer from Terraform

Exploring the HCP REST API

Summary

11

Automating Packer Builds

Technical requirements

Identifying common automation requirements

Exploring basic GitHub Actions support

Exploring GitLab CI pipeline support

Using HashiCorp Vault integration for automation

GitLab CI

GitHub Actions

Summary

12

Developing Packer Plugins

Technical requirements

Basics of Go

Sample plugin source

Building and testing your plugin

Protecting Packer from bad plugins

Summary

Grand conclusion

Index

Other Books You May Enjoy

Part 1: Packer’s Beginnings

This part offers a basic introduction to Packer, including the build process and what code options are available. Early versions of Packer used only JSON, but newer releases have added HCL2 as the new standard in Packer templates. In these chapters, we will lay the foundation for beginner-to-intermediate users of Packer.

This part has the following chapters:

Chapter 1, Packer FundamentalsChapter 2, Creating Your First TemplateChapter 3, Configuring Builders and SourcesChapter 4, The Power of ProvisionersChapter 5, Logging and Troubleshooting

1

Packer Fundamentals

Packer is a free and open source extensible software tool that takes your desired OS and container configurations and builds them simultaneously for the easy testing and management of complex system and application images and artifacts. If you ever find yourself in an environment where multiple custom system disks or cloud AMIs must be consistently maintained and adjusted to boot VMs or run containers, then Packer is here to simplify your life as you automate configuration through code.

This chapter is a very high-level introduction for those unfamiliar with Packer. It explains how Packer is not a service but a tool that can be manually run or inserted into an automation pipeline. It also describes how Packer can supplement Terraform to dramatically simplify anything from complex hybrid or multi-cloud deployments to on-premises private cloud or even local VMs on a development machine.

In this chapter, we will cover the following topics:

Packer architecture, which describes how the Packer binary is distributed and developed and how Packer works with templates, builders, and provisioners at a high levelHistory of Packer, which is important to understand why Packer was needed in the first place and what business problems it solvesWho uses Packer?, which lists what types of users Packer has today, including everything from small academic labs to large-scale enterprise organizations and software vendorsAlternatives to Packer, which is a section that describes industry alternatives and other tools that offer image management and how they compare to Packer at a high levelInstalling Packer, which covers how easy it is to install Packer on most environments, whether servers, cloud instances, or local laptopsHCL versus JSON, which is a very high-level description of JSON and HashiCorp Configuration Language (HCL) and how Packer supports either standard for templates

Technical requirements

For this chapter, you should have a basic understanding of JSON and HCL2 domain specific languages. You won't need to try any sample code for this chapter but if you choose to follow along you may use any device that supports running the Packer binary. This includes any laptop or small device running Linux, macOS, or Windows. As we focus a lot of examples on Linux, it may be useful to run Linux or use a cloud resource running Linux at minimum.

Packer architecture

Packer itself is a fairly simple binary written in Go. It supports plugins for various inputs and outputs. The plugins that translate your configuration and scripts into artifact outputs are called builders. Common builders include common hypervisors such as VMware, QEMU, VirtualBox, AWS, GCP, and Microsoft Azure. Builders also include multiple container image formats, including LXC, LXD, Docker, and Podman. Many plugins have been contributed by the community and we will cover how you can write your own in a future chapter.

The bit of code you write to tell Packer what to do is called a template. Early versions of Packer expected your template to be written in JavaScript Object Notation (JSON). As of Packer version 1.7.0, both JSON and HashiCorp Configuration Language version 2 (HCL2) are supported, with the latter being preferred. We will cover both formats and how you can migrate a JSON template into an HCL2 template shortly.

Provisioners are tasks or resources that should be applied to your image before packaging. By default, each builder in your template takes each provisioner. Take an example where you want to build a system image with your application across AWS, Azure, and GCP. All you need to do is define your list of builders for AWS, Azure, and GCP and include a single provisioner that uploads your application.

A build job is what runs the Packer build command with your template. Normally, this forks a parallel process for every builder you specify in your template. A build can happen simultaneously across VMware, AWS, Azure, GCP, or other builders while Packer tracks the results and reports any errors. When all builders finish or end in an error, the job is done and the Packer process terminates. Optionally, Packer may compress output images before terminating, to save space.

History of Packer

The origins of Packer can be found in HashiCorp’s Vagrant product. Vagrant was originally a Ruby project to select from a set of standard OS images, boot one or more VMs, and automatically configure them once booted. Vagrant allowed for rapid environments for development with an extensible framework to support multiple virtualization platforms, such as VirtualBox, VMware, and QEMU.

When managing multiple environments for multiple teams, one needs to strike a balance of build time versus runtime. Provisioning resources is quick and easy when everything comes in a pre-built package or artifact, but purpose-built artifacts for every use case take up quite a bit of storage. What resources will be common across an organization and which might be deployed in different ways when they are consumed? Building multiple gold images for Vagrant or cloud environments becomes a challenge at scale. Packer was built to simplify this and it works very efficiently. It can be run simply on your own computer or it can be inserted into automation jobs and pipelines. We will cover all of these use cases in this book and show you how easily Packer can simplify your image maintenance both locally and in the cloud. A team that needs identical images built across multiple regions, multiple clouds, and possibly even local infrastructure may require complex image management. Each region within each cloud may need multiple versions of an image to be maintained, based on the OS, applications deployed, and custom configuration. Keeping all environments consistent often creates exponential complexity. Imagine each line in this diagram represents a combination that requires an image to be built and maintained:

Figure 1.1 – Managing multiple applications across multiple environments can be complex

Many people will attempt to manage complex environments like this one using purely provisioning tools such as Vagrant and Terraform, which can actually result in more complexity in the end. A minor change to a Terraform provisioner can result in an entire environment being destroyed and rebuilt. It’s important to start with a good image strategy before provisioning to simplify things at runtime. Often, a single Packer template can be used to satisfy all of the preceding combinations.

Packer was also the first HashiCorp project written purely in Go, also known as Golang, the modern programming language created by Google. Go is an optimized compiled language that generates simple statically linked binaries using a community of open source projects. A lot of management tools like Packer tend to be written in a scripting language such as Python or Ruby so that they can be easily ported and customized. Even Vagrant was initially written in Ruby. Scripting languages such as Ruby tend to not perform as well as precompiled Go binaries. Scripting languages are also prone to dependency deprecation and complexity. If you download a Packer binary, everything you need to run is self-contained. You won’t run into an issue where an old OS version of glibc or Python prevents the binary from running. You also won’t have memory leaks or buffer vulnerabilities as Go manages its own memory via garbage collection. Golang has since been the language of choice for HashiCorp projects, including Vagrant, which was rewritten in Golang for consistency. If you don’t know how to write Go, there is no need to worry. You won’t need to write Go to use Packer unless you want to write a plugin or add a feature. We will cover how to do this in Chapter 12, Developing Packer Plugins.

You can also find books on Go from Packt here: https://www.packtpub.com/gb/tech/go?released=Available&language=Go.

Who uses Packer?

Packer is a purely open source tool for HashiCorp but that doesn’t mean that enterprise customers don’t use it. Packer is used to build images in private networks and public clouds around the world, covering many industries from investment banking to universities and students. Individuals and small teams often use Packer to maintain a set of disposable system images for mixed estates, including Mac, Windows, Linux, and serverless cloud applications. Large teams and organizations may use automation or continuous deployment pipelines for Packer to rebuild a set of images when certain events or edits occur. The beauty of Packer is it behaves the same whether it is running in a multi-cloud Fortune 500 firm or running on a laptop in a coffee shop. You can even run Packer on a low-power commodity ARM device such as Raspberry Pi. The difference between a coffee shop laptop and an enterprise deployment really comes down to security and best practices, which we’ll cover in Chapters 6-8.

The open source community has a great variety of sample templates, so you usually don’t need to start one from scratch. Search the Packer documentation page for samples, as well as GitHub. Unlike Vagrant, which has a public registry of source images, Packer requires the user to provide base images.

Terraform users find Packer valuable for any projects that use VM deployments in a hybrid cloud environment. Properly prepared images will dramatically ease VM provisioning with Terraform. More importantly, cloud-native tooling that may provision instances dynamically, such as autoscale groups or failover routines, will not inform Terraform about their activity. Having a proper VM autoscale group deployed with Terraform still requires a standard image for the cloud to scale.

Alternatives to Packer

Image management has been a challenge for years. Packer is certainly not the first tool to address these difficulties. Tools such as Solaris JumpStart or Red Hat Kickstart have been used to codify VM installation. These can be used in conjunction with Packer to build uniform images across platforms. Packer may use a kickstart to deploy a Linux platform from standard media but then use provisioners to deploy tooling identically across Linux and Windows environments. Docker Compose and Buildah are also modern tools for building specialized container images. Often, specialized community tools such as this can supplement Packer while letting Packer provide a more general-purpose building tool to bring complex mixed environments into one single template. Red Hat Enterprise Linux users have the option of leveraging Red Hat Satellite for platform standardization using a combination of Kickstart, Cobbler, and Puppet.

Historically, simple scripting has been used for early infrastructure as code strategies. If configuration can be scripted, it can be version controlled and used to build and test images captured with either virtualization tools or image tooling such as Norton Ghost.

Installing Packer

Packer is freely available via many options depending on your computer. You can download the full source code at any time via GitHub. In most cases, there is no need to compile your own release. Official binaries are available on HashiCorp’s release page: https://releases.hashicorp.com