Mastering AWS CloudFormation E-Book

Karen Tovmasyan

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 248

Veröffentlichungsjahr: 2020

Beschreibung

DevOps and the cloud revolution have forced software engineers and operations teams to rethink how to manage infrastructures. With this AWS book, you'll understand how you can use Infrastructure as Code (IaC) to simplify IT operations and manage the modern cloud infrastructure effectively with AWS CloudFormation.
This comprehensive guide will help you explore AWS CloudFormation from template structures through to developing complex and reusable infrastructure stacks. You'll then delve into validating templates, deploying stacks, and handling deployment failures. The book will also show you how to leverage AWS CodeBuild and CodePipeline to automate resource delivery and apply continuous integration and continuous delivery (CI/CD) practices to the stack. As you advance, you'll learn how to generate templates on the fly using macros and create resources outside AWS with custom resources. Finally, you'll improve the way you manage the modern cloud in AWS by extending CloudFormation using AWS serverless application model (SAM) and AWS cloud development kit (CDK).
By the end of this book, you'll have mastered all the major AWS CloudFormation concepts and be able to simplify infrastructure management.

Leseprobe

Mastering AWS CloudFormation

Plan, develop, and deploy your cloud infrastructure effectively using AWS CloudFormation

Karen Tovmasyan

BIRMINGHAM—MUMBAI

Mastering AWS CloudFormation

Copyright © 2020 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Vijin Boricha

Acquisition Editor: Meeta Rajani

Senior Editor: Arun Nadar

Content Development Editor: Pratik Andrade

Technical Editor: Dinesh Pawar

Copy Editor: Safis Editing

Project Coordinator: Neil Dmello

Proofreader: Safis Editing

Indexer: Rekha Nair

Production Designer: Nilesh Mohite

First published: May 2020

Production reference: 1070520

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78913-093-5

www.packt.com

To my mother, Susanna, and my father, Andranik, for everything they did for me. To my wife, Natalia, for being my beloved muse.

- Karen Tovmasyan

Packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionalsImprove your learning with Skill Plans built especially for youGet a free eBook or video every monthFully searchable for easy access to vital informationCopy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors

About the author

Karen Tovmasyan started his career journey when he was still a student at university. His first job – at one of Renault's factories in Russia – welcomed him and he spent his first year there in IT support, later joining the TechProjects team and having his first servers to manage, which was his first big migration project. A few years later, he joined a SaaS company as a DevOps engineer and in 2016, he moved to the Netherlands. He holds three AWS certifications and is an active member of the Russian-speaking AWS community, helping its members to embrace cloud adoption and make the most of AWS. He gives public talks at conferences and writes content for two personal blogs, on Medium and Telegram.

I want to thank my family, my friends, my readers, and all those who keep learning and improving themselves day after day.

About the reviewers

Anton Tokarev is now working at a company called Softeq with AWS services. He started his career in IT as a network engineer almost 10 years ago. He likes to write code in Python and Go. He has experience with start-ups, big companies, product development companies, and outsourcing.

Aleksandr Patrushev has worked in IT for more than 12 years as a system engineer and public speaker and every new step in his career has been a counterposition to the previous one: from x86 system engineer to IBM Power Systems system engineer at IBM, to VMware system engineer at Vmware, to AWS solution architect at AWS. Each time he jumped out of his comfort zone and ran toward new ideas and technologies. Currently, he is an AWS solution architect who helps companies of any size, from startups to large corporations, through the exciting process of adapting cloud technologies to meet business needs. He loves to find the most efficient solutions for complex projects and challenges. The opinions expressed in the book's review are his own.

I want to say thank you to the author of this book, Karen Tovmasyan, who decided to spend his time creating an interesting book that will help people to understand how to implement the IaC approach in simple language with examples.

Packt is searching for authors like you

If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Table of Contents

Preface

Who this book is forvii

What this book coversvii

To get the most out of this bookviii

Download the example code filesix

Code in Actionix

Download the color imagesx

Conventions usedx

Get in touchxi

Reviewsxi

Section 1: CloudFormation Internals

Chapter 1

CloudFormation Refresher

Technical requirements4

Understanding the internals of AWS CloudFormation4

Creating your first stack5

Understanding CloudFormation IAM permissions6

Drift detection11

Summary14

Questions14

Further reading14

Chapter 2

Advanced Template Development

Technical requirements16

Going through the internals of the template16

AWSTemplateFormatVersion16

Description16

Metadata17

Parameters17

Mappings20

Conditions21

Transform22

Resources22

Outputs23

Creating reusable templates24

Using conditional elements32

Deletion policies34

Referring to existing stacks37

AWS pseudo parameters41

AWS::AccountId41

AWS::NoValue43

AWS::Region44

AWS::StackId and AWS::StackName44

AWS::URLSuffix45

AWS::Partition46

Dynamic references with Parameter Store and Secrets Manager46

Summary50

Questions50

Further reading51

Section 2: Provisioning and Deployment at Scale

Chapter 3

Validation, Linting, and Deployment of the Stack

Technical requirements56

Validating the template56

Using a linter for best practices on templates58

Linting against specific regions59

Ignoring specific rules61

Creating custom rules61

Provisioning our stack65

Deploying stacks using Change Sets66

Handling errors71

Working with drifts73

Summary77

Questions77

Further reading78

Chapter 4

Continuous Integration and Deployment

Technical requirements80

Including a template in your application80

Running smoke tests on your stack81

Smoke testing for EC2 auto scaling groups82

Smoke testing VPC resources87

Best practices for the release management of CloudFormation stacks90

Always use version control systems90

Ensuring that your dependencies can be easily retrieved90

Keeping your code base clean91

Choosing a proper branching model91

Always perform a code review91

Having full test coverage91

Don't forget to use a CloudFormation IAM role92

Always perform smoke testing92

Using the correct CI/CD instruments92

Keeping up the good work92

Creating a CI/CD pipeline with CloudFormation and CodePipeline93

Scenario – core stack94

Summary107

Questions108

Further reading108

Chapter 5

Deploying to Multiple Regions and Accounts Using StackSets

Technical requirements110

The old-fashioned way of multi-regional and multi-account infrastructure management110

Introducing StackSets112

Permissions114

Deploying to multiple regions116

Using AWS Console116

Using awscli122

Best practices for StackSets124

Deploying to multiple accounts125

Preventing failures of multiple StackSet deployments using TAGs127

Summary132

Questions132

Further reading132

Chapter 6

Configuration Management of the EC2 Instances Using cfn-init

Technical requirements134

Introducing cfn-init134

Deploying your application to EC2 during stack creation138

Creating a Hello, World application138

Creating LNMP stack144

Using cfn-signal to inform CloudFormation about resource readiness149

Summary151

Questions152

Further reading152

Section 3: Extending CloudFormation

Chapter 7

Creating Resources outside AWS Using Custom Resources

Technical requirements156

Understanding custom resources156

The internals of the underlying Lambda function158

Writing and managing your own custom resource161

Creating databases in RDS using CRs161

Handling updates, deletions, and failures of CRs167

Deleting resource167

Updating resources171

Extra features for the custom database function172

Summary177

Questions177

Further reading177

Chapter 8

Dynamically Rendering the Template Using Template Macros

Technical requirements180

Understanding the use cases of the template macro180

Auto filling resource property values180

Adding extra resources182

Making resource declaration easier for developers182

Introducing the template macro183

Considerations184

Writing your own macro185

AMI ID filler185

Rendering the application template from a short declaration191

Summary199

Questions199

Further reading199

Chapter 9

Generating CloudFormation Templates Using AWS CDK

Technical requirements202

Introducing AWS CDK202

Facilitating template development with AWS CDK204

Writing your first template using AWS CDK and Python207

Preparing constructs207

Rendering core resources210

Rendering the web tier213

Rendering the storage tier220

Deploying CDK application221

Testing CDK applications223

Summary227

Questions228

Further reading228

Chapter 10

Deploying Serverless Applications Using AWS SAM

Technical requirements230

Introducing AWS SAM230

Understanding the differences between SAM and CloudFormation232

Writing your first serverless application with SAM233

Prerequisites234

Developing a Hello, World application234

Running SAM applications240

Examining logs with SAM243

Creating complex applications with SAM244

Summary252

Questions253

Further reading253

Chapter 11

What's Next?

The future of infrastructure as code256

Understanding the difference between Terraform and CloudFormation258

Provider support259

Declaration syntax260

Development and deployment methodologies261

Understanding the value of Cloud Development Kit262

Testing infrastructure263

Adding artifacts264

Summary265

Further reading265

Assessments

Chapter 1: CloudFormation Refresher267

Chapter 2: Advanced Template Development268

Chapter 3: Validation, Linting, and Deployment of the Stack268

Chapter 4: Continuous Integration and Deployment269

Chapter 5: Deploying to Multiple Regions and Accounts Using StackSets269

Chapter 6: Configuration Management of the EC2 Instances Using cfn-init270

Chapter 7: Creating Resources outside AWS Using Custom Resources270

Chapter 8: Dynamically Rendering the Template Using Template Macros270

Chapter 9: Generating CloudFormation Templates Using AWS CDK271

Chapter 10: Deploying Serverless Applications Using AWS SAM271

Other Books You May Enjoy

Section 1: CloudFormation Internals

In our first section, we will do a small refresher on CloudFormation. Later, we will dive deep into its core component – a template – and learn how to write universal, redundant, and reusable templates.

This section comprises the following chapters:

Chapter 1, CloudFormation RefresherChapter 2, Advanced Template Development

Chapter 1

CloudFormation Refresher

Cloud computing introduced a brand-new way of managing the infrastructure.

As the demand for the AWS cloud grew, the usual routine and operational tasks became troublesome. The AWS cloud allowed any type of business to rapidly grow and solve all the business needs regarding compute power; however, the need to maintain a certain stack of resources was hard.

DevOps culture brought a set of methodologies and ways of working, and one of those is called infrastructure as code. This process is about treating your infrastructure—network, virtual machines, storages, databases, and so on—as a computer program.

AWS CloudFormation was developed to solve this kind of problem.

You will already have some working knowledge of CloudFormation, but before we dive deep into learning advanced template development and how to provision at scale, use CloudFormation with CI/CD pipelines, and extend its features, let's quickly refresh our memory and look again at what CloudFormation is and how we use it.

In this chapter, we will learn the following:

The internals of AWS CloudFormationCreating and updating a CloudFormation stackManaging permissions for CloudFormationDetecting unmanaged changes in our stack

Technical requirements

The code used in this chapter can be found in the book's GitHub repository athttps://github.com/PacktPublishing/Mastering-AWS-CloudFormation/tree/master/Chapter1.

Check out the following video to see the Code in Action:

https://bit.ly/2WbU5Lh

Understanding the internals of AWS CloudFormation

AWS services consist of three parts:

APIBackendStorage

We interact with AWS by making calls to its API services. If we want to create an EC2 instance, then we need to perform a call, ec2:RunInstances.

When we develop our template and create a stack, we invoke the cloudformation:CreateStack API method. AWS CloudFormation will receive the command along with the template, validate it, and start creating resources, making API calls to various AWS services, depending on what we have declared for it.

If the creation of any resource fails, then CloudFormation will roll back the changes and delete the resources that were created before the failure. But if there are no mistakes during the creation process, we will see our resources provisioned across the account.

If we want to make changes to our stack, then all we need to do is update the template file and invoke the cloudformation:UpdateStack API method. CloudFormation will then update only those resources that have been changed. If the update process fails, then CloudFormation will roll the changes back and return the stack to the previous, healthy, state.

Now that we have this covered, let's start creating our stack.

Creating your first stack

I'm sure you've done this before.

We begin by developing our template first. This is going to be a simple S3 bucket. I'm going to use YAML template formatting, but you may use JSON formatting if you wish:

MyBucket.yaml

AWSTemplateFormatVersion: "2010-09-09"

Description: This is my first bucket

Resources:

  MyBucket:

    Type: AWS::S3::Bucket

Now we just need to create the stack with awscli:

$ aws cloudformation create-stack \

                     --stack-name mybucket\

                     --template-body file://MyBucket.yaml

After a while, we will see our bucket created if we go to the AWS console or run aws s3 ls.

Now let's add some public access to our bucket:

MyBucket.yaml

AWSTemplateFormatVersion: "2010-09-09"

Description: This is my first bucket

Resources:

  MyBucket:

    Type: AWS::S3::Bucket

    Properties:

      AccessControl: PublicRead

Let's run the update operation:

$ aws cloudformation update-stack \

                     --stack-name mybucket \

                     --template-body file://MyBucket.yaml

To clean up your workspace, simply delete your stack using the following command:

$ aws cloudformation delete-stack --stack-name mybucket

Let's now look at the CloudFormation IAM permissions.

Understanding CloudFormation IAM permissions

We already know that CloudFormation performs API calls when we create or update the stack. Now the question is, does CloudFormation have the same powers as a root user?

When you work with production-grade AWS accounts, you need to control access to your environment for both humans (yourself and your coworkers) and machines (build systems, AWS resources, and so on). That is why controlling access for CloudFormation is important.

By default, when the user runs stack creation, they invoke the API method cloudformation:CreateStack. CloudFormation will use that user's access to invoke other API methods during the stack creation.

This means that if our user has an IAM policy with an allowed action ec2:*, but attempts to create an RDS instance with CloudFormation, the stack will fail to create with an error, User is unauthorized to perform this action.

Let's try this. We will create an IAM role with ec2:*, assume that role, and try to create the same bucket stack:

Important note

We already have an IAM user Admin in our AWS account and we will add that user as a principal.

MyIamRole.yaml

AWSTemplateFormatVersion: "2010-09-09"

Description: "This is a dummy role"

Resources:

  IamRole:

    Type: AWS::IAM::Role

    Properties:

      AssumeRolePolicyDocument:

        Version: 2012-10-17

        Statement:

          - Sid: AllowAssumeRole

            Effect: Allow

            Principal:

              AWS:

                - !Join

                  - ""

                  - - "arn:aws:iam::"

                    - !Ref "AWS::AccountId"

                    - ":user/Admin"

            Action: "sts:AssumeRole"

      ManagedPolicyArns:

        - "arn:aws:iam::aws:policy/AmazonEC2FullAccess"

        - "arn:aws:iam::aws:policy/AWSCloudformationFullAccess"

Outputs:

  IamRole:

    Value: !GetAtt IamRole.Arn

If we create this stack, assume that role, and try to create the previous mybucket stack, it will fail to create with an error. Let's take a look:

$ aws cloudformation create-stack \

                     --stack-name iamrole \

                     --capabilities CAPABILITY_IAM \

                     --template-body file://IamRole.yaml

$ IAM_ROLE_ARN=$(aws cloudformation describe-stacks \

                                    --stack-name iamrole \

--query "Stacks[0].Outputs[?OutputKey=='IamRole'].OutputValue" \

--output text)

$ aws sts assume-role --role-arn $IAM_ROLE_ARN \

                      --role-session-name tmp

# Here goes the output of the command. I will store the access credentials in the env vars

$ export AWS_ACCESS_KEY_ID=…

$ export AWS_SECRET_ACCESS_KEY=…

$ export AWS_SESSION_TOKEN=…

$ aws cloudformation create-stack \

                     --stack-name mybucket \

                     --template-body file://MyBucket.yaml

We will see the following error on the AWS console:

Figure 1.1 – CloudFormation console – stack events

On the other hand, we cannot provide everyone with an AdminAccess policy, so we need to find a way to use CloudFormation with the necessary permissions while only letting CloudFormation use those permissions.

CloudFormation supports service roles. Service roles are the IAM roles that are used by different AWS services (such as EC2, ECS, Lambda, and so on). CloudFormation service roles are used by CloudFormation during stacks and StackSets operations—creation, update, and deletion:

Let's create a specific role for CloudFormation:

CfnIamRole.yaml

AWSTemplateFormatVersion: "2010-09-09"

Description: "This is a CFN role"

Resources:

  IamRole:

    Type: AWS::IAM::Role

    Properties:

      AssumeRolePolicyDocument:

        Version: 2012-10-17

        Statement:

          - Sid: AllowAssumeRole

            Effect: Allow

            Principal:

              Service: "cloudformation.amazonaws.com"

            Action: "sts:AssumeRole"

      ManagedPolicyArns:

        - "arn:aws:iam::aws:policy/AdministratorAccess"

Outputs:

  IamRole:

    Value: !GetAtt IamRole.Arn

We create this stack for the service role and obtain the CloudFormation role ARN:

$ aws cloudformation create-stack \

                     --stack-name cfniamrole \

                     --capabilities CAPABILITY_IAM \

                     --template-body file://CfnIamRole.yaml

$ IAM_ROLE_ARN=$(aws cloudformation describe-stacks \

                                    --stack-name cfniamrole \

--query "Stacks[0].Outputs[?OutputKey=='IamRole'].OutputValue" \

--output text)

Now we run the creation of the stack, which will use our role, specifying the Role ARN:

$ aws cloudformation create-stack \

                     --stack-name mybucket \

                     --template-body file://MyBucket.yaml \

                     --role-arn $IAM_ROLE_ARN

After a while, we can verify that our stack has been created, and we see our bucket!

$ aws s3 ls

# Output

2019-10-16 14:14:24 mybucket-mybucket-jqjpr6wmz19q

Before we continue, don't forget to clean your account:

$ for i in mybucket iamrole cfniamrole; do aws cloudformation delete-stack --stack-name $i ; done

Important note

Note that in the preceding example, we provide the CloudFormation role with an AdminPolicy (the one that is provided by AWS by default).

In production-grade systems, we want to allow CloudFormation only those privileges that are required for the stack.

There are two permission schemas that are being applied for CloudFormation roles:

We have a certain list of services that we can use (for example, EC2, RDS, VPC, DynamoDB, S3, and so on).Each template/stack combination will use only those services it needs—for example, if we declare Lambda functions with Simple Notification Service (SNS), then we should create the role with policies only for Lambda and SNS.

Drift detection

CloudFormation as a service often refers to the term state. The state is basically inventory information that contains a pair of values: the logical resource name and the physical resource ID.

CloudFormation uses its state to understand which resources to create or update. If we create a stack with a resource with a logical name foo, change the property of this resource (foo) in a template, and run an update, then CloudFormation will change the corresponding physical resource in the account.

CloudFormation has a set of limitations. For example, it will not update the stack if we do not introduce changes to it. If we perform manual changes to the resource, then CloudFormation will change them only when we make changes to the template.

Developers had to rethink their way of managing the infrastructure once they started using CloudFormation, but we will get to that in the later chapters. For now, we would like to show you a feature that doesn't solve problems of manual intervention, but at least notifies us when they happen. This feature is called drift detection.

For this example, we will use the same template (Dummy IAM Role) as we did in the previous section:

$ aws cloudformation create-stack \

                     --stack-name iamrole \

                     --template-body file://IamRole.yaml \

                     --capabilities CAPABILITY_IAM

After a while, we see our stack created:

Figure 1.2 – CloudFormation console

Note the link on the right called Drifts. If we follow that link, we will see the Drifts menu and under that Drift status: NOT_CHECKED. At the time of writing, we will have to run drift detection manually, so we need to run drift detection on our stack. After a while, we will see that everything is all right:

I'm going to run Detect stack drifts and verify that my stack is compliant:

Figure 1.3 – CloudFormation console – drifts

Now what we will do is add an extra policy to our role and rerun drift detection:

$ ROLENAME=$(aws cloudformation describe-stack-resources --stack-name iamrole --query "StackResources[0].PhysicalResourceId" --output text)

$ aws iam attach-role-policy --role-name $ROLENAME --policy-arn "arn:aws:iam::aws:policy/AdministratorAccess"

We can now detect drift again:

Figure 1.4 – CloudFormation console – drift detected

If we check our IamRole resource and click on View drift details, we will see what exactly has been changed and differs from CloudFormation's state:

Figure 1.5 – CloudFormation console – actual modification

Now we have two options: either roll back the change to the resource manually or add any dummy property to the template and run update-stack.

We've learned about CloudFormation drifts, how to run its drift detection, and the actions that must be taken afterward. But don't worry—we will revisit drifts again in the following chapters.

Summary

In this refresher chapter, we refreshed our memory as to what CloudFormation is, how we create and update stacks, why service roles are important, and how to implement them. We also remembered what drifts in CloudFormation are, when they occur, and how to detect them.

While this is an introductory chapter, we covered the fundamental building blocks of CloudFormation. In the following chapters, we will use service roles and drift detection again, but first, we need to deep dive into the internals of the CloudFormation template, which we are going to do in the next chapter.

Questions

Which API method is invoked when we create a CloudFormation stack?What is a CloudFormation service role?Which IAM policies are used if we do not specify the CloudFormation service role?How is the information about stack resources stored in CloudFormation?What happens if we delete the resource created by CloudFormation and try to create the same stack?What happens if we delete the resource created by CloudFormation and try to update the same stack?Why can't CloudFormation recreate the deleted resource?

Further reading

CloudFormation service roles: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-iam-servicerole.htmlDrift detection in CloudFormation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-stack-drift.html

Chapter 2