SAFe® for DevOps Practitioners E-Book

Robert Wen

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 469

Veröffentlichungsjahr: 2022

Beschreibung

Product development and release faces overlapping challenges due to the combined pressure of delivering high-quality products in shorter time-to-market cycles, along with maintaining proper operation and ensuring security in a complex high-tech environment. This calls for new ways of overcoming these challenges from design to development, to release, and beyond. SAFe® for DevOps Practitioners helps you use a DevOps approach with the Scaled Agile Framework and details how value streams help you resolve these challenges using examples and use cases.
The book begins by explaining how the CALMR approach makes DevOps effective in resolving product development roadblocks. Next, you’ll learn to apply value stream management to establish a value stream that enables product development flow, measure its effectiveness through appropriate feedback loops, and find ways of improving it. Finally, you’ll get to grips with implementing a continuous delivery pipeline that optimizes the value stream through four phases during release on demand. This book complements the latest SAFe DevOps courses, and you’ll find it useful while studying for the SAFe DevOps Practitioner (SDP) certification.
By the end of this DevOps book, you’ll have gained a clear understanding of how to achieve continuous execution and release on demand using DevOps and SAFe.

Leseprobe

SAFe® for DevOps Practitioners

Implement robust, secure, and scaled Agile solutions with the Continuous Delivery Pipeline

Robert Wen

BIRMINGHAM—MUMBAI

SAFe® for DevOps Practitioners

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Rahul Nair

Publishing Product Manager: Surbhi Suman

Senior Editor: Tanya D’cruz

Technical Editor: Arjun Varma

Copy Editor: Safis Editing

Project Coordinator: Shagun Saini and Prajakta Naik

Proofreader: Safis Editing

Indexer: Sejal Dsilva

Production Designer: Shyam Sundar Korumilli

Senior Marketing Coordinator: Nimisha Dua

Marketing Coordinator: Gaurav Christian

First published: December 2022

Production reference: 1251122

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80323-142-6

www.packt.com

To Jill and Charles, who have made my life better in so many ways.

And to Steve, a member of the family through years of friendship. Thanks so much for the support.

– Robert Wen

Foreword

Recently, I attended the 2022 DevOps Enterprise Summit and left with an interesting observation. Although a significant portion of the agenda continues to focus on deep technical and architectural practices, there’s a growing interest in social aspects including culture, leadership, and organizational change. My takeaway? While it’s true that us engineers value our technical tooling and techniques, a successful DevOps implementation requires more than just good tech.

The week after the summit, I read a preview of Robert Wen’s new book, SAFe® for DevOps Practitioners. I was delighted to learn that he had made the same observation: For DevOps to succeed, tooling and techniques must integrate with culture, leadership, and other practices. Organizations need to think of them holistically, as we do in SAFe.

Wen masterfully blends these two subjects. In one section, he discusses DORA metrics, the seminal research connecting DevOps practices to business performance. In another area, he introduces the Westrum model of organizational culture, and explains how leaders must model the right behaviors to transform the culture to support a DevOps transformation. In another section, he dives into Continuous Integration (CI) tooling and techniques for automated building, testing, and packaging.

For large enterprises to achieve business agility, there can be no choice between SAFe or DevOps. Success will require both SAFe guidance for business agility and the technical practices and tooling espoused by DevOps. SAFe® for DevOps Practitioners demonstrates how SAFe change agents and DevOps practitioners can work together for overall business success.

Wen makes this connection by structuring the content around three SAFe DevOps constructs:

Part 1, “CALMR”Part 2, SAFe value streams Part 3, the Continuous Delivery Pipeline (CDP)

Part 1 explores modern DevOps practices organized around CALMR, the acronym for Culture, Automation, Lean Flow, Measurement, and Recovery. Incorporating the guidance found in SAFe, Wen covers the principle-based elements of Culture and Lean Flow by including Westrum’s culture, Deming’s batch sizes, and Kersten’s project-to-product approach. He also provides insights into the deep technical practices and tooling required for automation, measuring, and recovery.

Part 2 focuses on value streams, a critical element in SAFe and Lean, including chapters defining them and measuring their performance. The first chapter presents guidance on identifying value streams in SAFe and the benefits of value stream mapping. The next chapter discusses the many ways to measure value stream performance through KPIs, DORA metrics, Mik Kersten’s Flow Framework, and SAFe’s three measurement domains.

Part 3 presents an extensive list of Agile, Lean, SAFe, and DevOps practices, organized by the four aspects and twelve practices of SAFe’s Continuous Delivery Pipeline (CDP). That organizing structure enables a deep, technical discussion of critical DevOps techniques, including Lean Startup, architecting for security, shift-left testing, chaos engineering, and many others. Bob puts them into a context that practitioners from both the SAFe and DevOps communities can easily understand.

I hope these brief descriptions give you a sense of the book’s remarkable breadth. It connects the SAFe principles and practices enabling business agility to the technical methods and tooling championed by DevOps. If you are a DevOps practitioner, you will connect with the book’s rich technical and tooling discussions and see how they align with SAFe’s principles and practices to enable business agility across the enterprise. If you are a SAFe practitioner, you will connect with the book’s organizing structure around CALMR and SAFe’s Continuous Delivery Pipeline (CDP) as you learn more about relevant DevOps technical tooling and techniques.

Finally, SAFe® for DevOps Practitioners can either be read from cover to cover or serve as a reference tool. Each chapter provides straightforward, detailed guidance from the relevant DevOps and SAFe bodies of knowledge. Readers (particularly those familiar with SAFe’s CALMR and CDP organizing structures) will find value in reading individual chapters and sections as a reference when needed.

Practitioners in the SAFe and DevOps communities help enterprises build some of the world’s most important systems. SAFe change agents enable business agility in most of the Global 2000, helping those enterprises survive and thrive in the digital age. DevOps practitioners in these organizations provide the tooling and technical processes that deliver digital products faster and more reliably. SAFe® for DevOps Practitioners shows how SAFe change agents and DevOps practitioners work together to accelerate business success.

Harry Koehnemann

SAFe® Fellow and Principal Consultant

Contributors

About the author

Robert Wen is a DevOps consultant at ReleaseTEAM. He specializes in Continuous Integration, Continuous Deployment, Value Stream Management, and Agile Development. He has over 20 years of experience in hardware and software development, working in industries such as defense, aerospace, healthcare, and telecommunications. During his career, he has been a coach as the Scrum Master for several Agile projects and a trainer for courses on the Scaled Agile Framework® (SAFe®). He holds the following standings and certifications: SAFe Program Consultant (SPC), Atlassian Certified Expert, Certified CloudBees Jenkins Platform Engineer, GitLab Professional Services Engineer, and Flow Framework Professional.

To the guides and inspirations on my learning journey.

For DevOps, thanks go to Jim and Daphne Herron, Bart Chrabski, and Joel Lanz at Island Training; Thad West, Rodney West, Becky King, Larry Cummings, and Tracy Walton at Isos Technology; and to the present day with Shawn Doyle, Karl Todd, and my coworkers at ReleaseTEAM.

For SAFe, thanks first go to Harry Koehnemann and Cindy VanEpps, who first guided me to SAFe. The journey continued with the help of Randy Smith, Jim Camden, Nate Crews, Abdou Mballey, Terrance Knecht, Sunny Rungasamy, Tojo Joseph, and all my former students.

I dedicate this book to the memory of Jack Caine. Jack, we miss you very much and hope you know how much you have taught us.

About the reviewers

Romano Roth graduated in 2001 as a computer scientist at FHNW. In the 20 years that he has worked as a software developer, software architect, and consultant for Zuhlke, he has built a comprehensive knowledge of software development, architecture, and processes. He has worked with a variety of platforms and technologies, consulting in the financial, insurance, cyber security, electricity, medical, and aviation sectors. DevOps is his big passion. He helps companies bring people, processes, and technology together so that they can continuously deliver value to their customers. He organizes DevOps Meetup Zürich and DevOpsDays Zürich. He also has his own YouTube channel on DevOps and he regularly speaks at conferences.

I’d like to thank my family and friends who understand the time and commitment it takes to work in this fast-changing industry. Working in this field would not be possible without the supportive DevOps community that has developed over the last several years. Thank you to all the trailblazers who make this field an exciting one to work in each and every day.

Martyn Coupland is a senior consultant at Inframon, a UK System Center consultancy company. He specializes in client and cloud technologies and has knowledge of the System Center suite. He spends time with customers, helping them design and implement System Center solutions. He has a background in support, and his career, like many others, started on the service desk in IT, performing first-line support. Apart from his experience with System Center, Martyn has experience in programming languages such as C# and web programming languages such as PHP, HTML, CSS, and JavaScript.

Table of Contents

Preface

1

Introducing SAFe® and DevOps

Challenges organizations face in product development

TTM pressures

Understanding customer wants

Security and compliance

Ensuring quality

Competition

Meeting the challenges

An introduction to Agile

The rise and fall of Waterfall

The emergence of Agile

The addition of Lean

An introduction to DevOps

DevOps tools and technology

DevOps people and processes

The DevOps movement begins

Scaling DevOps with SAFe®

Looking at CALMR

Mapping your value stream

Running your value stream through the Continuous Delivery Pipeline

Including security in the process

Summary

Questions

Further reading

Part 1: Approach – A Look at DevOps and SAFe® through CALMR

2

Culture of Shared Responsibility

A culture for organizational change

What kind of culture?

How do we change the culture?

Lean-Agile mindset

The SAFe House of Lean

Adjusting the Agile Manifesto

The SAFe core values

SAFe principles

Taking an economic view

Applying systems thinking

Assuming variability and preserving options

Building incrementally with fast, integrated learning cycles

Basing milestones on an objective evaluation of working systems

Visualizing and limiting WIP, reducing batch sizes, and managing queue lengths

Applying cadence – synchronizing with cross-domain planning

Unlocking the intrinsic motivation of knowledge workers

Decentralizing decision-making

Organizing around value

Value streams

Classic value streams

Operational and development value streams

Summary

Questions

Further reading

3

Automation for Efficiency and Quality

Pipelines and toolchains

Planning with Agile project management tools

Creating code and documentation

Continuous integration

Continuous integration versus continuous delivery versus continuous deployment

Orchestrating the change

Verifying quality

Packaging for deployment

Continuous deployment

Configuring environments with IaC

Releasing with configuration management and feature flags

Additional verification through advanced testing in the environment

Monitoring the environment

DevOps topologies

The system team

Summary

Questions

Further reading

4

Leveraging Lean Flow to Keep the Work Moving

Making the work visible

Specifying workflow with additional columns

Flagging impediments and urgent issues

Policies for specifying exit criteria

Limiting WIP

Keeping batch sizes small

Small batch sizes decrease cycle time

Small batch sizes decrease risk

Small batch sizes limit WIP

Small batch sizes improve performance

Finding the ideal batch size

Monitoring queues

Where are our queues?

Little’s Law and cycle time

Kingman’s Formula

Moving from project-based to product-based work

Project budgets versus value stream funding

Defined endpoint versus product life cycle

Cost centers versus business outcomes

Upfront risk identification versus spreading risks

Moving people to the work versus moving work to the people

Performing to plan versus learning

Misalignment versus transparent business objectives

Summary

Questions

Further reading

5

Measuring the Process and Solution

Measuring solution delivery

Cycle time

Lead time

WIP

Throughput

Blockers and bottlenecks

Measuring with cumulative flow diagrams

Looking at full stack telemetry

Application performance monitoring

Infrastructure monitoring

Network monitoring

Log management

Observability

Measuring the value proposition

The Innovation Accounting framework

Pirate metrics

The Google HEART framework

Fit for Purpose metrics

Net promoter score

Summary

Questions

Further reading

6

Recovering from Production Failures

Learning from failure

healthcare.gov (2013)

Atlassian cloud outage (2022)

Prevention – pulling the Andon Cord

SLIs, SLOs, and error budgets

Release engineering

Launch coordination engineering

Preparation – chaos engineering

Chaos engineering principles

Running experiments in chaos engineering

Problem-solving – enabling recovery

Incident management roles

The incident command post

The incident state document

Setting up clear handoffs

Perseverance – rolling back or fixing forward

Rolling back with blue/green deployment

Rolling back with feature flags

Rolling forward using the CI/CD pipeline

Summary

Questions

Further reading

Part 2: Implement – Moving Toward Value Streams

7

Mapping Your Value Streams

Aligning the organization’s mindset

Moving to SAFe

Moving to value streams

Setting the context for development value streams

Preparing for value stream identification

Creating the operational value stream

Finding solutions

Mapping the development value stream

Finding the process and workflow

Finding the people behind the process

Finding areas for improvement

Process step metrics

Value stream metrics

Identifying the future development value stream

The DevOps transformation canvas

Summary

Questions

Further reading

8

Measuring Value Stream Performance

Creating good measurements

Describing the goal or intended result

Understanding the method for measuring the goal

Selecting measures for each goal

Defining composite indices if needed

Defining targets or thresholds

Defining and documenting the selected measurements

DORA metrics

The DORA KPI metrics

DORA metric performance levels

Emerging trends from the State of DevOps report

Flow Framework® and Flow Metrics®

Flow Items

Flow Metrics

Measurements in SAFe

Measuring outcomes in SAFe

Measuring the Flow in SAFe

Measuring competency in SAFe

Summary

Questions

Further reading

9

Moving to the Future with Continuous Learning

Understanding continuous learning

Personal mastery

Mental models

Shared vision

Team learning

Systems thinking

Applying the Improvement Kata

Closing the Lean Improvement Cycle

Summary

Questions

Further reading

Part 3: Optimize – Enabling a Continuous Delivery Pipeline

10

Continuous Exploration and Finding New Features

Hypothesize customer value

Building with an MVP

Measuring with innovation accounting

Learning to pivot or persevere

The SAFe® Lean Startup Cycle

Collaboration and research

Collaboration with customers and stakeholders

Research activities

Architecting the solution

Architecting releasability

Designing security

Ensuring testability

Maintaining operations

Synthesizing the work

Completing the feature

Writing acceptance criteria using BDD

Prioritizing using WSJF

Preparing for PI planning

Summary

Questions

Further reading

11

Continuous Integration of Solution Development

Developing the solution

Breaking down into stories

Collaborative development

Building in quality by “shifting left”

Version control

Designing to the system

Building the solution package

Version control practices

Testing practices

Testing end to end

Equivalent test environments

Test automation

Management of test data

Service virtualization

Testing nonfunctional requirements

Moving to staging

Staging environments

Blue/green deployments

System demo

Summary

Questions

Further reading

12

Continuous Deployment to Production

Deploying to production

Increasing deployment frequency

Reducing risk

Verifying proper operation

Production testing

Test automation in production

Test data management

Testing NFRs

Monitoring the production environment

Full-stack telemetry

Visual displays

Federated monitoring

Responding and recovering when disaster strikes

Proactive detection

Cross-team collaboration

Chaos engineering

Session replay

Rollback and fix forward

Immutable architecture

Summary

Questions

Further reading

13

Releasing on Demand to Realize Value

Releasing value to customers

Decoupling releases by component architecture

Canary releases

Stabilizing and operating the solution

Site reliability engineering

Failover and disaster recovery

Continuous Security Monitoring

Architecting for operations

Measuring the value

Innovation accounting

Proving/disproving the benefit hypothesis

Learning from the outcomes

Pivot or persevere

Relentless improvement

Additional value stream mapping sessions

Summary

Questions

Further reading

14

Avoiding Pitfalls and Diving into the Future

Avoiding pitfalls

Understand the system

Start small

Start automating

Start automating the testing

Create environments, deployments, and monitoring

Measure, learn, and pivot

Emerging trends in DevOps

XOps

The Revolution model

Platform engineering

New technologies in DevOps

AIOps

GitOps

Frequently asked questions

What is DevOps?

What is the Scaled Agile Framework®?

Do I need to adopt SAFe to move to DevOps?

Is DevOps only for those companies that develop for cloud environments?

What’s the best tool for doing ___________?

How does DevSecOps fit into DevOps?

Summary

Further reading

Assessment Answers

Chapter 1 – Introducing SAFe® and DevOps

Chapter 2 – Culture of Shared Responsibility

Chapter 3 – Automation for Efficiency and Quality

Chapter 4 – Leveraging Lean Flow to Keep the Work Moving

Chapter 5 – Measuring the Process and Solution

Chapter 6 – Recovering from Production Failures

Chapter 7 – Mapping your Value Streams

Chapter 8 – Measuring Value Stream Performance

Chapter 9 – Moving to the Future with Continuous Learning

Chapter 10 – Continuous Exploration and Finding New Features

Chapter 11 – Continuous Integration of Solution Development

Chapter 12 – Continuous Deployment to Production

Chapter 13 – Releasing on Demand to Realize Value

Index

Other Books You May Enjoy

Preface

Both DevOps and the Scaled Agile Framework® (SAFe®) are about extending agility beyond the team.

DevOps looks to extend agility beyond the “traditional” boundary of development to include the deployment and release of the development to a production environment. This includes what to do if the new changes developed introduce failures in the production environment, activities typically done by operations teams.

SAFe® strives to extend an Agile mindset to products that require multiple teams to develop and maintain. In this paradigm, alignment and the ability to properly execute delivery and release are among the important factors for success. Both these factors are encouraged by DevOps. It is for this reason that to implement SAFe, you need to make sure that DevOps is implemented as well.

This book looks at the DevOps approach used in the SAFe. While there are practices specific to SAFe, many of the concepts, practices, and theories we introduce for DevOps are truly universal and are not limited to SAFe exclusively.

We hope you enjoy your learning journey!

Who this book is for

This book is intended for IT professionals such as DevOps and DevSecOps practitioners, SREs, and managers who are interested in implementing DevOps practices using the SAFe approach. Basic knowledge of DevOps and Agile Software Development Lifecycle (SDLC) and their methodologies will be helpful for using this book.

What this book covers

Chapter 1, Introducing SAFe® and DevOps, is a brief look at the history of how DevOps and SAFe came to be. We view the conditions that brought about Agile development, the evolution of Agile development to the DevOps movement, and the role SAFe can play in moving to DevOps.

Chapter 2, Culture of Shared Responsibility, covers the types of cultures that are present in organizations today, which are beneficial for DevOps. We also look at how to change your organization’s culture to one that is needed for DevOps.

Chapter 3, Automation for Efficiency and Quality, explores the automation and technology used by organizations to establish a Continuous Integration/Continuous Deployment (CI/CD) pipeline. We look at tools used for monitoring and measuring the pre-production and production environments. We then finish by discussing the teams responsible for setting it up.

Chapter 4, Leveraging Lean Flow to Keep the Work Moving, describes the principles and methods to accomplish a Lean Flow as part of SAFe. We examine the roles that the size of the work, the length of the backlog, how busy our workers are, and the differences between items of work play in the time it takes to complete the work.

Chapter 5, Measuring the Process and Solution, studies the potential measurements needed to ensure value, security, and reliability of the product under development. We look at the measurements that help identify whether teams have flow in their development. We explore monitoring and observability to find the metrics that ensure the solution is secure and reliable. Finally, we look to collecting metrics that assess the product’s end-user value.

Chapter 6, Recovering from Production Failures, outlines some methods to ensure reliability of the product in a customer-facing environment. We look at examples of famous production failures. We explore the discipline of Site Reliability Engineering (SRE), developed at Google to establish practices and ensure reliable environments. We finish our exploration by looking at Chaos Engineering, which strives to prepare for production failures by establishing experiments of failure in a production environment.

Chapter 7, Mapping Your Value Streams, takes a look at how to identify and establish value streams with a value stream identification workshop. We will explore how to prepare for the workshop and the mindset needed for moving to value streams. We then look at the steps needed to identify and map the Operational Value Stream. We finish by identifying and mapping the Development Value Stream.

Chapter 8, Measuring Value Stream Performance, delves into the metrics that are used to improve value streams. We explore the metrics that are organized by the DevOps Research and Assessment organization, known as the DORA metrics. We also explore Flow Metrics, a part of the Flow Framework created by Tasktop.

Chapter 9, Moving to the Future with Continuous Learning, examines how to become an organization that is continually learning. We explore the disciplines required for continuous learning as well as the practices from Lean thinking that encourage continuous learning such as the Improvement Kata.

Chapter 10, Continuous Exploration and Finding New Features, elaborates on the first phase of the Continuous Delivery Pipeline, Continuous Exploration. We explore the use of epics as hypotheses of potential customer value. We elaborate on the hypotheses by ensuring the architecture can allow for these new ideas and maintain the security and reliability of the product. We then look at decomposing the epics into features, ready for an Agile Release Train to develop.

Chapter 11, Continuous Integration of Solution Development, discusses the second phase of the Continuous Delivery Pipeline, Continuous Integration, including the start of the automation process. We look at the importance of testing, including the adoption of test-driven development and behavior-driven development. We explore the incorporation of automation in a CI/CD pipeline.

Chapter 12, Continuous Deployment to Production, provides an examination of the continued use of automation and practices in Continuous Deployment, the third stage of the Continuous Delivery Pipeline. We continue the exploration of automation through the CI/CD pipeline as it deploys to the production environment packages created in Continuous Integration. We also explore how testing continues in the production environment.

Chapter 13, Releasing on Demand to Realize Value, covers the last phase of the Continuous Delivery Pipeline, where customers receive new features through Release on Demand. We explore how teams continuously monitor the system to ensure the product is reliable and secure. We then look to see whether what is released really meets the customer’s needs.

Chapter 14, Avoiding Pitfalls and Diving into the Future, expounds on the new trends in DevOps in terms of process and technology and some tips and tricks to get you started on your journey. We start with helping you begin your journey to DevOps or SAFe.

To get the most out of this book

For this book, knowledge of the basics of the SDLC is beneficial. Knowledge of SAFe or DevOps is not required.

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/79pAN.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system.”

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Select System info from the Administration panel.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share your thoughts

Once you’ve read SAFe® for DevOps Practitioners, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere? Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application. 

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781803231426

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

1

Introducing SAFe® and DevOps

Developing products in many organizations—especially ones that work on software-based systems or complex systems involving both hardware and software and further enabled by networking technologies, known as cyber-physical systems—has changed over the past 10 to 20 years. Factors such as changes in technology, movement to geographically distributed or remote development, the push for faster time-to-market (TTM), understanding the customer needs, and pressures to reduce the occurrence and severity of production failures are opportunities, challenges, or a mixture of both that these organizations face.

To address these challenges and take advantage of the opportunities, mindsets derived from Lean manufacturing began to emerge and evolve. These mindsets, combined with practices from emerging frameworks, began to allow organizations to move past the challenges and improve business outcomes.

In this chapter, we’re going to highlight the historical challenges and popular mindsets and approaches that have allowed many organizations to overcome these obstacles. These challenges, approaches, and frameworks are described in the following topics:

Challenges organizations face in product developmentAn introduction to AgileAn introduction to DevOpsScaling DevOps with SAFe®

Challenges organizations face in product development

Product development today is enabled by a marriage of technology and society. Every product is a combination of hardware and software that is further enhanced by a connection to the internet. New product enhancements are only a software release away. We truly live in the age of software and digital.

It is against this backdrop that we look at challenges in product development and find that not only have these challenges not changed but the challenges also grow even more daunting thanks to the reliance on technology and software. Some of these classic challenges are presented here:

TTM pressuresUnderstanding customer wantsSecurity and complianceEnsuring qualityCompetition

These challenges don’t appear in isolation. Sometimes several challenges appear, or they appear all at once. Let’s examine how these challenges, alone or together, impede product development.

TTM pressures

TTM is the measure of the length of time it takes for a new product or new product feature to launch from an initial idea. It is usually seen as a measure of how innovative a company is.

A growing trend is that the length of TTM has decreased in recent years. Advances in technology have increased the pace of innovation. This increased pace of innovation has forced product development cycles to decrease from yearly cycles to 6-month or quarterly cycles. This trend will continue to happen and will force organizations to consider whether they can deliver features more frequently.

Understanding customer wants

Henry Ford is quoted as saying, “If I had asked people what they wanted, they would have said faster horses.” It often seems as if that statement is true today. Often, in the beginning, customers have no idea which features they want with a product. If a product requires long development cycles, the customer preferences may change, often to the point that what ultimately gets delivered is not what the customer needs or wants.

Often, what spurs a change in customer wants or requirements could be features offered by similar products provided by competitors. The added pressure from competitors provides a challenge: understand the desires of your customer and release a product or feature that meets those desires before your competition does.

Security and compliance

One challenge that organizations face doesn’t come from the marketplace. Products using software face growing threats from hackers and other bad actors that seek to take advantage of vulnerabilities in the software. Damages, if they are able to exploit these vulnerabilities, range from reputation to money in the form of ransomware payment or litigation.

In addition, because of the threats of these bad actors, regulations intended to ensure privacy and security have been enacted. Products may have to comply with region-specific (for example, the General Data Protection Regulation (GDPR)) or industry-specific (for example, the Health Insurance Portability and Accountability Act (HIPAA)) regulatory standards so that a customer’s confidential data is not exposed.

Ensuring quality

One thing that is important for organizations is maintaining quality while doing product development. Organizations that do not apply rigor in ensuring that products are developed with built-in quality soon find themselves open to other challenges. Rework means longer lead times and delays in bringing products to market. Customer experience with buggy software or a low-quality product may drive them to competitors’ products. Inadequate attention to quality may also allow security vulnerabilities to go unnoticed, allowing exploits to occur.

Vigilance toward maintaining quality during product development is ideally done by creating, setting up, and executing tests at all levels throughout development. Testing could even be done in pre-production and production environments and automated as much as possible. An approval/inspection-based approach only defers the discovery of problems until it may be too late or too costly to fix them.

Competition

Some of the challenges previously mentioned talked about the part that competition plays. The truth of the matter is that your competitors face the same challenges that you do. If your competition has found a way to master these challenges, they have a distinct advantage in the market.

But it’s important to remember that this race isn’t about being first. The challenge is to be first with the product or feature and be able to convey why this lines up with customer desires. A famous example comes from Apple. Apple was a couple of years behind other competitors in the marketplace for digital music players when it released the iPod. What made the iPod a runaway product sensation was the marketing that touted the memory size not in terms of megabytes (MB), but in terms of the number of songs. This simple message connected with the marketplace, beyond technology aficionados and music diehards, to even casual music listeners.

The incredibly successful launch of the iPod drove Apple on a path of innovation that catapulted it to its current place as one of several technology giants. The makers of the first commercial MPEG-1 Audio Layer 3 (MP3) player no longer provide support for their product.

Meeting the challenges

These challenges have plagued product development since the early history of man. The exact form of challenge, however, changes with every generation and with every shift in technology.

TTM cycles will always drive when to release products; however, with the aid of technology, these cycles are shrinking. A customer’s requirements may always remain a mystery, often even to the customer. Competition changes will ebb and flow, with the emergence of new disruptors and the disappearance of the also-rans.

As these ever-present challenges take on new forms, those organizations that have mastered these challenges do so through new mindsets and practices revolving around three areas: people, process, and technology.

A focus on people involves looking at the mindset, values, and principles that people hold in common to form an organizational culture. This culture is the most important counterpoint to challenges because it informs everybody on how they will meet these challenges.

With the culture established, a focus on the process implements practices that are modeled on the right mindset, values, and principles. Successful application of the practices promotes a feedback loop, encouraging the culture and reinforcing the mindset, values, and principles.

Finally, tools aid the process. They allow practices to be repeatable and automatic, strengthening the process, and allowing the application of the process to be successful.

The remainder of this book will highlight the combination of people, processes, and tools that have helped organizations meet these challenges seen in modern product development. These combinations are set up as frameworks, meant to be flexible enough to apply to different organizations in different industries. These combinations started with software development but warrant a look as the creation of software is prevalent in everyorganization today.

We begin our examination with a look at Agile development or the transition from large-scale delivery of software to incremental delivery of software in short design cycles.

An introduction to Agile

To understand the context in which these challenges lie, it is important to understand the dominant product development process, known colloquially as Waterfall. Waterfall was used for many years to develop cathedrals and rocket ships, but when the process was used for developing software, the strains of it were beginning to show, highlighting inadequacies against the challenges of meeting a shrinking TTM and satisfying customer needs. Something had to be done.

Next, let’s look at the emergence of Agile methods from the initial attempts to incorporate Lean thinking into software development, to the creation of the Agile Manifesto and the emergence of Agile practices and frameworks.

The rise and fall of Waterfall

The method known as Waterfall had its origins in traditional product development. Workers would divide up the work into specific phases, not moving to the next phase until the current phase was completed.

In 1970, Winston W. Royce first proposed a diagram and model for this method when product development moved to software, as depicted here:

Figure 1.1 – Waterfall diagram

Although Royce never advocated this approach and actually preferred a more incremental approach to development, his diagram caught on and many in the industry called the approach Waterfall because the arrows from one phase to the next resembled waterfalls.

In software development, this approach began to exhibit drawbacks. If a requirement, problem, or constraint appeared in the latter phases, the additional work drove the process backward, requiring an enormous amount of rework. Many times, the end customer would not have ideas on requirements early on, leading to rework or a final product that failed to meet customer expectations.

The delays introduced by rework also put pressure on fixed-time projects. To meet the deadlines, some of the later phases (often testing) would be curtailed or eliminated to deliver a product. With the lack of testing, errors or bugs would remain undiscovered until the product was released, resulting in low-quality software and low customer value.

TTM pressures on product development cycles reduced the time available to create new software or update existing software. The model was falling apart, but what could be done differently?

The emergence of Agile

In the early 21st century, other methods such as Extreme Programming (XP), Scrum, and Crystal began to emerge. These methods advocated incremental delivery, where a bit of the intended functionality would go through all stages (requirements, design, coding, and testing) in small design cycles, often no longer than a month. At the end of each design cycle, teams would solicit customer feedback, often incorporating that feedback into the next design cycle.

A representation of incremental delivery, containing short design cycles and delivery of packages of value, is shown in the following diagram:

Figure 1.2 – Incremental (Agile) development diagram

Between February 11 and 13, 2001, a group of software development experts, some of whom created XP, Scrum, and the Crystal method met in Snowbird, Utah to ponder which alternatives existed. What resulted from this weekend was a manifesto for Agile software development, simply called the Agile Manifesto: agilemanifesto.org.

The Agile Manifesto contains a set of values and a list of principles, but it must be noted that the authors of the manifesto talk about the values as a set of preferences. It is possible for Agile teams to have processes, tools, documentation, contracts, and plans. It’s only when those items interfere with the items on the left half of each statement of value that the team should re-evaluate the process, tool, or contract, and plan and adjust.

The value set shows what is important, stating the following:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The 12 principles of the Agile Manifesto elaborate and provide context to these values, stating the following:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for a shorter timescale.Business people and developers must work together daily throughout the project.Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.Working software is the primary measure of progress.Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.Continuous attention to technical excellence and good design enhances agility.Simplicity—the art of maximizing the amount of work not done—is essential.The best architectures, requirements, and designs emerge from self-organizing teams.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

The addition of Lean

Around the same time, others were looking at other ways of developing software with shorter TTM. They looked at applying principles from Lean manufacturing.

Lean manufacturing looks to apply practices to reduce waste. The methods were invented by Taiichi Ohno and used to create the Toyota Production System (TPS). Along with the removal of waste, Lean manufacturing looks to build in quality and strive for Kaizen, or continuous improvement.

The application of the principles found in Lean manufacturing was used by Mary and Tom Poppendieck to describe Lean software development in their book, Lean Software Development: An Agile Toolkit. These principles are summarized as follows:

Removing wasteEmphasizing feedback and learningWaiting until the last possible moment for decisionsFrequently deliveringMaking sure the team is empoweredMeeting users’ perceptions and expectationsEmployingsystemic thinking

In addition to the work by the Poppendiecks, David J. Anderson adapted Kanban, another tool from the TPS for software development at Microsoft. This adaptation of Kanban was modified as a framework of practices suitable for software development. Kanban soon rose in popularity not only as an alternative to Scrum or XP—many of the practices are used in conjunction with Scrum or XP to facilitate the execution of tasks.

Software development teams did see that the change to Agile development was producing results and overcoming challenges in product development, but the results were only seen in development and not in the overall organization. Clearly, changes needed to be made in other parts of the organization. Let’s examine the changes that development and operations made together in the DevOps movement.

An introduction to DevOps

As development teams began to adopt Agile methods and deliver software features incrementally, they faced a challenge in delivering value from the outside. Operations teams, the ones that maintain the development and production platforms where code executes, often do not release new packages from the development teams as they emerge. Rather, operations teams insist on collecting features and deploying them in specified release windows to minimize the risk that a single new change would bring down the production environment. But instead of minimizing the risk, it compounds the risk by straining the time allowed for the release windows, with mismatched configurations between development and production environments and untracked manual intervention in production environments. Bundling releases into release packages also moved delivery away from small increments to larger monoliths that may diminish customer value.

At this point, it becomes necessary to view the perspective of a typical operations team. Its job is to ensure that the organization’s production environment, the one that may be producing an organization’s revenue, is as operational as possible. Any change to that environment, even ones for new features, is seen as a risk to that stability.

In 2009, John Allspaw and Paul Hammond gave a talk titled 10+ Deploys per Day: Dev and Ops Cooperation at Flickr during the O’Reilly Velocity conference. In this talk, they outlined the methods they used to get an unheard-of 10 deployments a day. These methods still form the basic pillars of development-operations (DevOps) today. We will talk about the incorporation of the following:

Tools and technologyPeople and process

DevOps tools and technology

During the talk, Allspaw and Hammond identified technologies and what they did with them to align both the development and operations teams. But not only were the tools noteworthy—it was also how the teams used the tools together in a collaborative way.

These technologies includedthe following:

Automated infrastructureCommon version controlOne-button builds/deploymentFeature flagsShared metricsInstant messaging (IM) robots on shared channels

The use of tools and technologies continues to play a key role in DevOps today. We’ll explore how these tools enabled a quick release process into production and quick resolutions if problems happened in production.

Automated infrastructure

As software became more complex, the environments to execute them became more complex as well. Operations teams were faced with configuring servers that grew over time from tens to hundreds or thousands. The task required automation. Configuration management (CM) tools such as Chef and Puppet began to emerge.

CM allowed operations teams to standardize environmental configurations such as versions of an operating system, software applications, and code libraries. It allowed them to easily find those machines that did not have the standard configuration and correct them. As servers moved from physical hardware to virtual machines (VMs), CM allowed for the creation and maintenance of standard images that were differentiated by what role they played in the server environment.

CM also helps developers. Automated CM can make every server provisioned uniform between the multiple environments an organization has for software development and release. Consistency between development, staging, and production environments eliminated a problem colloquially known as “works on my machine” where even subtle differences between development, testing, and production environments led to the possibility that code would work on development but would fail in production.

Common version control

A version control systems (VCS) such as Git are is a popular tool in software development for managing source code. With version control, developers can make changes to source code on a private sandbox called a branch. When ready to share source code changes, developers can merge their changes back to the main branch of the source code repository, making sure those changes do not conflict with other changes from other developers. The VCS records all changes that happen to the source code, including those from other branches. Because the version control contains a comprehensive history of the source code’s evolution, it is possible to find a version of the source code from a specific point in time.

Soon, version control became important for storing more than source code. Testing tools required test scripts and test data that could be version-controlled as the tests changed. CM tools used text files to define ideal configurations for servers and VMs. Operations could also have scripts that would perform the configuration tasks automatically. All these could be version-controlled to record the evolution of the environment.

Making sure that both development and operations were not only using version control but using the same version control tool became important. Having all versions of the artifacts (code, tests, configuration files, scripting, and so on) developed allowed for an easy understanding of which version of which artifact was used in a release using tags or labels. A common version control tool ensured that one side (development or operations) was not denied access to view this understanding if a problem occurred.

One-button builds/deployment

Building a release from source code could be a time-intensive task for developers. They would have to pull their changes from version control, add in the required code libraries, compile the changes together into a build package, and upload that build package into an environment to test whether the changes worked. A smart developer would often automate these tasks by setting up build scripts, but could this process be easier?

Continuous integration (CI) tools such as Hudson (later named Jenkins) emerged that allowed developers to go through all the steps of a build process and execute build scripts by just pushing a button. A page could easily show not only build success but if a build failure occurred, it could also show in which step of the process that failure occurred. Automating the build through CI also made sure that the build process between developers was consistent by ensuring that all steps were followed and a step was not omitted.

Could that same consistency be applied to operations teams when they deployed releases? Continuous deployment (CD) tools take a build package and run tests against it in the current level environment, and if they pass, apply it to a specific environment. These tools can also connect with CM tools to create instances of the environment with the new build package “on the fly.” Any new deployment would be recorded showing who pushed the button, when it was pushed, and which artifacts and artifact changes were deployed to a particular environment.

Can a tool used for CI also be used for CD? This is a common way of implementing the same automation that can be used by both development and operations.

Feature flags

Flickr, the company Allspaw and Hammond worked for, was a photo-sharing and rating website. Its software differed from traditional desktop-based software because it was concerned with supporting only one release: the release on its production environment. The company did not have to worry about supporting multiple versions of the released software. This allowed it to have the main branch of its version code repository as the specific version it would support and examine if problems arose.

To handle problems introduced by buggy new features, it set up conditional branches in code called feature flags. Based on the value of a variable, the code for a new feature would be visible or invisible. The feature flag acts as an on-off switch, indicating which code is released, and thus visible, as illustrated in the following diagram:

Figure 1.3 – Illustration of a feature flag

Having feature flags in the code allowed for deployments to production environments to be more flexible. Newly deployed code could be present in production but not be seen until thoroughly tested. “Dark launches” could result from this where operations could evaluate the performance of the new feature with the existing software against production data and load. Test customers could evaluate new features in a subset of the production environment where those feature flags were activated. Finally, the behavior of the environment could be quickly changed by changing the value of the feature flag(s), propagating the change through CI and CD, and allowing the change in production. This method of recovery is called roll-forward or fix-forward.

Shared metrics

To ensure stability, operations collects the performance of every environment and reviews the metrics such data collection produces. These metrics can be displayed as specific views on a dashboard. Dashboard views not only give an indication of performance now but also allow operations to identify trends and take corrective action.

Flickr made these dashboards not only for operations but also for developers as well. Developers could see the performance of the application in the context of the environment. Allowing developers access to this contextual data ensured they could see the effects of their new features and whether those features provided value.

Shared metrics also allowed for adaptive feedback loops to occur in the environment. The performance metric could be evaluated by the application, and the evaluation could generate a notification that additional resources would be required.

IM robots on shared channels

Communication between development and operations was paramount. The use of standard email was discouraged in favor of instant messaging and chat mechanisms that allowed for ongoing real-time communications of the systems between development and operations. Notifications about development events such as build status and operations events—for example, deployment status, system alerts, and monitoring messages—could be inserted into the channel by chat robots to give both development and operations personnel notice of specific events that occurred. Chats could also be searchable to provide a timeline of events for troubleshooting when problems arose.

DevOps people and processes

It’s worth noting that other organizations besides Flickr were using the same tools and technologies. What made a difference to Flickr was how the people from distinct groups worked together in shared processes to leverage the tools and technologies. These people and processes formed an organizational culture that allowed them to deploy rapidly.

Allspaw and Hammond made note of specific touchpoints during the talk. These included the following:

RespectTrustLearning from failureNo “fingerpointing”

Having these touchpoints form that organizational culture was just as important as the application of tools and technology.

Respect

It was important that people from different groups within Flickr operated from a place of respect. That respect meant moving past stereotypes about developers or operations people and looking at common goals.

The respect extended to other people’s expertise, opinions, and recommendations. There’s a fundamental understanding that different people have different backgrounds and experiences that shape their opinions and responsibilities. A key part of problem-solving is to listen to those different perspectives that may give different and better solutions to a problem. Understanding the differing responsibilities allows you to understand another person’s perspective.

Another important extension of that respect that Allspaw and Hammond highlighted was not just giving a response but understanding the reasons and motivations others would have for solving these problems. It’s not enough to answer a question—you should also understand why the question is being asked before giving an answer. Allowing everyone to understand the context allows the group to create unique solutions to problems.

To have this respect shown, there must be transparency. Hiding information between groups does not allow for the free exchange required to create innovative solutions to problems. Also, at some point, whatever you hide will be found out, creating conflict.

An important part of respect is empathy. Knowing what effects to operations there may be from a code change is important before having that discussion with operations personnel. This allows room for any hidden assumptions to be unearthed and for creative solutions to flow.

Trust

Armed with transparency and empathy to build respect, people from one group need to trust the other groups. If a development person has that understanding of what impact to operations their feature will have, it is then incumbent on them to have that conversation with operations personnel to confirm those impacts or at least make them aware of potential impacts.

Conversely, operations people need to have developers involved to discuss together what effects any infrastructure changes will have on current or future features.

Ultimately, this comes to an understanding that everyone should trust that everyone else is doing their best for the good of the business.

Examples of the manifestation of trust are not only the sharing of data through version control, IM chat mechanisms, and metrics/dashboards, but also lie in the construction of shared runbooks and escalation plans that are created when readying a new release. The construction of these plans allows discussion to flow on risks, impacts, and responsibilities.

Finally, including mechanisms to allow the other group to operate is an important part of leveraging that trust. For developers, that meant setting up controls in software for operations people to manipulate. For operations people, that meant allowing appropriate access to the production environment so that developers could directly see the effects new changes had in the production environment.

Learning from failure

Failures will happen. How an organization deals with that failure is the difference between a successful organization and an organization that will not remain operational for long. Successful organizations focus more on how to respond to failures, foreseen and unforeseen, more than expending energy to prevent the next failure.

Preparation for responding to failure is a responsibility that falls on everyone. Each person, developer, or operations team member must know how they would react in an emergency. Ways of practicing emergencies include having junior employees “shadow” senior employees to see how they would react. At Flickr, those junior employees were put in the same “what-if” scenario as the exact outage occurred to see which solutions they could develop.

No “fingerpointing”

At Flickr, they discovered that when people were afraid of getting blamed for production failures, the first reaction would be to try to individually fix the problem, find who to blame, or conceal the evidence. That always led to a delay in finding a solution to the problem. They instituted a no fngerpointing rule.

The results were dramatic. Resolution times to fix problems rapidly decreased. The focus then shifted from who caused the problem to what the solution was.

The DevOps movement begins

The response to Allspaw and Hammond’s talk was swift and impactful. People started to look at ways to better align development and operations. Patrick Debois, having missed the O’Reilly Velocity conference where Allspaw and Hammond gave their talk, organized the first DevOpsDays conference in Ghent, Belgium to keep the conversation happening. The conversation continues and has become a movement through successive DevOpsDays conferences, messages on Twitter highlighted with “#DevOps,” blogs, and meetups.

The response drives the creation of new tools for version control, change management, CI, CD, CM, automated testing, and artifact management. Technology evolves from VMs to containers to reduce the differences between development, test, staging, and production environments.

The DevOps movement continues to grow. As with the adoption of Agile, DevOps is open to all and decentralized. There is no one way to “do DevOps.” DevOps can be applied to environments of any type such as legacy mainframes, physical hardware, cloud environments, containers, and Kubernetes clusters. DevOps works in any industry, whether finance, manufacturing, or aerospace.

Since the original talk by Allspaw and Hammond, organizations that have adopted DevOps principles and practices have seen incredible gains in deployment frequency, while also being able to reduce the probability of production failures and recovery times when an errant production failure does occur. According to the 2021 State of DevOps report, “elite” organizations can release on-demand, which may happen multiple times a day. This is 973 times more frequent than organizations rated as “low.” Elite organizations are also a third less likely to release failures and are 6,570 times faster at recovering from a failure should it occur.

Some organizations are medium-to-large-sized companies working in industries such as finance, aerospace, manufacturing, and insurance. The products they create may be systems of systems of systems. They may not know how to incorporate Agile and DevOps approaches. For these companies, one framework to consider is the Scaled Agile Framework® (SAFe®).

Scaling DevOps with SAFe®

SAFe® is one of the more popular adopted platforms used to incorporate the Agile mindset and practices according to recent