27,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Achieve streamlined, rapid production with enterprise-level DevOps Awarded DevOps 2017 Book of the Year, The DevOps Adoption Playbook provides practical, actionable, real-world guidance on implementing DevOps at enterprise scale. Author Sanjeev Sharma heads the DevOps practice for IBM; in this book, he provides unique guidance and insight on implementing DevOps at large organizations. Most DevOps literature is aimed at startups, but enterprises have unique needs, capabilities, limitations, and challenges; "DevOps for startups" doesn't work at this scale, but the DevOps paradigm can revolutionize enterprise IT. Deliver high-value applications and systems with velocity and agility by adopting the necessary practices, automation tools, and organizational and cultural changes that lead to innovation through rapid experimentation. Speed is an advantage in the face of competition, but it must never come at the expense of quality; DevOps allows your organization to keep both by intersecting development, quality assurance, and operations. Enterprise-level DevOps comes with its own set of challenges, but this book shows you just how easily they are overcome. With a slight shift in perspective, your organization can stay ahead of the competition while keeping costs, risks, and quality under control. * Grasp the full extent of the DevOps impact on IT organizations * Achieve high-value innovation and optimization with low cost and risk * Exceed traditional business goals with higher product release efficiency * Implement DevOps in large-scale enterprise IT environments DevOps has been one of IT's hottest trends for the past decade, and plenty of success stories testify to its effectiveness in organizations of any size, industry, or level of IT maturity, all around the world. The DevOps Adoption Playbook shows you how to get your organization on board so you can slip production into the fast lane and innovate your way to the top.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 620
Veröffentlichungsjahr: 2017
Ähnliche
The DevOps Adoption Playbook
A Guide to Adopting DevOps in a Multi-Speed IT Enterprise
Sanjeev Sharma
The DevOps Adoption Playbook: A Guide to Adopting DevOps in a Multi-Speed IT Enterprise
Published by John Wiley & Sons, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com
Copyright © 2017 by John Wiley & Sons, Inc., Indianapolis, Indiana
Published simultaneously in Canada
ISBN: 978-1-119-30874-4
ISBN: 978-1-119-31052-5 (ebk)
ISBN: 978-1-119-31076-1 (ebk)
Manufactured in the United States of America
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or website is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make. Further, readers should be aware that Internet websites listed in this work may have changed or disappeared between when this work was written and when it is read.
For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.
Library of Congress Control Number: 2016962068
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. IBM, the IBM Press logo, UrbanCode, uDeploy, System z, Rational, IBM Watson, WebSphere, Bluemix, InfoSphere, Optim, PureApplication, DB2, SoftLayer, and Blue Box are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. A current list of IBM trademarks is available on the web at “copyright and trademark information” as www.ibm.com/legal/copytrade.shtml. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.
To my wife Ritika, for always motivating me to do more, be more, and never be satisfied with the status quo. And my children, Saransh and Shreya, for being the ones I am motivated to do and be more for.
About the Author
Sanjeev Sharma is an internationally known DevOps and cloud transformation thought leader, technology executive, and published author. Sanjeev’s industry experience includes tenures as CTO and Worldwide Technical Sales Leader, Acquisition Integration Technical Leader, and IT Architect. As an IBM Distinguished Engineer, Sanjeev is recognized at the highest levels of the exclusive core of technical leaders at IBM.
Sanjeev provides core leadership to drive the adoption of cutting-edge solutions, architectures, and strategies for DevOps and the cloud. His experience as the Global CTO for DevOps Technical Sales at IBM, combined with his deep insight and ability to understand both business and IT needs, drives a unique perspective for any business. This perspective allows Sanjeev to advise and mentor C-level and senior technical executives on achieving DevOps and cloud transformations, across industries and geographies.
Sanjeev is a frequent speaker on the international tech scene, as a cloud and DevOps expert. He regularly publishes articles, blog posts, and videos for leading tech publications and his own blog, at http://bit.ly/sdarchitect. Sanjeev tweets as @sd_architect.
About the Technical Editor
Lee Reid has more than 30 years’ experience in software engineering, architecture, product development, technology innovation, and team leadership in both manufacturing and information technology domains. Lee is an engineering graduate of General Motors Institute (BME) and the University of Michigan (MSE) and holds four U.S. patents. He has recently transitioned into higher education, where he leads IT and is introducing Lean and DevOps practices at St. Norbert College.
Credits
Project EditorAdaobi Obi Tulton
Technical EditorLee Reid
Production EditorRebecca Anderson
Copy EditorMarylouise Wiack
Production ManagerKatie Wisor
Manager of Content Development & AssemblyMary Beth Wakefield
Marketing ManagerLorna Mein
Professional Technology & Strategy DirectorBarry Pruett
Business ManagerAmy Knies
Executive EditorJim Minatel
Project Coordinator, CoverBrent Savage
ProofreaderKim Wimpsett
IndexerJ&J Indexing
Cover DesignerWiley
Cover Image©traffic_analyzer/Getty Images
Acknowledgments
This book is an effort to put to paper countless conversations and (sometimes heated) discussions and debates on DevOps and IT optimization and innovation that I have had with my customers, co-workers, and peers in the DevOps community. Through these conversations and discussions, dozens of people have contributed to this book, not to mention all those whose blogs, articles, books, webinars, videos, meetings, and presentations I learned from.
The key contributors include my fellow DevOps subject matter experts and technology thought leaders at IBM. These include (in alphabetical order by first name):
Al Wagner
Albert Ho
Alex Abi Khaled
Ana Lopez-Mancisidor
Andy Moynahan
Ann Marie Somerville
Anshu Kak
Anujay Bidla
Ava Hakim
Bala Rajaraman
Bernie Coyne
Bill Higgins
Bob Bogan
Brian Naylor
Chris Lazzaro
Chris Lucca
C. J. Paul
Claudette Hickey
Cliff Utstein
Dan Berg
David Curbishley
David Leigh
David Ziskind
Dibbe Edwards
Eric Minick
Erik Anderson
Greg Wunderle
Hayden Lindsey
Helen Dai
Jagan Karuturi
James Pierce
Jeff Crume
Jim Fieseler
Jim Moffitt
John Lanuti
John Wiegand
Kay Johnson
Kedar Walimbe
Kristof Kloeckner
Kyle Brown
Leigh Williamson
Mahendra Pingale
Maneesh Goyal
Mark Borowski
Mark Meinschein
Mark Roberts
Mark Tomlinson
Meenagi Venkat
Michael Elder
Michael Samano
Mike McNamee
Mustafa Kapadia
Paul Bahrs
Paul Meharg
Peter Eeles
Peter Spung
Randy Newell
René Bostic
Rick Weaver
Rob Cuddy
Robbie Minshall
Roger Snook
Rosalind Radcliffe
Sal Vella
Saleem Padani
Steve Abrams
Steve Kagan
Steven Boone
Sudhakar Frederick
Swati Moran
Tony Doyle
Tim Hahn
Tim Pouyer
Varban Vassilev
Wendy Toh
Some contributors who were formerly at IBM include the following:
Alan Sanie
Ashok Reddy
Bowman Hall
David Grimm
David Myers
Jan Svoboda
Mike Lundblad
Murray Cantor
Steven Pogue
Walker Royce
Several key customers, business partners, and experts also contributed, as real-life examples of leaders who led DevOps transformations at their own companies and organizations. Their stories from the trenches are the best sources of lessons learned. In many cases, they were at the other end of the conversations that led to the lessons learned and practices documented in this book. Because I met most of these people in a professional capacity as an IBM employee, I cannot list them all here. I will list the few who also co-presented at conferences, meetings, and webinars with me, or co-authored articles or blogs with me. They include the following (along with their current employers):
Alan Shimel,
DevOps.com
Antony Morris, Monitise
Ben Chodroff, CloudOne
Brad Schick, Skytap
Carmen DeArdo, Nationwide Insurance
Chris Lepre, Wells Fargo
Gareth Evans, Monitise
James Governor, RedMonk
Jayne Groll, DevOps Institute
John Comas, NBCUniversal Media
John Kosco, Blue Agility
J. P. Morgenthal, CSC
Mark Howell, Lloyds Banking Group
Tapabrata “Topo” Pal, Capital One
I would be remiss to not acknowledge separately Gene Kim, the über-guru of DevOps, with his contributions through his books and his DevOps Enterprise Summit Conference. I personally had multiple opportunities to talk to him one-on-one, including a video interview I recorded in 2014 (https://youtube/6QK2Mt-KPo4).
I would also like to give a special thanks to Lee Reid. I have worked with Lee for more than a decade. He was also my “partner in crime,” leading the Worldwide DevOps architect team at IBM for two years. We developed the DevOps Value Stream Mapping workshop techniques together, and I personally bounced tons of ideas off of him. It was only fitting that I had the opportunity to leverage Lee’s talents and mind, despite his having left IBM for St. Norbert College, by having him be the technical editor of this book. There is no way the book would have made it to its final refined and well-structured form without Lee’s insights, critique, and feedback.
Lastly, I would like to thank the wonderful editing staff at Wiley: Adaobi Obi Tulton, whose skills certainly live up to her Jedi-sounding name, and Marylouise Wiack for her complete mastery of language and prose (and yes, punctuation—my nemesis). The book is light-years ahead of what I originally put to paper because of their hard work and painstaking correction of my meek attempt to put words together into coherent sentences.
CONTENTS
Introduction
A Playbook for Adopting DevOps at Scale
Disrupt or Be Disrupted
Defining DevOps
Who Is This Book For?
The Sports Analogies
Companion Website
Notes
Chapter 1 DevOps: An Overview
DevOps: Origins
DevOps: Roots
DevOps: Practices
DevOps: Culture
Summary
Notes
Chapter 2 Adopting DevOps
Developing the Playbook
Summary
Chapter 3 Developing a Business Case for a DevOps Transformation
Developing the Business Case
Completing the Business Model Canvas
Customer Segments
Value Propositions
Channels
Customer Relationships
Revenue Streams
Key Resources
Key Activities
Key Partnerships
Cost Structures
Summary
Chapter 4 DevOps Plays for Optimizing the Delivery Pipeline
DevOps as an Optimization Exercise
Core Themes
Note
Chapter 5 DevOps Plays for Driving Innovation
Optimize to Innovate
The Uber Syndrome
Innovation and the Role of Technology
Core Themes
Play: Build a DevOps Platform
Play: Deliver Microservices Architectures
Play: Develop an API Economy
Play: Organizing for Innovation
Summary
Notes
Chapter 6 Scaling DevOps for the Enterprise
Core Themes
Play: DevOps Center of Competency
Play: Developing Culture of Innovation at Scale
Play: Developing a Culture of Continuous Improvement
Play: Team Models for DevOps
Play: Standardization of Tools and Processes
Play: Security Considerations for DevOps
Play: DevOps and Outsourcing
Summary
Notes
Chapter 7 Leading DevOps Adoption in the Enterprise
Play: DevOps as a Transformation Exercise
Play: Developing a Culture of Collaboration and Trust
Play: DevOps Thinking for the Line of Business
Play: Starting with Pilot Projects
Play: Rearing Unicorns on an Aircraft Carrier
Appendix Case Study: Example DevOps Adoption Roadmap
Organization Background
Roadmap Structure
Adoption Roadmap
Notes
Bibliography
Introduction
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Appendix
EULA
List of Illustrations
Chapter 1
Figure 1-1
Delivery pipeline bottlenecks
Figure 1-2
Continuous integration
Figure 1-3
A delivery pipeline
Figure 1-4
Continuous delivery
Figure 1-5
Example of test virtualization
Chapter 2
Figure 2-1:
A simple value stream map
Figure 2-2:
Value stream mapping a delivery pipeline
Figure 2-3:
Productivity dip
Chapter 3
Figure 3-1:
Business Model Canvas
Figure 3-2:
Building a DevOps adoption business case
Chapter 4
Figure 4-1
Reduced cycle time drives faster feedback.
Figure 4-2
Addressing the impedance mismatch caused by “water-Scrum-fall”
Figure 4-3
Achieving agility across the delivery pipeline
Figure 4-4
Integrated delivery pipeline
Figure 4-5
End-to-end traceability of a defect
Figure 4-6
Multi-Speed IT with multiple delivery pipelines
Figure 4-7
CI across components and applications
Figure 4-8
Continuous integration drives the delivery pipeline.
Figure 4-9
Deploying application components
Figure 4-10
Deployment process, with rollback (in IBM UrbanCode Deploy)
Figure 4-11
Deployment automation “blueprints”
Figure 4-12
OpenStack Heat pattern (in IBM UrbanCode Deploy’s Blueprint Designer)
Figure 4-13
The full stack
Figure 4-14
Continuous integration to continuous delivery
Figure 4-15
Test service virtualization
Figure 4-16
Hygieia, from Capital One
Figure 4-17
Hygieia—pipeline view
Figure 4-18
Release management with multiple delivery pipelines in IBM UrbanCode Release
Figure 4-19
LinkedIn mobile app architecture
Figure 4-20
Application delivery handoff to engineering
Figure 4-21
A big data and analytics reference architecture
Figure 4-22
Analytics Solutions Unified Method (ASUM) phases
Chapter 5
Figure 5-1
Redbox grocery kiosk in Washington, DC, 2002 (Imgur, 2013)
Figure 5-2
Multi-Speed IT touchpoints
Figure 5-3
Netflix Simian Army (image source: github.com/Netflix)
Figure 5-4
Integrated delivery pipeline
Figure 5-5
Cloud consumption models
Figure 5-6
IaaS versus PaaS
Figure 5-7
IaaS capability stack
Figure 5-8
OpenStack Heat supporting multiple clouds
Figure 5-9
DevOps services
Figure 5-10
Bluemix DevOps delivery pipeline (IBM)
Figure 5-11
Scaling with microservices
Figure 5-12
Re-architecting to a microservices architecture
Figure 5-13
Repackaging Java apps into microservices
Figure 5-14
ESPN public APIs (ESPN Developer Center, 2015)
Figure 5-15
OpenStack networking API (OpenStack.org, 2016)
Chapter 6
Figure 6-1
Squads, tribes, chapters, and guilds (Image by Shreya Sharma)
Figure 6-2
IBM open toolchain
Figure 6-3
Building a DevOps tool chain using services on IBM Bluemix
Figure 6-4
API security with API management tools
Figure 6-5
Software supply chain
Guide
Cover
Table of Contents
Chapter
Pages
iii
v
vii
ix
xi
xii
xiii
xxiii
xxiv
xxv
xxvi
xxvii
xxviii
xxix
xxx
xxxi
xxxii
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
Introduction
WHAT’S IN YOUR PLAYBOOK?
In April 2016, the Villanova Wildcats played the North Carolina Tar Heels in the 2016 NCAA Basketball National Championship Game. The game was the greatest ever, and it all came down to one last play, with just 4.7 seconds left on the clock. Joel Berry II hit a three-pointer to tie the game at 74 apiece, and Villanova Coach Jay Wright called his final timeout. In the NCAA, you have to go down the entire length of the court after a timeout. Immediately, Kris Jenkins of Villanova inbounded the ball to point guard Ryan Arcidiacono. Arcidiacono dribbled down the court past Berry, but it was the design of the play on both sides that made for the great ending. UNC played a 1-3-1 man-on-man press to Arcidiacono, to hopefully force a turnover, but if Arcidiacono got past Berry, they had Justin Jackson, Isaiah Hicks, and Brice Johnson, all who could stop the three-point shot. Villanova had designed a play to make sure that Arcidiacono got the ball off the inbound in a position in which he could get past half-court and find someone on the three-point line. Arcidiacono got past Berry, UNC collapsed, and Arcidiacono popped it back to Jenkins on the three-point line to get an almost-uncontested three-pointer to win the championship, and boy did it pay off. #Win!
—By Saransh Sharma (Sharma, 2016)
A Playbook for Adopting DevOps at Scale
Teams that excel do so not just because they have the best members, the best tools, the best training, the best processes, or the best leaders and coaches. They excel because they, as a team, have all of the above but also know what to do when they face various situations and challenges. They have a playbook of potential solutions (plays) for a variety of scenarios.
When faced with a unique situation or challenge, the players and coaches come together as a team to pick the right play from the playbook, and then, most importantly, they execute it. My alma mater, Villanova University, won the national championship when it came down to the final play, with seconds on the clock, because they had plays they had practiced before. They read the situation, picked the right play, and executed with precision to win. Had they not had the play that would catch North Carolina off guard, there may have been a different outcome.
In the same way, IT organizations need plays to execute. For day-to-day application delivery and operations, these so-called plays are captured in their development, delivery, and operational processes. IT organizations that succeed have good processes and execute them with excellence. However, transforming IT organizations is another story. Most organizations struggle with transformations, not having well-defined, winning plays that can overcome cultural and organizational inertia. This book captures a set of proven, repeatable plays for adopting DevOps at the enterprise scale and for transforming a large, complex, distributed IT organization to adopt DevOps.
These plays come from my experience over several years in the trenches as I helped dozens of organizations, of myriad sizes and maturity levels, in a variety of industries and geographical locations, to adopt DevOps. Since the early days of DevOps, as the Worldwide CTO of DevOps Technical Sales and Adoption at IBM, I have had a front-row seat to see the evolution and maturation of DevOps from a set of practices pioneered by startups to a cultural and technological transformation effort in large enterprises. I was a pioneer and thought leader for DevOps at IBM, and I became the face of DevOps to IBM’s clients. This book distills patterns of success that I have observed among hundreds of clients working, struggling, and succeeding at adopting DevOps at organization or enterprise-wide scale.
Adopting DevOps in a small, co-located organization, without a lot of cultural memory, is not difficult. Even in large organizations, small teams—the proverbial two-pizza1 teams—regularly succeed at attaining the business results promised by DevOps. In most organizations, you see many such efforts made, and most succeed. It is taking that success from an individual, isolated team level and scaling it to an enterprise that is a challenge. It is like having a series of small dance squads around the organization. However, these dance squads are all unique; some are doing the salsa, some jazz, some are ballroom dancing, while yet others are doing something my daughter tells me is “hip-hop.” They cannot combine and grow to a massive dance ensemble that can perform at the next halftime show, filling up the entire paying field of a football stadium. For that they need to be dancing not only to the same music but also be performing the same dance form, in unison. Similarly, small unique teams cannot impact the entire organization. These teams need to make the effort to standardize practices, processes, platforms, and tools in order to allow them to be replicated in other parts of the organization.
The organization, in turn, needs to set the right environment for DevOps adoption by sponsoring transformation efforts, by enabling change to rigid legacy processes, and by a top-down push to overcome cultural inertia.
NOTE A bottom-up practitioner-led effort allows extremely productive individual teams to adopt DevOps and thrive. A top-down executive leadership-led effort enables these individual successes to scale.
The engagement of the business is imperative for these scaling efforts to succeed. IT organizations exist to deliver the capabilities a business needs in order to deliver business value to its customers. The business is asking the IT organization for optimization—to be more agile, to be resilient to change, to be more responsive, to do more with less, to be more productive, to increase throughput, to deliver faster, to deliver at higher quality, to be reactive to the market, to accelerate past the competition, to keep up with an ever-changing regulatory and compliance regime, and, yes, to reduce expenses.
In addition, it may also be asking for innovation—to allow the company to enter new markets, to enable exponential growth, to engage and grow the customer base, to be responsive to customers’ needs, and, again, to reduce expenses. Delivering on these asks (hopefully not all of them at the same time) is what drives the need for change. It is what creates the motivation to work toward achieving the benefits that come from adopting DevOps.
NOTE You do not just adopt DevOps because it is cool. You need to have a business reason. The need for agility or velocity is the number-one reason that DevOps exists. The maturing and wide adoption of DevOps over the last few years is a reflection of today’s dynamic marketplace, of customers’ expectations.
Thus, in order for IT to undergo a transformation, this change has to improve and enhance IT’s ability to deliver business capabilities in a manner that, in turn, improves and enhances the business value delivered. Proper partnering between the business and IT is imperative so that the transformation IT undergoes by adopting DevOps delivers what the business needs the most by properly balancing optimization and innovation. Business goals have to drive why IT transforms, which will in turn drive how IT transforms.
This book will categorize DevOps adoption plays as follows:
DevOps for optimization
DevOps for innovation
Scaling DevOps adoption for the enterprise
Driving DevOps transformation in the enterprise
It will include lessons learned, examples, success patterns, and anti- patterns for each adoption play. Like a sports playbook, this book is designed to deliver certain plays that can be executed for different scenarios and situations—depending upon your organization’s current maturity and state—when it comes to transforming to a high-performance application delivery organization by adopting DevOps. An organization will need to take those plays and execute them in a tactical manner, based on the projects and teams that are adopting DevOps. Just as no battle plan ever survived contact with the enemy, these plays will need to be executed with an action plan or a broader adoption roadmap designed for each organization.
Furthermore, no organization is monolithic or homogenous in nature. Some parts of the organization may be more mature in some areas, but less mature in others. Some teams and groups may have already achieved agility and velocity, whereas others may be experiencing tremendous cultural inertia, all within the same organization, sometimes in the same building; they all need to work together in order to get scale.
An organization may have an innovation lab that is using modern agile and DevOps practices, while core systems teams may still be delivering in a rigid waterfall manner. Thus, these patterns of adoption will apply differently to different parts of the same organization and will need to be customized to suit the needs of various teams. To help with this customization effort, this book will also apply the technique of value stream mapping, used for decades as a component of Lean practices, that can be used to develop a custom adoption roadmap from these plays for an organization’s business goals, current maturity, and capabilities.
Disrupt or Be Disrupted
We live in an era of massive change. In 1960, the average life expectancy of a Fortune 500 company was 75 years. Today, the average lifespan is just 15 years and declining further. So what gives? You only have to look as far as what is referred to as the Uber effect to understand why so many companies fail. A company led by a founder who is not from the taxi industry, Uber disrupted a centuries-old industry with the touch of a button on a mobile app; they have made cab service part of the on-demand, service economy, where consumers get what they want, when they want it, with no delays. New IT capabilities—accelerated by the intersection of new approaches like Agile and DevOps and technologies like cloud and microservices—are allowing startups armed with no more than services on a cloud and a mobile app to Uber large, established organizations with massive IT investments, valuable infrastructure, and experienced people.
NOTE The fastest-growing transportation company in the world does not own any vehicles (Uber); the fastest-growing hospitality company providing living space for rent owns no property (Airbnb); the fastest-growing media company in the world produces no media (Facebook); the largest encyclopedia in the world has no staff writers (Wikipedia). Disruption is real.
So, ask yourself, is your organization a disruptor or a disruptee? The reality is, most organizations are the latter, putting IT organizations under more pressure today than ever before. Whether it’s the fear of being Ubered by the competition or business demands to pick up the pace, IT organizations face a balancing act of ensuring the optimized operation of core applications and of being innovative. However, the truth is, innovation and maintaining the efficiencies of legacy systems can co-exist. While the prospect of competing with born-on-the-web companies like Uber and Airbnb may seem daunting, adopting DevOps at scale across your organization can enable your IT team to become more agile, efficient, and innovative. Adopting DevOps can put your IT in a position to become the enabler of change at your organization so it can fend off the disrupters; this in turn allows it to become the enabler that lets your organization become the disrupter. In today’s technology-driven world, IT capabilities have become the key differentiators between the disrupter and the disrupted.
Defining DevOps
Before I begin to delve into the core capabilities and practices that you need to adopt and the various plays you need to execute in order to adopt DevOps in an organization, it is essential that you understand the definition of the term DevOps.
DevOps, like any new technology or tech-related movement that is adopted in industry, has become an overloaded buzzword. Everyone talks about it, not everyone knows what it is all about, and worst of all, many of those who claim to do it are really doing a terrible job. There are some excellent examples of companies that have excelled and are at the leading edge of the DevOps movement—the often-cited Etsy, Flickr, Facebook, and Netflix come to mind. But even here, there is contention and debate as to what is truly the best approach to DevOps. Netflix says what they do is NoOps, with developers taking over Ops responsibilities. Yet others counter that such a situation would lead to anarchy.
Such debate is to be expected as the industry refines what DevOps is as it evolves. As I will discuss at length in this book, there are different approaches to adopting DevOps, and each organization should look at adopting the right capabilities and practices of DevOps, based on their individual risk-value and business drivers balance. In fact, this adoption needs to start at a project level and then be scaled across the enterprise, leveraging techniques that will be described in this book.
As I mentioned previously, there are as many definitions of DevOps, or at least opinions of what DevOps really is, as there are blogs and tech “experts.” There is the perspective of DevOps where the developer is king; DevOps where continuous delivery is the driver; DevOps where it all hinges on the cloud and one cannot have true DevOps without a cloud; and DevOps where DevOps equals microservices. So, let’s start with the definition listed on a (fairly) neutral source—Wikipedia (Wikipedia, 2016):
DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.
It is important to note that the Wikipedia definition has also evolved over time, as DevOps has matured. For comparison, here is the definition listed on Wikipedia in 2013:
DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and Information Technology (IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.
This evolution of the Wikipedia definition is indicative of the evolution of DevOps and how the industry views DevOps. Other than the replacement of the esoteric portmanteau, which had everyone looking it up on Dictionary.com, the key points to note are as follows:
Replacement of software development method by culture, movement, or practice.
Addition of the reference to
automation.
Change of the end-goal from “rapidly producing software products and services” to “building, testing, and releasing software, which can happen rapidly, frequently, and more reliably.” Thus, the goal of DevOps changes from being just speed, to being speed, reliability, and quality.
Of course, I would be remiss not to mention the most concise definition of DevOps ever written; this was seen on a T-shirt at the O’Reilly Velocity Conference in 2013:
DevOps—taking the SH out of IT!
Who Is This Book For?
A sports team does not just have the players who take to the field on game day; it also has everything from coaches, assistant coaches, team management, team executives, trainers, doctors, nutritionists, physiotherapists, equipment managers, all the way to ball carriers and water servers. All are essential, and all need to excel in their roles and how they work together as a team, for the team to perform at its highest capacity. The same way, DevOps is not just about development and operations practitioners. It requires all the stakeholders in the application delivery pipeline to transform how they work, how they collaborate and communicate, how they work together like a high performance team.
This book is for all the team members in an organization who are stakeholders in the application delivery pipeline—from line of business owners, to analysts, architects, designers, developers, testers, quality assurance (QA) practitioners, automation engineers, infrastructure engineers, operations practitioners, database administrators, system administrators, documentation writers, project managers, product owners, all the way to c-suite executives. These roles may vary by organization, and many will need to evolve and transform what they do and how they do it as the organization adopts DevOps. This book is designed to benefit them all.
The application of each play discussed will impact each stakeholder role differently—some will see significant change in their role and how they interact with others, and some will see none at all. Just all the ball carriers and water servers are typically not impacted by which plays a team runs, but they are still stakeholders who can impact team performance if they do not perform as expected. The same way certain roles are supporting roles in IT too. Other roles, like key stakeholders who directly work with artifacts and processes that are a part of the application delivery pipeline, will benefit significantly from the plays in the book. They are the players and the direct supporting staff who play the game or support those who do, enabling them to perform at peak performance capacity.
Chapter 1 is an overview of DevOps. It documents the evolution of DevOps from its origins to today. It defines and describes all the common practices and capabilities that make up DevOps. It sets the stage with the broad definition of DevOps and of a DevOps transformation, which are used as the premise of this book.
Chapter 2 is focused on the leaders on the team: the coaches, the team captain, the senior players who form the core of the team. It focuses on how to assess the playing conditions and the competition to develop and select the right set of plays—the playbook for the team. It is for the IT management, project and program managers, product owners, team leads, senior practitioners, DevOps coaches, and anyone who aspires to be one of them.
Chapter 3 provides guidance on how to build a business case for a DevOps Transformation, allowing for the right sponsorship and investments to ensure success.
Chapters 4 through 6 are the actual plays. They are categorized as follows:
Chapter 4—DevOps Plays for Optimization
: Plays to optimize the application delivery pipeline to maximize efficiency, by eliminating waste
Chapter 5—DevOps Plays for Innovation:
Plays to make the application delivery pipeline fast and agile to support the ability to experiment, to drive innovation
Chapter 6—DevOps Plays for scaling DevOps in the enterprise:
Plays to scale DevOps adoption across the organization, an organization that is large, complex, distributed and is not homogenous in its maturity
Chapter 7 is a chapter dedicated to the executive leadership driving the DevOps adoption. Like the general managers and executives of a sports teams, executives make the executive decisions and set the direction and culture of the organization. They are the ones who need to make the decision to undertake a DevOps transformation. They need to make the necessary investment and sponsor the transformation. They will need to know how to make the business case and determine the return on investment for such a transformation. They need to drive the transformation, from the front, across the enterprise.
The book also has one appendix. It is an example of a DevOps Transformation adoption roadmap, developed for a fictitious bank by delivering a value stream mapping exercise.
The book purposefully has tried to remain tool and platform agnostic. While several examples of tools, platforms, and technologies—both commercial and open source—are made throughout the book, they are done to as current examples of tooling and platforms available in the market to enable automation. Tools are necessary to automate processes, making them fast, repeatable, scalable, and error free. However, tools and platforms are continuously evolving and getting replaced by newer and better ones. It is hence a futile effort to recommend or even document available tools and platforms. The goal is to highlight capabilities, while remaining as tool and platform agnostic as possible to remain relevant even as tools and platforms available evolve.
The Sports Analogies
Individual commitment to a group effort—that is what makes a team work, a company work, a society work, a civilization work.
—Vince Lombardi, legendary American football coach
There is nothing that transcends culture, language, and geographic borders than sports. If you have any doubts, just watch the reruns of the recently concluded Olympics in Rio. Analogies from sports are also very relevant to application development and delivery, as they are both team events. While developing or delivering a new application or service may not require the physical conditioning an Olympics gold medalist does, they do require the leadership, communication, collaboration, and trust that any team sport needs.
I also have a personal passion for sports. Right from my childhood I grew up in a household that had a love for sports. My maternal grandfather was an Olympian and national sports figure in India. He played for the Indian National Hockey team in his youth and later was a sports executive with the Indian National Football (Soccer) team at the 1952 Helsinki Olympic Games. He also had the opportunity to run with the Olympic torch for the 1964 Tokyo Olympics, as the torch passed through India. He remained a sports executive for domestic soccer tournaments for several decades after that, including when I was a child. Growing up with an Olympic torch in the family home gives one a high respect for sportsmen and women.
Major Lachhman Singh running with the ‘64 Tokyo Olympics Torch (Source: Singh Family Personal Collection)
In the book I have taken examples, quotes, and players’ and coaches’ experiences from multiple sports and mapped them to the Plays of DevOps Adoption. The parallels are intended to make the plays more relatable and understandable and hopefully the book more interesting to read.
Companion Website
This books comes with a companion website where I will continue to post updates and new content, including case studies, presentations, videos, and outtakes from the book. Check it out at http://devopsadoptionplaybook.com.
Note
1
Amazon CEO Jeff Bezos claims that a team that cannot be fed with two pizzas is too big to be a productive team.
CHAPTER 1DevOps: An Overview
RANT OF A DEVELOPMENT MANAGER
So, the developer completes writing code for a new service by Monday afternoon. She builds the code, runs unit tests, and delivers the code to the integration stream so it gets included in the continuous integration (CI) build. To get her service tested, before leaving for work, she opens a ticket with the Quality Assurance (QA) team.
Tuesday morning, the QA team comes in and sees the ticket assigned to them. A tester gets the ticket and emails the developer asking for the deployment instructions. As there is no deployment automation, the developer responds saying she will deploy the service to the QA environment herself. Tuesday afternoon, they get on a conference call to deploy the code. The developer discovers that test environment is not compatible with her code. They need a new environment. Tuesday evening, the tester opens a ticket with the operations (Ops) team for a new environment, with the new specs.
Wednesday morning, the Ops team assigns the ticket to an engineer who looks at the specs and sees a firewall port change. As he leaves for lunch, he opens a ticket with the security team to approve the port change. Wednesday afternoon, the security team assigns the ticket to a security engineer, who approves the change. Wednesday evening, the Ops engineer receives the approval and starts building the new environment. He needs to manually build new Virtual Machines (VMs), with an Operating System (OS), app server, database, and web server.
Thursday morning, the server build is done, and the ticket is closed. The tester emails the developer again to deploy the new service. The developer deploys the service, and the tester starts walking through the test scripts, which pass. He now needs to run a regression test but needs additional test data to re-run tests. Thursday afternoon he opens a ticket to request new test data with the production support team.
Friday morning, the production support team assigns a database analyst (DBA) to extract test data from production. But now it’s Friday afternoon. Everyone knows DBAs don’t work on Friday afternoons. Monday morning, the tester gets the test data from the DBA. It takes him 20 minutes to run the regression tests and discover a defect. He returns the ticket to the developer—a full week after the code was written and built. A full week of coding has now been done on top of that code, not knowing it was defective. We are now another week behind.
What’s scary about this story is that when I tell it to my peers in other companies, they shake their heads not in empathy but in amazement as to how efficient we are compared to them!
—Yet another frustrated development manager
DevOps: Origins
The DevOps movement began with a seminal talk given by John Allspaw and Paul Hammond (both at Flickr/Yahoo at that time) at the O’Reilly Velocity 2009 conference. The talk was entitled “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr.”1 Ten deploys a day was considered unprecedented. Their approach was eventually referred to as DevOps by Patrick Debois, when he organized the first DevOpsDays event in Ghent, Belgium, the same year.
While the name caught on and started getting tremendous interest, the traction was initially limited to startups, more specifically, organizations delivering web applications. These applications were created by developers (the Dev) who typically delivered changes and updates to their web apps in a very rapid manner. The main hurdle they faced was that of operations (the Ops), which were slow in deploying those changes, as they had rigid and rigorous change management processes.
The goal of the DevOps movement was to address this impedance mismatch between the Dev and Ops teams; to bridge the chasm between them; and to foster more communication, collaboration, and trust. At its heart, it was a cultural movement, focused on changing the cultural differences between Dev and Ops, along with automation to make application delivery faster, more efficient, and eventually, continuous. In 2010, Jez Humble, then at ThoughtWorks, took DevOps to practitioners throughout the industry with his book Continuous Delivery, codifying some of the practices that make up the core of DevOps and making DevOps adoption tangible and available to all.
Still, DevOps was seen as something done by the unicorns—the startups and the upstarts, organizations at the cutting edge of innovation, without large, complex legacy systems to maintain. It had not yet gone mainstream with the large enterprises. However, these large enterprises were seeing what the startups were achieving with DevOps, and were trying to determine how to adapt DevOps for their own needs. Organizations like IBM were beginning to dabble with deployment automation, and with visual architecting of environments, and even stitching these two capabilities together. At the same time, well-established companies in the build automation space, like UrbanCode, started pivoting into DevOps with the release of uDeploy, thus establishing a new category of tools to enable continuous delivery. Other companies in the automation space, like Nolio, joined in with their own competitive offerings. In parallel, coming from the Ops and infrastructure as code side, companies like Opscode (now called Chef) and Puppet Labs were gaining traction (Opscode with Chef, and Puppet Labs with Puppet).
The real growth for DevOps into large enterprises began in 2012, with companies like IBM jumping into the fray with their first, albeit short-lived, continuous delivery experiment with SmartCloud Continuous Delivery. Several consulting firms, like ThoughtWorks and IBM, also started to offer consulting services for organizations, especially large enterprises looking to adopt DevOps, and helping to translate what worked for the unicorns so that it could work for enterprises. IBM and CA Technologies announced their formal entrance into the DevOps world by acquiring UrbanCode and Nolio, respectively (and coincidently on the same day in April 2013). However, the biggest turning point for the DevOps movement since its inception came later, in 2013, with the publication of Gene Kim’s book, The Phoenix Project. This book, inspired by and modeled after the historic The Goal by Eliyahu M. Goldratt, became the must-read book for the modern-day implementation of Lean practices and Goldratt’s Theory of Constraints in the IT world, just as Goldratt’s book had been a few decades earlier for the manufacturing world. Kim truly took DevOps mainstream with his book, as well as subsequent work he has done with the State of DevOps Report that he publishes every year, with Jez Humble and Puppet Labs.
DevOps: Roots
Where does DevOps come from? While I have already outlined its origin story, the true roots of DevOps predate Allspaw, Debois, Humble, and Kim by almost a century. You have to go way back to the 1910s and look at the origins of the Lean movement.
The Lean movement started in manufacturing with Henry Ford and his adoption of Lean for flow management in the Model T production lines. This work was further extended, refined, and codified by Kiichiro Toyoda and Taiichi Ohno at Toyota starting in the 1930s and really accelerating after World War II. Their work was both refined and influenced by Dr. William E. Deming in the 1950s, who proposed the Plan–Do–Check–Act (or Adjust) cycle (PDCA), to continuously improve manufacturing quality. Based on this core approach, the Lean manufacturing movement aimed to both continuously improve the product being manufactured and reduce waste in the manufacturing process. Lean was further refined in the works of James P. Womack and Daniel T. Jones when they published The Machine that Changed the World in 1990 and (required reading for everyone) Lean Thinking in 1996 (Lean.org, 2016).
DEMING ON LEAN THINKING AND CONTINUOUS IMPROVEMENT
Dr. W. Edwards Deming taught that by adopting appropriate principles of management, organizations can increase quality and simultaneously reduce costs (by reducing waste, rework, staff attrition and litigation while increasing customer loyalty). The key is to practice continual improvement and think of manufacturing as a system, not as bits and pieces.
—Dr. Deming’s Management Training (Deming, 1998)
In 2001 came Agile, a group of 17 thought leaders, including Alistair Cockburn and Martin Fowler, who created The Agile Manifesto.2 The core principles of the manifesto were to get away from the rigid, waterfall-oriented, documentation-heavy world of software development, which was resulting in most software development projects being late, over budget, or abject failures.
Their goal was to move to an iterative approach where there was constant interaction with the customer, end-user, or a surrogate who represented them. They wanted to move away from measuring progress through major rigid milestones such as Requirements Documentation, which brought code no closer to being delivered than the day before. Other goals were to use real running code (working software) as the true measure of progress; to look at planning as being adaptive to real progress; and to create requirements that did not need to be written in stone up front, but would evolve and be refined as the applications were being developed.
Agile was refined with the development of methodologies like XP, Scrum, and, more recently, Scaled Agile Framework or SAFe. Today, Agile is used by both large and small organizations to deliver projects of all sizes and technologies.
Agile was the precursor to, and became the core driver for, the need for DevOps. As developers started delivering code faster, that code needed to be tested faster; it also needed to be deployed to Dev and test servers, and eventually to production, more often. The Ops teams were not set up for this, which resulted in a major bottleneck being created at the Dev-to-test handoff, due to lack of availability of the right test environments as and when needed and, more importantly, at the hands of production at release time. Production release remained a major undertaking, with “release weekends” that typically lasted beyond the weekend.
THE RELEASE WEEKEND
I remember when I was working as a developer at a financial services institution in the early ’90s. (We called them banks back then.) On release weekends, much to my chagrin, we were asked to show up at work on Friday mornings with our sleeping bags in hand. We were expected to stay there through the weekend. There were multiple conference rooms set up with conference call bridges open to get every team in communication with each other. One conference room was set up like a war room with the project leader coordinating all the stakeholders off a massive spreadsheet. The management did their best to create a party atmosphere, but that faded right after the first few hours. We were communicating with the Ops people for the first time. We were handing off our code to people who had never seen the code before. They were deploying code into environments we had no visibility into, using scripts and tools we had no familiarity with. It would be chaos the whole weekend. Lots of delivered food and stale coffee, and nothing seemed to work as planned. And the traders we supported, they were smart. They always planned their team outing or picnics on the Monday following. They knew nothing would work. And they were right. Fortunately, we only did this twice a year. Even more fortunately for me personally, I worked there for only two releases.
The rapid development of code in short iterations amplified the need for better collaboration and coordination between Dev and Ops teams. The frequent failure of release to production exposed the need for providing developers with access to production-like environments. The major inefficiencies in the entire process were exposed by making just one part of the process—developing code—more efficient, which created major bottlenecks with test and Ops. If you think of the application development and delivery process as an assembly line in a factory, speeding up an operation of just one of the stations to increase the number of widgets it produces does not help the overall delivery speed if the downstream stations are still operating at a slower speed. It just creates more of a backlog for them. (See Figure 1-1.) This was not just a challenge for Ops, but for all the stakeholders in the delivery life cycle.
Figure 1-1: Delivery pipeline bottlenecks
The focus now turned toward minimizing cycle time—the time from the inception of a requirement, or user story, to the time that capability is in the hands of the customer, or at least is integrated, tested, and ready to be deployed to the customer. This resulted in the development of the two core capabilities of DevOps: continuous integration (already a core competency of Agile) and continuous delivery. I will discuss both capabilities in detail shortly. This extension of Agile beyond the Dev-test cycle—including the Ops team in the delivery cycle, as a part of the process, and not in a separate silo that was not engaged until the code was ready for release—became the core principle of DevOps.
Addressing Dev versus Ops
Dev and Ops have traditionally lived in different silos, with misaligned, even opposing priorities. Development (Dev) is tasked with creating innovation and getting it into the hands of users as soon as possible. Operations (Ops) is tasked with making sure that the users have access to a stable, fast, and responsive system. While Dev and Ops’ ultimate goal is to make the user a satisfied (and potentially a happy, paying) customer of the systems they provide, their views of how to do it tend to be inherently antithetical. No developer wants to intentionally produce a buggy system that would cause the application to crash while a user is using it. No operations person wants developers to not produce updates with new, exciting features and capabilities. It is how they go about it that is different. This is a classic symptom of what is referred to as water-Scrum-fall (Forrester, 2011). Developers want, and are expected to produce, new features quickly. Operations want, and are expected to produce, a stable system, at all times.
Dev versus Ops
Before the advent of Agile, in the purely waterfall-oriented paradigm, when developers and operations lived in truly isolated worlds, these opposing priorities were not that much of an issue. Developers and operations worked on a schedule that was marked by limited interactions, only at release times. Developers knew when the release date was, and they could only release new features then. If they did not create a new feature by the release date, they would have to wait for the next release train. Operations knew when the train would come to town. They would have enough time to test the new features before deploying them, and they could take days (weekends) to deploy them out to customers. For large systems, they could even deploy in a phased manner spread over long periods of time. Stability was maintained.
Agile changed all that. With continuous integration (CI), developers were now deploying their features daily. There was no release train to wait for; it was a conveyer belt (pipeline) that ran all the time. The developers now wanted their features up and running—in the Dev environment, in the test environment, and finally in production (Prod)—at the same frequency at which they produced and integrated them. They wanted Ops to accommodate all these new releases.
Ops now had to deal with not one release every so often but a continuous barrage of CI builds. These builds may or may not have been deployment-ready, but they had to be managed by Ops and deployed to test, and eventually production, environments. Ops now cared more about quality. Developers and testers cared about how quickly they could get Dev and test environments and whether or not those environments were production-like. They did not want to test the code they were delivering on environments that did not function and behave like production environments. Thus, Ops could no longer take days to provision and configure new environments—for Dev, test, and eventually Prod. They had to do all of this while still maintaining stability and reliability of production systems.
CYCLE TIME?
If you have two-week Scrums but it takes three weeks to get a new test server, how long are your Scrums?
Dev and Ops
The solution to this battle between Dev and Ops is what DevOps addresses: to achieve the balance between innovation and stability and between speed of delivery and quality. To achieve this, both Dev and Ops need to improve how they operate and align.
The Dev View The previous section may give the impression that Ops needs to change more than Dev, but Dev also needs to make several changes:
Dev needs to work with Ops to understand the nature of the production systems their applications will be running on. What are the standards for the production systems (environment patterns) and how should their applications perform on them? Within what constraints do the applications need to operate? Dev now needs to understand system and enterprise architectures.
Dev needs to get more involved in testing. This means not just making sure that their code is bug-free but also testing the application to see how it will perform in production. This requires Dev to work closely with Quality Assurance (QA) and to test their application in a production-like system. (I’ll discuss production-like systems later in this chapter.)
Dev also needs to learn how to monitor deployed applications and understand the metrics Ops cares about. They need to able to decipher how processes interact and how one process can cause another one to slow down or even crash. They need to understand how changes to their code will impact the entire production system and not just their own application.
Dev needs to communicate and collaborate better with Ops.
The Ops View Ops needs to be able to provision new environments rapidly, and they need to architect their systems to absorb rapid change.
Ops needs to know what code is coming and how it may impact their system. This requires them to be involved with Dev, right from understanding requirements and system specs of the applications being developed. This process is referred to in Lean and DevOps as
shift left.
They need to make sure that their systems can accommodate these applications as they are enhanced.
Ops needs to automate how they manage their systems. Rapid change with stability cannot be achieved without automation. Automation will allow not only rapid change but also rapid rollbacks, if something does break.
