Becoming an Agile Software Architect E-Book

Rajesh R V

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 428

Veröffentlichungsjahr: 2021

Beschreibung

Many organizations have embraced Agile methodologies to transform their ability to rapidly respond to constantly changing customer demands. However, in this melee, many enterprises often neglect to invest in architects by presuming architecture is not an intrinsic element of Agile software development.
Since the role of an architect is not pre-defined in Agile, many organizations struggle to position architects, often resulting in friction with other roles or a failure to provide a clear learning path for architects to be productive.
This book guides architects and organizations through new Agile ways of incrementally developing the architecture for delivering an uninterrupted, continuous flow of values that meets customer needs. You'll explore various aspects of Agile architecture and how it differs from traditional architecture. The book later covers Agile architects' responsibilities and how architects can add significant value by positioning themselves appropriately in the Agile flow of work. Through examples, you'll also learn concepts such as architectural decision backlog,the last responsible moment, value delivery, architecting for change, DevOps, and evolutionary collaboration.
By the end of this Agile book, you'll be able to operate as an architect in Agile development initiatives and successfully architect reliable software systems.

Leseprobe

Becoming an Agile Software Architect

Strategies, practices, and patterns to help architects design continually evolving solutions

Rajesh R V

BIRMINGHAM—MUMBAI

Becoming an Agile Software Architect

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Aaron Lazar

Publishing Product Manager: Shweta Bairoliya

Senior Editor: Nitee Shetty

Content Development Editor: Tiksha Lad

Technical Editor: Rashmi Subhash Choudhari

Copy Editor: Safis Editing

Project Coordinator: Francy Puthiry

Proofreader: Safis Editing

Indexer: Tejal Daruwale Soni

Production Designer: Shankar Kalbhor

First published: February 2021

Production reference: 1170221

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80056-384-1

www.packt.com

To my lovely wife, Saritha, and our kids, Nikhil and Aditya; they sacrificed their interests to support my passion.

To my parents, Ramachandran Nair and Vasanthakumari, and extended family members; their enthusiasm fueled my energy.

Contributors

About the author

Rajesh R V is a seasoned IT architect with over 20 years of extensive experience in various technologies.

As the head of architecture at the Emirates Group, Rajesh has helped the company transform, reshape, and re-organize into a high-performing organization by adopting SAFe and Agile architecture practices.

Rajesh has a deep passion for technology and architecture. He also architected the Open Travel Platform (OTP) that earned the Emirates Group the prestigious 2011 Red Hat Innovation Award.

He previously wrote the best-selling books, Spring Microservices and, Spring 5.0 Microservices, and reviewed Service-Oriented Java Business Integration by Packt.

I want to thank everyone at Packt and the reviewers; their in-depth knowledge of this subject helped improve the quality. Thanks to all who supported and encouraged me on my journey to become an Agile software architect.

About the reviewers

With more than a decade of experience in both fields, Gaurav Mishra is an expert in user interface (UI) development and user experience (UX) design. He is comfortable working with any type of technology, and employs a growth mindset.

He has mentored many students around the world, providing workshops and training in UI development, UX design, and Drupal.

With a passion for building products and services from scratch, Gaurav has ultimately played a key role in the success of many organizations. He likes to challenge the status quo to bring out the best from a team and to reshape an organization's culture.

In his free time, Gaurav enjoys all genres of music from Indian classical to club.

Ken Cochrane is an experienced technologist, architect, author, and leader. He has extensive professional experience leading software development teams, developing and implementing scalable web solutions, and delivering high-quality, highly functional web applications that are currently in use by millions of people worldwide. Ken was previously a member of the founding teams of Docker and CashStar. He is now currently the senior director of enterprise architecture for WEX, a global leader in financial technology solutions. Ken coauthored the book Docker Cookbook - Second Edition, published by Packt in 2018. He currently resides in Southern Maine with his wife, Emily, and two sons, Zander and Maddox.

I would like to thank my wife, Emily, and my two boys, Zander and Maddox, for giving me the time to work on this book; also, my parents for buying me my first computer, and letting me spend so much of my free time on it.

Table of Contents

Preface

Section 1: Understanding Architecture in the Agile World

Chapter 1: Looking through the Agile Architect's Lens

Chapter 2: Agile Architecture – The Foundation of Agile Delivery

Technical requirements

The journey leading to Agile software development

What is Agile software development?

Is Lean different from Agile?

Five benefits of Agile software development

Agile development and traditional architecture – an oxymoron?

Architecture as a continuum

Traditional development methods frontload architecture efforts

Agile architecture – architecting with Agile practices

Balancing speed and sustainability

Developing a little architecture upfront, but how much?

Agile architecture principles

Architecture needs a collective effort

Architecture is continuous

Architecture is Lean and efficient

Test and learn architecture early and often

Architecture enables agility

Comparing different enterprise Agile frameworks

Scaled Agile Framework (SAFe)

Disciplined Agile (DA)

Large-Scale Scrum (LeSS)

Measuring Agile architecture maturity

Lessons learned from Snow in the Desert

Circumventing key challenges in adopting Agile architecture

Agile architecture process flow and poster

Summary

Further reading

Section 2: Transformation of Architect Roles in Agile

Chapter 3: Agile Architect – The Linchpin of Success

Technical requirements

Understanding the challenging environment of architects

A myriad of architect roles have impacted clarity

Architects slow down delivery

Self-organizing teams propel the no architect movement

The birth of the accidental architect

Architects are vital in Agile – the question is who plays that role?

Organizing Agile architects – none, volunteered, designated, or dedicated

Different roles played by Agile architects

Essential skills for Agile architects to be relevant

Explaining the Agile architect's behavior and duties with metaphors

The changing behavior of the Agile architect

The challenging duties of the Agile architect

How different frameworks handle architect roles

Scaled Agile Framework

Disciplined Agile

Large-Scale Scrum

Learnings from Snow in the Desert

Summary

Further reading

Chapter 4: Agile Enterprise Architect – Connecting Strategy to Code

Technical requirements

The need for change in lean-Agile EA

EA frameworks focus on describing enterprises

The city planning metaphor is no longer valid

EA operates without purpose

Digging into the current state is an evil

EA disconnected from its intended purpose

Showing value first is a trap

Enterprise architects are non-technical

Understanding the principles and duties

The principles for Agile enterprise architects to succeed

The duties of Agile enterprise architects

The EA repository – the gold dust problem

Measuring enterprise architects

The enterprise architect role in Agile frameworks

The enterprise architect role in Scaled Agile Framework

The enterprise architect role in Disciplined Agile

The enterprise architect role in Large-Scale Scrum

Enterprise architects at Snow in the Desert

The profile of an enterprise architect

Understanding the portfolio flow

Summary

Further reading

Chapter 5: Agile Solution Architect – Designing Continuously Evolving Systems

Technical requirements

Solution architects – the busy bees of agile teams

A solution architect's mindset – the periscope in action

The overlapping roles of enterprise and solution architects

Maximizing value and eliminating flow barriers

Starting with business value

Focusing on sustainable quality

Reducing reject cycles

Minimizing lead time

Reducing rework

The duties of solution architects

Developing an intentional architecture

Preparing for emergent design

Enabling continuous evolution

Measuring the success of solution architects

Summary

Further reading

Section 3: Essential Knowledge to Become a Successful Agile Architect

Chapter 6: Delivering Value with New Ways of Working

Technical requirements

Understanding business value

Showing the value of architecture is difficult

Linking architecture activities to business backlogs

Determining the business value of architecture

Allocating capacity for prioritization

Making the work transparent

Looking ahead of development

Working with a pragmatic mindset

Understanding the last responsible moment

Using eventual integrity

Using a risk- and cost-driven approach

Ensuring anti-viscosity

Keeping options open

Using a hypothesis-based solution

Using real options theory

Using set-based concurrent engineering

Delivering early value with MVA

Managing tech debt

Summary

Further reading

Chapter 7: Technical Agility with Patterns and Techniques

Amplifying agility with technical excellence

Adopting software craftsmanship

Enhancing quality with technical agility

Building technical agility with patterns and techniques

Architecting for change

Driving technical agility with simplicity

Evolving with isolation by design

Legacy modernization architecture

Developing good code with engineering excellence

Applying coding principles, techniques, and patterns

Good coding practices

Continuous refactoring

Twelve-factor principles

Understanding enterprise integration

Developing for testability

Test-driven development

Treating infrastructure like software with the cloud

Monitoring everything with full stack telemetry

Outcome-based monitoring with progressive enhancement

Moving to product-centric monitoring

Building observable systems

Summary

Further reading

Chapter 8: DevOps and Continuous Delivery to Accelerate Flow

Embracing DevOps culture

Enhancing security with DevSecOps

Crossing over with site reliability engineering

Enabling flow with continuous delivery

Benefiting from CD

Architecting for continuous delivery

Measuring the effectiveness of continuous delivery

Implementing continuous delivery for legacy applications

Adopting continuous integration

Improving development and source code management

Automating quality assurance

Automatically deploying to production

Launching in dark mode for early feedback

Using feature flags to release features selectively

Releasing on demand

Progressively releasing with canary releases

Collecting early feedback with A/B testing

Architecting for release on demand

Securing systems by design

Summary

Further reading

Chapter 9: Architecting for Quality with Quality Attributes

Understanding software quality

Improving quality needs systems thinking

Incrementally applying quality with a build-to-adapt strategy

Building in quality to eliminate waste

Adopting a quality attribute model

Documenting quality attributes

Using the scaled Agile approach for specifying quality attributes

Using a quality attribute scenario

Using the quality life cycle in Agile software development

Discovering and refining quality attributes

Aligning stakeholders to quality attributes

Conducting a quality attribute workshop

Refining QASes and playback

Modeling and simulation

Applying architecture trade-offs

Using the architecture trade-off analysis method

Using the solution architecture review method

Developing quality attributes

Assessing system quality

Integrating quality checks in development

Using fitness functions

Evaluating using the Well-Architected Framework

Summary

Further reading

Chapter 10: Lean Documentation through Collaboration

Persisting knowledge with documentation

Understanding the reasons for documentation

Why the traditional way of documenting is no good

Using Lean-Agile ways of documentation

Evolutionary collaboration over documentation

Just barely enough documentation when required

How much documentation is sufficient?

Adhering to the principles of documentation

Documenting software architecture

Different proposals for documenting architecture

Documentation ecosystem at Snow in the Desert

Applying Model-Based Software Engineering

Summary

Further reading

Chapter 11: Architect as an Enabler in Lean-Agile Governance

Understanding architecture governance

Challenges with traditional governance

Bringing agility with Lean-Agile governance

Embracing Lean-Agile governance

Principles of Lean-Agile governance

Benefits of Lean governance

Balancing autonomy with agility

Determining the level of decentralization of decisions

Implementing Goldilocks governance – a case study

Documenting architecture decisions

Ensuring psychological safety

Measuring the quality of architecture decisions

Summary

Further reading

Section 4: Personality Traits and Organizational Influence

Chapter 12: Architecting Organizational Agility

Correlating business agility with IT agility

Understanding business and IT flow of value

Improving the flow of value in IT

Organizing people around value

Examining the need for architecting the organization and teams

Identifying flow and organizing teams with flow decomposition

Assigning systems to flows

Giving autonomy to flow teams

Comparing capability-centric versus flow-centric teams

Validating the organization of teams

Structuring teams within flow teams

Implementing communities of practice

Stepping into next-generation IT

Summary

Further reading

Chapter 13: Culture and Leadership Traits

Understanding the need for change 

Examining culture in high-performing organizations 

Understanding the behavior of high-performing teams 

Choosing the right leadership

Personality traits of an architect

Selflessness for relentless support

Respect, equality, and humility

Mindfulness for managing wellbeing

Curious to acquire new knowledge

Growth mindset for positive thinking

Intrinsic motivation for commitment

Creativity to explore new opportunities

Self-esteem to protect personal value

Empathy for emotional connection

Executive presence for gaining attention

Interpersonal traits of Agile architects

Collaborating for collective ownership

Resolving conflict for superior quality outcome

Communicating well using storytelling

Nurturing and growing talent for a competent workforce

Motivating for a better outcome

Leading by example and demonstrating servant leadership

Feedback for continuous improvement

Psychological safety for trust and transparency

Leading the change by creating multipliers

Caring for building a sense of belonging

Summary

Further reading

Why subscribe?

Other Books You May Enjoy

Preface

Enterprises worldwide are significantly accelerating their digital transformation programs to achieve the state of business agility. Building a strong foundation to resist change is no longer a viable strategy. In today's increasingly turbulent market conditions, business agility is the holy grail for many enterprises to be more productive and responsive. Organizations embracing business agility place Agile software development at the center of their IT strategy to bring immediate value.

Agile development practices, underpinned by Lean-Agile principles, focus on the speed of delivery to support a high rate of business changes by eliminating impediments to flow. However, a substantial focus on speed challenges many of the traditional architecture principles and practices. This book guides and enlightens you to understand how architecture development practices can be effectively readjusted in Agile software development projects without compromising speed and quality. We will also explore how to position architects appropriately in Agile software projects to deliver the maximum possible value.

This book focuses on two key architect roles – enterprise architect and solution architect – and explains their duties in Agile software development projects with numerous examples.

The book then provides a series of strategies, best practices, and patterns to deliver value without compromising on delivery speed or quality. Later in this book, we will cover the critical role of architects in architecting organizations for agility. At the end of this book, we will cover several personal and interpersonal qualities required for architects' success.

Who this book is for

This book is for architects currently working on Agile development projects or aspiring to work on Agile software delivery, irrespective of the methodology they are using. You will also find this book useful if you're a senior developer or a budding architect looking to understand the Agile architect's role by embracing Agile architecture strategies and a Lean-Agile mindset.

What this book covers

Chapter 1, Looking through the Agile Architect's Lens, provides a framework and navigation tool to explore this book easily.

Chapter 2, Agile Architecture – The Foundation of Agile Delivery, explains the concepts and principles of Agile architecture and compares it with traditionalarchitecture.

Chapter 3, Agile Architects – The Linchpin to Success, uses several metaphors to reinforce Agile architects' roles and responsibilities.

Chapter 4, Agile Enterprise Architect – Connecting Strategy to Code, establishes the duties of modern enterprise architects in Agile software development.

Chapter 5, Agile Solution Architect – Designing Continuously Evolving Systems,focuses on solutions architects' operating methods in Agile software development projects.

Chapter 6, Delivering Value with New Ways of Working, provides techniques for architects to be successful in Agile delivery environments.

Chapter 7, Technical Agility with Patterns and Techniques, introduces several patterns and practices to achieve technical agility.

Chapter 8, DevOps and Continuous Delivery to Accelerate Flow, explores the importance of architects in DevOps and continuous delivery.

Chapter 9, Architecting for Quality with Quality Attributes, discusses various quality models, tools, and approaches to ensure that teams deliver quality products to customers.

Chapter 10, Lean Documentation through Collaboration, talks about alternate approaches to documentation and familiarizes you with the required documentation concepts.

Chapter 11, Architect as an Enabler in Lean-Agile Governance, blows apart the myths around governance in Agile software development and introduces Lean governance principles.

Chapter 12, Architecting Organizational Agility, draws attention to the need to architect organizations around the flow of work.

Chapter 13, Culture and Leadership Traits, discusses the need to change by introducing new personal and interpersonal qualities required for architects.

To get the most out of this book

To understand the concepts covered in this book easily, you need to have prior knowledge of basic Agile development practices. Besides this, software architecture fundamentals are essential as the book builds on the foundational architecture practices.

Download the example code files

You can download the supporting files for this book from GitHub at https://github.com/PacktPublishing/Becoming-an-Agile-Software-Architect. In case there's an update to these assets, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here:

https://static.packt-cdn.com/downloads/9781800563841_ColorImages.pdf

Conventions used

There are a number of text conventions used throughout this book.

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "A scenario from Snow in the Desert's Automatic Vehicle Tracking System (AVTS) is captured as an example in the preceding diagram, mapping Source, Stimulus, Artifact, Environment, Response, and Response measure. A mini-QAS is a simplified version that eliminates Environment and Artifact."

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

Section 1: Understanding Architecture in the Agile World

In this section, you will gain clarity on various aspects of Agile architecture, how it differs from traditional architecture practices, and how the Agile architecture helps organizations to deliver a continuous sustainable flow of work.

This section contains the following chapters:

Chapter 1, Looking through the Agile Architect's Lens Chapter 2,Agile Architecture – The Foundation of Agile Delivery

Chapter 1: Looking through the Agile Architect's Lens

"Less is more."

– Ludwig Mies van der Rohe (Architected Barcelona Pavilion, Expo 1929)

Agile software development is an effective practice that helps organizations continuously deliver the maximum possible business value with organizational agility. The 14th Annual State of Agile Report 2020 shows that an astounding 95% of the 1,121 respondents adopted Agile software development practices. The sharp increase in Agile adoption indicates why organizations cannot shy away from embracing Agile software development practices in their business. Most of these organizations adopt Agile software delivery practices for accelerating software delivery by successfully managing the changing priorities of the business. Architecting systems using Agile values, principles, and practices are incredibly crucial to improve productivity and deliver quality products.

This book introduces the Agile Architect's Lens—a comprehensive representation of the knowledge of key segments and leading practices for Agile architects to successfully operate in a Lean-Agile organization. The Agile Architect's Lens has twelve focal points, as shown in the diagram. Each focal point that can you see in the following figure captures tried and tested approaches, patterns, and guidelines for architects to design continually evolving solutions:

Figure 1.1 – The Architect's Lens

The Agile Architect's Lens act as a single pane of glass, guiding light, and topic navigator for readers to easily explore this book.

Agile architecture is a set of practices, principles, and guidelines for developing architecture as an intrinsic element of Agile software development. Using Agile architecture concepts, Agile teams can confidently drive sustainable, fit-for-purpose solutions to meet customer needs. We thus begin by focusing on the Agile Architecture focal point of the Agile Architect's Lens.

Agile architects need special skills to operate efficiently in Agile organizations. The collaborative nature of Agile software development needs a new set of metaphors to understand expectations from Agile architects. A well-architected solution enables organizations to deliver value with the shortest sustainable lead time, which significantly enhances business operations. Our next logical focus will therefore be on the Agile Architect focal point, followed by covering two specific architect roles crucial to Agile development projects – the Enterprise Architect and Solution Architect focal points of the Agile Architect's Lens.

Agile architects are responsible for bridging strategies and providing a shared vision to all stakeholders in the continuous delivery flow. Agile architects have to acquire technical knowledge in specific areas to address commonly occurring challenges efficiently. Hence, the knowledge kitty has to have the right blend of process excellence and technical excellence. We will therefore shift our attention next to covering the Value Delivery, Patterns and Techniques, DevOps and Automation, Built-in Quality, Evolutionary Collaboration, and Safety Nets focal points of the Agile Architect's Lens.

In addition to the knowledge kitty, Agile architects need to undergo a radical transformation in behavior and mindset to achieve operational excellence. Servant leadership is one of the foremost characteristics that Agile architects need to have, but it is just a starting position. In our journey ahead, we expand on the Culture and Leadership Traits focal point of the Agile Architect's Lens.

Business agility is one of the key motivations for many organizations to opt for digital transformation. The lean practices of Agile software delivery can boost organizational agility. However, this needs considerable effort to organize systems and people around values to be nimble enough to respond quickly to changes in business strategies. Architecting organizations around values is an essential aspect of organizational agility. We cover this in our final focal point of the Agile Architect's Lens, Architecting Organizational Agility.

In summary, digital transformation is a continuous activity for many organizations. Agile software delivery is the foundation on which the organizational agility essential to support successful transformations can be built. Architecture, and especially architects, play an extremely critical role not just in Agile software delivery but also in architecting organizations for agility. While technical excellence is paramount for architects, working beyond traditional boundaries with an intrinsically motivated team of multi-disciplinary individuals requires a significant shift in mentality.

Let's start our exciting journey by exploring our first focal point, Agile Architecture, in our next chapter.

Chapter 2: Agile Architecture – The Foundation of Agile Delivery

"Simplicity is the ultimate sophistication."

– Leonardo da Vinci

In this era of innovative and disruptive digital transformation, Agile software development is no longer a marketing buzzword. Agile software development methodologies have enabled many organizations at scale (at the required size) to significantly accelerate their ambitious journey to accomplish and sustain organizational agility. Agile architecture and the new Agile ways of developing architecture and design are incredibly important in delivering an uninterrupted, continuous flow of values to meet the needs and wants of customers. However, architecture and architects need a radical reshaping to be relevant in the world of Agile software development.

This chapter takes you through the evolution of software development and analyzes the lessons learned, which help in reinforcing the vital need of architecture in Agile software development initiatives.

In this chapter, we're going to cover the following main topics:

The journey leading to Agile software developmentAgile development and traditional architecture – an oxymoron?Agile and architecture – the battle between speed and sustainabilityComparing different scaling Agile frameworksMeasuring Agile architecture Lessons learned from Snow in the Desert

This chapter focuses on the Agile Architecture focal point of the Agile Architect Lens, as shown in the following diagram:

Figure 2.1 – The Agile Architect Lens – focal point

Technical requirements

Additional materials such as posters and Agile architecture assessment parameters referenced in this chapter are available to download from https://github.com/PacktPublishing/Becoming-an-Agile-Software-Architect/tree/master/Chapter2.

The journey leading to Agile software development

Interestingly, unlike many other natural evolutions, software development has gone through many forms, shapes, and, notably, cycles over the years. One of the significant parameters in software development evolution has been the risk of failure. The likelihood of risk dramatically reduced as a result of this evolution.

Software development is a young discipline, with only just under seven decades of legacy. Grady Booch, Chief Scientist at IBM, in his speech History of Software Engineering, at the Association for Computing Machinery, observed that the first mention of software was by John Turkey back in 1952.

The Agile method of software development has become increasingly popular in the last decade, but the traces of Agile development are deeply rooted in history. The paper Iterative and Incremental Development: A Brief History published by IEEE highlighted the existence of Agile development even before that. The article stated that, back in the 1960s, NASA's Project Mercury ran with an iterative approach of very short, half-day cycles and used test-driven development practices. The more interesting observations came from the Software Engineering report at a conference sponsored by the NATO science committee held back in 1968. Kinslow, a computer systems consultant, articulated very well in this paper that the software design process is an iterative one. Ross, another attendee of the conference from MIT, also highlighted the deadly symptom of traditional software development as first specifying what you are going to do and then doing it. Both Kinslow and Ross were pointing toward what is today called the Agile software development approach.

Most of the software development during the 1960s followed the waterfall approach, and the earlier statements came from frustration at the failures and challenges of waterfall-style software development. Sequentially staged software development was first proposed by Dr. Winston W. Royce back in the 1970s, which appeared in an IEEE publication, Managing Development of Large Software Systems. Later, the term "waterfall" was introduced by Bell and Thayer in the second International Conference on Software Engineering referring to Royce's approach. Royce, in his paper, explained a step-by-step approach, as shown in the following diagram, as a better way of software development:

Figure 2.2 – Sequential approach to software development

The most interesting part is that Royce called out the risk of this sequential step-by-step approach in the same paper, highlighting that the testing phase only occurs at the end of the development cycle.

As Royce expected and clearly articulated, the biggest challenge in the waterfall model of software development is the risk associated with late validation. As shown in the following diagram, the risk is much higher in the waterfall approach since we confirm the design too early in the development cycle, with almost no means for immediate feedback. When we start receiving feedback, it is too late as the cost of rework is going to be unimaginably huge:

Figure 2.3 – Risk of not meeting customer expectations

The most attractive aspect of iterative software development is managing potential risks and, therefore, better management of the cost of delays and reworks.

Even though there were many traces of Agile software development before the 1990s, Agile software development practices eventually became popular as an aftereffect of the failures of many software development projects in the 1990s. As a result of this hue and cry, an alternate set of lightweight methods emerged, such as Scrum, Extreme Programming (XP), Feature Driven Design (FDD), and a few more. Later, in 2001, a group of Agile software development practitioners joined together and developed a manifesto for Agile software development, documenting a set of values and principles.

What is Agile software development?

The most important document for Agile software development is the manifesto for Agile software development. The manifesto is positioned as uncovering better ways of developing software, but it does not prescribe any approach or process for Agile software development.

Agile software development is a software development practice based on a set of values and principles defined in the manifesto for Agile software development. Manifesto statements are pretty simple, abstract, and general, for the right reasons. But the marketing buzz around it led to many interpretations and complicated explanations.

One of the most radically simple models to explain Agile is The Heart of Agile, proposed by Alistair Cockburn, one of the signatories of the manifesto for Agile software development. He summarized Agile with four imperatives: Collaborate, Deliver, Reflect, and Improve, as shown on the left-hand side of the following diagram:

Figure 2.4 – The Heart of Agile and PDCA cycles

The meaning of collaborate, deliver, reflect, and improve is as follows:

Collaborate: Collaboration in this context is a mix of trust, culture, motivation, and the act of collaboration between people. Deliver: Deliver means delivering business value with a continuous, uninterrupted flow, that essentially helps to earn more revenue and learn things such as market potential, among others. Reflect: Reflect is about collecting and examining subjective data from people and processes, and objective information from the ecosystem of the business and its customers. Improve: Improve is about solving problems based on past learnings and experimenting, to explore further and acquire new knowledge.

The Heart of Agile is pretty much similar to the Plan-Do-Check-Act (PDCA) cycle applied in the context of Agile, as shown on the right-hand side of the preceding diagram. PDCA was first introduced by Walter Shewhart in his book Statistical Method from the Viewpoint of Quality Control back in 1939. It is also called the Shewhart cycle and the Deming cycle. PDCA is a continual process improvement model for organizations to plan actions, execute those actions, study deviations from the plan, and act on what has been learned from the study.

In reality, Agile software development means two key things:

Decompose problems in to small bite-size chunks so that PDCA can be applied repeatedly.Continually optimize the PDCA cycle time, without compromising purpose and quality.

We see these two key Agile philosophies in the following diagram:

Figure 2.5 – Two key concepts of Agile: small batch size and cycle time

The PDCA cycle time can be improved dramatically by continuously monitoring and optimizing activities across different steps and, more importantly, diligently applying Lean principles.

Is Lean different from Agile?

We always use the words Lean and Agile almost interchangeably. But what is Lean? How is it related to Agile?

Back in the 1990s, when many large software projects started failing, many industry gurus looked at the reasons for failure and opportunities to improve. Up until then, software engineering was always compared with the construction industry as they both used engineering practices. However, there is a substantial difference between these two. In the construction industry, there is always a detailed plan. Once the architect signs off the plan, it goes for approval from the authorities before engineers start construction. Unfortunately, we had replicated this in software engineering with a stage-gate process with audits and approvals.

Finally, when things fell apart, the hunt for best practices extended to adjacent industries. The manufacturing industry was doing quite well with Lean manufacturing principles. The Toyota Production System (TPS) was leading this space, focusing considerably on enumerating and eradicating wastage, shortening lead times, relentless improvements, moving work to the fixed capacity workforce, using just-in-time production, and delaying critical decisions.

The following diagram shows key parts of the Lean manufacturing process, with a focus on minimizing lead time from order to cash:

Figure 2.6 – Concepts of Lean manufacturing

As you will have noticed, Agile and Lean have many characteristics in common. The reason is, many Agile software development frameworks got inspired by the Lean thinking of manufacturing and adopted many of those practices into software development even before the manifesto for Agile software development was constructed in 2001.

Agile software development has the best of both worlds: all Lean principles are also reflected in Agile software development practices. Effectively, many of those principles help to reduce the PDCA cycle time in Agile software development, which brings enormous benefits to software development, businesses, and customers.

Five benefits of Agile software development

There are many benefits when adopting Agile software development at scale. Some of the benefits, such as a set of motivated individuals and an empowered team, are considered to be indirect benefits, essentially contributing to one of the direct benefits.

The top five benefits of Agile-based software development are as follows:

Reduced risk: The iterative development, faster release cycles, visible and transparent progression, and short feedback loops increase predictability, reduce risk, and eliminate unnecessary cost. Better productivity: Agile methodologies focus on continuous sustainable delivery of business values to maximize customer satisfaction by removing forces that impose delays in the software delivery flow. It also helps the organization to organize teams systematically without burdening them with management overheads. Improved quality: The people-centric approach, with a high degree of collaboration together with fast learning cycles of inspect, feedback, and adapt mechanisms improve the quality and maintainability of products. Shorter cycle time: The quick turnaround nature of Agile development helps businesses to respond quickly ahead of the competition. The test and learn approach in Agile assists faster and cheaper innovation cycles. Business agility: The ability to smoothly manage frequent priority changes in business as a mechanism by design, dramatically improves business agility to respond rapidly to changes in trading conditions and market dynamics.

As we see here, Agile brings many benefits, and Agile architecture inherits principles and practices from Agile software delivery to develop architecture as an integrated software development activity.

Agile software development is most important for organizations to successfully swim through the increasingly difficult environment. We will explore Agile software development and how architecture fits into the flow of work in the next section.

Agile development and traditional architecture – an oxymoron?

The perception of paradox came out of contrasting approaches adopted by Agile software development and the waterfall model. While Agile software developments are heavily rooted in iterative development, traditional developments follow rigid sequential steps.

To comprehend the cause of this perception, it is imperative to define what we mean by software architecture and its purpose.

Architecture as a continuum

Architecture is often misunderstood and misused in the software industry. There are many architecture definitions referenced in the document What is Your Definition of Architecture, published by the Software Engineering Institute at Carnegie Mellon University.

The very first definition of architecture came from Fred Brooks, writing about computer architecture in his book, Planning a Computer System: Project Stretch 1962, inspired by structural engineering:

"Computer architecture, like other architecture, is the art of determining the needs of the user of a structure and then designing to meet those needs as effectively as possible within economic and technological constraints."

A more concise definition of software architecture is from IEEE-1471-2000:

"Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution."

The IEEE definition is the most logical and straightforward definition of architecture and has been followed widely ever since it was defined.

The Rational Unified Process (RUP) provides a more elaborate definition of architecture as follows:

A set of significant decisions about the organization of a software system, the selection of the structural elements and their interfaces by which the system is composed together with their behavior as specified in the collaboration among those elements, the composition of these elements into progressively larger subsystems, the architectural style that guides this organization, these elements, and their interfaces, their collaborations, and their composition. Software architecture is not only concerned with structure and behavior, but also with usage, functionality, performance, resilience, reuse, comprehensibility, economic and technological constraints and tradeoffs, and aesthetics.

– RUP (Kruchten, 1998)

Martin Fowler, in the IEEE journal Who Needs an Architect?, defined architecture as important stuff that is expensive to change, whereas Grady Booch called it significant decisions, where the cost of change measures significance.

There is no debate on these definitions of architecture itself in the context of Agile software development methodologies. However, any architecture definition that treats architecture as "the foundation", "hard to change", "things that you need to know upfront", or "get it right early" are up for debate and no longer stack up in Agile development.

The purpose of architecture is to shape a solution that meets near-term customer needs, and address any technical risks that could hinder operations. A simple definition of software architecture in the context of Agile development is: architecture is a series of irreversible decisions for the development of a sustainable technology solution. The keywords in this definition are the following:

Irreversible: Irreversible decisions are significant decisions that are hard to change without incurring considerable cost. For example, you decided to build an order management system on an off-the-shelf ERP. It is almost impossible to reverse that decision halfway through without an expensive change.Sustainable: Ability to maintain the state of the solution beyond immediate needs with an optimal run cost. Technology solution: A working software component or a hardware component or, in most scenarios, a mix of both, consisting of one or more products or services.Decisions: Decisions are related to defining the right solution for a given purpose, and the organization of the solution, context, elements, and relationships between various elements.

The preceding architecture definition from RUP touched on behavior. Behavior could be anything from the messaging style of an interface, such as synchronous or asynchronous, to the flow of data represented with a dynamic interaction diagram. This often leads to another question: whether the behavior is a question of architecture or design.

The distinction between architecture and design is not always clear. Simon Brown, who introduced C4 models, observed that the "significant decisions are architecture, and everything else is design." Grady Booch's viewpoint is that "all architecture is design, but not all design is architecture." The definition from Tom Hollander, Microsoft evangelist, is more appropriate in the Agile software development context. Hollander described architecture as "a continuum where coarse-grained is architecture and fine-grained is design."

Architecture can be represented as a continuum, as shown in the following diagram:

Figure 2.7 – Architecture represented as a continuum

As shown, there is no real cut-off point between architecture and design or high-level and low-level. The number of people who touch architecture also increases as we go from left to right in the continuum. However, in the traditional waterfall, both architecture and design activities are performed as part of the upfront design stage.

Traditional development methods frontload architecture efforts

Traditional architecture is an upfront activity that is deterministic and finite. In traditional projects, architects had the time and luxury of clearly knowing the requirements at the very early phase of the project to define the architecture upfront. This approach commits technologies and design decisions too early, which are unvalidated until close to going live. Often this is done without understanding the purpose and the business drivers, and, more importantly, draws unnecessary attention to many imaginary requirements. Neal Ford from ThoughtWorks observed that "spending too much time upfront won't guarantee that you will know everything that you need to know about the system to future proof architecture."

The following diagram shows the timelines for architecture decisions in a typical waterfall project as well as in Agile software development:

Figure 2.8 – Architecture decision timelines in waterfall and Agile

As shown in the diagram, the Agile architecture approach flattens the architecture efforts over time compared to the upfront decisions in traditional development.

Big Upfront Design (BUFD) is treated as one of the anti-patterns when developing architecture in Agile software development projects. In iterative development, upfront architecture is unfeasible as the requirements are not known until we are close to the end of the iteration. Architecturally significant requirements may surface even at a later stage, which in effect impact the resiliency of the architecture and cause significant rework.

Overengineering is a symptom of upfront designs. In the early 2000s, the abstract database factory was a common pattern, as many architects believed that they needed to cater to multiple databases such as MySQL, Oracle, and so on, but in reality, the need does not exist. Overengineering makes the system significantly complex and expensive to develop, operate, and evolve.

Before Agile, architecture was treated as a solid foundation, and therefore the notion of architecture decisions had to be taken upfront in the project life cycle. As a result, architects used to spend a lot of time creating abstractions. The notion of an architecture foundation came from building physical architecture where foundations are critical and have to be strong. Alistair Cockburn observed that "some technologies exist today to replace the foundation of buildings without even moving the tenants out."

In my own experience from one of the largest successful home-grown projects, back in early 2000, we spent a good initial 6 months for requirements analysis. During this period, as architects, we spent time and effort in building an architecture foundation and a development framework. We had to do that as EJB was underperforming, and Spring was just in its nascent stage. We committed all decisions upfront, including the selection of technologies. It was successful in the end, mainly because of the spirited mindset of the team. Two years down the line, when we deployed in production, the technologies were already obsolete, which severely impacted the longevity of the solution.

One of the key lessons from this project resonates with Dietzler's law, which we see in the following diagram:

Figure 2.9 – Dietzler's law applied to abstraction

Creating an abstraction upfront will make the developer's life easier for 80% of the customer requirements, 10% is hard to achieve, and the last unanticipated 10% is impossible to achieve. However, customers always want 100%. The last 10% will break the abstraction and contribute to architecture complexities.

Developing architecture in Agile projects is more demanding than in traditional waterfall projects and therefore needs special attention and treatment. In applying the lessons learned from the failed story of Kodak, where it failed to embrace innovations, teams must fearlessly adopt Agile architecture.

We have learned that traditional architecture is not suitable for Agile software development. We will explore the importance of architecture in Agile and the reasons why it is termed the foundation of Agile software development in the following sections.

Agile architecture – architecting with Agile practices

Agile architecture is a combination of doing architecture using Agile methodologies, as well as building resilient architecture to meet evolving customer needs. Agile architecture treats architecture as a continuous flow of decisions delivering sustainable solutions that maximize business value. It is a disciplined way of iteratively developing and managing architecture in line with the values, principles, and practices of Agile software development.

The art of Agile architecture is described as follows:

Anticipating, discovering, and stacking architecture decisions with an overarching intentSystematically solving problems by considering optionsCareful prioritization to minimize the cost of delayContinually evolving and improving solutionsTransparently evaluating and addressing possible technical risksMaintaining velocity from demand to delivery

Agile architecture avoids large upfront design activities and provides guardrails for feature teams to avoid offshoot from the right directions without slowing down delivery flow.

Balancing speed and sustainability

Agile means quick and easy, whereas good architecture provides sustainability; Agile architecture balances speed and sustainability. There is a notion that Agile is all about fast and cheap, and everything else is secondary. Often development teams want to go just too fast. But developing sustainable solutions needs checks and balances. Fred Brooks made an interesting observation in his book The Mythical Man-Month:

"Good cooking takes time; some tasks cannot be hurried without spoiling the result."

Kevlin Henney, the author of the book 97 Things Every Programmer Should Know, shared the analogy of a speeding car. Driving a car at 150 km an hour is always great, but only if in the right direction. Otherwise, it creates more damage than benefits as backtracking will consume time and money. Kevlin Henney also shared another great metaphor of the four forces that keep aircraft in the air and compared them with Agile software development. A modified version of this metaphor is shown in the following diagram:

Figure 2.10 – Speed and sustainability – aircraft metaphor

Traveling faster without balancing sustainability will create more tech debts, which eventually drag the project back. This is where sustainability, and hence architecture, is immensely important. The architecture keeps the required checks and balances to ensure delivery is heading in the right direction by continually managing technical debts and preparing the architecture for the next set of iterations.

In Agile development, BUFD is not acceptable, but at the same time, having no upfront architecture is also not the right thing to do. Simon Brown, in his book Software Architecture for Developers, quoted Dave Thomas's observation that "the big upfront design is dumb but no design upfront is even dumber."

Developing a little architecture upfront, but how much?

While big upfront architecture is an anti-pattern for Agile architecture, it is inevitable to have some elements of architecture upfront. In earlier days, an example of such an upfront decision was the technology stack and programming language. However, with the evolution of microservices, this is no longer an issue.

How much architecture upfront depends on the nature, size, and complexity of the project. In larger and critical projects, not doing enough architecture upfront leads to higher exposure to risk and may lead to failure. Philippe Kruchten observed this in the IEEE paper Software architecture and Agile software development: a clash of two cultures? as ignoring the architecture focus for certain classes of systems will lead projects to hit a wall and collapse.

In addition to nature, size, and complexity, there are several other forces proposed by Michael Waterman, James Noble, and George Allan in the IEEE paper How Much Up-Front? A Grounded Theory of Agile Architecture. The paper emphasizes the importance of considering technical and environmental factors such as the organizational context and domain-related parameters, as well as social factors such as the experience and skills of the architects.

When there are multiple solution options, the Value, Cost, Risk (VRC) model can calculate the upfront design score. The one that is highest is the solution with the right upfront design. The formula is shown in the following diagram:

Figure 2.11 – VRC model for calculating the amount of upfront architecture

A good metaphor to explain how much upfront design to have is building an airport runway. Usually, massive investments are directly proportional to revenue generation. In the case of airports, the expansion will by and large depend on the growth in passenger volumes. This metaphor is shown in the following diagram:

Figure 2.12 – Extending the runway based on immediate needs

Let's say the initial plan for the first year is to only handle Cessna-type aircraft, which only need a runway length of 2,000 feet. There is no point in building an 8,000-foot runway upfront as it is not going to add any business value. In the second year, the airport wants to handle narrow-body aircraft. This means we need to widen and lengthen the runway to 7,000 feet. We do this incrementally while operations are in progress. In the third year, the airport wants to accept wide-body aircraft. This means we will have to further widen and extend the runway to 8,000 feet. Again, runway upgrades will be done incrementally while aircraft movements are in progress. In this metaphor, we need to have an architecture intent to build an 8,000-foot runway, which can accept wide-body aircraft over 3 years, but we build incrementally based on what the airport needs at each point, which balances the present and the future.

Simon Brown summarized the elements of upfront design quite nicely in his article Contextualizing just enough up front design, published at codingthearchitecture.com. He observed that upfront architecture brings clarity on the big picture, how significant elements fit together, and identified the key risks to be mitigated, such as quality attributes, environment, and so on.

A disciplined approach, following a set of principles for developing Agile architecture, helps to bring agility without leading to heavy rework. We will discuss this in the next section.

Agile architecture principles

Agile architecture revolves around five core principles. Culture is one of the most important factors in Agile architecture and is considered as embedded in every principle.

The following diagram captures the five key Agile architecture principles:

Figure 2.13 – Agile architecture principles

These five principles act as guiding forces for any successful transformation to Agile architecture. As we know, Agile software development practice does not prescribe anything, so the implementations can take different shapes.

Architecture needs a collective effort

Architecture development needs collective effort from all team members. One of the principles from the manifesto for Agile software development is "the best architectures, requirements, and design emerge from self-organizing teams." This also resonates with the Heart of Agile, where Collaborate is one of the keywords. Agile architecture moves away from prescriptive to collaborative architecture development with no more throwing architecture over the fence to developers.

Three key aspects around this principle are as follows:

Figure 2.14 – Agile collaboration triangle

Let's understand each of these principles:

Collective intelligence: In Agile architecture, architecture is collaboratively developed by all team members. The collective intelligence of all the team members is far better than an individual's knowledge and intelligence. Collective ownership: Collaborative development brings collective ownership. The nature of human beings is they take better care of things that they own. The team strives for success for those decisions taken collectively by them. Collective knowledge: Collaboratively built solutions eliminate the need for extensive knowledge sharing, documentation, and communication. This also improves the resiliency of the team in case they lose an individual's service.

This principle increases the cohesiveness of the team, which results in better quality, reduced maintenance cost, faster delivery cycles, and reduced risk of losing knowledge.

Architecture is continuous

If architecture is a continuum, then architecture development is a continuous activity. One of the principles of the manifesto for Agile software development emphasizes "customer satisfaction through early and continuous delivery of valuable software."

A continuous architecture needs an intent called intentional architecture and an incremental evolution called emergent design, as shown in the following diagram:

Figure 2.15 – Intentional architecture and emergent design

The concepts of intentional architecture and emergent design are as follows:

Intentional architecture: Intentional architecture defines the big picture and shared understanding of the purpose, vision, contextual guardrails, and guidelines that the development teams must know to protect, align, and balance the customer's and organization's interests. It also provides economic conditions for trade-offs and sets shared goals when multiple teams are working on a solution. Emergent design: With intentional architecture in place, development teams evolve the design of the solution continuously over several iterations to meet the immediate needs of the customer. The emergent design is a natural outcome of the response to new customer demands, as well as addressing findings coming from constant refactoring efforts.

The bridge girder erection machine is a great real-world metaphor for intentional architecture and emergent design. The working principle for the bridge girders is that pillars are already developed, and girders are built as the machine moves forward. The pillars represent intentional architecture, and girders represent the emergent design.