Event-Driven Architecture in Golang E-Book

Michael Stack

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 435

Veröffentlichungsjahr: 2022

Beschreibung

Event-driven architecture in Golang is an approach used to develop applications that shares state changes asynchronously, internally, and externally using messages. EDA applications are better suited at handling situations that need to scale up quickly and the chances of individual component failures are less likely to bring your system crashing down. This is why EDA is a great thing to learn and this book is designed to get you started with the help of step-by-step explanations of essential concepts, practical examples, and more.
You’ll begin building event-driven microservices, including patterns to handle data consistency and resiliency. Not only will you learn the patterns behind event-driven microservices but also how to communicate using asynchronous messaging with event streams. You’ll then build an application made of several microservices that communicates using both choreographed and orchestrated messaging.
By the end of this book, you’ll be able to build and deploy your own event-driven microservices using asynchronous communication.

Leseprobe

Event-Driven Architecture in Golang

Building complex systems with asynchronicity and eventual consistency

Michael Stack

BIRMINGHAM—MUMBAI

Event-Driven Architecture in Golang

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Gebin George

Publishing Product Manager: Gebin George

Senior Editor: Rounak Kulkarni

Technical Editor: Pradeep Sahu

Copy Editor: Safis Editing

Project Coordinator: Manisha Singh

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Prashant Ghare

Developer Relations Marketing Executive: Sonakshi Bubbar

Business Development Executive: Bhanu Rangani

First published: November 2022

Production reference: 1281022

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80323-801-2

www.packt.com

To my father, William Stack, for introducing me to the world of computers and instilling in me a passion for technology. To my wife, Kate, for enduring my long hours and weekends locked away working on the book. To my son, Malcolm, most of all, since I doubt I could have finished without his unceasing support, drawings, messages, and words of encouragement.

– Michael Stack

Contributors

About the author

Michael Stack is a solutions architect who has over 20 years of expertise in the software development industry. Throughout the course of his career, he has developed a variety of applications, including company blogs and intranets, festival ticketing systems, multiplayer games, and national park management software. Currently, his principal focus is on the development of microservices and other distributed applications. He is passionate about using Go and has been doing so for the better part of a decade for both personal and professional projects.

I would like to thank everyone who supported and encouraged me, especially my wife, Kate, my father, my son, and Packt.

About the reviewers

Dimas Yudha Prawira is a father, engineer, public speaker, tech community leader, and tech organizer. He has 18 years of experience with software engineering and has worked for various companies, including PT Telekomunikasi Indonesia (Telkom), Kudo x Grab, OVO, and RCTI+. He loves the Go programming language, Java, and talking about code. He is most interested in microservices, SOA, APIs, software architecture, embedded systems, and more.

When he is not working, he uses his time to read books, watch movies, or just play with his family.

Samantha Coyle is a Go backend engineer, speaker, and mentor with a love for all things Go and open source. She spends her days developing Go microservices, bringing in new features, observability, improved testing, and best practices. She is a software engineer for the Internet of Things (IoT) group at Intel, where she enables healthcare solutions using Go and Open Source technology at the edge and has worked on computer vision-based smart city solutions and industrial applications. Samantha explores thought leadership avenues, including reviewing Go textbooks, speaking at GopherCon, Grace Hopper Conference, and Open Source Summit in 2022, attaining her CKAD certification, and volunteering to mentor early career professionals to grow as Go engineers.

Table of Contents

Preface

Part 1: Event-Driven Fundamentals

1

Introduction to Event-Driven Architectures

Technical requirements

An exchange of facts

Event notifications

Event-carried state transfer

Event sourcing

Core components

Wrap-up

The MallBots application

The pitch

Application services

API gateway services

Clients

A quick note about hexagons

Benefits of EDA

Resiliency

Agility

User experience (UX)

Analytics and auditing

Challenges of EDA

Eventual consistency

Dual writes

Distributed and asynchronous workflows

Debuggability

Getting it right

Summary

2

Supporting Patterns in Brief

Domain-driven design

DDD misconceptions

So, what is it all about then?

How is it useful for EDA?

Domain-centric architectures

An evolving solution

Hexagonal architecture applied

Testing

A rulebook, not a guidebook

Should you use domain-centric architectures?

How is it useful for EDA?

Command and Query Responsibility Segregation

The problem being solved

Applying CQRS

When to consider CQRS

CQRS and event sourcing

Task-based UI

Application architectures

Monolithic architecture

Microservices

Recommendation for green field projects

Summary

Further reading

3

Design and Planning

Technical requirements

What are we building?

Finding answers with EventStorming

What is EventStorming?

Big Picture EventStorming

Design-level EventStorming

Understanding the business

Recording architectural decisions

Summary

Further reading

Part 2: Components of Event-Driven Architecture

4

Event Foundations

Technical requirements

A tour of MallBots

The responsibilities of the monolith

Module code organization

User interface

Running the monolith

A focus on event-driven integration and communication patterns

Taking a closer look at module integration

Using external data

Commanding external components

Types of events

Domain events

Event sourcing events

Integration events

Refactoring side effects with domain events

What about the modules not using DDD?

Summary

5

Tracking Changes with Event Sourcing

Technical requirements

What is event sourcing?

Understanding the difference between event streaming and event sourcing

Adding event sourcing to the monolith

Beyond basic events

Adding the event sourcing package

Using just enough CQRS

A group of stores is called a mall

A group of products is called a catalog

Taking note of the little things

Connecting the domain events with the read model

Recapping the CQRS changes

Aggregate event stream lifetimes

Taking periodic snapshots of the event stream

Using snapshots

Summary

6

Asynchronous Connections

Technical requirements

Asynchronous integration with messages

Integration with notification events

Integration with event-carried state transfer

Eventual consistency

Message-delivery guarantees

Idempotent message delivery

Ordered message delivery

Implementing messaging with NATS JetStream

The am package

The jetstream package

Making the Store Management module asynchronous

Modifying the monolith configuration

Updating the monolith application

Providing to the modules the JetStreamContext

Publishing messages from the Store Management module

Receiving messages in the Shopping Baskets module

Verifying we have good communication

Summary

7

Event-Carried State Transfer

Technical requirements

Refactoring to asynchronous communication

Store Management state transfer

Customer state transfer

Order processing state transfer

Payments state transfer

Documenting the asynchronous API

Adding a new order search module

Building read models from multiple sources

Creating a read model record

Summary

8

Message Workflows

Technical requirements

What is a distributed transaction?

Why do we need distributed transactions?

Comparing various methods of distributed transactions

The 2PC

The Saga

Implementing distributed transactions with Sagas

Adding support for the Command and Reply messages

Adding an SEC package

Converting the order creation process to use a Saga

Adding commands to the saga participants

Implementing the create order saga execution coordinator

Summary

9

Transactional Messaging

Technical requirements

Identifying problems faced by distributed applications

Identifying problems in synchronous applications

Identifying problems in asynchronous applications

Examining potential ways to address the problem

The singular write solution

Exploring transactional boundaries

How the implementation will work

The di package

Updating the Depot module with dependency containers

Using an Inbox and Outbox for messages

Implementing a messages inbox

Implementing a messages outbox

Summary

Part 3: Production Ready

10

Testing

Technical requirements

Coming up with a testing strategy

Unit tests

Integration tests

Contract tests

End-to-end tests

Testing the application and domain with unit tests

Table-driven testing

Creating and using test doubles in our tests

Testing dependencies with integration testing

Incorporating the dependencies into your tests

Running tests with more complex setups

Testing ProductCacheRepository

Breaking tests into groups

Testing component interactions with contract tests

Consumer expectations

Provider verifications

Not building any silos

Contract testing with Pact

REST consumer and provider example

Message consumer and provider example

Testing the application with end-to-end tests

Relationship with behavior-driven development

E2E test organization

Making executable specifications out of our features

What to test or not test

Summary

11

Deploying Applications to the Cloud

Technical requirements

Turning the modular monolith into microservices

Refactoring the monolith construct

Updating the composition root of each module

Making each module run as a service

Updates to the Dockerfile build processes

Updates to the Docker Compose file

Adding a reverse proxy to the compose environment

Fixing the gRPC connections

Installing the necessary DevOps tools

Installing every tool into a Docker container

Installing the tools into your local system

Using Terraform to configure an AWS environment

Preparing for the deployment

A look at the AWS resources we are deploying

Deploying the infrastructure

Viewing the Kubernetes environment

Deploying the application to AWS with Terraform

Getting to know the application resources to be deployed

Deploying the application

Tearing down the application and infrastructure

Summary

12

Monitoring and Observability

Technical requirements

What are monitoring and observability?

The three pillars of observability

How tracing works

Instrumenting the application with OpenTelemetry and Prometheus

Adding distributed tracing to the application

Adding metrics to the application

Viewing the monitoring data

Summary

Index

Other Books You May Enjoy

Preface

Companies are adopting event-driven architecture (EDA) as their web applications grow in size and complexity. Applications that communicate using events are easier to develop and scale. Adding or developing your application around real-time interactions becomes easier with EDA.

Direct point-to-point communication between microservices inevitably leads to the development of a distributed monolith, which is just a monolith with extra and unnecessary complexity. EDA is an architecture that helps organizations to decouple microservices and avoid developing another distributed monolith.

Choosing a new architecture for your next application or deciding to refactor an existing one can be fraught with known and unknown challenges. It is my intention and this book’s goal to provide you with enough examples and knowledge to give you a great head start should you decide to take the development of an EDA.

In this book, we will discuss and cover EDA concepts and related topics with the help of a small modular monolith demonstration application. We will use this application to take a journey through the concepts and topics to convert the synchronous mechanisms used by the application into asynchronous communication mechanisms.

Who this book is for

This architecture book is for developers working with microservices, or those architecting and designing new applications that will be built with microservices. Intermediate-level knowledge of Go is required to make the most of the examples and concepts in this book. Developers with a background in any programming language and experience working with microservices should still find the concepts and explanations useful.

What this book covers

Chapter 1, Introduction to Event-Driven Architectures, introduces EDA.

Chapter 2, Supporting Patterns in Brief, covers helpful patterns such as domain-driven design, domain-centric architectures, and application architectures.

Chapter 3, Design and Planning, explores the ways to discover the capabilities and features of an application using EventStorming and other methods.

Chapter 4, Event Foundations, introduces the Mallbots modular monolith application and domain events.

Chapter 5, Tracking Changes with Event Sourcing, introduces event sourcing and leads you through the development of event-sourced aggregates.

Chapter 6, Asynchronous Connections, covers adding asynchronous communication using event messages.

Chapter 7, Event-Carried State Transfer, expands on the use of message-based communication between components.

Chapter 8, Message Workflows, covers the concept of distributed transactions and introduces orchestrated sagas.

Chapter 9, Transactional Messaging, explores the use of message inboxes and outboxes to reduce data loss.

Chapter 10, Testing, discusses the concept of a testing strategy and leads you through testing an event-driven application.

Chapter 11, Deploying Applications to the Cloud, covers the use of infrastructure as code and deploying an application as microservices.

Chapter 12, Monitoring and Observability, discusses how to monitor a distributed application and make it observable with logging, metrics, and distributed tracing.

To get the most out of this book

This book is written with the expectation that you can execute the demonstration application to understand and view the code changes that have been made in each chapter as the application is refactored. To get the most out of the book, it is recommended you read the chapters in order, as the chapters will reference code that has been modified in the previous chapter.

Software/hardware covered in the book

Operating system requirements

Go 1.18+

Windows, macOS, or Linux

Docker 20.10.x

Windows, macOS, or Linux

NATS 2.4

Windows, macOS, or Linux

Most of the development for this book was done in Windows 10, but the code was tested to run in Windows Subsystem for Linux 2 (WSL 2) in Ubuntu 20.04 and tested to run on a Mac. You are expected to run the application and its dependencies within a Docker compose environment. Instructions to use Docker are given wherever possible to minimize installing new software on your machine.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

You can follow the author on GitHub (https://github.com/stackus) or make a connection with them on LinkedIn (https://www.linkedin.com/in/stackmichael).

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Event-Driven-Architecture-in-Golang. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/qgf1O.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “If all the participants have responded positively, then the coordinator will send a COMMIT message to all of the participants and the distributed transaction will be complete.”

A block of code is set as follows:

BEGIN; -- execute queries, updates, inserts, deletes … PREPARE TRANSACTION 'bfa1c57a-d99d-4d74-87a9-3aaabcc754ee';

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

func NewCommandHandlers(     app application.App,) ddd.CommandHandler         [ddd.Command] {     return commandHandlers{         app: app,

Any command-line input or output is written as follows:

--- PASS: TestApplication_AddItem (0.00s)

    --- PASS: TestApplication_AddItem/NoBasket (0.00s)

    --- PASS: TestApplication_AddItem/NoProduct (0.00s)

    --- PASS: TestApplication_AddItem/NoStore (0.00s)

    --- PASS: TestApplication_AddItem/SaveFailed (0.00s)

    --- PASS: TestApplication_AddItem/Success (0.00s)

PASS

Bold: Indicates a new term, an important word, or words that you see on screen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “The Customers module remains uncoupled from the Order Processing module because we do not have any explicit ties to the Order Processing module in this handler.”

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Event-Driven Architecture in Golang, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere? Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781803238012

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Event-Driven Fundamentals

This first part will provide an understanding of what event-driven architecture (EDA) is and the benefits of using it for your next application. We will also be introduced to the application we will be working with. This part will also cover some helpful patterns that can be helpful for the adoption and development of EDA. Then, it will introduce methods for planning an application using EventStorming.

This part consists of the following chapters:

Chapter 1, Introduction to Event-Driven ArchitectureChapter 2, Supporting Patterns in BriefChapter 3, Design and Planning

1

Introduction to Event-Driven Architectures

Event-driven architecture (EDA) is the foundational design of an application’s communication of state changes around an asynchronous exchange of messages called events. The architecture allows applications to be developed as a highly distributed and loosely coupled organization of components. Probably predominantly, the most well-known arrangement of components today is the microservices architecture for applications.

Our world is made up of events—they’re happening everywhere around us. A simple act of waking up in the morning becomes an event the instant it occurs. The same goes for the act of purchasing a book. Whether or not it was recorded that way in some database, somewhere, it was considered an event. Since it has occurred, several other operations might have sprung from it.

Just as companies looked at microservices a decade ago to address issues such as web-scale, EDA is gaining in interest and proponents of continuing that journey to help with global-scale.

It is my goal in this chapter to introduce you to the concepts and components of EDA and its applications that we will be using to demonstrate what EDA has to offer. We’ll also be taking a grounded look at the benefits and reasons to use EDA and the challenges you’re likely to encounter when starting a new greenfield project or adding select concepts and components to an existing project.

Whether you’re looking to start a new project with an event-driven approach or looking to break up a monolithic application into modules or further into microservices, this book will give you the information and patterns necessary to implement EDA where you need it.

In this chapter, we’re going to cover the following main topics:

An exchange of factsThe MallBots applicationBenefits of EDAChallenges of EDA

Technical requirements

We will be developing using Go and using Docker to run our application within containers. Visit the following to locate installers for your operating system:

Go installers can be found at https://go.dev/doc/installDocker installers can be found at https://docs.docker.com/desktop/

Go 1.17 or higher is required to run the code from this book.

An exchange of facts

Three different uses or patterns exist that can be called EDA individually or altogether, as follows:

Event notificationsEvent-carried state transferEvent sourcing

In this book, we will be covering each of these patterns, going over their uses and both when to use them and when you might not.

Event notifications

Events can be used to notify something has occurred within your application. A notification event typically carries the absolute minimum state, perhaps even just the identifier (ID) of an entity or the exact time of the occurrence of their payload. Components that are notified of these events may take any action they deem necessary. Events might be recorded locally for auditing purposes, or the component may make calls back to the originating component to fetch additional relevant information about the event.

Let’s see an example of PaymentReceived as an event notification in Go, as follows:

type PaymentReceived struct {     PaymentID string }

Here is how that notification might be used:

Figure 1.1 – PaymentReceived as an event notification

Figure 1.1 shows the PaymentReceived notification being received by two different services. While ServiceA only needed to be notified of the event, ServiceB will require additional information and must make a call back to the Payments service to fetch it.

Event-carried state transfer

Event-carried state transfer is an asynchronous cousin to representational state transfer (REST). In contrast with REST’s on-demand pull model, event-carried state transfer is a push model where data changes are sent out to be consumed by any components that might be interested. The components may create their own local cached copies, negating any need to query the originating component to fetch any information to complete their work.

Let’s see an example of PaymentReceived as an event-carried state transfer, as follows:

type PaymentReceived struct {     PaymentID    string     CustomerID   string     OrderID      string     Amount       int }

In this example for event-carried state transfer, we’ve included some additional IDs and an amount collected, but more detail could be added to provide as much detail as possible, as illustrated in the following diagram:

Figure 1.2 – PaymentReceived as an event-carried state change

When the PaymentReceived event is sent with additional information, it changes how downstream services might react to it. We can see in Figure 1.2 that ServiceB no longer needs to call the Payments service because the event it has received already contains everything it requires.

Event sourcing

Instead of capturing changes as irreversible modifications to a single record, those changes are stored as events. These changes or streams of events can be read and processed to recreate the final state of an entity when it is needed again.

When we use event sourcing, we store the events in an event store rather than communicating them with other services, as illustrated in the following diagram:

Figure 1.3 – Payment data recorded using event sourcing

In Figure 1.3, we see the entire history of our data is kept as individual entries in the event store. When we need to work with a payment in the application, we would read all the entries associated with that record and then perform a left fold of the entries to recreate the final state.

Core components

You will observe that four components are found at the center of all event patterns, as illustrated in the following diagram:

Figure 1.4 – Event, queue, producer, and consumer

Event

At the heart of EDA is the event. In EDA terms, it is an occurrence that has happened in the application. The event itself is in the past and it is an immutable fact. Some examples of events are customers signing up for your services, payments being received for orders, or failed authentication attempts for an account.

With EDA, the consumers of these events may know nothing about what caused the production of these events or have any relationship or connection with them, but only with the event itself.

In most languages, events are simple value objects that contain state. An event is equal to another if all the attributes are the same. In Go, we would represent an event with a simple struct, such as this one for PaymentReceived:

type PaymentReceived struct {     PaymentID string     OrderID   string     Amount    int }

Events should carry enough data to be useful in capturing the change in the application state that they’re meant to communicate. In the preceding example, we might expect that this event is associated with some payment, and the specific payment is identified by the queue name or as some metadata passed along with the event instead of the PaymentID field in the body of the event being necessary.

The amount of information required to include in an event’s payload matters to all events, the event notification, the event-carried state transfer, and for the changes recorded with event sourcing.

Queues

Queues are referred to by a variety of terms, including bus, channel, stream, topic, and others. The exact term given to a queue will depend on its use, purpose, and sometimes vendor. Because events are frequently—but not always—organized in a first-in, first-out (FIFO) fashion, I will refer to this component as a queue.

Message queues

The defining characteristic of a message queue is its lack of event retention. All events put into a message queue have a limited lifetime. After the events have been consumed or have expired, they are discarded.

You can see an example of a message queue in the following diagram:

Figure 1.5 – Message queue

A message queue is useful for simple publisher/subscriber (pub/sub) scenarios when the subscribers are actively running or can retrieve the events quickly enough.

Event streams

When you add event retention to a message queue, you get an event stream. This means consumers may now read event streams starting with the earliest event, from a point in the stream representing their last read position, or they can begin consuming new events as they are added. Unlike message queues, which will eventually return to their default empty state, an event stream will continue to grow indefinitely until events are removed by outside forces, such as being configured with a maximum stream length or archived based on their age.

The following diagram provides an example of an event stream:

Figure 1.6 – Event stream

When you need retention and the ability to replay events, an event stream should be used instead of a message queue.

Event stores

As the name implies, an event store is an append-only repository for events. Potentially millions of individual event streams will exist within an event store. Event stores provide optimistic concurrency controls to ensure that each event stream maintains strong consistency. In contrast to the last two queue examples, an event store is typically not used for message communication.

You can see an example of an event store in the following screenshot:

Figure 1.7 – Event store

Event stores are used in conjunction with event sourcing to track changes to entities. The top three rows of Figure 1.7 depict the event-sourcing example events from Figure 1.3.

Producers

When some state in the application has changed, the producer will publish an event representing the change into the appropriate queue. The producer may include additional metadata along with the event that is useful for tracking, performance, or monitoring. The producers of the events will publish it without knowing what the consumers might be listening to. It is essentially a fire-and-forget operation.

Consumers

Consumers subscribe to and read events from queues. Consumers can be organized into groups to share the load or be individuals reading all events as they are published. Consumers reading from streams may choose to read from the beginning of a stream, read new events from the time they started listening, or use a cursor to pick up from where they left the stream.

Wrap-up

Equipped with the types of events we will be using and the knowledge of the components of the patterns involved, let’s now look at how we’ll be using them to build our application.

The MallBots application

We’re going to be building a small application that simulates a retail experience coupled with some futuristic shopping robots. We will be building the backend services that power this application. A high-level view of the components involved is shown here:

Figure 1.8 – High-level view of the MallBots application components

The pitch

“We have developed incredible robots to save the time of people shopping at the mall. Customers will now have access to a kiosk that would facilitate the selection of items from available stores that customers do not wish to visit. After completing their selections, the customer is free to do other shopping or directly visit the depot and wait for their items to be brought in by robots. The customer may pay when they arrive at the depot or may choose to wait for all items to arrive before doing so. After both are done, the transaction is complete, and the customer takes their items and goes on their merry way.”

Application services

Starting with the four services—Orders, Stores, Payments, and Depot—on the right of Figure 1.8, we have the application services. These will all use events to communicate new states for triggers and notifications and will both publish them and subscribe to them. They will also have GRPCapplication programming interfaces (APIs) to support the API gateway layer.

API gateway services

The API gateway layer displayed down the center of Figure 1.8 will support a RESTful API for the customer kiosks, a management user interface (UI) with WebSocket subscriptions for the staff to use, and finally, a gRPC streams API for the robots. The API gateways are implemented as a demonstration of the Backend for Frontend(BFF) pattern.

The administrative BFF and the automation API gateways will create subscriptions to application events to allow delivery of state changes to clients. Note that we will not be developing API gateway services in this book.

Clients

Finally, on the left of Figure 1.8 are the expected clients, as outlined in more detail here:

Customer kiosks, placed near or at mall entrances for ease of useAn administrative web application for staff to manage the application data, process customer pickups, and take paymentShopper bot clients that perform autonomous shopping tasks for the busy customers

A quick note about hexagons

You’re going to be seeing a lot of hexagons in the diagrams of this book. The services in Figure 1.8 all have some combinations of synchronous and asynchronous communication or connections, and all are drawn as hexagons, as depicted in the following diagram:

Figure 1.9 – Hexagonal representation of a service

The API gateway and application services are all represented as hexagons with inputs (such as the API and event subscriptions, shown on the left) and the outputs (the database and event publications, on the right). This is a visual presentation of hexagonal architecture, and we will be talking more about that in Chapter 2, Supporting Patterns in Brief.

Benefits of EDA

An EDA brings several benefits to your application when compared to an application that uses only synchronous or point-to-point (P2P) communication patterns.

Resiliency

In a P2P connection as shown in the following diagram, the calling component, Orders, is dependent on the called component, Depot, being available. If the called component cannot process the operation in time, or if the called component has a fault, then that error will propagate back to the caller. Worse is a chain or tree of calls that end up with a fault somewhere far away from the original caller, causing the entire operation to fail.

If the Depot service is not responding or is failing to respond on time, then the Orders service may fail to pass on information regarding new orders:

Figure 1.10 – P2P communication

In an EDA application, the components have been loosely coupled and will be set up with an event broker between them, as shown in the following diagram. A crash in an event consumer will have no impact on the event producer. Likewise, other faults (internal to the consumer) that cause it to temporarily be unable to process events again have no impact:

Figure 1.11 – Brokered event communication

Considering the example case of the Depot service becoming overrun with work, causing it to get backed up, orders submitted by the Orders service will be processed, just a little slower. The Orders service will be unaffected and continue to take orders as they come in. Better still, if the Depot service is down entirely, then it may only cause a longer delay until it can be restarted or replaced, and the Orders service continues.

Agility

An event-driven application can be more agile in its development. Less coordination between teams is required when introducing new components to an application. The new feature team may drop in the new component without having to socialize any new API with any of the other teams.

The organization can more easily experiment with new features as an aside. A small team can stand up a new component without disrupting the work of other teams or the flow of existing processes. When the experiment is over, the team can just as easily remove the component from the application.

We can imagine that, at some point, an Analytics service could be introduced to the application. There are two ways this new service could be added. The first way is with a synchronous API (as shown in Figure 1.12) and the second is with an asynchronous event consumer (as shown in Figure 1.13).

When they choose to use the API, the team will need to coordinate with existing teams to potentially add new logic to capture data and new calls to their service. Completing this task will now require scheduling with one or more teams and will become dependent on them, as illustrated in the following diagram:

Figure 1.12 – New P2P service

Components that communicate using events make it easier for new components and processes to come online without requiring coordination with the teams in charge of existing components, as shown in the following diagram:

Figure 1.13 – New brokered event service

Now, when the Analytics service team has finished its work of picking which events to consume and captures the data that it needs, it can then add it to the application immediately.

If event streams are part of your EDA application, this also has the advantage of providing new components with a complete history of events to spin up with.

User experience (UX)

With Internet of Things (IoT) devices exploding in number and millions of people having phones in their hands, users expect to be notified of the latest news and events the instant they happen. An event-driven application is already sending updates for orders, shipment notifications, and more. The organization may extend this to users more easily than a traditional synchronous-first application might allow.

Analytics and auditing

Whether you’re using event notifications, event-carried state transfer, or event sourcing, you will have ample opportunity to plug in auditing for the small changes that occur in your system. Likewise, if you’re interested in building on analytics to your application to gather business intelligence (BI) for your marketing and product teams, often one or both are an afterthought, and in a traditional or non-EDA application, you may not have the data or can only recreate a partial picture.

Challenges of EDA

Adopting EDA patterns for your application brings along some challenges that must be overcome for the application to succeed.

Eventual consistency

Eventual consistency is a challenge for any distributed application. Changes in the application state may not be immediately available. Queries may produce stale results until the change has been fully recorded. An asynchronous application might have to deal with eventual consistency issues, but without a doubt, an event-driven application certainly will.

Dual writes

Not entirely a challenge of event-driven applications alone, dual write refers to any time you’re changing the application state in two or more places during an operation. For an event-driven application, this means we are making a change locally to a database, and then we’re publishing an event either about the event or the event itself. If the events we intend to publish never make it to the event broker, then our state changes cannot be shared, and post-operation operations will never happen.

For this challenge, we have a solution that will have us publish our events into the database alongside the rest of the changes to keep the state change atomic.

This allows a second record of to-be-published events to be created, and even adds additional resiliency on top of what we got from using an event broker between components, as illustrated in the following diagram:

Figure 1.14 – Outbox pattern

We will learn more about this challenge and solution when I introduce you to the Outbox pattern in Chapter 6, Asynchronous Connections.

Distributed and asynchronous workflows

Our third challenge involves performing complex workflows across components using events, making the workflow entirely asynchronous. When each component is coupled this way, we experience eventual consistency. Each component may not have the final state of the application when queried, but it will eventually.

This creates an issue for the UX and one for the collaboration of the components of the application involved with the operation. Each will need to be evaluated on its own to determine the correct solution for the problem.

UX

The asynchronous nature of the operation would obviously make it difficult to return a final result to the user, so the choice becomes how to handle this limitation. Solutions include but are not limited to fetching the result using polling on the client, delivering the result asynchronously using WebSockets, or creating the expectation the user should check later for the result.

Component collaboration

There are two patterns we can use to bring components together to manage workflows, as illustrated in the following diagram:

Figure 1.15 – Workflow choreography and orchestration

Choreography: The components each individually know about the work they must do, and which step comes nextOrchestration: The components know very little about their role and are called on to do their part by a centralized orchestrator

We will dive into the differences, some of the details to consider in choosing one over the other, and more in Chapter 8, Message Workflows.

Debuggability

Synchronous communication or P2P involves a caller and callee. This method of communication has the advantage of always knowing what was called and what made the call. We can include a request ID or some other unique ID (UID) that is passed on to each callee.

One of the disadvantages of EDA is being able to publish an event and not necessarily knowing if anything is consuming that event and if anything is done with it. This creates a challenge in tracing an operation across the application components.

We might see multiple operations unrelated to one another spring up from the same event. The process to trace back to the originating event or request becomes harder as a result. For an event-driven application, the solution is to expand on the solution used for P2P-only applications, and we will see crumbs of this solution throughout the book and discuss it in more detail in Chapter 12, Monitoring and Observability.

Testing the application using several forms of tests will be covered in Chapter 10, Testing.

Getting it right

It can be challenging for teams to think in terms of events and asynchronous interactions. Teams will need to look much more closely and know the application that they’re building better to see the small details that sometimes make up events. In Chapter 2, Supporting Patterns in Brief, we will look at some patterns that teams can use to break down the complexities of an application, and how to make managing and maintaining event-driven applications easier in the long run.

In Chapter 3, Design and Planning, we will cover tools that teams can use to break down an application into behaviors and the events associated with each one.

Big Ball of Mud with events

A Big Ball of Mud (BBoM) is an anti-pattern, where an application is haphazardly designed or planned. We can end up with one in our event-driven application just as easily with events as without and perhaps even more easily if we do not do a good job identifying behaviors and events.

Summary

In this chapter, we were introduced to EDA, the types of events, and the core components involved with each event pattern you would find in an event-driven application. I covered some of the advantages from which you could benefit with an event-driven approach, and I introduced the challenges that will be encountered with this pattern.

In the next chapter, we will cover a range of patterns that will be used in the development of the demonstration application and why we might find them useful in conjunction with the development of an event-driven application.

2

Supporting Patterns in Brief

There are a lot of software patterns we might use or come across in the development of an event-driven application. Event-driven architecture should not be the first tool you reach for in your toolbox.

We’ve been introduced to event-driven architectures, and now we’ll see the patterns that work together with EDA to support excellent event-driven application design and development. These helpful patterns may not always be successful but using them in the right places and in moderation will improve your production time and reduce your bug rates.

In this chapter, we’re going to cover the following main topics:

Domain-driven designDomain-centric architecturesCommand and Query Responsibility SegregationApplication architectures

Domain-driven design

Domain-driven design (DDD) is a very large and complex topic, with entire books devoted to the use and implementation of the many patterns and methodologies that are brought together. I won’t try to fit all of it into this chapter, much less this section, so we’ll be taking a high-level look at the key strategic patterns that are useful to us as we design and develop event-driven applications. As for the tactical patterns, we’ll be seeing examples of their use throughout the rest of the book.

Going deeper into DDD

For an in-depth look at DDD, I can recommend both Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans, as an original introduction to the topic, and Implementing Domain-Driven Design by Vaughn Vernon, for the expansion of the topic and a deeper dive into the strategic patterns of DDD. Finally, Patterns, Principles, and Practices of Domain-Driven Design by Scott Millett with Nick Tune rounds out the recommendations with its very deep and lengthy look at DDD.

DDD misconceptions

The philosophies, methodologies, and patterns of DDD are well-suited for the development of event-driven applications. Before getting into DDD, I would like to cover a couple of misconceptions about it that developers might hold.

Misconception one – DDD is a set of coding patterns

For most developers, their first exposure to DDD might be seeing an entity, value object, or some other pattern such as the repository that is being used in a code base they’ve worked on, or from some web tutorial covering a pattern or two. Regardless of the number of patterns they see, it is still an incomplete picture of what DDD is. Most DDD is never explicitly shown in the code, and a good amount of DDD comes into the picture before the first line is ever written.

Misconception two – DDD is enterprise-level or leads to overengineered applications

DDD prescribes no specific architecture to use, and it neither instructs you how to organize your code for any given programming language nor enforces any rule that you must use in every corner of your application. DDD does not force you or your team to utilize a specific architecture, pattern, or code structure; that is something you are doing. The strategic patterns will actually assist you in identifying areas of the problem domain where you should not need to devote a lot of development time and resources.

Both misconceptions are centered around the use and a perceived overuse of the tactical patterns of DDD. As developers, we’re technically minded people; we will search for a technical solution or a better way to do something when faced with a challenging or novel problem. What we’ve learned or used will find its way into our conversations when we include the names of the patterns. If all we seek out or share with others are the tactical patterns of DDD, then it’s inevitable that we will miss out on the design philosophies and strategic patterns, only to turn around to complain that DDD has doomed another project.

So, what is it all about then?

DDD is about modeling a complex business idea into software by developing a deep understanding of the problem domain. This understanding is then used to break up the problem into smaller, more manageable pieces. The two key patterns of DDD at play here are the ubiquitous language and bounded contexts.

Alignment and agreement on the goals

To find success with DDD, collaboration must exist between domain experts and developers. There should be meetings where business ideas and concepts are sketched and diagrammed to be gone over from top to bottom and thoroughly discussed. The results of these discussions are then modeled and discussed further to weed out any incorrect understanding of implicit details.

This is not a process you do once before writing any code. Complex systems are living entities in a way, and they change and evolve. When new features are being considered, the same people should meet to discuss how these will be added to the domain model.

Speaking the same language

When domain experts come together with developers, discussions could fall apart if the parties involved cannot come to an understanding of a concept by having different ideas about what is being said or read. The Ubiquitous Language (UL) principle requires every domain-specific term to have a single meaning within a bounded context. By using a shared language, a better understanding of the domain can flourish. The domain experts have their jargon and the developers theirs. It is preferable to use the terms spoken by the domain experts, and it is these terms that will be used to name and describe the domain models.

This is a core principle of DDD and a very important one too, but it doesn’t come easy. Words that should be simple and have an obvious meaning may suddenly appear to have lost all meaning during discussions. Words may begin to develop a depth, which should highlight to everyone involved the importance of developing a UL and using it everywhere and always.

To hammer the point home, use the UL everywhere in code. It should drive the names of your function names, the structs, the variables, and the processes that you develop. When you sign off on the completion of some task or are given a bug to fix, the UL should always be used. This keeps the UL aligned across an organization.

When the UL is being spoken but confusion starts to appear, it could be a sign that the domain model is undergoing an evolution, and it might be a good time to have a meeting with the domain experts and developers again.

Tackling the complexity

The complexity of the problem domain can be reduced by breaking the domain into subdomains so that we’re dealing with more manageable chunks of the problem. Each new domain we identify falls into one of three types:

Core domains: Critical components of the application that are unique or provide a competitive advantage to the business. These get the most focus, the most money, and the best developers. A core domain is not always obvious and can evolve or change with the business.Supporting domains: The utility components that provide functionality that supports the core business. You might consider using an off-the-shelf solution if what is being provided and developed by a team is not specific enough to the business.Generic domains: Components that are unrelated to the core business but necessary for it to function. Email, payment processing, reporting, and other common commodity solutions fall into this domain type. It wouldn’t make sense to devote teams to develop this functionality when so many solutions exist.

As a business changes in response to competition or other factors, it is possible over time for the type associated with a domain to change or for the domain to split into two or more new domains.

Using a core domain chart to chart the business differentiation and model complexity for each domain in our MallBots application, we end up with the following:

Figure 2.1 – A core domain chart for the MallBots domains

In Figure 2.1, we’ve identified that the depot has the highest value to the business, is going to be rather complex, and is going to be our core domain. Taking orders and managing the store’s inventory is important to the business, but it has no differentiators and provides supporting functionality only. Payments exist simply because they must, so we’ve decided to integrate with a third-party SaaS to handle our money, which makes our last domain generic.

Modeling

The domain model is a product of the collaboration between domain experts and developers using the UL. What goes into the model should be limited to the data and behaviors that are relevant to the problem domain, not everything possible in an attempt at modeling reality. The point of a domain model is to solve problems identified in the domain.

Eric Evans suggests experimenting with several models and not getting stuck too long on minutia. You are trying to pull out what is important from the conversation with the domain experts. Listen for connecting words to identity processes and behaviors, titles and positions to identify actors, and, of course, the names of things to identify data. This should be captured on a large surface such as a whiteboard or a large roll of paper or a blank wall if you’re doing EventStorming. We will talk more about using EventStorming as a method to develop a domain model more in Chapter 3, Design and Planning.

The model should be free of any technical complexities or concerns, such as mentioning any databases or inter-process communication methods and should only be focused on the problem domain.

Defining boundaries

Every model belongs to a bounded context, which is a component of the application. Because the model belongs to this context, care needs to be taken in keeping it safe from outside influences or enabling external control.

You have broken down the complexity into multiple domains and discovered the models hidden within your software. The boundaries that we see forming from our discovery efforts will be around the business capabilities in our application. Examples of business capabilities for the MallBots application are the following:

Order managementPayment processingDepot operationsStore inventory management

All the domains should not have a singular view of any given model; they should be concerned with the parts that are relevant to a particular bounded context.

Every bounded context has its own UL, which should be taken to mean terms that might have different meanings when contexts change across an application. The products that are picked out by a customer will exist in several domains and, depending on the context, have completely different models, with different purposes and attributes. When the domain experts and developers discuss products, they will need to include the context to which they’re referring. They could be talking about the inventory for a store, the line items in an order, or fulfillment and delivery at the depot.

A bounded context