Microservices with Go E-Book

Alexander Shuiskov

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 376

Veröffentlichungsjahr: 2022

Beschreibung

This book covers the key benefits and common issues of microservices, helping you understand the problems microservice architecture helps to solve, the issues it usually introduces, and the ways to tackle them.
You’ll start by learning about the importance of using the right principles and standards in order to achieve the key benefits of microservice architecture. The following chapters will explain why the Go programming language is one of the most popular languages for microservice development and lay down the foundations for the next chapters of the book. You’ll explore the foundational aspects of Go microservice development including service scaffolding, service discovery, data serialization, synchronous and asynchronous communication, deployment, and testing. After covering the development aspects, you’ll progress to maintenance and reliability topics. The last part focuses on more advanced topics of Go microservice development including system reliability, observability, maintainability, and scalability. In this part, you’ll dive into the best practices and examples which illustrate how to apply the key ideas to existing applications, using the services scaffolded in the previous part as examples.
By the end of this book, you’ll have gained hands-on experience with everything you need to develop scalable, reliable and performant microservices using Go.

Leseprobe

Microservices with Go

Building scalable and reliable microservices with Go

Alexander Shuiskov

BIRMINGHAM—MUMBAI

Microservices with Go

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Gebin George

Publishing Product Manager: Kunal Sawant

Senior Editor: Nisha Cleetus

Content Development Editor: Yashi Gupta

Technical Editor: Pradeep Sahu

Copy Editor: Safis Editing

Project Coordinator: Deeksha Thakkar

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Roshan Kawale

Business Development Executive: Debadrita Chatterjee

Marketing Coordinators: Rayyan Khan and Deepak Kumar

First published: October 2022

Production reference: 2081122

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80461-700-7

www.packt.com

I dedicate this book to my family and to my source of passion, inspiration, and love, Vera

– Alexander Shuiskov

Contributors

About the author

Alexander Shuiskov is a staff software engineer at Uber working on the observability and reliability of thousands of Uber microservices. He is an expert in alerting and monitoring solutions. Before Uber, Alexander worked on microservice development at multiple tech companies, including Booking.com and eBay.

Alexander holds an MSc in computer science from the Ural State University, as well as a degree in economics and management.

Outside of work, Alexander loves street photography, abstract art, sound production, and many sports, including cycling and running.

I want to thank all the people who supported me throughout this journey and provided their valuable feedback: my friends, family, colleagues, and my loving partner, Vera. I’m also grateful for the chance to work on this book: at first, I thought I would never have time for it, but curiosity and passion took precedence over all uncertainty. Never say never!

About the reviewer

Sadman Khan is currently a senior engineer at Uber in the observability department. He co-authored the alerting engine that has been critical to keeping Uber reliable. He graduated from the University of Waterloo, Canada, with a bachelor’s degree in computer engineering. He has previously worked at Coinbase and LinkedIn.

Table of Contents

Preface xiii

Part 1: Introduction

1

Introduction to Microservices 1

What is a microservice?1

Motivation to use microservices2

Pros and cons of microservices4

Benefits of microservices 4

Common issues of microservices6

When to use microservice architecture7

Role of Go in microservice development8

Summary9

Further reading9

Part 2: Foundation

2

Scaffolding a Go Microservice 13

Technical requirements13

Go basics14

Core principles14

Writing idiomatic Go code15

Interfaces17

Tests18

Context18

Project structure19

Private packages19

Public packages20

Executable packages20

Other commonly used directories20

Common files20

Best practices21

Scaffolding an example application21

Movie application22

Application code structure25

Movie metadata service27

Movie service42

Summary52

Further reading53

3

Service Discovery 55

Technical requirements55

Service discovery overview55

Registry57

Service discovery models58

Service health monitoring59

Service discovery solutions59

HashiCorp Consul60

Kubernetes60

Adopting service discovery60

Preparing the application61

Implementing the discovery logic64

Using the discovery logic70

Summary76

Further reading76

4

Serialization 77

Technical requirements77

The basics of serialization78

Popular serialization formats80

Using Protocol Buffers84

Best practices for serialization90

Summary91

Further reading91

5

Synchronous Communication 93

Technical requirements93

Introduction to synchronous communication94

Go RPC frameworks and libraries95

Defining a service API using Protocol Buffers96

Implementing gateways and clients100

Metadata service100

Rating service105

Movie service107

Summary113

Further reading113

6

Asynchronous Communication 115

Technical requirements115

Asynchronous communication basics115

Benefits and challenges of asynchronous communication116

Techniques and patterns of asynchronous communication118

Using Apache Kafka for messaging120

Adopting Kafka for our microservices122

Asynchronous communication best practices130

Versioning130

Leveraging partitioning131

Summary132

Further reading132

7

Storing Service Data 133

Technical requirements133

Introduction to databases134

Using MySQL to store our service data137

Summary145

Further reading146

8

Deployment with Kubernetes 147

Technical requirements147

Preparing application code for deployments148

Deployment basics148

Application configuration149

Deploying via Kubernetes151

Introduction to Kubernetes152

Setting up our microservices for Kubernetes deployments153

Deployment best practices158

Automated rollbacks159

Canary deployments159

Replace with Continuous Deployment (CD)160

Summary160

Further reading161

9

Unit and Integration Testing 163

Technical requirements163

Go testing overview164

Unit tests168

Mocking170

Implementing unit tests173

Integration tests175

Testing best practices187

Using helpful messages187

Avoiding the use of Fatal in your logs188

Making a comparison using a cmp library189

Summary190

Further reading190

Part 3: Maintenance

10

Reliability Overview 193

Technical requirements193

Reliability basics194

Achieving reliability through automation195

Communication error handling195

Graceful shutdown205

Achieving reliability through development processes and culture208

On-call process208

Incident management210

Reliability drills212

Summary213

Further reading213

11

Collecting Service Telemetry Data 215

Technical requirements215

Telemetry overview216

Collecting service logs217

Choosing the logging library220

Using logging features221

Storing microservice logs224

Logging best practices225

Collecting service metrics229

Storing metrics231

Popular Go metrics libraries233

Emitting service metrics233

Metrics best practices236

Tracing237

Tracing tools239

Collecting tracing data with the OpenTelemetry SDK240

Summary249

Further reading250

12

Setting Up Service Alerting 251

Technical requirements251

Alerting basics252

Alerting use cases253

Introduction to Prometheus254

Setting up Prometheus alerting for our microservices257

Alerting best practices265

Summary266

Further reading266

13

Advanced Topics 267

Technical requirements267

Profiling Go services268

Creating microservice dashboards272

Frameworks278

Storing microservice ownership data280

Securing microservice communication with JWT282

JWT basics283

Implementing authentication and authorization with JWTs284

Summary289

Further reading289

Index 291

Other Books You May Enjoy 302

Preface

Since its first release, the Go programming language has gained popularity among all types of software developers. Simple language syntax, ease of use, and a rich set of libraries made Go one of the primary languages for writing different kinds of software, from small tools to large-scale systems consisting of hundreds of components.

Among the primary Go use cases is microservice development – the development of individual applications, called microservices, that can play various roles, from processing payments to storing user data. Organizing a large system as a set of microservices often brings multiple advantages, such as increasing the development and deployment speed, but also brings multiple types of challenges. Among such challenges are service discovery and communication, integration testing, and service monitoring.

In this book, we will illustrate how to implement Go microservices and establish communication between them, how to enable the deployment of individual microservices and secure their interactions, and how to store and retrieve service data and provide service APIs, enabling other applications to use our microservices. You will learn about some of the industry’s best practices related to all of these topics and get a detailed overview of the possible challenges along the way, as well as the possible benefits. The knowledge that you will gain by reading this book will help you to both create new microservices and efficiently maintain the existing ones. I hope this journey will be exciting for you!

Who this book is for

This book is for all types of developers: from people interested in learning how to write microservices in Go to seasoned professionals who want to take the next step in mastering the art of writing scalable and reliable microservice-based systems. The first two parts of the book, covering the development of microservices, would be useful to developers who are just starting their experience with Go or those who are interested in the best practices of organizing the Go application code base according to the industry’s best standards; the final part would be useful for all developers, even the most experienced ones, as it provides great insights on maintaining and operating microservices at scale.

What this book covers

Chapter 1, Introduction to Microservices, will cover the key benefits of and common issues with a microservice architecture, helping you to understand which problems microservices solve and which challenges they usually introduce. The chapter emphasizes the role of the Go programming language in microservice development and lays down the foundation for the rest of the book.

Chapter 2, Scaffolding a Go Microservice, will introduce the you to the main principles of the Go programming language and provide the most important recommendations for writing Go code. It will cover the process of setting up the right structure to organize the microservice code in Go and introduce the you to an example application consisting of three microservices. Finally, the chapter will illustrate how to scaffold the code for each of the example microservices. The example microservices implemented in this chapter are going to be used throughout the book, with each chapter adding new features to them.

Chapter 3, Service Discovery, will talk about the problem of service discovery and illustrates how different services can find each other in a microservice environment. It will cover the most popular service discovery tools and walk the you through the steps of adding service discovery logic to the example microservices from the previous chapter.

Chapter 4, Serialization, will bring us to the concept of data serialization, which is required for understanding upcoming chapters covering microservice communication. The you will be introduced to the Protocol Buffers data format, which is going to be used for encoding and decoding the data transferred between our example microservices. The chapter will provide examples of how to define serializable data types and generate code for them, and how to use the generated code in Go microservices.

Chapter 5, Synchronous Communication, will cover the topic of synchronous communication between microservices. It will illustrate how to define service APIs using the Protocol Buffers format and introduce the you to gRPC, a service communication framework. The chapter will wrap up with examples of how to implement microservice gateways and clients and perform remote calls between our microservices.

Chapter 6, Asynchronous Communication, will talk about asynchronous communication between microservices. It will introduce the you to a popular asynchronous communication tool, Apache Kafka, and provide examples of sending and receiving messages, using it for our example microservices. The chapter will wrap up with an overview of the best practices for using asynchronous communication in microservice environments.

Chapter 7, Storing Service Data, will cover the topic of persisting service data in databases. The you will learn about the common types of databases and the benefits they bring to software developers. The chapter will walk the you through the process of implementing the logic for storing service data in a MySQL database.

Chapter 8, Deployment with Kubernetes, will talk about service deployment and provide an overview of a popular deployment and orchestration platform, Kubernetes. The chapter will illustrate how to prepare service code for deployment and how to deploy it using Kubernetes. The chapter will include the best practices for deploying microservice applications.

Chapter 9, Unit and Integration Testing, will describe the common techniques of testing Go microservice code. It will cover the basics of Go unit and integration testing and demonstrate how to test the microservice code from the previous chapters. The chapter will wrap up with the industry’s best practices for writing and organizing tests.

Chapter 10, Reliability Overview, will introduce the you to the topic of system reliability and describe the core principles, instruments, and industry best practices for building reliable and highly available microservices. It will illustrate how to automate service responses to various types of failures, as well as how to establish the processes for keeping service reliability under control.

Chapter 11,Collecting Service Telemetry Data, will provide a detailed overview of modern instruments and solutions for collecting service telemetry data, such as logs, metrics, and traces. The chapter will provide lots of detailed examples of collecting all different types of telemetry data and list some of the best practices for working with them.

Chapter 12, Setting up Service Alerting, will illustrate how to set up automated incident detection and notification for microservices, using the telemetry data collected in the previous chapter. It will introduce the you to a popular alerting and monitoring tool, Prometheus, and show how to set up Prometheus alerts for our example microservices.

Chapter 13, Advanced Topics, will wrap up the last part of the book and cover some of the advanced topics in microservice development, such as profiling, dashboarding, frameworks, service ownership, and security. The chapter will include some examples of setting up secure communication between Go microservices using the JWT protocol.

To get the most out of this book

I suggest you get some familiarity with Go by implementing a few applications, such as simple web services. Familiarity with a Docker tool would be a plus because we will be using it for running some of the tools that our microservices will be using. Finally, I strongly suggest implementing, running, and playing with the example microservices that we will be implementing so that all your knowledge will be cemented by practice.

Software/hardware covered in the book

Operating system requirements

Go 1.11 or above

Windows, macOS, or Linux

Docker

Windows, macOS, or Linux

grpcurl

Windows, macOS, or Linux

Kubernetes

Windows, macOS, or Linux

Prometheus

Windows, macOS, or Linux

Jaeger

Windows, macOS, or Linux

Graphviz

Windows, macOS, or Linux

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/microservices-with-go. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book.

You can download it here: https://packt.link/1fb2C.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system.”

A block of code is set as follows:

package main import (     "encoding/json"     "fmt"     "os"     "time"     "github.com/confluentinc/confluent-kafka-go/kafka"     "movieexample.com/rating/pkg/model" )

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

    if err := p.Produce(&kafka.Message{ TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny}, Value:          []byte(encodedEvent), }, nil); err != nil {         return err     }     return nil

Any command-line input or output is written as follows:

mysql movieexample -h localhost -P 3306 --protocol=tcp -u root -p

Bold: Indicates a new term, an important word, or words that you see on screen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Select System info from the Administration panel.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Preface

Share Your Thoughts

Once you’ve read Microservices with Go, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/978-1-80461-700-7

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Introduction

This part provides an overview of the microservice architecture model. It covers the key benefits and common issues of microservices, helping the readers to understand which problems the microservice architecture helps to solve and which issues it usually introduces. The single chapter in this part focuses on the role of the Go programming language in microservice development and lays down the foundation for the rest of the book.

This part contains the following chapter:

Chapter 1, Introduction to Microservices

1

Introduction to Microservices

In this chapter, you will be introduced to microservices and the motivation behind them. You will understand the key benefits and common issues of the microservice architecture model and learn when to use it, as well as getting some microservice development best practices. This knowledge will help you establish a solid foundation for reading the next chapters and give you some ideas on what challenges you may face with microservices in the future.

In this chapter, we will cover the following topics:

What is a microservice?Motivation to use microservicesPros and cons of microservicesWhen to use microservice architectureRole of Go in microservice development

What is a microservice?

Companies worldwide have used the microservice architecture model so widely that it has almost become a default way of software development. Those companies have tens, hundreds, and even thousands of microservices at their disposal.

So, what exactly is the microservice model?

The microservice architecture model is organizing an application as a collection of services, called microservices, each of which is further responsible for a certain part of application logic, usually defined by a particular business capability.

As an example, consider an online marketplace application. The application may have multiple features, including search, shopping cart, payments, order history, and many more. Each feature can be so different that the code may (and, in certain cases, should) be completely independent of the rest of the application. In this example, search and payments technically have nothing in common. In the microservice architecture model, each component would be an independent service playing its own role in the system.

Organizing each part of the application as a separate service is not necessarily a requirement. As with any architecture model or any aspect of software development, engineers need to be careful with choosing a particular approach or solution – doing an initial analysis and understanding the solution under the given conditions.

Before we proceed to the key benefits and downsides of microservices, let's see what challenges you could face when the application is not separated into multiple services.

Motivation to use microservices

In order to understand the motivation behind using the microservice architecture, it is very important to see the opposite approach – when the application is built and executed as a single program. Such applications are called monolithic applications or monoliths.

Monolithic architecture is, in most ways, the simplest model to implement since it does not involve splitting the application into multiple parts that need to coordinate with each other. This can provide you with major advantages in many cases, such as the following:

Small code base: Splitting an application into multiple independent parts may significantly increase the size of the code base by introducing extra logic required for communication between the components.Application logic is still loosely defined: It is very common that parts of the application or the entire system go through major structural or logical changes, especially at the very early stages of development. This might be caused by a sudden change of requirements, priorities, changes in the business model, or a different approach to development. During the early stages of development, iterating fast can be critical not only to the development process, but also to the entire company.Narrow scope of the application: Not every service requires a decomposition and division into separate parts. Consider a service for generating random passwords – it has a single logical feature and, in most cases, it would be unnecessary to split it into multiple parts.

In all of the preceding cases, monolithic architecture would be a better fit for the application. However, at some point, services get too big to remain monolithic. Developers start experiencing the following issues:

Large application size and slow deployments: At a certain point, an application can become so big that it can take minutes or even hours to build, start, or deploy.Inability to deploy a particular part of the application independently: Not being able to replace a part of a large application can easily become a bottleneck, slowing down the development and release process.Higher blast radius: If there is a bug in a certain function or library widely used across the application code, it is going to affect all parts of the system at once, potentially causing major issues.Vertical scalability bottleneck: The more logic the application has, the more resources it needs in order to run. At a certain point, it can get hard or impossible to scale the application up even further, given the possible limits on CPU and RAM.Interference: Certain parts of the application can heavily load CPU, I/O, or RAM, causing delays for the rest of the system.Unwanted dependencies between components: Having the entire application represented as a single executable leaves room for unnecessary dependencies between the components. Imagine a developer refactoring a code base, and making a change suddenly affects some important parts of the system, such as payments. Having more isolation between the components gives more protection against such issues.Security: A possible security issue in the application may result in unauthorized access to all components at once.

In addition to the possible issues we just described, different components may have different requirements, such as the following:

Resources and hardware requirements: Certain components are more CPU-intensive or memory-intensive and may perform I/O operations at a higher rate. Separating such components may reduce the load on the entire system, increasing system availability and reducing latency.Deployment cadence: Some parts of the system mostly remain unchanged while others require multiple deployments per day.Deployment monitoring and automated testing: Certain components may require stricter checks and monitoring and can be subject to slower deployments due to multi-step rollouts.Technologies or programming languages: It is not uncommon that different parts of the system can be written in different programming languages or use fundamentally different technologies, libraries, and frameworks.Independent APIs: Components may provide fully independent APIs.Code review process: Some components may be subject to a stricter code review process and additional requirements.Security: Components may have different security requirements and may require additional isolation from the rest of the application for security reasons.Compliance: Some parts of the system may be subject to stricter compliance requirements. For example, handling personally identifiable information (PII) for users from a certain region can put stricter requirements on the entire system. Logical separation of such components helps to reduce the scope of work required to keep the system compliant.

With all the preceding issues described, we can see that at a certain point monolithic applications can become too big for a one-size-fits-all model. As the application grows, certain parts of it may start becoming independent and have different requirements, benefiting from a logical separation from the rest of the application.

In the next section, we are going to see how splitting the application into microservices can solve the aforementioned problems and which aspects of it you should be careful with.

Pros and cons of microservices

In order to understand how to get the best results from using microservices and which issues to be aware of, let's review the pros and cons of the microservice model.

Benefits of microservices

As previously described, different application components may have fundamentally different requirements and at certain points diverge so much that it would be beneficial to separate them. In this case, microservice architecture provides a clear solution by decoupling the parts of the system.

Microservices provide the following benefits to developers:

Faster compilation and build time: Faster build and compilation time may play a key role in speeding up all development processes.Faster deployments, lower deployable size: When each part of the system is deployed separately, the deployable size can get so significantly smaller that individual deployments can take just a fraction of the time compared to monolithic applications.Custom deployment cadence: The microservice model solves the problem of following a custom deployment schedule. Each service can be deployed independently and follow its own schedule.Custom deployment monitoring: Some services can perform more critical roles in the system than others and may require more fine-grained monitoring and extra checks.Independent and configurable automated testing: Services may be configured to perform different automated tests as a part of the build and deployment pipeline. Additionally, the scope of checks can be reduced for individual microservices, that is, we don't need to perform tests for the entire application, which may take longer.Cross-language support: It is no longer required to run an application as a single executable, so it is possible to implement different parts of the system using different technologies, finding the best fit for each problem.Simpler APIs: Fine-grained APIs are one of the key aspects of microservice development and having clear and efficient APIs helps to enforce the right composition of the system.Horizontal scaling: Microservices are easier and often cheaper to scale horizontally. Monolithic applications are usually resource-heavy and running them on numerous instances could be quite expensive due to high hardware requirements. Microservices, however, can be scaled independently. So, if a particular part of the system requires running on hundreds or thousands of servers, other parts don't need to follow the same requirements.Hardware flexibility: Splitting an application often means reducing the hardware requirements for most parts of the system. It provides more flexibility in choosing the hardware or cloud providers to execute applications.Fault isolation: Service decoupling provides an efficient safety mechanism to prevent major issues on partial system failures.Understandability: Services are easier to understand and maintain due to lower code base sizes.Cost optimization: Running most application components on lower-grade instances compared to expensive high-resource monolithic instances may result in significant cost savings for the company.Distributed development: Removing the coupling between the components helps achieve more independence in code development, which can play an important role in distributed teams.Ease of refactoring: In general, it is much easier to perform refactoring for microservices due to the lower scope of changes and independent release and testing processes, which helps detect possible issues and reduce the scope of failures.Technological freedom: With microservice architecture, it is much easier to switch to new technologies given that each service is smaller in size and is structurally independent of the others. This can play a key role in companies with an open and experimental development culture, helping find the right solutions for particular problems and keep their technological stack up to date.Independent decision-making: Developers are free to choose programming languages, libraries, and tools that fit their needs the best. This does not, however, imply that there should be no standardization, but it is often highly beneficial to achieve a certain degree of freedom for distributed decision-making.Removing unnecessary dependencies: It is easy to miss detecting unwanted dependencies between the components of a monolithic application given the tighter coupling of the components. Microservice architecture helps you notice unwanted dependencies between components and restricts the use of certain services to particular parts of the application.

As we can see, microservices bring a high degree of flexibility and help to achieve a higher level of independence between the components. These aspects may be instrumental to the success of a large development team, allowing them to build and maintain independent components separately. However, any model comes at its own cost, and in the next section, we are going to see the challenges you could face with a collection of microservices.

Common issues of microservices

As with any solution, microservice architecture has its own issues and limitations. Some issues with microservice architecture include the following:

Higher resource overhead: When an application consists of multiple components, instead of sharing the same process space, there is a need to communicate between the components that involve higher network use. This puts more load on the entire system and increases traffic, latency, and I/O usage. In addition, the total CPU and RAM are also higher due to the extra overhead of running each component separately.Debugging difficulty: Troubleshooting and debugging are often more difficult when you deal with multiple services. For example, if multiple services process a request that fails, a developer needs to access the logs of multiple services in order to understand what caused the failure.Integration testing: Separating a system requires building a large set of integration tests and other automated checks that would monitor the compatibility and availability of each component.Consistency and transactions: In microservice applications, the data is often scattered across the system. While this helps to separate the independent parts of the application, it makes it harder to do transactional and atomic changes in the system.Divergence: Different services may use different versions of libraries, which may include incompatible or outdated ones. Divergence makes it harder to perform system upgrades and resolve various issues, including software vulnerability fixes.Tech debt addressability: It is much harder to address tech debt in a distributed system where each component is owned by a different team.Observability: Managing multiple applications brings additional challenges in collecting and using the system events and messages, including logs, traces, and metrics. Developers need to make sure all such signals are collected for all applications and are available for analysis, including all necessary contextual information to debug any issues and locate the root cause of the issue among the target services.Possible duplication, overlapping functionality: In a highly distributed development environment, it is not uncommon to have multiple components performing similar roles in the system. It is important to set clear boundaries within the system and decide in advance which particular roles the components are assigned.Ownership and accountability: Ownership becomes a major aspect of the development process when there are many different teams maintaining and developing independent components. It is crucial to define clear ownership contracts to address the development requests, security and support issues, and all other types of maintenance work.

As we have just illustrated, the microservice model comes at a cost and you should expect that you will need to solve all these challenges at a certain point. Being aware of the possible challenges and being proactive in solving them is the key to success – the benefits that we have described earlier can easily outweigh the possible issues.

In the next section, we are going to summarize when to use the microservices and learn some best practices for working with them.

When to use microservice architecture

We have covered the benefits and common issues of microservices, providing a good overview of the applicability of using the microservice architecture model in the application. Let's summarize the key points of using the microservice model, which are the following:

Don't introduce microservices too early: Don't use the microservice architecture too early if the product is loosely defined or can go through significant changes. Even when developers know the exact purpose of the system, there are high chances of various changes in the early stages of the development process. Starting from a monolithic application – and splitting it over time once there are clearly defined business capabilities and boundaries – helps reduce the amount of work and establish the right interfaces between the components.No size fits all: Each company is unique and the final decision should depend on many factors, including the size of the team, its distribution, and geography. A small local team may be comfortable working with a monolithic application, whereas a geographically distributed team may highly benefit from splitting the application into multiple microservices to achieve higher flexibility.

Additionally, let's summarize the best practices of using the microservice architecture model for applications, which are the following:

Design for failure: In a microservice architecture, there are many interactions between the components, most of which are happening via remote calls and events. This increases the chance of various failures, including network timeouts, client errors, and many more. Build the system thinking of every possible failure scenario and different ways to proceed with it.Embrace automation: Having more independent components requires much stricter checks in order to achieve stable integration between the services. Investing in solid automation is absolutely necessary in order to achieve a high degree of reliability and ensure all changes are safe to be deployed.Don't ship hierarchy: It is a relatively common practice to split the application into services based on the organizational structure, where each team may be responsible for its own service. This model works well if the organizational structure perfectly aligns with the business capabilities of the microservices, but quite often this is not the case. Instead of using a service-per-team model, try to define the clear domains and business capabilities around which the code is structured and see how the components interact with each other. It is not easy to achieve perfect composition, but you will be highly rewarded for it.Invest in integration testing: Make sure you have comprehensive tests for the integrations between your microservices performing automatically.Keep backward compatibility in mind: Always remember to keep your changes backward compatible to ensure that new changes are safe to deploy. Additionally, use techniques such as versioning, which we are going to cover in the next chapter of the book.

At this point, we have covered the key aspects of microservice development, and you have learned its benefits and the challenges you may face. Before we proceed to the next chapter, let's cover one more topic to ensure we are ready for the journey into microservice development. Let's get familiar with the Go programming language and its role in microservice development.

Role of Go in microservice development

Over the last decade, the Go programming language has become one of the most popular languages for application development. There have been many factors contributing to its success, including its simplicity, ease of writing network applications, and an ability to easily develop parallel and concurrent applications.

Additionally, the larger developer community has played a key role in raising its popularity across all types of developers. The Go community is welcoming to everybody, from people just starting their journeys into programming to seasoned experts with decades of experience building different types of applications.

The Go standard library provides a set of packages that can often be enough for building a complete web application or an entire service, sometimes without even requiring any external dependencies. Many developers have been fascinated by the ease of writing applications and tools performing network calls, data serialization and encoding, file processing, and many other types of common operations.

This simplicity, paired with fast and efficient compilation into native binaries as well as rich tooling, made it one of the primary languages for writing web tools and services. The high adoption of the Go language for web service development made it one of the primary choices for writing microservices across the industry.

The biggest advantages of Go for microservice development include the following:

Smooth learning curve: As one of the critical aspects of application development in growing teams, the simplicity of the Go language helps reduce the onboarding time for new, inexperienced developers.Explicit error handling: While being a hot topic in the Go community, error handling in Go encourages explicit handling of all application errors. It aligns with one of the key principles of microservice development of designing applications for failure.Useful standard library: The Go standard library includes lots of packages that can be used in production-grade systems without requiring external solutions.Community support: The Go community is among the biggest in the industry and the most popular libraries get enough support and maintenance.Ease of writing concurrent code: Concurrent calls are very common in microservice application logic - microservices often call multiple other services and combine their results. Writing concurrent code in Golang can be a fairly trivial task when utilizing the built-in sync package and core language features, such as channels and Goroutines.

The growth of the Go community has increased the rate of development of additional libraries for the language and resulted in the creation of the entire ecosystem of tools, powering application logging, debugging, and implementations of all widely used networking protocols and standards. The community keeps growing and the rate of new releases is only accelerating.

Summary

We have discussed the key aspects of the microservice development model, including the motivation to use it, the common benefits, and the possible challenges. You have learned that the microservice model brings many advantages, helping you achieve a higher degree of flexibility. It also comes with its own costs, which often include additional complexity and a lack of uniformness in the system. Microservice architecture requires you to think about these problems proactively in order to address them before they become big issues.

In the next chapter, we are going to start our journey into microservice development with the Go language. You will learn the important basics of the Go programming language and we will scaffold our microservices, which we are going to improve throughout the rest of the book.

Further reading

A collection of resources on microservice development: https://microservices.io/Overview of the microservice architecture model: https://martinfowler.com/articles/microservices.html15 best practices for building microservices: https://www.bmc.com/blogs/microservices-best-practices/

Part 2: Foundation

This part covers the foundational aspects of Go microservice development, such as service discovery, data serialization, synchronous and asynchronous communication, deployment, and testing. You will learn how to scaffold Go microservices, establish communication between them, store service data, and implement service APIs, as well as many other important aspects of microservice development.

This part contains the following chapters:

Chapter 2, Scaffolding a Go MicroserviceChapter 3, Service DiscoveryChapter 4, SerializationChapter 5, Synchronous CommunicationChapter 6, Asynchronous CommunicationChapter 7, Storing Service DataChapter 8, Deployment with KubernetesChapter 9, Unit and Integration Testing

2

Scaffolding a Go Microservice

In this chapter, we will finally start scaffolding our microservice code. The goal of this chapter is to establish a solid foundation for writing Go microservices and setting the right structure for future changes. While Go makes it relatively easy to write small applications, there are multiple challenges that engineers may face along the way, including the following:

How to set the right project structure to make it easier to evolve and maintain the code baseHow to write idiomatic Go code that is going to be consistent with the largest Go code basesHow to separate the components of a microservice and wire them together

In this chapter, we are going to address each of these challenges. First, you will be introduced to the key aspects of writing idiomatic and conventional Go code. You will learn important recommendations for writing and organizing your code base, as well as how to set up the proper code structure for your services. Then, we are going to introduce you to an example application, which will consist of three microservices that we are going to use throughout the book. In the following chapters, we will add additional features to these services, illustrating all the important areas of microservice development.

In this chapter, we will cover the following topics:

Go basicsProject structureScaffolding an example application

Technical requirements

To complete this chapter, you need to have Go 1.11 or above. If you don’t have Go installed, you can download it from the official website at go.dev/dl.

You can find the code examples for this chapter on GitHub: https://github.com/PacktPublishing/microservices-with-go/tree/main/Chapter02.

Go basics

Go is a great language for writing microservices. It is relatively easy to learn and has a pretty smooth learning curve, making onboarding new engineers easier. While you may have already had some experience with Go, one of the purposes of this book is to provide enough information to all types of developers—from beginners to highly experienced professionals.

In this section, we are going to summarize important concepts of the language. If you already have experience with Go, you can still quickly scan through this part. It also includes some useful recommendations and best practices commonly missed even by experienced engineers.

Core principles

Before we proceed to look at the basics of Go, I’m going to share with you some fundamental principles that will help you make decisions when writing and organizing your code. These principles include the following:

Always follow the official guidelines. It is not uncommon for us engineers to have strong opinions about various styling and coding practices. However, in any developer community, consistency is more important than individual opinions. Make sure you get familiar with the most fundamental Go programming guidelines, written by the Go team:Effective Go—an official set of guidelines for Go developers: https://go.dev/doc/effective_goGo code review comments: Another useful source of information on Go development, covering various aspects, including code style, naming, and error handling: https://github.com/golang/go/wiki/CodeReviewCommentsFollow the style used in the standard library. The standard Go library, which comes with any Go installation, is the best source of code examples and comments. Get familiar with some of the packages from the library, such as context and net. Following the coding style used in these packages will help you to write consistent, readable, and maintainable code, regardless of who will be using it later.Do not try to apply the ideas from other languages to Go. Instead, understand the philosophy of Go and see the implementation of the most elegant Go packages—you can check the net package for some good examples: https://pkg.go.dev/net.

Now, as we are aligned on the core principles, let’s move on to the key recommendations for writing conventional and idiomatic Go code.

Writing idiomatic Go code

This section summarizes the key topics described in the Effective Go document. Following the suggestions provided in this section will help you to keep your code consistent with the official guidelines.

Naming

Naming is one of the most important aspects of Go development. Writing Go code in an idiomatic way requires an understanding of its core naming principles:

Exported names start with an uppercase character.When a variable, struct, or interface is imported from another package, its name includes a package name or alias, for example, bytes.Buffer.Since references include package names, you should not prefix your names with the package name. If the package name is xml, use the name Reader, not XMLReader