25,99 €
YOUR PRACTICAL, HANDS-ON GUIDE TO WRITING APPLICATIONS USING GO Google announced the Go programming language to the public in 2009, with the version 1.0 release announced in 2012. Since its announcement to the community, and the compatibility promise of the 1.0 release, the Go language has been used to write scalable and high-impact software programs ranging from command-line applications and critical infrastructure tools to large-scale distributed systems. It's speed, simplicity, and reliability make it a perfect choice for developers working in various domains. In Practical Go - Building Scalable Network + Non-Network Applications, you will learn to use the Go programming language to build robust, production-ready software applications. You will learn just enough to building command line tools and applications communicating over HTTP and gRPC. This practical guide will cover: * Writing command line applications * Writing a HTTP services and clients * Writing RPC services and clients using gRPC * Writing middleware for network clients and servers * Storing data in cloud object stores and SQL databases * Testing your applications using idiomatic techniques * Adding observability to your applications * Managing configuration data from your applications You will learn to implement best practices using hands-on examples written with modern practices in mind. With its focus on using the standard library packages as far as possible, Practical Go will give you a solid foundation for developing large applications using Go leveraging the best of the language's ecosystem.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 500
Veröffentlichungsjahr: 2021
Cover
Title Page
Introduction
What Does This Book Cover?
Reader Support for This Book
Getting Started
Installing Go
Choosing an Editor
Installing Protocol Buffer Toolchain
Installing Docker Desktop
Guide to the Book
Go Refresher
Summary
CHAPTER 1: Writing Command-Line Applications
Your First Application
Writing Unit Tests
Using the Flag Package
Improving the User Interface
Updating the Unit Tests
Summary
CHAPTER 2: Advanced Command-Line Applications
Implementing Sub-commands
Making Your Applications Robust
Summary
CHAPTER 3: Writing HTTP Clients
Downloading Data
Deserializing Received Data
Sending Data
Working with Binary Data
Summary
CHAPTER 4: Advanced HTTP Clients
Using a Custom HTTP Client
Customizing Your Requests
Implementing Client Middleware
Connection Pooling
Summary
CHAPTER 5: Building HTTP Servers
Your First HTTP Server
Setting Up Request Handlers
Testing Your Server
The Request Struct
Attaching Metadata to a Request
Processing Streaming Requests
Streaming Data as Responses
Summary
CHAPTER 6: Advanced HTTP Server Applications
The Handler Type
Sharing Data across Handler Functions
Writing Server Middleware
Writing Tests for Complex Server Applications
Summary
CHAPTER 7: Production-Ready HTTP Servers
Aborting Request Handling
Server-Wide Time-Outs
Implementing Graceful Shutdown
Securing Communication with TLS
Summary
CHAPTER 8: Building RPC Applications with gRPC
gRPC and Protocol Buffers
Writing Your First Service
A Detour into Protobuf Messages
Multiple Services
Error Handling
Summary
CHAPTER 9: Advanced gRPC Applications
Streaming Communication
Receiving and Sending Arbitrary Bytes
Implementing Middleware Using Interceptors
Summary
CHAPTER 10: Production-Ready gRPC Applications
Securing Communication with TLS
Robustness in Servers
Robustness in Clients
Connection Management
Summary
CHAPTER 11: Working with Data Stores
Working with Object Stores
Working with Relational Databases
Summary
APPENDIX A: Making Your Applications Observable
Logs, Metrics, and Traces
Emitting Telemetry Data
Summary
APPENDIX B: Deploying Applications
Managing Configuration
Distributing Your Application
Deploying Server Applications
Summary
Index
Copyright
Dedication
About the Author
About the Technical Editor
Acknowledgments
End User License Agreement
Chapter 1
Table 1.1: Parsing of command-line arguments via flag
Chapter 2
Figure 2.1: The main application looks at the command-line arguments and inv...
Figure 2.2: The main package implements the root command. A sub-command is i...
Chapter 5
Figure 5.1: Request processing by an HTTP server
Figure 5.2: Any package can register a handler function with the
DefaultServ
...
Figure 5.3: Each incoming request is handled by a new goroutine.
Figure 5.4: A context is created for every incoming request and destroyed wh...
Figure 5.5: From left to right: An incoming HTTP request triggers a long-run...
Chapter 6
Figure 6.1: Request processing by an HTTP server when using a custom handler...
Figure 6.2: Request processing by an HTTP server when using an
http.HandlerF
...
Figure 6.3: Request processing by an HTTP server when using a wrapped
ServeM
...
Figure 6.4: Request processing by an HTTP server when using multiple middlew...
Chapter 7
Figure 7.1: Aborting the request processing when the time-out handler has ki...
Figure 7.2: The different time-outs that play a role when handling an HTTP r...
Figure 7.3: Interaction between the
Shutdown()
and
ListenAndServe()
methods...
Chapter 8
Figure 8.1: Functioning of an RPC-based service architecture
Figure 8.2: Parts of a protobuf language specification
Figure 8.3: Creating the gRPC server with the
Users
service
Figure 8.4: Directory structure of the
Users
service
Figure 8.5: Comparison of a real network listener with one created using
buf
...
Chapter 9
Figure 9.1: Streaming communication pattern
Figure 9.2: Protobuf
oneof
field and the equivalent generated Go type
Figure 9.3: Interceptors and streaming communication
Chapter 10
Figure 10.1: Functioning of an RPC-based service architecture
Chapter 11
Figure 11.1: Architecture of the example scenario
Figure 11.2: Creating a bucket in MinIO
Figure 11.3: Entity relationship diagram for the package server database
Cover
Table of Contents
Title Page
Copyrigt
Dedication
About the Author
About the Technical Editor
Acknowledgments
Introduction
Getting Started
Begin Reading
APPENDIX A: Making Your Applications Observable
APPENDIX B: Deploying Applications
Index
End User License Agreement
i
xvii
xviii
xix
xxi
xxii
xxiii
xxiv
xxv
xxvi
xxvii
xxviii
xxix
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
375
376
377
378
379
380
381
382
383
384
385
386
ii
iii
v
vii
ix
387
NOTE A glossary of relevant terms is available for free download from the book's web page: https://www.wiley.com/go/practicalgo.
Amit Saha
Google announced the Go programming language to the public in 2009, with the version 1.0 release announced in 2012. Since its announcement to the community, and the compatibility promise of the 1.0 release, the Go language has been used to write scalable and high-impact software programs ranging from command-line applications and critical infrastructure tools to large-scale distributed systems. The Go language has made a huge contribution to the growth of a number of modern software success stories. For a number of years, my personal interest in Go has been due to its, for the lack of a better word, boring nature—that's what I like about it. It felt like it combined the power of the second programming language I learned, C, with the batteries-included approach of another favorite language of mine, Python. As I have written more programs using the Go language, I have learned to appreciate its focus on providing all the necessary tools and features to write production-quality software. I have often found myself thinking, “Will I be able to implement this failure-handling pattern in this application?” Then I look at the standard library package documentation, and the answer has always been a resounding “Yes!” Once you have grasped the fundamentals of Go, with almost zero effort on your part as the software developer, the result is a highly performant application out of the box.
My goal in this book is to showcase the various features of the Go language and the standard libraries (along with a few community-maintained packages) by developing various categories of applications. Once you have refreshed or learned the language fundamentals, this book will help you take the next step. I have adopted a writing style where the focus is on using various features of the language and its libraries to solve the particular problem at hand—one that you care about.
You will not find a detailed walk-through of a language feature or every feature of a certain package. You will learn just enough to build a command-line tool, a web application, or a gRPC application. I focus on a strictly chosen subset of the fundamental building blocks for such applications to provide a compact and actionable guide. Hence, you may find that the book doesn't cover the more higher-level use cases that you may want to learn about. That is intentional, as the implementation of those higher-level use cases is often dependent on domain-specific software packages, and hence no single book can do justice to recommending one without missing out on another. I also strive to use standard library packages as far as possible for writing the applications in the book. This is again done to ensure that the learning experience is not diluted. Nonetheless, I hope that the building blocks you learn about in the book will provide you with a solid foundation to leverage higher-level libraries to build your applications.
This book teaches you concepts and demonstrates patterns to build various categories of applications using the Go programming language. We focus on command-line applications, HTTP applications, and gRPC applications.
The Getting Started chapter will help you set up your Go development environment, and it lays down some conventions for the rest of the book.
Chapter 1 and Chapter 2 discuss building command-line applications. You will learn to use the standard library packages to develop scalable and testable command-line programs.
Chapter 3 and Chapter 4 teach you how to build production-ready HTTP clients. You will learn to configure time-outs, understand connection pooling behavior, implement middleware components, and more.
Chapters 5 through 7 discuss building HTTP server applications. You will learn how to add support for streaming data, implement middleware components, share data across handler functions, and implement various techniques to improve the robustness of your applications.
Chapters 8 through 10 delve deep into building RPC applications using gRPC. You will learn about Protocol Buffers, implement various RPC communication patterns, and implement client-side and server-side interceptors to perform common application functionality.
In Chapter 11, you will learn to interact with object stores and relational database management systems from your applications.
Appendix A briefly discusses how you can add instrumentation into your applications.
Appendix B provides some guidelines around deploying your applications.
Each group of chapters is mostly independent from the other groups. So feel free to jump to the first chapter of a group; however, there may be references to a previous chapter.
Within each group, however, I recommend reading the chapters from beginning to end, as the chapters within a group build upon the previous chapter. For example, if you are keen to learn more about writing HTTP clients, I suggest reading Chapter 3 and Chapter 4 in that order.
I also encourage you to write and run the code yourself as you work through the book and to attempt the exercises as well. Writing the programs yourself in your code editor will build that Go muscle, as it certainly did for me while writing the programs in the book.
You can find links to the source code and resources related to the book at https://practicalgobook.net. The code from the book is also posted at https://www.wiley.com/go/practicalgo.
If you believe that you've found a mistake in this book, please bring it to our attention. At John Wiley & Sons, we understand how important it is to provide our customers with accurate content, but even with our best efforts an error may occur. To submit your possible errata, please email it to our Customer Service Team at [email protected] with the subject line “Possible Book Errata Submission.”
To start off, we will install the necessary software needed for the rest of the book. We will also go over some of the conventions and assumptions made throughout. Finally, I will point out key language features you will use in the book and resources to refresh your knowledge about them.
The code listings in this book work with Go 1.16 and above. Follow the instructions at https://go.dev/learn/ to install the latest version of the Go compiler for your operating system. It usually involves downloading and running a graphical installation process for Windows or macOS. For Linux, your distribution's package repository may contain the latest version already, which means that you can use your package manager to install the Go compiler as well.
Once you have it installed, no further configuration is necessary to run the programs that you will write throughout the book. Verify that you have everything set up correctly by running the command go version from your terminal program. You should see an output telling you which Go version is installed and the operating system and architecture. For example, on my MacBook Air (M1), I see the following:
$ go version
go version go1.16.4 darwin/arm64
If you can see an output like the above, you are ready to continue with the next steps.
If you don't yet have a favorite Go editor/integrated development environment (IDE), I recommend Visual Studio Code (https://code.visualstudio.com/download). If you are a Vim user, I recommend the vim-go extension (https://github.com/fatih/vim-go).
For some chapters in the book, you will need the Protocol Buffers (protobuf) and gRPC tools for Go installed. You will install three separate programs: the protobuf compiler, protoc, and the Go protobuf and gRPC plug-ins, protoc-gen-go and protoc-gen-go-grpc, respectively.
To install the compiler, run the following steps for Linux or macOS:
Download the latest release (3.16.0 at the time of this book's writing) file from
https://github.com/protocolbuffers/protobuf/releases
, corresponding to your operating system and architecture. Look for the files in the
Assets
section. For example, for Linux on a x86_64 system, download the file named
protoc-3.16.0-linux-x86_64.zip
. For macOS, download the file named
protoc-3.16.3-osx-x86_64.zip
.
Next, extract the file contents and copy them to your
$HOME/.local
directory using the
unzip
command:
$ unzip protoc-3.16.3-linux-x86_64.zip -d $HOME/.local.
Finally, add the
$HOME/.local/bin
directory to your
$PATH
environment variable:
$ export PATH="$PATH:$HOME/.local/bin"
in your shell's initialization script, such as
$HOME/.bashrc
for Bash shell and
.zshrc
for Z shell.
Once you have completed the preceding steps, open a new terminal window, and run the command protoc --version :
$ protoc --version
libprotoc 3.16.0
If you see output like the one above, you are ready to move on to the next step.
To install the protobuf plug-in for Go, protoc-gen-go (release v1.26), run the following command from a terminal window:
$ go install google.golang.org/protobuf/cmd/[email protected]
To install the gRPC plug-in for Go, protoc-gen-go-grpc (release v1.1) tool, run the following command:
$ go install google.golang.org/grpc/cmd/[email protected]
Then add the following to your shell's initialization file ($HOME/.bashrc or $HOME/.zshrc) :
$ export PATH="$PATH:$(go env GOPATH)/bin"
Open a new terminal window, and run the following commands:
$ protoc-gen-go --version
protoc-gen-go v1.26.0
$ protoc-gen-go-grpc --version
protoc-gen-go-grpc 1.1.0
If you see an output like above, the tools have been installed successfully.
NOTE You will need to open a Windows PowerShell window as an administrator to run the steps.
To install the protocol buffers compiler, run the following steps:
Download the latest release (3.16.0 at the time of this book's writing) file from
https://github.com/protocolbuffers/protobuf/releases
, corresponding to your architecture. Look for a file named
protoc-3.16.0-win64.zip
in the
Assets
section.
Then create a directory where you will store the compiler. For example, in
C:\Program Files
as follows:
PS C:\> mkdir 'C:\Program Files\protoc-3.16.0'
.
Next, extract the downloaded
.zip
file inside that directory. Run the following command while you are inside the directory where you downloaded the
.zip
file:
PS C:\> Expand-Archive.\protoc-3.16.0-win64\ -DestinationPath 'C:\Program Files\protoc-3.16.0
’.
Finally, update the Path environment variable to add the above path:
PS C:\> [Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\Program Files\protoc-3.16.0\bin", "Machine").
Open a new PowerShell window, and run the command protoc --version :
$ protoc --version
libprotoc 3.16.0
If you see an output like the one above, you are ready to move on to the next step.
To install the protobuf compiler for Go, protoc-gen-go tool (release v1.26), run the following command from a terminal window:
C:\> go install google.golang.org/protobuf/cmd/[email protected]
To install the gRPC plug-in for Go, protoc-gen-go-grpc (release v1.1) tool, run the following command:
C:\> go install google.golang.org/grpc/cmd/[email protected]
Open a new Windows PowerShell Window, and run the following commands:
$ protoc-gen-go --version
protoc-gen-go v1.26.0
$ protoc-gen-go-grpc --version
protoc-gen-go-grpc 1.1.0
If you see an output like the one above, the tools have been installed successfully.
For the last chapter in the book, you will need the ability to run applications in software containers. Docker Desktop (https://www.docker.com/get-started) is an application that allows us to do that. For macOS and Windows, download the installer from the above website corresponding to your operating system and architecture, and follow the instructions to complete the installation.
For Linux, the installation steps will vary depending on your distribution. See https://docs.docker.com/engine/install/#server for detailed steps for your specific distribution. I also recommend that for ease of use (not recommended for production environments), you configure your docker installation to allow non-root users to run containers without using sudo .
Once you have followed the installation steps for your specific operating system, run the following command to download a docker image from Docker Hub and run it to ensure that the installation has been successfully completed:
$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
109db8fad215: Pull complete
Digest: sha256:0fe98d7debd9049c50b597ef1f85b7c1e8cc81f59c8d
623fcb2250e8bec85b38
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be
working correctly.
..
That completes our software installation for the book. Next, we will quickly cover some conventions used throughout the book.
In the following sections, you will learn various bits and pieces of information that will help you get the most out of the book. First, I discuss the choice of the module path for the code listings.
In this book, all applications will start by initializing a module as the first step. This will translate to running the go command, go mod init <module path>. Throughout the book, I have used a “placeholder” module path, which is github.com/username/<application-name>. Thus, in applications where we have written our module to consist of more than one package, the import path looks like this: github.com/username/<application-name>/<package>.
You can use these module paths if you are not planning to share these applications. If you plan to share your applications, or develop them further, you are encouraged to use your own module path, which is pointing to your own repository, likely a Git repository hosted on https://bitbucket.org, https://github.com or https://gitlab.com. Simply substitute username by your own username in the repository hosting service. It's also worth noting that the code repository for the book, https://github.com/practicalgo/code, contains the module path as github.com/practicalgo/code/<chap1>/<application-name>, in other words, an actual path that exists rather than a placeholder path.
You will be required to execute command-line programs throughout the book. For Linux and macOS, the default terminal program running your default shell is sufficient. For Windows, I assume that you will be using the Windows PowerShell terminal instead of the default command-line program. Most of the command-line executions are shown as executed on a Linux/macOS terminal, indicated by the $ symbol. However, you also should be able to run the same command on Windows. Wherever I have asked you to execute a command to create a directory or copy a file, I have indicated the commands for both Linux/macOS and Windows, where they are different.
I have used some terms throughout the book that may be best clarified here to avoid ambiguity and set the right expectations.
Both terms, robustness and resiliency, express the ability of an application to handle unexpected scenarios. However, these terms differ in their expected behavior under these circumstances as compared to their normal behavior. A system is robust if it can withstand unexpected situations and continue to function to some degree. This will likely be suboptimal behavior, as compared to normal behavior. On the other hand, a system is resilient if it continues exhibiting its normal behavior, potentially taking a finite amount of time before being able to do so. I put forward the following examples from the book to illustrate the difference.
In Chapter 2, you will learn to enforce time-outs for command-line application functionality that is executing a user-specified program. By enforcing time-outs, we avoid the scenario where the application continues to hang indefinitely because of bad user output. Since we configure an upper bound on how long we want to allow the user-specified command to be executed, we will exit with an error when this duration expires before the command could be completed. This is not the normal behavior of the application—that we should wait for the command to complete—but this suboptimal behavior is necessary to allow the application to recover from an unexpected situation, such as the user-specified command taking longer than expected. You will find similar examples throughout, notably when sending or receiving network requests in Chapters 4, 7, 10, and 11. We will refer to these techniques as introducing robustness in our applications.
In Chapter 10, you will learn to handle transient failures in your gRPC client applications. You will write your applications in a manner in which they can tolerate temporary failures that are likely to be resolved soon. We refer to this as introducing resilient behavior in our applications. However, we also introduce an upper time limit, which we allow to resolve the potentially temporary failure. If this time limit is exceeded, we consider that the operation cannot be completed. Thus, we introduce robustness as well.
To summarize, resiliency and robustness both aim to handle unexpected situations in our applications, and this book uses these terms to refer to such techniques.
I use the term production readiness in the book as all steps that you should think about as you develop your application but before you deploy it to any kind of a production environment. When the production environment is your own personal server where you are the only user of your application, the techniques that you will learn will likely be sufficient. If the production environment means that your application will perform critical functionality for your users, then the techniques in this book should be the absolute baseline and a starting point. Production readiness consists of a vast body of often domain-specific techniques across various dimensions—robustness and resiliency, observability, and security. This book shows you how to implement a small subset of these topics.
The code listings in the book use various standard library packages and a few third-party packages. The descriptions of the various functions and types are limited to the contextual usage. Knowing where to look when you want to find out more about a package or function is important to get the most out of the book. The key reference documentation for all standard library packages is https://pkg.go.dev/std. When I import a package as net/http, the documentation for that package will be found at the path https://pkg.go.dev/net/http. When I refer to a function such as io.ReadAll(), the function reference is the package io 's documentation at https://pkg.go.dev/io.
For third-party packages, the documentation is available by going to the address https://pkg.go.dev/<import path>. For example, the Go gRPC package is imported as google.golang.grpc. Its reference documentation is available at https://pkg.go.dev/google.golang.org/grpc.
I recommend going through the topics in “A Tour of Go,” at https://tour.golang.org/list, to serve as a refresher of the various features that we will be using to implement programs in the book. These include for loops, functions, methods, struct and interface types, and error values. Additionally, I want to highlight the key topics that we will use extensively, along with references to learn more about them.
We will be using struct types defined by the standard library and third-party packages, and we will also be defining our own. Beyond defining objects of struct types, we will be working with types that embed other types—other struct types and interfaces. The section “Embedding” in the “Effective Go” guide (https://golang.org/doc/effective_go#embedding) describes this concept. We will also be making use of anonymous struct types when writing tests. This is described in this talk by Andrew Gerrand, “10 things you (probably) don't know about Go”: https://talks.golang.org/2012/10things.slide#1.
To use the various library functions and to write testable applications, we will be making extensive use of interface types. For example, we will be making extensive use of alternative types that satisfies the io.Reader and io.Writer interfaces to write tests for applications that interface with the standard input and output.
Learning to define a custom type that satisfies another interface is a key step to writing Go applications, where we plug in our functionality to work with the rest of the language. For example, to enable sharing data across HTTP handler functions, we will define our own custom type implementing the http.Handler interface.
The section on interfaces in “A Tour of Go,” https://tour.golang.org/methods/9, is useful to get a refresher on the topic.
We will be using goroutines and channels to implement concurrent execution in our applications. I recommend going through the section on Concurrency in “A Tour of Go”: https://tour.golang.org/concurrency/1. Pay special attention to the example use of select statements to wait on multiple channel communication operations.
We will be using the standard library's testing package exclusively for writing all of the tests, and we will use Go test to drive all of the test executions. We have also used the excellent support provided by libraries such as net/http/httptest to test HTTP clients and servers. Similar support is provided by gRPC libraries. In the last chapter, we will use a third-party package, https://github.com/testcontainers/testcontainers-go, to create local testing environments using Docker Desktop.
In some of the tests, especially when writing command-line applications, we have adopted the style of “Table Driven Tests,” as described at https://github.com/golang/go/wiki/TableDrivenTests, when writing the tests.
In this introduction to the book, you installed the software necessary to build the various applications to be used in the rest of the book. Then I introduced some of the conventions and assumptions made throughout the remainder of the book. Finally, I described the key language features with which you will need to be familiar to make the best use of the material in the book.
Great! You are now ready to start your journey with Chapter 1, where you will be learning how to build testable command-line applications.
