34,79 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Fachliteratur
- Sprache: Englisch
Microservice architecture is sweeping the world as the de facto pattern to build web-based applications. Golang is a language particularly well suited to building them. Its strong community, encouragement of idiomatic style, and statically-linked binary artifacts make integrating it with other technologies and managing microservices at scale consistent and intuitive. This book will teach you the common patterns and practices, showing you how to apply these using the Go programming language.
It will teach you the fundamental concepts of architectural design and RESTful communication, and show you patterns that provide manageable code that is supportable in development and at scale in production. We will provide you with examples on how to put these concepts and patterns into practice with Go.
Whether you are planning a new application or working in an existing monolith, this book will explain and illustrate with practical examples how teams of all sizes can start solving problems with microservices. It will help you understand Docker and Docker-Compose and how it can be used to isolate microservice dependencies and build environments. We finish off by showing you various techniques to monitor, test, and secure your microservices.
By the end, you will know the benefits of system resilience of a microservice and the advantages of Go stack.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 474
Veröffentlichungsjahr: 2017
Ähnliche
BIRMINGHAM - MUMBAI
Building Microservices with Go
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: July 2017
Production reference: 2100119
ISBN 978-1-78646-866-6
www.packtpub.com
Credits
Author
Nic Jackson
Copy Editor
Karuna Narayan
Reviewers
Magnus Larsson
Erik Lupander
Project Coordinator
Vaidehi Sawant
Commissioning Editor
Kunal Parikh
Proofreader
Safis Editing
Acquisition Editor
Karan Sadawana
Indexer
Francy Puthiry
Content Development Editor
Zeeyan Pinheiro
Graphics
Jason Monteiro
Technical Editor
Vivek Pala
Production Coordinator
Nilesh Mohite
About the Author
Nic Jackson is a developer advocate working for HashiCorp.com; he has over 20 years, experience in software development and leading software development teams. A huge fan of mobile application and microservice architecture, he is constantly looking out for the most efficient way to reuse code and improve development flow. In his spare time, Nic coaches and mentors at Coder Dojo teaches at Women Who Go and GoBridge, speaks and evangelizes good coding practice, process, and technique.
About the Reviewers
Magnus Larsson has been in the IT business since 1986. He is an experienced architect in areas such as distributed systems, systems integration, and SOA. He is currently engaged in exploring the benefits of modern technologies such as microservices, container technologies, reactive frameworks, and mobile solutions.
Magnus has a special dedication to the open source community for Java and has been active in various projects over the years. He enjoys exploring other languages and currently finds the Go language very interesting for building microservices. He is also a frequent speaker at conferences such as Cadec, Spring I/O, Jfokus, and jDays.
Magnus lives outside Gothenburg, Sweden, with his family. When time permits, he enjoys cross-country skiing, which must be done either on roller skis or indoors, in the Gothenburg area for most of the year.
He has worked for large corporations such as Digital Equipment Corporation, AstraZeneca, and Ericsson Hewlett Packard Telecom over the years. In 2000, Magnus co-founded Callista Enterprise AB, a Swedish-based consultancy company specialized in software architecture.
Erik Lupander is a software architect and developer with over 15 years of professional experience. A lifelong computer and software enthusiast, he wrote his first GW-BASIC programs at the age of 7 back in the mid-80s.
Erik holds an M.Sc. in applied informatics from the University of Gothenburg and has worked in a variety of roles in the software industry ever since. While JVM-based languages and architecture has been his bread and butter, he is a polyglot software craftsman at heart who, among other technologies, has embraced the Go programming language and microservice architecture.
Erik has spoken at software conferences such as Scandev (2012), dev:mobile (2014), and Cadec (2016, 2017) about topics ranging from OpenGL ES to Golang and microservices.
He lives just outside Gothenburg, Sweden, with his wife and two children. Aside from family, computers, and software, he enjoys alpine skiing, golf, and running, and he is an avid supporter of IFK Gothenburg.
Erik is currently employed by Callista Enterprise AB, a Swedish-based consultancy specialized in software architecture. His previous employers include Siemens Medical Solutions, Epsilon IT, University of Gothenburg, and Squeed AB.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1786468662.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Title Page
Copyright
Building Microservices with Go
Credits
About the Author
About the Reviewers
www.PacktPub.com
Why subscribe?
Customer Feedback
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
Introduction to Microservices
Building a simple web server with net/http
Reading and writing JSON
Marshalling Go structs to JSON
Unmarshalling JSON to Go structs
Routing in net/http
Paths
Convenience handlers
FileServer
NotFoundHandler
RedirectHandler
StripPrefix
TimeoutHandler
Static file handler
Creating handlers
Context
Background
WithCancel
WithDeadline
WithTimeout
WithValue
Using contexts
RPC in the Go standard library
Simple RPC example
RPC over HTTP
JSON-RPC over HTTP
Summary
Designing a Great API
RESTful APIs
URIs
URI format
URI path design for REST services
Collections
Documents
Controller
Store
CRUD function names
HTTP verbs
GET
POST
PUT
PATCH
DELETE
HEAD
OPTIONS
URI query design
Response codes
2xx Success
200 OK
201 Created
204 No Content
3xx Redirection
301 Moved Permanently
304 Not Modified
4xx Client Error
400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found
405 Method Not Allowed
408 Request Timeout
5xx Server Error
500 Internal Server Error
503 Service Unavailable
HTTP headers
Standard request headers
Authorization - string
Date
Accept - content type
Accept-Encoding - gzip, deflate
Standard response headers
Returning errors
Accessing APIs from JavaScript
JSONP
CORS
RPC APIs
RPC API design
RPC message frameworks
Gob
Thrift
Protocol Buffers
JSON-RPC
Filtering
Versioning APIs
Semantic versioning
Versioning formats for REST APIs
Versioning formats for RPC APIs
Naming for RPC
Object type standardization
Dates
Durations
Intervals
Documenting APIs
REST based-based APIs
Swagger
API Blueprint
RAML
RPC based-based APIs
Summary
Introducing Docker
Introducing Containers with Docker
Installing Docker
Running our first container
Docker volumes
Union filesystem
Mounting volumes
Docker ports
Removing a container starting with an explicit name
Docker networking
Bridge networking
Host networking
No network
Overlay network
Custom network drivers
Weaveworks
Project Calico
Creating custom bridge networks
Creating a bridge network
Connecting containers to a custom network
Writing Dockerfiles
Building application code for Docker
FROM
MAINTAINER
EXPOSE
COPY
ENTRYPOINT
CMD
Good practice for creating Dockerfiles
Building images from Dockerfiles
Docker build context
Docker Ignore files
Running Daemons in containers
Docker Compose
Installing Docker Compose on Linux
Service startup
Specifying the location of a compose file
Specifying a project name
Summary
Testing
The testing pyramid
Outside-in development
Unit tests
httptest.NewRequest
httptest.NewRecorder
Dependency injection and mocking
Code coverage
Behavioral Driven Development
Testing with Docker Compose
Benchmarking and profiling
Benchmarks
Profiling
Summary
Common Patterns
Design for failure
Patterns
Event processing
Event processing with at least once delivery
Handling Errors
Dead Letter Queue
Idempotent transactions and message order
Atomic transactions
Timeouts
Back off
Circuit breaking
Health checks
Throttling
Service discovery
Server-side service discovery
Client-side service discovery
Load balancing
Caching
Premature optimization
Stale cache in times of database or downstream service failure
Summary
Microservice Frameworks
What makes a good microservice framework?
Micro
Setup
Code generation
Tooling (CI/CD, cross platform)
Maintainable
Format (REST/RPC)
Patterns
Language independence
Ability to interface with other frameworks
Efficiency
Quality
Open source
Security
Support
Extensibility
What we learned about Micro
Kite
Setup
Code generation
Tooling
Maintainable
Format
Patterns
Language independence
Efficiency
Quality
Open source
Security
Support
Extensibility
Summing up Kite
gRPC
Setup
Code generation
Tooling
Maintainable
Format
Patterns
Language independence
Efficiency
Quality
Open source
Security
Support
Extensibility
A few lines about gRPC
Summary
Logging and Monitoring
Logging best practices
Metrics
Types of data best represented by metrics
Naming conventions
Storage and querying
Software as a Service
Self-hosted
Grafana
Logging
Distributed tracing with Correlation IDs
Elasticsearch, Logstash, and Kibana (ELK)
Kibana
Exceptions
Panic and recover
Panic
Recover
Summary
Security
Encryption and signing
Symmetric-key encryption
Public-key cryptography
Digital signatures
X.509 digital certificates
TLS/SSL
External security
Layer 2 or 3 firewalls
Web application firewall
API Gateway
DDoS protection
Types of DDoS attack
UDP fragment attack
UDP flood
DNS
NTP
CHARGEN
SYN flood
SSDP
ACK
Application security
Prevention
Detection
Response
Recovery
Confused deputy
How an attacker could bypass the firewall
Scenario
Attack
Input validation
Fuzzing
TLS
Generating private keys
Generating X.509 certificates
Securing data at rest
Physical machine access
OWASP
Never storing session tokens in a URL
Cross-site scripting (XSS) and cross-site request forgery (CRSF)
Insecure direct object references
Authentication and authorization
Password hashing
Adding a little seasoning
Dictionary attacks
Adding a pepper
bcrypt
Benchmarks
JWTs
Format of a JWT
Secure messages
Shared secrets
Asymmetric encryption with large messages
Maintenance
Patching containers
Software updates
Patching application code
Logging
Summary
Event-Driven Architecture
Differences between synchronous and asynchronous processing
Synchronous processing
Asynchronous processing
Types of asynchronous messages
Pull/queue messaging
Push messaging
Command Query Responsibility Segregation (CQRS)
Domain-driven design
What is DDD?
Technical debt
Anatomy of DDD
Strategic design
Tactical design
Ubiquitous language
Bounded contexts
Context mapping
Software
Kafka
NATS.io
AWS SNS/SQS
Google Cloud Pub/Sub
Summary
Continuous Delivery
What is continuous delivery?
Manual deployment
The benefits of continuous delivery
Aspects of continuous delivery
Reproducibility and consistency
Artifact storage
Automation of tests
Automation of integration tests
Infrastructure as code
Security scanning
Static code analysis
Smoke tests
End-to-end tests
Monitoring
Continuous delivery process
Overview
What is container orchestration?
Options for container orchestration
What is immutable infrastructure?
Terraform
Providers
Terraform config entry point
VPC module
Output variables
Creating the infrastructure
Example application
Continuous delivery workflow
Build
Testing
Benchmarking
Static code analysis
Integration tests
Deployment
Smoke tests
Monitoring/alerting
Complete workflow
Summary
Preface
Microservice architecture is sweeping the world as the de facto pattern for building web-based applications. Golang is a language particularly well suited to building them. Its strong community, encouragement of idiomatic style, and statically-linked binary artifacts make integrating it with other technologies and managing microservices at scale consistent and intuitive. This book will teach you the common patterns and practices, and show you how to apply these using the Go programming language.
It will teach you the fundamental concepts of architectural design and RESTful communication, and introduce you to the patterns that provide manageable code that is supportable in development and at scale in production. We will provide you with examples of how to put these concepts and patterns into practice with Go.
Whether you are planning a new application or working in an existing monolith, this book will explain and illustrate with practical examples how teams of all sizes can start solving problems with microservices. It will help you understand Docker and Docker Compose, and how they can be used to isolate microservice dependencies and build environments. We will conclude by showing you various techniques to monitor, test, and secure your microservices.
By the end, you will know the benefits of the system resilience of a microservice and the advantages of the Go stack.
What this book covers
Chapter 1, Introduction to Microservices, looks at what makes the Go language suitable for building microservices and takes a look at the standard library that has all the components required to build a basic microservice. Looking at the standard elements first will give you a grounding and make you appreciate how some of the frameworks that we will discuss later can be incredibly useful.
Chapter 2, Designing a Great API, looks at what makes a good API. We will introduce both REST and RPC, explaining the differences between them. We will also examine the best practices for writing and versioning APIs.
Chapter 3, Introducing Docker, explains how you can wrap your application into a Docker image and how you can use Docker and Docker Compose as part of your development workflow. We will see how it is possible to build a small lightweight image for your application and some good practices for using Docker and writing Dockerfiles.
Chapter 4, Testing, will introduce the various techniques to ensure that your microservices are of the highest quality. We will look at unit testing, behavioral testing, and performance testing, providing you with practical advice and knowledge of the core testing frameworks.
Chapter 5, Common Patterns, introduces some of the standard patterns often employed in microservice architecture. We will take an in-depth look at load balancing, circuit breaking, service discovery, and the autopilot pattern to see what a Go-specific implementation for this would look like.
Chapter 6, Microservice Frameworks, builds on frameworks that implement many of the common features needed for a microservice. We will compare and contrast these through examples of their usage.
Chapter 7, Logging and Monitoring, examines essential techniques to ensure that your service is behaving correctly, and when it does not, ensures you have all the information at your disposal for successful diagnostics and debugging. We will look at using StatsD for simple metrics and timings, how to handle log file information, and approaches to logging more detailed data and platforms such as NewRelic, which provides a holistic overview of your service.
Chapter 8, Security, takes a look at authentication, authorization, and security for your microservice. We will look at JWT and how you can implement middleware for validating your requests and keeping things secure. We will also look at the bigger picture, looking at why you should implement TLS encryption and a principle of no trust between your services.
Chapter 9, Event-Driven Architecture, discusses that a common pattern to allow your microservices to collaborate using Events; you will learn about two of the most common eventing patterns and see how you can implement them in Go. We will also look at the introduction of Domain-Driven Design and how the use of a ubiquitous language can help your software development process.
Chapter 10, Continuous Delivery, discusses the concepts behind continuous delivery. We will then examine in detail a continuous delivery setup for one of the simple applications we created earlier in the book.
What you need for this book
Go compiler for running your Go codes successfully. You can find it on https://golang.org. You also need Docker for container applications. Dockers are available on https://www.docker.com/.
Who this book is for
If you are looking to apply techniques to your own projects by taking your first steps into microservice architecture, this book is for you.
Conventions
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows:
"The @SpringBootApplication annotation replaces the different annotation required in the Spring framework."
A block of code is set as follows:
@SpringBootApplication@EnableZuulProxypublic class ApiGatewayExampleInSpring{ public static void main(String[] args) { SpringApplication.run(ApiGatewayExampleInSpring.class, args); }}
Any command-line input or output is written as follows:
mvn spring-boot:run
New terms and important words are shown in bold.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the
SUPPORT
tab at the top.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on
Code Download
.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Building-Microservices-with-Go. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.
Introduction to Microservices
First, we are going to look at how easy it is to create a simple web server with a single endpoint using the net/http package. Then, we will move on to examine the encoding/json package to see just how easy Go makes it for us to use JSON objects for our requests and our responses. Finally, we will look at how routing and handlers work and how we can manage context between these handlers.
Building a simple web server with net/http
The net/http package provides all the features we need to write HTTP clients and servers. It gives us the capability to send requests to other servers communicating using the HTTP protocol as well as the ability to run a HTTP server that can route requests to separate Go funcs, serve static files, and much more.
To begin we should ask the question, what technical book would be complete without a simple hello world example? I say none and this is exactly where we will begin.
In this example, we are going to create an HTTP server with a single endpoint that returns static text represented by the JSON standard, this will introduce the basic functions of the HTTP server and handlers. We will then modify this endpoint to accept a request that is encoded in JSON and using the encoding/json package return a response to the client. We will also examine how the routing works by adding a second endpoint that returns a simple image.
By the end of this chapter, you will have a fundamental grasp of the basic packages and how you can use them to quickly and efficiently build a simple microservice.
Building a web server in Go is incredibly easy thanks to the HTTP package, which is distributed as part of the standard library.
It has everything you need to manage routing, dealing with Transport Layer Security (TLS), which we will cover in Chapter 8, Security, support for HTTP/2 out of the box, and the capability to run an incredibly efficient server that can deal with a huge number of requests.
The source code for this chapter can be found on GitHub at http://github.com/building-microservices-with-go/chapter1.git, all the examples in this and subsequent chapters will reference the source extensively so if you have not already done so, go and clone this repo before continuing.
Let's look at the syntax for creating a basic server then we can walk through the packages in more depth:
Example 1.0 basic_http_example/basic_http_example.go
09 func main() { 10 port := 8080 11 12 http.HandleFunc("/helloworld", helloWorldHandler) 13 14 log.Printf("Server starting on port %v\n", 8080) 15 log.Fatal(http.ListenAndServe(fmt.Sprintf(":%v", port), nil)) 16 } 17 18 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 19 fmt.Fprint(w, "Hello World\n") 20 }
The first thing we are doing is calling the HandleFunc method on the http package. The HandleFunc method creates a Handler type on the DefaultServeMux handler, mapping the path passed in the first parameter to the function in the second parameter:
func HandleFunc(pattern string, handler func(ResponseWriter, *Request))
In line 15 we start the HTTP server, ListenAndServe takes two parameters, the TCP network address to bind the server to and the handler that will be used to route requests:
func ListenAndServe(addr string, handler Handler) error
In our example, we are passing the network address :8080" this means we would like to bind the server to all available IP addresses on port 8080.
The second parameter we are passing is nil, this is because we are using the DefaultServeMux handler, which we are setting up with our call to http.HandleFunc. In Chapter 3, Introducing Docker, you will see the use of this second parameter when we introduce more sophisticated routers, but for now we can ignore it.
If the ListenAndServe function fails to start a server it will return an error, the most common reason for this is that you may be trying to bind to a port that is already in use on the server. In our example, we are passing the output of ListenAndServe straight to log.Fatal(error), which is a convenience function equivalent to calling fmt.Print(a ...interface{}) followed by a call to os.Exit(1). Since ListenAndServe blocks if the server starts correctly we will never exit on a successful start.
Let's quickly run and test our new server:
$ go run ./basic_http_example.go
You should now see the application output:
2016/07/30 01:08:21 Server starting on port 8080
What if you do not see the preceding output and instead see something like the following?
2016/07/19 03:51:11 listen tcp :8080: bind: address already in use exit status 1
Take another look at the signature of ListenAndServe and the way we are calling it. Remember what we were saying about why we were using log.Fatal?
If you do get this error message it means that you are already running an application on your computer that is using port 8080, this could be another instance of your program or it could be another application. You can check that you do not have another instance running by checking the running processes:
$ ps -aux | grep 'go run'
If you do see another go run ./basic_http_example.go then you can simply kill it and retry. If you do not have another instance running, then you probably have some other software that is bound to this port. Try changing the port on line 10 and restart your program.
To test the server, open a new browser and type in the URI http://127.0.0.1:8080/helloworld and if things are working correctly you should see the following response from the server:
Hello World
Congratulations, that's the first step into microservice mastery. Now that we have our first program running, let's take a closer look at how we can return and accept JSON.
Reading and writing JSON
Thanks to the encoding /json package, which is built into the standard library encoding and decoding JSON to and from Go types is both fast and easy. It implements the simplistic Marshal and Unmarshal functions; however, if we need them, the package also provides Encoder and Decoder types that allow us greater control when reading and writing streams of JSON data. In this section, we are going to examine both of these approaches, but first let's take a look at how simple it is to convert a standard Go struct into its corresponding JSON string.
Marshalling Go structs to JSON
To encode JSON data, the encoding/json package provides the Marshal function, which has the following signature:
func Marshal(v interface{}) ([]byte, error)
This function takes one parameter, which is of type interface, so pretty much any object you can think of since interface represents any type in Go. It returns a tuple of ([]byte, error), you will see this return style quite frequently in Go, some languages implement a try catch approach that encourages an error to be thrown when an operation cannot be performed, Go suggests the pattern (return type, error), where the error is nil when an operation succeeds.
In Go, unhandled errors are a bad thing, and whilst the language does implement Panic and Recover, which resemble exception handling in other languages, the situations where you should use these are quite different (see The Go Programming Language, Kernaghan). In Go, the panic function causes normal execution to stop and all deferred function calls in the Go routine are executed, the program will then crash with a log message. It is generally used for unexpected errors that indicate a bug in the code and good robust Go code will attempt to handle these runtime exceptions and return a detailed error object back to the calling function.
This pattern is exactly what is implemented with the Marshal function. In the instance that Marshal cannot create a JSON encoded byte array from the given object, which could be due to a runtime panic, then this is captured and an error object detailing the problem is returned to the caller.
Let's try this out, expanding on our existing example, instead of simply printing a string from our handler, let's create a simple struct for the response and return this instead.
Example 1.1 reading_writing_json_1/reading_writing_json_1.go
10 type helloWorldResponse struct { 11 Message string 12 }
In our handler, we will create an instance of this object, set the message, then use the Marshal function to encode it to a string before returning.
Let's see what that will look like:
23 func helloWorldHandler(w http.ResponseWriter, r *http.Request) { 24 response := helloWorldResponse{Message: "HelloWorld"} 25 data, err := json.Marshal(response) 26 if err != nil { 27 panic("Ooops") 28 } 29 30 fmt.Fprint(w, string(data)) 31 }
Now, when we run our program again and refresh our browser, we see the following output rendered in valid JSON:
{"Message":"Hello World"}
This is awesome; however, the default behavior of Marshal is to take the literal name of the field and use this as the field in the JSON output. What if I prefer to use camel case and would rather see "message", could we just rename the field in the helloWorldResponse struct?
Unfortunately we can't, as in Go, lowercase properties are not exported, Marshal will ignore these and will not include them in the output.
All is not lost as the encoding/json package implements struct field attributes that allow us to change the output for the property to anything we choose.
Example 1.2 reading_writing_json_2/reading_writing_json_2.go
10 type helloWorldResponse struct { 11 Message string `json:"message"` 12 }
Using the struct field's tags, we can have greater control of how the output will look. In the preceding example, when we marshal this struct the output from our server would be:
{"message":"Hello World"}
This is exactly what we want, but we can use field tags to control the output even further. We can convert object types and even ignore a field altogether if we need to:
type helloWorldResponse struct {// change the output field to be "message" Message string `json:"message"` // do not output this field Author string `json:"-"` // do not output the field if the value is empty Date string `json:",omitempty"` // convert output to a string and rename "id" Id int `json:"id, string"` }
Channel, complex types, and functions cannot be encoded in JSON; attempting to encode these types will result in an UnsupportedTypeError being returned by the Marshal function.
It also can't represent cyclic data structures; if your stuct contains a circular reference then Marshal will result in an infinite recursion, which is never a good thing for a web request.
If we want to export our JSON prettily formatted with indentation, we can use the MarshallIndent function, this allows you to pass an additional parameter of string to specify what you would like the indent to be. Two spaces right, not a tab?
func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error)
The astute reader might have noticed that we are decoding our struct into a byte array and then writing that to the response stream, this does not seem to be particularly efficient and in fact it is not. Go provides Encoders and Decoders, which can write directly to a stream, since we already have a stream with the ResponseWriter then let's do just that.
Before we do, I think we need to look at the ResponseWriter a little to see what is going on there.
The ResponseWriter is an interface that defines three methods:
// Returns the map of headers which will be sent by the // WriteHeader method. Header() // Writes the data to the connection. If WriteHeader has not // already been called then Write will call // WriteHeader(http.StatusOK). Write([]byte) (int, error) // Sends an HTTP response header with the status code. WriteHeader(int)
If we have a ResponseWriter interface, how can we use this with fmt.Fprint(w io.Writer, a ...interface{})? This method requires a Writer interface as a parameter and we have a ResponseWriter interface. If we look at the signature for Writer we can see that it is:
Write(p []byte) (n int, err error)
Because the ResponseWriter interface implements this method, it also satisfies the interface Writer and therefore any object that implements ResponseWriter can be passed to any function that expects Writer.
Amazing, Go rocks, but we have not answered our question, Is there any better way to send our data to the output stream without marshalling to a temporary object before we return it?
The encoding/json package has a function called NewEncoder this returns us an Encoder object that can be used to write JSON straight to an open writer and guess what; we have one of those:
func NewEncoder(w io.Writer) *Encoder
So instead of storing the output of Marshal into a byte array, we can write it straight to the HTTP response.
Example 1.3 reading_writing_json_3/reading_writing_json_3.go:
func helloWorldHandler(w http.ResponseWriter, r *http.Request) { response := HelloWorldResponse{Message: "HelloWorld"} encoder := json.NewEncoder(w) encoder.Encode(&response) }
We will look at benchmarking in a later chapter, but to see why this is important we have created a simple benchmark to check the two methods against each other, have a look at the output.
Example 1.4 reading_writing_json_2/reading_writing_json_2.go:
$go test -v -run="none" -bench=. -benchtime="5s" -benchmem
BenchmarkHelloHandlerVariable-8 20000000 511 ns/op 248 B/op 5 allocs/op
BenchmarkHelloHandlerEncoder-8 20000000 328 ns/op 24 B/op 2 allocs/op
BenchmarkHelloHandlerEncoderReference-8 20000000 304 ns/op 8 B/op 1 allocs/op
PASS
ok github.com/building-microservices-with-go/chapter1/reading_writing_json_2 24.109s
Using Encoder rather than marshalling to a byte array is nearly 50% faster. We are dealing with nanoseconds here so that time may seem irrelevant, but it isn't; this was two lines of code. If you have that level of inefficiency throughout the rest of your code then your application will run slower, you will need more hardware to satisfy the load and that will cost you money. There is nothing clever in the differences between the two methods all we have done is understood how the standard packages work and chosen the correct option for our requirements, that is not performance tuning, that is understanding the framework.
Routing in net/http
Even a simple microservice will need the capability to route requests to different handlers dependent on the requested path or method. In Go this is handled by the DefaultServeMux method which is an instance of ServerMux. Earlier in this chapter, we briefly covered that when nil is passed to the handler parameter for the ListenAndServe function then the DefaultServeMux method is used. When we call the http.HandleFunc("/helloworld", helloWorldHandler) package function we are actually just indirectly calling http.DefaultServerMux.HandleFunc(…).
The Go HTTP server does not have a specific router instead any object which implements the http.Handler interface is passed as a top level function to the Listen() function, when a request comes into the server the ServeHTTP method of this handler is called and it is responsible for performing or delegating any work. To facilitate the handling of multiple routes the HTTP package has a special object called ServerMux, which implements the http.Handler interface.
There are two functions to adding handlers to a ServerMux handler:
func HandlerFunc(pattern string, handler func(ResponseWriter, *Request)) func Handle(pattern string, handler Handler)
The HandleFunc function is a convenience function that creates a handler who's ServeHTTP method calls an ordinary function with the func(ResponseWriter, *Request) signature that you pass as a parameter.
The Handle function requires that you pass two parameters, the pattern that you would like to register the handler and an object that implements the Handler interface:
type Handler interface { ServeHTTP(ResponseWriter, *Request) }
Paths
We already explained how ServeMux is responsible for routing inbound requests to the registered handlers, however the way that the routes are matched can be quite confusing. The ServeMux handler has a very simple routing model it does not support wildcards or regular expressions, with ServeMux you must be explicit about the registered paths.
You can register both fixed rooted paths, such as /images/cat.jpg, or rooted subtrees such as /images/. The trailing slash in the rooted subtree is important as any request that starts with /images/, for example /images/happy_cat.jpg, would be routed to the handler associated with /images/.
If we register a path /images/ to the handler foo, and the user makes a request to our service at /images (note no trailing slash), then ServerMux will forward the request to the /images/ handler, appending a trailing slash.
If we also register the path /images (note no trailing slash) to the handler bar and the user requests /images then this request will be directed to bar; however, /images/ or /images/cat.jpg will be directed to foo:
http.Handle("/images/", newFooHandler())http.Handle("/images/persian/", newBarHandler())http.Handle("/images", newBuzzHandler())/images => Buzz/images/ => Foo/images/cat => Foo/images/cat.jpg => Foo/images/persian/cat.jpg => Bar
Longer paths will always take precedence over shorter ones so it is possible to have an explicit route that points to a different handler to a catch all route.
We can also specify the hostname, we could register a path such as search.google.com/ and /ServerMux would forward any requests to http://search.google.com and http://www.google.com to their respective handlers.
If you are used to a framework based application development approach such as using Ruby on Rails or ExpressJS you may find this router incredibly simple and it is, remember that we are not using a framework but the standard packages of Go, the intention is always to provide a basis that can be built upon. In very simple cases the ServeMux approach more than good enough and in fact I personally don't use anything else. Everyone's needs are different however and the beauty and simplicity of the standard packages makes it incredibly simple to build your own route as all is needed is an object which implements the Handler interface. A quick trawl through google will surface some very good third party routers but my recommendation for you is to learn the limitations of ServeMux first before deciding to choose a third-party package it will greatly help with your decision process as you will know the problem you are trying to solve.
Convenience handlers
The net/http package implements several methods that create different types of convenience handlers, let's examine these.
FileServer
A FileServer function returns a handler that serves HTTP requests with the contents of the filesystem. This can be used to serve static files such as images or other content that is stored on the file system:
func FileServer(root FileSystem) Handler
Take a look at the following code:
http.Handle("/images", http.FileServer(http.Dir("./images")))
This allows us to map the contents of the file system path ./images to the server route /images, Dir implements a file system which is restricted to a specific directory tree, the FileServer method uses this to be able to serve the assets.
NotFoundHandler
The NotFoundHandler function returns a simple request handler that replies to each request with a 404 page not found reply:
func NotFoundHandler() Handler
RedirectHandler
The RedirectHandler function returns a request handler that redirects each request it receives to the given URI using the given status code. The provided code should be in the 3xx range and is usually StatusMovedPermanently, StatusFound, or StatusSeeOther:
func RedirectHandler(url string, code int) Handler
StripPrefix
The StripPrefix function returns a handler that serves HTTP requests by removing the given prefix from the request URL's path and then invoking h handler. If a path does not exist, then StripPrefix will reply with an HTTP 404 not found error:
func StripPrefix(prefix string, h Handler) Handler
TimeoutHandler
The TimeoutHandler function returns a Handler interface that runs h with the given time limit. When we investigate common patterns in Chapter 6, Microservice Frameworks, we will see just how useful this can be for avoiding cascading failures in your service:
func TimeoutHandler(h Handler, dt time.Duration, msg string) Handler
The new handler calls h.ServeHTTP to handle each request, but if a call runs for longer than its time limit, the handler responds with a 503 Service Unavailable response with the given message (msg) in its body.
The last two handlers are especially interesting as they are, in effect, chaining handlers. This is a technique that we will go into more in-depth in a later chapter as it allows you to both practice clean code and also allows you to keep your code DRY.
I may have lifted most of the descriptions for these handlers straight from the Go documentation and you probably have already read these because you have read the documentation right? With Go, the documentation is excellent and writing documentation for your own packages is heavily encouraged, even enforced, if you use the golint command that comes with the standard package then this will report areas of your code which do not conform to the standards. I really recommend spending a little time browsing the standard docs when you are using one of the packages, not only will you learn the correct usage, you may learn that there is a better approach. You will certainly be exposed to good practice and style and you may even be able to keep working on the sad day that Stack Overflow stops working and the entire industry grinds to a halt.
Static file handler
Whilst we are mostly going to be dealing with APIs in this book, it is a useful illustration to see how the default router and paths work by adding a secondary endpoint.
As a little exercise, try to modify the code in reading_writing_json_5/reading_writing_json_5.go to add an endpoint /cat, which returns the cat picture specified in the URI. To give you a little hint, you are going to need to use the FileServer function on the net/http package and your URI will look something like http://localhost:8080/cat/cat.jpg.
Did it work the first time or did you forget to add the StripPrefix handler?
Example 1.7 reading_writing_json_6/reading_writing_json_6.go:
21 cathandler := http.FileServer(http.Dir("./images")) 22 http.Handle("/cat/", http.StripPrefix("/cat/", cathandler))
In the preceding example, we are registering a StripPrefix handler with our path /cat/. If we did not do this, then the FileServer handler would be looking for our image in the images/cat directory. It is also worth reminding ourselves about the difference with /cat and /cat/ as paths. If we registered our path as /cat then we would not match /cat/cat.jpg. If we register our path as /cat/, we will match both /cat and /cat/whatever.
Creating handlers
We will now finish off our examples here by showing how you can create a Handler rather than just using HandleFunc. We are going to split the code that performs the request validation for our helloworld endpoint and the code that returns the response out into separate handlers to illustrate how it is possible to chain handlers.
Example 1.8 chapter1/reading_writing_json_7.go:
31 type validationHandler struct { 32 next http.Handler 33 } 34 35 func newValidationHandler(next http.Handler) http.Handler { 36 return validationHandler{next: next} 37 }
The first thing we need to do when creating our own Handler is to define a struct field that will implement the methods in the Handlers interface. Since in this example, we are going to be chaining handlers together, the first handler, which is our validation handler, needs to have a reference to the next in the chain as it has the responsibility for calling ServeHTTP or returning a response.
For convenience, we have added a function that returns us a new handler; however, we could have just set the next field. This method, however, is better form as it makes our code a little easier to read and when we need to pass complex dependencies to the handler using a function to create, it keeps things a little neater:
37 func (h validationHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) {38 var request helloWorldRequest39 decoder := json.NewDecoder(r.Body)4041 err := decoder.Decode(&request)42 if err != nil {43 http.Error(rw, "Bad request", http.StatusBadRequest)44 return45 }4647 h.next.ServeHTTP(rw, r)48 }
The previous code block illustrates how we would implement the ServeHTTP method. The only interesting thing to note here is the statement that begins at line 44. If an error is returned from decoding the request, we write a 500 error to the response, the handler chain would stop here. Only when no error is returned do we call the next handler in the chain and we do this simply by invoking its ServeHTTP method. To pass the name decoded from the request, we are simply setting a variable:
53 type helloWorldHandler struct{} 54 55 func newHelloWorldHandler() http.Handler { 56 return helloWorldHandler{} 57 } 58 59 func (h helloWorldHandler) ServeHTTP(rw http.ResponseWriter, r *http.Request) { 60 response := helloWorldResponse{Message: "Hello " + name} 61 62 encoder := json.NewEncoder(rw) 63 encoder.Encode(response) 64 }
The helloWorldHandler type that writes the response does not look too different from when we were using a simple function. If you compare this to example 1.6, you will see that all we really have done is remove the request decoding.
Now the first thing I want to mention about this code is that it is purely to illustrate how you can do something, not that you should do something. In this simple case, splitting the request validation and response sending into two handlers adds a lot of needless complexity and it is not really making our code DRYer. The technique, however, is useful. When we examine authentication in a later chapter, you will see this pattern as it allows us to centralize our authentication logic and share it among handlers.
Context
The problem with the previous pattern is that there is no way that you can pass the validated request from one handler to the next without breaking the http.Handler interface, but guess what Go has us covered. The context package was listed as experimental for several years before finally making it in to the standard package with Go 1.7. The Context type implements a safe method for accessing request-scoped data that is safe to use simultaneously by multiple Go routines. Let’s take a quick look at this package and then update our example to see it in use.
Background
The Background method returns an empty context that has no values; it is typically used by the main function and as the top-level Context:
func Background() Context
WithCancel
The WithCancel method returns a copy of the parent context with a cancel function, calling the cancel function releases resources associated with the context and should be called as soon as operations running in the Context type are complete:
func WithCancel(parent Context) (ctx Context, cancel CancelFunc)
WithDeadline
The WithDeadline method returns a copy of the parent context that expires after the current time is greater than deadline. At this point, the context's Done channel is closed and the resources associated are released. It also passes back a CancelFunc method that allows manual cancellation of the context:
func WithDeadline(parent Context, deadline time.Time) (Context, CancelFunc)
WithTimeout
The WithTimeout method is similar to WithDeadline except you pass it a duration for which the Context type should exist. Once this duration has elapsed, the Done channel is closed and the resources associated with the context are released:
func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc)
WithValue
The WithValue method returns a copy of the parent Context in which the val value is associated with the key. The Context values are perfect to be used for request-scoped data:
func WithValue(parent Context, key interface{}, val interface{}) Context
Why not attempt to modify example 1.7 to implement a request scoped context. The key could be in the previous sentence; every request needs its own context.
Using contexts
You probably found that rather painful, especially if you come from a background in a framework such as Rails or Spring. Writing this kind of code is not really something you want to be spending your time on, building application features is far more important. One thing to note however is that neither Ruby or Java have anything more advanced in their base packages. Thankfully for us, over the seven years that Go has been in existence, many excellent people have done just that, and when looking at frameworks in Chapter 3, Introducing Docker, we will find that all of this complexity has been taken care of by some awesome open source authors.
