Building Distributed Applications in Gin E-Book

Mohamed Labouardy

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 277

Veröffentlichungsjahr: 2021

Beschreibung

Gin is a high-performance HTTP web framework used to build web applications and microservices in Go. This book is designed to teach you the ins and outs of the Gin framework with the help of practical examples.
You’ll start by exploring the basics of the Gin framework, before progressing to build a real-world RESTful API. Along the way, you’ll learn how to write custom middleware and understand the routing mechanism, as well as how to bind user data and validate incoming HTTP requests. The book also demonstrates how to store and retrieve data at scale with a NoSQL database such as MongoDB, and how to implement a caching layer with Redis. Next, you’ll understand how to secure and test your API endpoints with authentication protocols such as OAuth 2 and JWT. Later chapters will guide you through rendering HTML templates on the server-side and building a frontend application with the React web framework to consume API responses. Finally, you’ll deploy your application on Amazon Web Services (AWS) and learn how to automate the deployment process with a continuous integration and continuous delivery (CI/CD) pipeline.
By the end of this Gin book, you will be able to design, build, and deploy a production-ready distributed application from scratch using the Gin framework.

Leseprobe

Building Distributed Applications in Gin

A hands-on guide for Go developers to build and deploy distributed web apps with the Gin framework

Mohamed Labouardy

BIRMINGHAM—MUMBAI

Building Distributed Applications in Gin

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Kunal Chaudhari

Acquisition Editor: Denim Pinto

Publishing Product Manager: Ashish Tiwari

Senior Editor: Rohit Singh

Content Development Editor: Ananya Endow

Technical Editor: Gaurav Gala

Copy Editor: Safis Editing

Project Coordinator: Deeksha Thakkar

Proofreader: Safis Editing

Indexer: Vinayak Purushotham

Production Designer: Joshua Misquitta

Published: June 2021

Production reference: 2200821

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80107-485-8

www.packt.com

I want to thank my wife, Mounia. You've always supported me, always patiently listened while I struggled to get this done, and always made me believe I could finish this. I love you.

Thanks as well to all the folks at Packt who worked with me on the production and promotion of the book. It was truly a team effort.

Contributors

About the author

MohamedLabouardy is the CTO and cofounder of Crew.work, and a DevSecOps evangelist. He is the founder of Komiser.io, a regular conference speaker, and the author of several books about serverless and distributed applications.

You can find him on Twitter (@mlabouardy).

This book is dedicated to my parents, even though they will never know unless someone tells them about this.

About the reviewer

Dmitry Goryunov was born in 1987 in Tomsk, a really nice city in Siberia that many talented engineers call their home. His first coding experience was writing macros for Ultima Online at the age of 12.

Ever since 2007, throughout his entire career, he has enjoyed building distributed software systems. Currently, he lives in Berlin and works for Zalando SE, where he is solving MLOps challenges for natural language processing systems.

Table of Contents

Preface

Section 1: Inside the Gin Framework

Chapter 1: Getting Started with Gin

Technical requirements

What is Gin?

Setting up the Go environment

Linux/FreeBSD10

Windows11

MacOS13

Integrated development environment15

Installing Go tools18

Installing and configuring Gin

Dependency management in Golang

Writing a custom HTTP handler

Summary

Questions

Further reading

Section 2: Distributed Microservices

Chapter 2: Setting Up API Endpoints

Technical requirements

Exploring API functionality

Defining the data model43

HTTP endpoints46

Implementing HTTP routes

POST /recipes46

GET /recipes51

PUT /recipes/{id}56

DELETE /recipes/{id}60

GET /recipes/search63

Writing the OpenAPI Specification

Installing Go Swagger66

Swagger metadata67

Summary

Questions

Further reading

Chapter 3: Managing Data Persistence with MongoDB

Technical requirements

Running a MongoDB Server

Installing Docker CE80

Running a MongoDB container83

Configuring Go's MongoDB driver

Exploring MongoDB queries

The InsertMany operation90

The Find operation94

The InsertOne operation95

The UpdateOne operation98

Designing the project's layout

Caching an API with Redis

Running Redis in Docker107

Optimizing MongoDB queries110

Performance benchmark122

Summary

Questions

Further reading

Chapter 4: Building API Authentication

Technical requirements

Exploring authentication

Using API keys129

Introducing JWTs

Sign-in HTTP handler137

Renewing a JWT 143

Hashing and salting passwords147

Persisting client sessions and cookies

Authenticating with Auth0

Building an HTTPS server

Self-signed certificates163

Summary

Questions

Further reading

Chapter 5: Serving Static HTML in Gin

Technical requirements

Serving static files

Rendering HTML templates

Creating the view templates187

Creating reusable templates194

Building a self-contained web application

Bundling static files202

Building a SPA

Getting started with React205

Exploring React components207

Resolving cross-origin requests212

Summary

Questions

Further reading

Chapter 6: Scaling a Gin Application

Technical requirements

Scaling workloads with a message broker

Deploying RabbitMQ with Docker234

Exploring the Producer/Consumer pattern237

Scaling horizontally with Docker replicas

Using Docker multi-stage builds248

Scaling services with Docker Compose250

Using the NGINX reverse proxy

Caching assets with HTTP cache headers

Setting HTTP caching headers270

Summary

Further reading

Section 3: Beyond the Basics

Chapter 7: Testing Gin HTTP Routes

Technical requirements

Testing Gin HTTP handlers

Generating code coverage reports

Performing integration tests with Docker286

Discovering security vulnerabilities

Gosec289

Securing Go modules with Snyk291

Running Postman collections

Scripting in Postman297

Running collections with Newman304

Summary

Questions

Further reading

Chapter 8: Deploying the Application on AWS

Technical requirements

Deploying on EC2 instance

Launching an EC2 instance310

SSL offloading with an application load balancer319

Deploying on Amazon ECS

Storing images in a private repository324

Creating an ECS cluster328

Deploying on Kubernetes with Amazon EKS

Configuring kubectl339

Migrating a Docker Compose workflow to Kubernetes340

Summary

Questions

Further reading

Chapter 9: Implementing a CI/CD Pipeline

Technical requirements

Exploring CI/CD practices

Continuous integration348

Continuous deployment349

Continuous delivery349

Building a CI workflow

Pipeline as Code352

Maintaining multiple runtime environments

Implementing continuous delivery

Improving the feedback loop with Slack384

Summary

Questions

Further reading

Chapter 10: Capturing Gin Application Metrics

Technical requirements

Exposing Gin metrics with Prometheus

Instrumenting a Gin application395

Monitoring server-side metrics

Creating a Grafana notification channel414

Streaming Gin logs to the ELK platform

Deploying the ELK stack with Docker419

Writing Grok expressions428

Updating the Gin logging format432

Summary

Further reading

Conclusion

Assessments

Chapter 1 – Getting started with Gin

Chapter 2 – Setting up API Endpoints

Chapter 3 – Managing Data Persistence with MongoDB

Chapter 4 – Building API Authentication

Chapter 5 – Serving Static HTML in Gin

Chapter 7 – Testing Gin HTTP Routes

Chapter 8 – Deploying the Application on AWS

Chapter 9 – Implementing a CI/CD Pipeline

Other Books You May Enjoy

Section 1: Inside the Gin Framework

In this part, we will cover the hype and performance benefits of the Gin Framework. This part will also cover how to write a simple Gin-based application. As such, it includes the following chapter:

Chapter 1, Getting Started with Gin

Chapter 1: Getting Started with Gin

This chapter will give you a foundational understanding of what the Gin framework is, how it works, and its features. We'll also supply guidelines for setting up the Go runtime and development environment. Moreover, we'll discuss the advantages of using Gin as a web framework for building distributed applications. We will finish this chapter by learning to write our first Gin-based web application.

In this chapter, we will cover the following topics:

What is Gin?Go runtime and integrated development environment (IDE)Go modules and dependency managementWriting a Gin web application

By the end of this chapter, you will be able to build a basic HTTP server with the Gin web framework.

Technical requirements

To follow along with this chapter, you will need the following:

Some programming experience. The code in this chapter is pretty simple, but it helps to know something about Go.A tool to edit your code with. Any text editor you have will work fine. Most text editors have good support for Go. The most popular are Visual Studio Code (VSCode) (free), GoLand (paid), and Vim (free).A command terminal. Go works well using any Terminal on Linux and Mac, and on PowerShell or CMD in Windows.

The code bundle for this chapter is hosted on GitHub at https://github.com/PacktPublishing/Building-Distributed-Applications-in-Gin/tree/main/chapter01.

What is Gin?

Before deep diving into the Gin web framework, we need to understand why Go is a top choice when it comes to building scalable and distributed applications.

Go (also referred to as Golang) is an open source programming language, developed by Robert Griesemer, Rob Pike, and Ken Thompson within Google in 2007. It is a compiled, statically typed language designed to enable users to easily write reliable, scalable, and highly efficient applications. The key features of Go are as follows:

Simple and consistent: Go has a rich set of library packages with powerful standard libraries for testing, error management, and concurrency.Fast and scalable: Go is a general-purpose programming language developed for the multi-core reality of today's computers. It has built-in concurrency with Goroutines and channels. Goroutines provide lightweight, threaded execution. Declaring a Goroutine is as simple as adding the go keyword before a function.Efficient: Go provides efficient execution and compilation. Go is also statically linked, which means that the compiler invokes a linker in the last step that resolves all library references. This means we would get one binary executable after compiling a Go program with no external dependencies. Moreover, it offers efficient memory utilization with a built-in garbage collector (Go exhibits many similarities with low-level programming languages such as C or C++).Community and support: Go is backed by Google and has an ever growing ecosystem and numerous contributors to the language on GitHub. Moreover, many online resources (tutorials, videos, and books) are available for getting started with Go.

Go has become hugely popular among enterprises and the open source community. Based on the StackOverflow Developer Survey 2020 (https://insights.stackoverflow.com/survey/2020), Go is in the top 5 of the most loved programming languages:

Figure 1.1 – Most loved programming languages according to the StackOverflow Survey 2020

Golang is known to be the number one choice when it comes to building large-scale, complex tools and cloud-based applications. The following image highlights the main open source projects that have been developed using Go:

Docker: A solution that's used to create, deploy, and run applications using containers.Kubernetes: A container orchestration platform for managing containers across a fleet of nodes/machines.Etcd: A reliable distributed key-value store used to store data for a distributed system or application.InfluxDB: A scalable time-series database designed to handle high write and query loads.CoreOS: A lightweight operating system designed to deploy container-based applications.Terraform: An infrastructure-as-code tool for building, changing, and versioning cloud infrastructure.CockroachDB: A cloud-native SQL database for data-intensive applications.Consul: A distributed store with service discovery, service mesh, and health check monitoring capabilities:

Figure 1.2 – Open source tools powered by Go

As we can see, Go is a solid language for distributed systems and infrastructure tools. Docker, Kubernetes, Prometheus, and others are built using Go.

Go is also known for building web applications of all shapes and sizes. This is partly due to the fantastic work that has been put into making the standard library clean, consistent, and easy to use. Perhaps one of the most important packages for any budding Go web developer is the net/http package. This package allows you to build HTTP servers in Go with its powerful compositional constructs.

To build a web application, you'll need to build an HTTP server. The client (for example, a browser) makes an HTTP request with some information; the server then processes that request and returns a response. The response can be in JSON, XML, or HTML format:

Figure 1.3 – HTTP client-server communication

This pattern of request-response is one of the key focal points in building web applications in Go.

While the net/http package allows you to craft a web application easily, the routing mechanism is not so powerful, especially for complex applications. That's where a web framework comes into play. The top Golang web frameworks are listed in the following table:

Gin is possibly the most used and largest running Go web framework. The framework has already harvested 48,210 stars and 5,100 forks in GitHub, which shows that the framework is very popular. This modular framework can be extended easily with minimal fuss. It is great to use because many components can be reused with a direct net/http package.

Important note

Another strong but conservative framework is Gorilla/Mux. It has one of the biggest online communities with many resources on the internet to teach you how to build end-to-end web applications.

According to the official documentation https://gin-gonic.com/docs/, Gin is described as follows:

"Gin is an HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster. If you need smashing performance, get yourself some Gin".

Gin is a minimalistic web framework suitable for building web applications, microservices, and RESTful APIs. It reduces boilerplate code by creating reusable and extensible pieces of code: you can write a piece of middleware that can be plugged into one or more request handlers. Moreover, it comes with the following key features:

Well documented: The documentation for Gin is broad and comprehensive. Most tasks that you will need to do relating to the router can be found easily in the docs.Simplicity: Gin is a pretty minimalistic framework. Only the most essential features and libraries are included, with little to no boilerplate to bootstrap applications, making Gin a great framework for developing highly available REST APIs.Extensible: The Gin community has created numerous pieces of well-tested middleware that make developing for Gin a charm. Features include compression with GZip, authentication with an authorization middleware, and logging with external solutions such as Sentry.Performance: Gin runs 40x faster than Martini and runs comparatively well compared to other Golang frameworks. The following is the results of a benchmark I ran against multiple Go libraries:

Figure 1.4 – Golang web framework benchmarks

Important note

This benchmark was performed on a macOS High Sierra, 2.7 GHz Intel Core i7, 16 GB DDR3 computer, with Go 1.15.6 as the runtime environment.

That being said, before you can write your first line of Go code, you'll need to set up the environment. Let's start by installing Go.

Setting up the Go environment

At the time of writing this book, the latest version of Go is Go 1.15.6. To install Go, you can either download or use the official binary distributions, or you can install Go from the source code (https://github.com/golang/go).

Important note

The official binary distributions are available for FreeBSD (release 8 and above), Linux (2.6.23 and above), macOS (Snow Leopard and above), and Windows (XP and above). Both the 32-bit (386) and 64-bit (amd64) x86 processor architectures are supported. For FreeBSD and Linux, the ARM processor architecture is also supported.

To install Go, download the distribution package from the https://golang.org/dl/ web page, as shown here, and choose the file that's appropriate for your platform:

Figure 1.5 – Golang available packages

Once you have the distribution package, install Go according to your platform of choice. We will cover this in the following sections.

Linux/FreeBSD

To install Go on Linux or FreeBSD, you must download go.-.tar.gz. The latest Go for Linux on a 64-bit architecture is go1.15.6.linux-amd64.tar.gz:

wget -c https://golang.org/dl/go1.15.6.linux-amd64.tar.gz   //64bit

wget -c https://golang.org/dl/go1.15.6.linux-386.tar.gz     //32bit

Download the archive and extract it into the /usr/local folder. Then, run the following command as root or through sudo:

tar -C /usr/local -xzf go1.15.6.linux-amd64.tar.gz

Add /usr/local/go/bin to the PATH environment variable. You can do this by adding the following line to $HOME/.profile or /etc/profile (for a system-wide installation):

export PATH=$PATH:/usr/local/go/bin

Verify that you've installed Go by opening a command prompt and typing the following command:

go version

This command should display the installed version of Go:

Figure 1.6 – Installed version of Go

Let's move on to see how to set up a Go environment on Windows.

Windows

To install Go on Windows, you can either use the MSI installer or the ZIP archive. Installing from MSI is easier. The latest Go for Windows on a 64-bit architecture is go1.15.6.windows-amd64.msi. You will then need to execute one of the following commands based on your system:

wget -c https://golang.org/dl/go1.15.6.windows-amd64.msi   //64bit

wget -c https://golang.org/dl/go1.15.6.windows-386.msi     //32bit

Open the MSI file you downloaded and follow the prompts to install Go. By default, the installer will place Go at C:\Go and set up C:\Go\bin in your PATH environment variable. You can change the location as needed:

Figure 1.7 – Golang installation wizard

After installing Go, you will need to close and reopen any open command prompts so that changes to the environment that have been made by the installer are reflected in the command prompt.

Important note

Using the ZIP archive is easy as well. Extract the files into a directory (for example, C:\Go) and add the bin subdirectory to your PATH variable.

Once installed, click on the Start menu. Then, in the menu's search box, type cmd. After, press the Enter key. In the command prompt window that appears, type the go version command, as shown here:

Figure 1.8 – Installed version of Go

You'll see go version go1.15.6 windows/amd64, as shown in the preceding screenshot. With that, you're all set up!

MacOS

For MacOS, you can download the appropriate PKG file; that is, go1.15.6.darwin-amd64.pkg (at the time of writing this book). Once downloaded, run through the installation wizard. The package will install the distribution to /usr/local/go and place the /usr/local/go/bin directory in your PATH environment variable:

Figure 1.9 – Installing Go on MacOS

You'll need to restart your terminal, or run this command in your Terminal:

source ~/.profile

Alternatively, you can use Homebrew to install Go. This can be as simple as doing the following:

brew install [email protected]

The Terminal window will give you feedback regarding the installation process of Go. It may take a few minutes before the installation is complete. Verify that you've installed Go by opening a command prompt and typing the go version command.

Important note

In the future, to update Go, you can run the following commands to update Homebrew and then update Go. You don't have to do this now as you've just installed the latest version:

brew update

brew upgrade golang

Now that you've installed Go, you need to set it up properly. Go development tools are designed to work with code maintained in public repositories, and the model is the same, regardless of whether you're developing an open source program or something else. Go code is developed in a workspace. A workspace is made up of three directories, namely the following:

bin: This will contain all you Go executable binaries.src: This will store your source files, which are organized in packages, with one subdirectory in the src directory representing one package.pkg: This will store your package objects.

The default directory for the Go workspace is the home directory with a go subdirectory or $HOME/go. Issue the following command to create the directory structure for your Go workspace:

mkdir -p $HOME/go/{bin,src,pkg}

The -p option tells mkdir to create all the parents in the directory, even if they don't currently exist. Using {bin, src, pkg} creates a set of arguments for mkdir and tells it to create the bin, src, and pkg directories. This will ensure the following directory structure is now in place:

The go, bin and  src folders should be at the same level (remove extra spaces from go and src folders, so the folders are aligned with bin folder)

$HOME

└── go

├── bin

└── src

Next, you need to set the GOPATH environment variable, as follows:

export GOPATH=$HOME/go

export PATH=$PATH:$GOPATH/bin

You can verify your $PATH has been updated by using the echo command and inspecting the output:

echo $PATH

You should see your $GOPATH/bin in your home directory. If you were logged in as USER, you will see /Users/USER/go/bin in the path:

/Users/USER/go/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

With the Go workspace and GOPATH created, we can go ahead and set up the development environment.

Integrated development environment

Throughout this book, I will be using an IDE to write RESTful API and services. Using an IDE can boost your productivity since it provides rich features such as autocompletion, highlighting code, a powerful built-in debugger, and custom extensions. There are many IDEs available. In this book, I will be using VSCode.

To install VSCode, download the appropriate package based on your system from https://code.visualstudio.com/download:

Figure 1.10 – VS Code – available packages

Note

Mac users can also use Brew to install VSCode with the following command:

brew install --cask visual-studio-code

Once downloaded, run the setup wizard and follow the instructions. Once the installation is complete, launch VSCode. You will be presented with the following start screen:

Figure 1.11 – VSCode user interface

VSCode supports all the popular programming languages and Git integration by default. You can also install extensions to extend the functionality of VSCode. The VS Code marketplace contains a huge list of free community plugins and extensions. To enable support for Golang, you need to install an extension called Go by navigating to the Extensions tab from the left sidebar:

Figure 1.12 – Golang extension for VSCode

Click on the Install button and then restart VSCode for the changes to take effect.

Installing Go tools

Next, we will install the following Go tools, which are a set of packages that help improve your development workflow and overall experience while writing code:

Important Note

A complete list of the available Go tools can be found at https://pkg.go.dev/golang.org/x/tools.

To install the tools, click on View | Command Pallete, or use the Ctrl + Shift + P shortcut and type goinstall update/tools:

Figure 1.13 – Available Go tools on VSCode

Check all the dependencies and click on OK. It will take some time to download all the dependencies:

Figure 1.14 – Go tools installation

With Go installed and configured on your computer, you are now ready to install the Gin framework.

Installing and configuring Gin

Gin is a third-party package. To install Gin in Go projects, we need to use the go get command. The command takes the URL of the package to be installed as an argument. Issue the following command to install the gin package from GitHub:

go get github.com/gin-gonic/gin

Note

If you're running Go 1.16 and above, you need to disable Go modules via the GO111MODULE=off option.

When checking out the gin package, the go get command creates a Gin directory in the $GOPATH/src path. The directory will contain the source code of the Gin framework:

Figure 1.15 – Gin package source code

Begin by creating the hello-world project directory under $GOHOME/src/hello-world or any directory that seems appropriate:

mkdir -p $GOHOME/src/hello-world

cd $GOHOME/src/hello-world

Open the folder with VSCode and create a main.go file inside the project folder that contains the following content:

package main

import "github.com/gin-gonic/gin"

func main() {

   router := gin.Default()

   router.GET("/", func(c *gin.Context) {

       c.JSON(200, gin.H{

           "message": "hello world",

       })

   })

   router.Run()

}

The first line, package main, indicates that this is the main module in this project. The import section is for importing the gin package. This package provides us with the router variable, which is declared right below import and the API context to be used while we send the response in our main function.

Next, we create an HTTP GET method on the root (/) resource and define a function to be called when HTTP requests hit the root endpoint. The function sends a JSON response with a status code of 200 (OK) with a body of "message": "test successful".

Finally, we must deploy the router on port 8080 using the router.Run() method. The following diagram summarizes how an HTTP request is processed in Gin:

Figure 1.16 – Parsing incoming HTTP requests with Gin

To run the app, execute the following command from the terminal session:

go run main.go

All the files and commands executed henceforth will be within this directory. If you followed the setup process, you should see the following output in your Terminal:

Figure 1.17 – Gin server logs

Point your favorite browser to http://localhost:8080. You should see a "hello world" message:

Figure 1.18 – Hello world example

Awesome – you have successfully started an HTTP server in Go with the Gin framework.

Back to the terminal, Gin will trace the HTTP requests:

Figure 1.19 – Tracing incoming HTTP requests

You can use a cURL command to issue an HTTP request:

curl -X GET http://localhost:8080

Alternatively, you can use an advanced REST client such as Postman. You can download the right version based on your platform from the following URL: https://www.getpostman.com/apps.

Once it has downloaded, run the wizard and open Postman. Set the fields as follows:

HTTP method: GETURL: http://localhost:8080Headers: Set Content-Type to application/json

The request should be configured like so:

Figure 1.20 – GET request with the Postman client

It's worth mentioning that by default, the HTTP server is listening on port 8080. However, if the port is being used by another application, you can define a different port by adding an argument to the Run method:

r.Run(":5000")

This command will run the server on port 5000, as shown in the following screenshot:

Figure 1.21 – Running the Gin server on port 5000

Note that the port parameter needs to be passed as a string, prepended by colon punctuation.

You should now be familiar with the basics of building and running a simple web application. In the next few sections, we will cover how to enhance those functionalities with third-party packages. But before we do that, let's cover how to manage Go dependencies.

Dependency management in Golang

For now, the code is stored locally. However, it's recommended to store the source code in a remote repository for versioning. That's where a solution such as GitHub comes into play. Sign up for a free account at https://github.com. Then, create a new GitHub repository called hello-world:

Figure 1.22 – New GitHub repository

Next, initialize the repository with the following commands:

git init

git remote add origin https://github.com/mlabouardy/hello-world.git

Commit the main.go file to the remote repository by executing the following commands:

git add .

git commit -m "initial commit"

git push origin master

Your repository should now look like this:

Figure 1.23 – Versioning main.go in Git

We can stop here. However, if you're working within a team, you will need some way to ensure all team members are using the same Go version and packages. That's where Go modules come into the picture. Go modules were introduced in 2018 to make dependency management a lot easier.

Note

Starting with Go 1.16, Go modules are the default way to manage external dependencies.