Go Recipes for Developers E-Book

Burak Serdar

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 354

Veröffentlichungsjahr: 2024

Beschreibung

With its simple syntax and sensible conventions, Go has emerged as the language of choice for developers in network programming, web services, data processing, and other settings. This practical guide helps engineers leverage Go through up-to-date recipes that solve common problems in day-to-day programming. Drawing from three decades of distributed systems engineering and technical leadership at companies like Red Hat, Burak Serdar brings battle-tested expertise in building robust, scalable applications.
He starts by covering basics of code structure, describing different approaches to organizing packages for different types of projects. You’ll discover practical solutions to engineering challenges in network programming, dealing with processes, databases, data processing pipelines, and testing. Each chapter provides working solutions and production-ready code snippets that you can seamlessly incorporate into your programs while working in sequential and concurrent settings. The solutions leverage the more recent additions to the Go language, such as generics and structured logging. Most of the examples are developed using the Go standard library without any third-party packages.
By the end of this book, you’ll have worked through a collection of proven recipes that will equip you accelerate your Go development journey.

Leseprobe

Go Recipes for Developers

Top techniques and practical solutions for real-life Go programming problems

Burak Serdar

Go Recipes for Developers

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Kunal Sawant

Publishing Product Manager: Samriddhi Murarka

Book Project Manager: Prajakta Naik

Lead Editor: Kinnari Chohan

Technical Editor: Vidhisha Patidar

Copy Editor: Safis Editing

Proofreader: Kinnari Chohan

Indexer: Pratik Shirodkar

Production Designer: Alishon Mendonca

DevRel Marketing Coordinator: Sonia Chauhan

First published: December 2024

Production reference: 2171224

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83546-439-7

www.packtpub.com

Contributors

About the author

Burak Serdar is a software engineer with over 30 years of experience designing and developing distributed applications. He has used Go to create backend software, data processing platforms, interactive applications, and automation systems. Burak has worked for both startups and large corporations as an engineer and technical lead. He holds B.Sc. and M.Sc. degrees in Electrical and Electronics Engineering, as well as an M.Sc. degree in Computer Science.

About the reviewer

Dylan Meeus is a software engineer with over a decade of experience in various functional and non-functional programming languages. He has used Go to develop systems across diverse domains, including healthcare, machine learning frameworks, and digital signal processing software. Dylan developed a passion for functional programming while learning Haskell and has applied this knowledge to traditionally non-functional languages like Java. In recent years, he has spoken at various Go- and Java-oriented conferences, such as GopherCon and Devoxx.

Table of Contents

Preface

1

Project Organization

Modules and packages

Technical requirements

Creating a module

How to do it...

Creating a source tree

How to do it...

Building and running programs

How to do it...

Importing third-party packages

How to do it...

Importing specific versions of packages

How to do it...

Working with the module cache

How to do it...

Using internal packages to reduce an API surface

How to do it...

Using a local copy of a module

How to do it...

Working on multiple modules – workspaces

How to do it...

Managing the versions of your module

How to do it...

Summary and further reading

2

Working with Strings

Creating strings

How to do it...

Formatting strings

How to do it...

How it works...

Combining strings

How to do it...

How it works...

Working with string cases

How to do it...

How it works...

There’s more...

Working with encodings

How to do it...

How it works...

Iterating bytes and runes of strings

How to do it...

How it works...

Splitting

How to do it...

How it works...

Reading strings line by line, or word by word

How to do it...

How it works...

Trimming the ends of a string

How to do it...

Regular expressions

Validating input

Searching patterns

Extracting data from strings

How to do it...

How it works...

Replacing parts of a string

How to do it...

Templates

Value substitution

Iteration

Variables and scope

There’s more – nested loops and conditionals

Dealing with empty lines

How to do it...

Template composition

How to do it...

How it works...

Template composition – layout templates

How to do it...

How it works...

There’s more...

3

Working with Date and Time

Working with Unix time

How to do it...

Date/time components

How to do it...

Date/time arithmetic

How to do it...

How it works...

Formatting and parsing date/time

How to do it...

Working with time zones

How to do it...

How it works...

Storing time information

How to do it...

Timers

How to do it...

How it works...

Tickers

How to do it...

How it works...

4

Working with Arrays, Slices, and Maps

Working with arrays

Creating arrays and passing them around

Working with slices

Creating slices

Creating a slice from an array

How to do it...

How it works...

Appending/inserting/deleting slice elements

How to do it...

How it works...

Implementing a stack using a slice

How to do it...

Working with maps

Defining, initializing, and using maps

How to do it...

Implementing a set using a map

How to do it...

How it works...

Composite keys

How to do it...

How it works...

Thread-safe caching with maps

Simple cache

How to do it...

Cache with blocking behavior

How to do it...

How it works...

5

Working with Types, Structs, and Interfaces

Creating new types

Creating a new type based on an existing type

How to do it...

Creating type-safe enumerations

How to do it...

Creating struct types

How to do it...

Extending types

Extending a base type

How to do it...

How it works...

Initializing structs

How to do it...

Defining interfaces

Interfaces as contracts

How to do it...

Factories

How to do it...

Defining interfaces where you use them

How to do it...

How it works...

Using a function as an interface

How to do it...

How it works...

Discovering capabilities of data types at runtime – testing "implements" relationship

How to do it...

How it works...

Testing whether an interface value is one of the known types

How to do it...

Ensuring a type implements an interface during development

How to do it...

Deciding whether to use a pointer receiver or value receiver for methods

How to do it...

How it works...

Polymorphic containers

How to do it...

How it works...

Accessing parts of an object not directly exposed via the interface

How to do it...

Accessing the embedding struct from the embedded struct

How to do it...

Checking whether an interface is nil

How to do it...

How it works...

6

Working with Generics

Generic functions

Writing a generic function that adds numbers

Declaring constraints as interfaces

Using generic functions as accessors and adapters

Returning a zero value from a generic function

Using type assertion on generic arguments

Generic types

Writing a type-safe set

An ordered map – using multiple type parameters

7

Concurrency

Doing things concurrently using goroutines

Creating goroutines

Running multiple independent functions concurrently and waiting for them to complete

Communicating between goroutines using channels

Sending and receiving data using channels

Sending data to a channel from multiple goroutines

Collecting the results of concurrent computations using channels

Working with multiple channels using the select statement

Canceling goroutines

Detecting cancelation using nonblocking select

Sharing memory

Updating shared variables concurrently

8

Errors and Panics

Returning and handling errors

How to do it...

How it works...

Wrapping errors to add contextual information

How to do it...

Comparing errors

How to do it...

How it works...

Structured errors

How to do it...

How it works...

Wrapping structured errors

How to do it...

How it works...

Comparing structured errors by type

How to do it...

How it works...

Extracting a specific error from the error tree

How to do it...

How it works...

Dealing with panics

Panicking when necessary

How to do it...

Recovering from panics

How to do it...

How it works...

Changing return value in recover

How to do it...

How it works...

Capturing the stack trace of a panic

How to do it...

9

The Context Package

Using context for passing request-scoped data

How to do it...

How it works...

There’s more...

Using context for cancellations

How to do it...

How it works...

Using context for timeouts

How to do it...

How it works...

There’s more...

Using cancellations and timeouts in servers

How to do it...

10

Working with Large Data

Worker pools

Capped worker pools

Fixed-size worker pools

Connection pools

Pipelines

Simple pipeline without fan-out/fan-in

Pipeline with worker pools as stages

Pipeline with fan-out and fan-in

Working with large result sets

Streaming results using a goroutine

11

Working with JSON

Marshaling/unmarshaling basics

Encoding structs

How to do it...

Dealing with embedded structs

How to do it...

Encoding without defining structs

How to do it...

Decoding structs

How to do it...

Decoding with interfaces, maps, and slices

How to do it...

Other ways of decoding numbers

How to do it...

Dealing with missing and optional values

Omitting empty fields when encoding

How to do it...

Dealing with missing fields when decoding

How to do it...

Customizing JSON encoding/decoding

Marshaling/unmarshaling custom data types

How to do it...

Custom marshaling/unmarshaling of object keys

How to do it...

Dynamic field names

How to do it...

Polymorphic data structures

Custom unmarshaling with two passes

How to do it...

Streaming JSON data

Streaming an array of objects

How to do it...

Parsing an array of objects

How to do it...

Other ways of streaming JSON

Security considerations

How to do it...

12

Processes

Running external programs

How to do it...

Passing arguments to a process

Expanding arguments

Running the command via the shell

Processing output from a child process using a pipe

How to do it...

Providing input to a child process

How to do it...

Changing environment variables of a child process

How to do it...

Graceful termination using signals

How to do it...

13

Network Programming

TCP networking

Writing TCP servers

How to do it...

How it works...

Writing TCP clients

How to do it...

Writing a line-based TCP server

How to do it...

Sending/receiving files using a TCP connection

How to do it...

Writing a TLS client/server

How to do it...

A TCP proxy for TLS termination and load-balancing

How to do it...

Setting read/write deadlines

How to do it...

Unblocking a blocked read or write operation

How to do it...

How it works...

Writing UDP clients/servers

How to do it...

Working with HTTP

Making HTTP calls

How to do it...

Running an HTTP server

How to do it...

HTTPS – setting up a TLS server

How to do it...

Writing HTTP handlers

How to do it...

Serving static files on the file system

How to do it...

Handling HTML forms

How to do it...

Writing a handler for downloading large files

How to do it...

Handling HTTP uploaded files and forms as a stream

How to do it...

14

Streaming Input/Output

Readers/writers

Reading data from a reader

Writing data to a writer

Reading from and writing to a byte slice

Reading from and writing to a string

Working with files

Creating and opening files

Closing a file

Reading/writing data from/to files

Reading/writing from/to a specific location

Changing the file size

Finding the file size

Working with binary data

How to do it...

Copying data

Copying files

Working with the filesystem

Working with filenames

Creating temporary directories and files

Reading directories

Working with pipes

Connecting code expecting a reader with code expecting a writer

Intercepting a reader using TeeReader

15

Databases

Connecting to a database

How to do it...

Running SQL statements

Running SQL statements without explicit transactions

Running SQL statements with transactions

Running prepared statements within a transaction

How to do it...

Getting values from a query

How to do it...

Dynamically building SQL statements

Building UPDATE statements

How to do it...

Building WHERE clauses

How to do it...

16

Logging

Using the standard logger

Writing log messages

Controlling format

Changing where to log

Using the structured logger

Logging using the global logger

Writing structured logs using different levels

Changing log level at runtime

Using loggers with additional attributes

Changing where to log

Adding logging information from the context

17

Testing, Benchmarking, and Profiling

Working with unit tests

Writing a unit test

How to do it...

Running unit tests

How to do it...

Logging in tests

How to do it...

Skipping tests

How to do it...

Testing HTTP servers

How to do it...

Testing HTTP handlers

How to do it...

Checking test coverage

How to do it...

Benchmarking

Writing benchmarks

How to do it...

Writing multiple benchmarks with different input sizes

How to do it...

Running benchmarks

How to do it...

Profiling

How to do it…

See also

Index

Other Books You May Enjoy

Preface

Go, with its straightforward syntax and pragmatic conventions, has solidified its position as the language of choice for developers tackling network programming, web services, data processing, and beyond. This book is designed to empower engineers by providing up-to-date, practical recipes for solving common programming challenges.

The journey begins with foundational principles, including effective approaches to organizing packages and structuring code for various project types. From there, the book delves into real-world engineering challenges, offering practical solutions in network programming, process management, database interactions, data pipelines, and testing. Each chapter presents working solutions and production-ready code snippets, tailored for both sequential and concurrent programming environments.

Leveraging Go’s most recent language features—such as generics and structured logging—the recipes in this book primarily rely on the Go standard library, ensuring minimal reliance on third-party packages and maximizing compatibility.

By the end of this book, you’ll have a wealth of proven, hands-on solutions to accelerate your Go development journey and tackle the complexities of modern software engineering with confidence.

Who this book is for

This book is intended for developers with a basic understanding of the Go language. More experienced developers can also use it as a reference, offering practical examples that can be applied to a variety of use cases.

What this book covers

Chapter 1, Project Organization, covers modules, packages, source tree organization, importing packages, versioning modules, and workspaces.

Chapter 2, Working with Strings, contains recipes showing how to work with strings, internationalization, encoding, regular expressions, parsing, and generating formatted text using templates.

Chapter 3, Working with Date and Time, shows how to work with date, time, and duration values correctly with time zone considerations, formatting/parsing date and time values, performing periodic tasks, and scheduling functions to run later.

Chapter 4, Working with Arrays, Slices, and Maps, introduces the basic container types that are the building blocks for many data structures.

Chapter 5, Working with Types, Structs, and Interfaces, shows how to define new types, extending existing types to share functionality, interfaces, and their uses. In particular, this chapter includes the two approaches to using interfaces, namely, interfaces as contracts and defining interfaces where they are used.

Chapter 6, Working with Generics, introduces the basic recipes for writing generic functions and generic types with examples.

Chapter 7, Concurrency, includes basic recipes to write concurrent programs using goroutines and channels. Mutual exclusion using mutexes is also discussed here.

Chapter 8, Errors and Panics, shows generating errors, passing errors around, handling them, and organizing errors in a project. It also discusses how to generate and deal with panics.

Chapter 9, The Context Package, introduces the Go’s Context which is useful for controlling request lifecycle and passing request-scoped values within an application in a concurrent program.

Chapter 10, Working with Large Data, includes recipes for working with large amounts of data in a concurrent setting using worker pools and concurrent pipelines.

Chapter 11, Working with JSON, includes recipes for encoding and decoding JSON, marshaling/unmarshaling simple and complex data types, working with custom serialization logic, encoding/decoding polymorphic structures, and streaming JSON data.

Chapter 12, Processes, shows how to run and interact with external programs, working with environment variables, working with pipes, and graceful termination using signals.

Chapter 13, Network Programming, gives recipes for TCP and UDP servers and clients, working with TLS, deadlines, HTTP client/servers, request multiplexing, and HTML forms.

Chapter 14, Streaming Input/Output, includes recipes using reads and writers, working with files and the file system, and pipes.

Chapter 15, Databases, shows how to interact with an SQL database using the standard library packages in a secure way.

Chapter 16, Logging, has recipes showing the common uses of the standard library log and slog packages.

Chapter 17, Testing, Benchmarking, and Profiling, gives recipes on writing and running unit tests, testing HTTP servers, benchmarking, and profiling

To get the most out of this book

You need a recent version of Go (anything newer than 1.22 will do) integrated with your favorite development environment. Some of the example programs use Docker.

If you are using the digital version of this book, we advise you to type the code yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Go-Recipes-for-Developers. In case there’s an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Note the capitalization of InitDB.”

A block of code is set as follows:

ctx:=context.Background() cancelable, cancel:=context.WithCancel(ctx) defer cancel()

Tips or important notes

Appear like this.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it..., How it works...).

To give clear instructions on how to complete a recipe, use these sections as follows:

Getting ready

This section tells you what to expect in the recipe and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Go Recipes for Developers, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/978-1-83546-439-7

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

1

Project Organization

This chapter is about how you can start a new project, organize a source tree, and manage the packages you need to develop your programs. A well designed project structure is important because when other developers work on your project or try to use components from it, they can quickly and easily find what they are looking for. This chapter will first answer some of the questions you may have when you are starting a new project. Then, we will look at how you can use the Go package system, work with standard library and third-party packages, and make it easy for other developers to use your packages.

This chapter includes the following recipes:

Creating a moduleCreating a source treeBuilding and running programsImporting third-party packagesImporting specific versions of packagesUsing internal packages to reduce API surfaceUsing a local copy of a moduleWorkspacesManaging the versions of your module

Modules and packages

First, a few words about modules and packages would be helpful. A package is a cohesive unit of data types, constants, variables, and functions. You build and test packages, not individual files or modules. When you build a package, the build system collects and also builds all dependent packages. If the package name is main, building it will result in an executable. You can run the main package without producing a binary (more specifically, the Go build system first builds the package, produces the binary in a temporary location, and runs it). To use another package, you import it. Modules help with organizing multiple packages and the resolution of package references within a project. A module is simply a collection of packages. If you import a package into your program, the module containing that package will be added to go.mod, and a checksum of the contents of that module will be added to go.sum. Modules also help you to manage versions of your programs.

All files of a package are stored under a single directory on the filesystem. Every package has a name declared using the package directive, shared by all source files in it. The package name usually matches the directory name containing the files, but this is not necessarily so. For example, the main package is not usually under a directory named main/. The directory of the package determines the package’s “import path.” You import another package into your current package using the import <importPath> statement. Once you import a package, you use the names declared in that package using its package name (which is not necessarily the directory name).

A module name points to the location where the module contents are stored in a version control system on the Internet. At the time of writing, this is not a hard-and-fast requirement, so you can actually create module names that do not follow this convention. This should be avoided to prevent potential future incompatibilities with the build system. Your module names should be part of the import paths for the packages of those modules. In particular, module names whose first component (the part before the first /) does not have . are reserved for the standard library.

These concepts are illustrated in Figure 1.1.

Figure 1.1 – Modules and packages

The module name declared in go.mod is the repository path where the module can be found.The import path in main.go defines where the imported package can be found. The Go build system will locate the package using this import path, and then it will locate the module containing the package by scanning the parent directories of the package path. Once the module is found, it will be downloaded to the module cache.The package name defined in the imported module is the package name you use to access the symbols of that package. This can be different from the last component of the import path. In our example, the package name is example, but the import path for this package is github.com/bserdar/go-recipes-module.The Example function is located in the example package.The example package also imports another package contained in the same module. The build system will identify this package to be part of the same module and resolve the references, using the downloaded version of the module.

Technical requirements

You will need a recent version of Go on your computer to build and run the examples in this chapter. The examples in this book were tested using Go version 1.22. The code from this chapter can be found at https://github.com/PacktPublishing/Go-Recipes-for-Developers/tree/main/src/chp1.

Creating a module

When you start working on a new project, the first thing to do is to create a module for it. A module is how Go manages dependencies.

How to do it...

Create a directory to store a new module.Under that directory, use go mod init <moduleName> to create the new module. The go.mod file marks the root directory of a module. Any package under this directory will be a part of this module unless that directory also has a go.mod file. Although such nested modules are supported by the build system, there is not much to be gained from them.To import a package in the same module, use moduleName/packagePath. When moduleName is the same as the location of the module on the internet, there are no ambiguities about what you are referring to.For the packages under a module, the root of the module is the closest parent directory containing a go.mod file. All references to other packages within a module will be looked up in the directory tree under the module root.Start by creating a directory to store the project files. Your current directory can be anywhere on the filesystem. I have seen people use a common directory to store their work, such as $HOME/projects (or \user\myUser\projects in Windows). You may choose to use a directory structure that looks like the module name, such as $HOME/github.com/mycompany/mymodule (or \user\myUser\github.com\mycompany\mymodule in Windows). Depending on your operating system, you may find a more suitable location.

Warning

Do not work under the src/ directory of your Go installation. That is the source code for the Go standard library.

Tip

You should not have an environment variable, GOPATH; if you have to keep it, do not work under it. This variable was used by an older mode of operation (Go version <1.13) that is now deprecated in favor of the Go module system.

Throughout this chapter, we will be using a simple program that displays a form in a web browser and stores the entered information in a database.

After creating the module directory, use go mod init. The following commands will create a webform directory under projects and initialize a Go module there:

$ cd projects $ mkdir webform $ go mod init github.com/examplecompany/webform

This will create a go.mod file in this directory that looks like this:

module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform go 1.21.0

Use a name that describes where your module can be found. Always use a URL structure such as the <host>.<domain>/location/to/module format (e.g., github.com/bserdar/jsonom). In particular, the first component of the module name should have a dot (.) (the Go build system checks this).

So, even though you can name the module something such as webform or mywork/webform, do not do so. However, you can use something such as workspace.local/webform. When in doubt, use the code repository name.

Creating a source tree

Once you have a new module, it is time to decide how you are going to organize the source files.

How to do it...

There are several established conventions, depending on the project:

Use a standard layout, such as https://github.com/golang-standards/project-layout.A library with a narrow focus can put all the exported names at the module root, with implementation details optionally stored under internal packages. A module that produces a single executable with relatively few or no reusable components can also use the flat directory structure.

For a project like ours that produces an executable, the structure laid out in https://github.com/golang-standards/project-layout fits. So, let’s follow that template:

webform/   go.mod   cmd/     webform/       main.go   web/     static/   pkg/     ...   internal/     ...   build/     ci/     package/   configs/

Here, the cmd/webform directory will contain the main package. As you can see, this is one instance where the package name does not match the directory it is in. The Go build system will create executables using the directory name, so when you build the main package under cmd/webform, you get an executable named webform. If you have multiple executables built within a single module, you can accommodate them by creating a separate main package under a directory matching the program name, under the cmd/ directory.

The pkg/ directory will contain the exported packages of the program. These are packages that can be imported and reused in other projects.

If you have packages that are not usable outside this project, you should put them under the internal/ directory. The Go build system recognizes this directory and does not allow you to import packages under internal/ from other packages that are outside the directory containing the internal/ directory. With this setup, all the packages of our webform program will have access to the packages under internal/, but it will be inaccessible to packages importing this module.

The web/ directory will contain any web-related assets. In this example, we will have a web/static directory containing static web pages. You can also add web/templates to store server-side templates if you have any.

The build/package directory should have packaging scripts and configuration for cloud, container, packaging systems (dep, rpm, pkg, etc.).

The build/ci directory should have continuous integration tool scripts and configurations. If the continuous integration tool you are using requires its files to be in a certain directory other than this, you can create symbolic links, or simply put those files where the tool needs them instead of /build/ci.

The configs/ directory should contain the configuration file templates and default configurations.

You can also see projects that have the main package under the module root, eliminating the cmd/ directory. This is a common layout when the module has only one executable:

webform/   go.mod   go.sum   main.go   internal/     ...   pkg/     ...

Then there are modules without any main package. These are usually libraries that you can import into your projects. For example, https://github.com/google/uuid contains the popular UUID implementation using a flat directory structure.

Building and running programs

Now that you have a module and a source tree with some Go files, you can build or run your program.

How to do it...

Use go build to build the current packageUse go build ./path/to/package to build the package in the given directoryUse go build <moduleName> to build a moduleUse go run to run the current main packageUse go run ./path/to/main/package to build and run the main package in the given directoryUse go run <moduleName/mainpkg> to build and run the module’s main under the given directory

Let’s write the main function that starts an HTTP server. The following snippet is cmd/webform/main.go:

package main import (     "net/http" ) func main() {     server := http.Server{         Addr:    ":8181",         Handler: http.FileServer(http.Dir("web/static")),     }     server.ListenAndServe() }

Currently, main only imports the standard library’s net/http package. It starts a server that serves the files under the web/static directory. Note that for this to work, you have to run the program from the module root:

$ go run ./cmd/webform

Always run the main package; avoid go run main.go. This will run main.go, excluding any other files in the main package. It will fail if you have other .go files that contain helper functions in the main package.

If you run this program from another directory, it will fail to find the web/static directory; because it is a relative path, it is resolved relative to the current directory.

When you run a program via go run, the program executable is placed in a temporary directory. To build the executable, use the following:

$ go build ./cmd/webform

This will create a binary in the current directory. The name of the binary will be determined by the last segment of the main package – in this case, webform. To build a binary with a different name, use the following:

$ go build -o wform ./cmd/webform

This will build a binary called wform.

Importing third-party packages

Most projects will depend on third-party libraries that must be imported into them. The Go module system manages these dependencies.

How to do it...

Find the import path of the package you need to use in your project.Add the necessary imports to the source files you use in the external package.Use the go get or go mod tidy command to add the module to go.mod and go.sum. If the module was not downloaded before, this step will also download the module.

Tip

You can use https://pkg.go.dev to discover packages. It is also the place to publish documentation for the Go projects you publish.

Let’s add a database to our program from the previous section so that we can store the data submitted by the web form. For this exercise, we will use the SQLite database.

Change the cmd/webform/main.go file to import the database package and add the necessary database initialization code:

package main import (     "net/http"     "database/sql"     _ "modernc.org/sqlite"     "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/     webform/pkg/commentdb" ) func main() {     db, err := sql.Open("sqlite", "webform.db")     if err != nil {         panic(err)     }     commentdb.InitDB(db)     server := http.Server{         Addr:    ":8181",         Handler: http.FileServer(http.Dir("web/static")),     }     server.ListenAndServe() }

The _ "modernc.org/sqlite" line imports the SQLite driver into the project. The underscore is the blank identifier, meaning that the sqlite package is not directly used by this file and is only included for its side effects. Without the blank identifier, the compiler would complain that the import was not used. In this case, the modernc.org/sqlite package is a database driver, and when you import it, its init() functions will register the required driver with the standard library.

The next declaration imports the commentdb package from our module. Note that the complete module name is used to import the package. The build system will recognize the prefix of this import declaration as the current module name, and it will translate it to a local filesystem reference, which, in this case, is webform/pkg/commentdb.

On the db, err := sql.Open("sqlite", "webform.db") line, we use the database/sql package function, Open, to start a SQLite database instance. sqlite names the database driver, which was registered by the imported _ "modernc.org/sqlite".

The commentdb.InitDB(db) statement will call a function from the commentdbpackage .

Now, let’s see what commentdb.InitDB looks like. This is the webform/pkg/commentdb/initdb.go file:

package commentdb import (     "context"     "database/sql" ) const createStmt=`create table if not exists comments ( email TEXT, comment TEXT)` func InitDB(conn *sql.DB) {     _, err := conn.ExecContext(context.Background(), createStmt)     if err != nil {         panic(err)     } }

As you can see, this function creates the database tables if they have not been created yet.

Note the capitalization of InitDB. If the first letter of a symbol name declared in a package is a capital letter, that symbol is accessible from other packages (i.e., it is exported). If not, the symbol can only be used within the package it is declared (i.e., it is not exported). The createStmt constant is not exported and will be invisible to other packages.

Let’s build the program:

$ go build ./cmd/webform   cmd/webform/main.go:7:2: no required module provides package modernc.org/sqlite; to add it:       go get modernc.org/sqlite

You can run go get modernc.org/sqlite to add a module to your project. Alternatively, you can run the following:

$ go get

That will get all the missing modules. Alternatively, you can run the following:

$ go mod tidy

go mod tidy will download all missing packages, update go.mod and go.sum with updated dependencies, and remove references to any unused modules. go get will only download missing modules.

Importing specific versions of packages

Sometimes, you need a specific version of a third-party package because of API incompatibilities or a particular behavior you depend on.

How to do it...

To get a specific version of a package, specify the version label: $ go get modernc.org/[email protected] get the latest release of a specific major version of a package, use this: $ go get gopkg.in/yaml.v3

Alternatively, use this:

$ go get github.com/ory/dockertest/v3To import the latest available version, use this: $ go get modernc.org/sqliteYou can also specify a different branch. The following will get a module from the devel branch, if there is one: $ go get modernc.org/sqlite@develAlternatively, you can get a specific commit: $ go get modernc.org/sqlite@a8c3eea199bc8fdc39391d5d261eaa3577566050

As you can see, you can get a specific revision of a module using the @revision convention:

$ go get modernc.org/[email protected]

The revision part of the URL is evaluated by the version control system, which, in this case, is git, so any valid git revision syntax can be used.

Tip:

You can find which revision control systems are supported by checking out the src/cmd/go/alldocs.go file under your Go installation.

That also means you can use branches:

$ go get modernc.org/sqlite@master

Tip

The https://gopkg.in service translates version numbers to URLs compatible with the Go build system. Refer to the instructions on that website on how to use it.

Working with the module cache

The module cache is a directory where the Go build system stores downloaded module files. This section describes how to work with the module cache.

How to do it...

The module cache is, by default, under $GOPATH/pkg/mod, which is $HOME/go/pkg/mod when GOPATH is not set:

By default, the Go build system creates read-only files under the module cache to prevent accidental modifications.To verify that the module cache is not modified and reflects the original versions of modules, use this: go mod verifyTo clean up the module cache, use this: go clean -modcache

The authoritative source for information about the module cache is the Go Modules Reference (https://go.dev/ref/mod)

Using internal packages to reduce an API surface

Not every piece of code is reusable. Having a smaller API surface makes it easier for others to adapt and use your code. So, you should not export APIs that are specific to your program.

How to do it...

Create internal packages to hide implementation details from other packages. Anything under an internal package can only be imported from the packages under the package containing that internal package – that is, anything under myproject/internal can only be imported from the packages under myproject.

In our example, we placed the database access code into a package where it can be accessed by other programs. However, it does not make sense to expose the HTTP routes to others, as they are specific to this program. So, we will put them under the webform/internal package.

This is the internal/routes/routes.go file:

package routes import (     "database/sql"     "github.com/gorilla/mux"     "net/http" ) func Build(router *mux.Router, conn *sql.DB) {     router.Path("/form").         Methods("GET").HandlerFunc(func(w http.ResponseWriter, r         *http.Request) {         http.ServeFile(w, r, "web/static/form.html")     })     router.Path("/form").         Methods("POST").HandlerFunc(func(w http.ResponseWriter, r         *http.Request) {         handlePost(conn, w, r)     }) } func handlePost(conn *sql.DB, w http.ResponseWriter, r *http.Request) {     email := r.PostFormValue("email")     comment := r.PostFormValue("comment")     _, err := conn.ExecContext(r.Context(), "insert into comments     (email,comment) values (?,?)",     email, comment)     if err != nil {         http.Error(w, err.Error(), http.StatusInternalServerError)         return     }     http.Redirect(w, r, "/form", http.StatusFound) }

Then, we change the main.go file to use the internal package:

package main import (     "database/sql"     "net/http"     "github.com/gorilla/mux"     _ "modernc.org/sqlite"     "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/     webform/internal/routes"     "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/     webform/pkg/commentdb" ) func main() {     db, err := sql.Open("sqlite", "webform.db")     if err != nil {         panic(err)     }     commentdb.InitDB(db)     r := mux.NewRouter()     routes.Build(r, db)     server := http.Server{         Addr:    ":8181",         Handler: r,     }     server.ListenAndServe() }

Using a local copy of a module

Sometimes, you will work on multiple modules, or you download a module from a repository, make some changes to it, and then want to use the changed version instead of the version available on the repository.

How to do it...

Use the replace directive in go.mod to point to the local directory containing a module.

Let’s return to our example – suppose you want to make some changes to the sqlite package:

Clone it: $ ls   webform $ git clone [email protected]:cznic/sqlite.git $ ls   sqlite   webformModify the go.mod file under your project to point to the local copy of the module. go.mod becomes the following: module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform go 1.22.1 replace modernc.org/sqlite => ../sqlite require (     github.com/gorilla/mux v1.8.1     modernc.org/sqlite v1.27.0 ) ...You can now make changes in the sqlite module on your system, and those changes will be built into your application.

Working on multiple modules – workspaces

Sometimes you need to work with multiple interdependent modules. A convenient way to do this is by defining a workspace. A workspace is simply a set of modules. If one of the modules within a workspace refers to a package in another module in the same workspace, it is resolved locally instead of that module being downloaded over the network.

How to do it...

To create a workspace, you have to have a parent directory containing all your work modules: $ cd ~/projects $ mkdir ws $ cd wsThen, start a workspace using this: $ go work init

This will create a go.work file in this directory.

Place the module you are working on into this directory.

Let’s demonstrate this using our example. Let’s say we have the following directory structure:

$HOME/   projects/     ws/        go.work        webform        sqlite

Now, we want to add the two modules, webform and sqlite, to the workspace. To do that, use this:

$ go work use ./webform $ go work use ./sqlite

These commands will add the two modules to your workspace. Any sqlite reference from the webform module will now be resolved to use the local copy of the module.

Managing the versions of your module

Go tooling uses the semantic versioning system. This means that the version numbers are of the X.Y.z form, broken down as follows:

X is incremented for major releases that are not necessarily backward compatible.Y is incremented for minor releases that are incremental but backward-compatiblez is incremented for backward-compatible patches

You can learn more about semantic versioning at https://semver.org.

How to do it...

To publish a patch or minor version, tag the branch containing your changes with the new version number: $ git tag v1.0.0 $ git push origin v1.0.0If you want to publish a new release that has an incompatible API with the previous releases, you should increment the major versions of that module. To release a new major version of your module, use a new branch: $ git checkout -b v2

Then, change your module name in go.mod to end with /v2, and update all references in the source tree to use the /v2 version of the module.

For example, let’s say you released the first version of the webform module, v1.0.0. Then, you decided you would like to add new API endpoints. This would not be a breaking change, so you simply increment the minor version number – v1.1.0. But then it turns out some of the APIs you added were causing problems, so you removed them. Now, that is a breaking change, so you should publish v2.0.0 with it. How can you do that?

The answer is, you use a new branch in the version control system. Create the v2 branch:

$ git checkout -b v2

Then, change go.mod to reflect the new version:

module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform/v2 go 1.22.1 require (   ... )

If there are multiple packages in the module, you have to update the source tree so that any references to packages within that module also use the v2 version.

Commit and push the new branch:

$ git add go.mod $ git commit -m "New version" $ git push origin v2