Network Automation with Go E-Book

Nicolas Leiva

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 520

Veröffentlichungsjahr: 2023

Beschreibung

Go’s built-in first-class concurrency mechanisms make it an ideal choice for long-lived low-bandwidth I/O operations, which are typical requirements of network automation and network operations applications.
This book provides a quick overview of Go and hands-on examples within it to help you become proficient with Go for network automation. It’s a practical guide that will teach you how to automate common network operations and build systems using Go.
The first part takes you through a general overview, use cases, strengths, and inherent weaknesses of Go to prepare you for a deeper dive into network automation, which is heavily reliant on understanding this programming language. You’ll explore the common network automation areas and challenges, what language features you can use in each of those areas, and the common software tools and packages. To help deepen your understanding, you’ll also work through real-world network automation problems and apply hands-on solutions to them.
By the end of this book, you’ll be well-versed with Go and have a solid grasp on network automation.

Leseprobe

Network Automation with Go

Learn how to automate network operations and build applications using the Go programming language

Nicolas Leiva

Michael Kashin

BIRMINGHAM—MUMBAI

Network Automation with Go

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Associate Group Product Manager: Mohd Riyan Khan

Publishing Product Manager: Mohd Riyan Khan

Content Development Editor: Nihar Kapadia

Technical Editor: Rajat Sharma

Copy Editor: Safis Editing

Project Coordinator: Manisha Singh

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Roshan Kawale

Marketing Coordinator: Nimisha Dua

First published: November 2022

Production reference: 1021222

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80056-092-5

www.packt.com

For my wife Catalina and daughter Renata, whose constant support and encouragement made this book possible.

– Nicolas Leiva

To the memory of my Mother, for her love, support and inspiration.

– Michael Kashin

Contributors

About the authors

Nicolas Leiva is a staff solutions architect at Red Hat. In his role, he helps customers of all business sizes to automate the provisioning and operation of IT infrastructure, services, and applications. Prior to that, he worked in the networking industry for 15 years, becoming a Cisco Certified Design Expert (CCDE) and Cisco Certified Internetwork Expert (CCIE). He is passionate about writing open source software in Go with a keen interest in cloud technologies.

I want to thank my parents, Rafael and Maria Estela, for guiding me in the right direction to get here and be the person I am today.

Michael Kashin is a cloud infrastructure solutions architect, currently working in the networking business unit of NVIDIA. Throughout his career, he held multiple roles ranging from network operations, through software development, to systems architecture and design. He enjoys breaking the boundaries between different disciplines and coming up with creative solutions to satisfy business needs and solve technical problems in the most optimal way. He is a prolific open source contributor and writer, with much of his work focused on cloud-native infrastructure, automation, and orchestration.

I want to thank my wife for putting up with me spending endless hours writing this book, and for looking after the kids all this time.

About the reviewers

John McGovern holds a first-class Honours Bachelor of Science degree (BSc Hons) in cybersecurity and networking from Glasgow Caledonian University. He is a technical instructor for CBT Nuggets and has developed a wide library of training around network automation, with a focus on Python programming, including Python-based libraries such as Nornir, Scrapli, and NAPALM. John is also a regular panelist on the Network Automation Hangout podcast and an active contributor to the Cisco DevNet Code Exchange program.

I would like to primarily thank the authors, Nicolas and Michael, for all of their great work in creating this fantastic learning resource. You both are incredible. I would also like to thank Stephen Hendry and Kamil Stachura for all of the help, guidance, and camaraderie over these last few years automating networks. And lastly, to Carl Montanari, Dmitry Figol, and Roman Dodin – you guys are the best; thank you for everything you do.

Chris Luke has been engineering networks since the early days of the dial-up internet, using home-grown software development as a tool to improve the reliability of those services. Chris is currently at Comcast, modernizing the configuration management and automation practices of the core network. He also leads a program teaching network engineers how to bring code to the day job.

Chris is currently a maintainer for the OpenConfig project and a committer for the FD.io VPP project. He has previously been the chairperson of the OpenDaylight Advisory Group and has contributed networking code to FreeBSD and Linux, as well as to the Bird and Quagga routing daemons and many other open source projects.

Table of Contents

Prefacexiii

Part 1: The Go Programming Language

1

Introduction3

Technical requirements4

Networking and Go4

Why Go?5

Non-technical reasons5

The future of Go8

Technical reasons8

Go for networking13

Go versus Python16

Code execution16

Type system17

Performance17

Ease of use17

Memory management18

Syntax18

Failure handling18

Concurrency19

Community 19

Installing Go on your computer20

Windows20

Mac20

Linux21

Summary21

Further reading21

2

Go Basics25

Technical requirements26

What is Go?26

Go Proverbs29

Go source code files30

Packages31

Go modules32

Importing packages33

Comments34

Names36

Executing your Go code37

Running Go programs online38

The Go Playground38

The Go Play Space40

A look into the Future42

The Go tool43

Build44

Run46

Mod46

Get49

Install49

Fmt50

Test50

Env52

Summary53

Further reading53

3

Getting Started with Go55

Technical requirements55

Go’s type system56

Basic types57

Container types63

User-defined types69

Arithmetic, comparison, and logical operators71

Arithmetic operators71

Logical operators77

Comparison operators78

Control flow79

for loops79

Conditional statements81

goto statements83

Functions85

Function arguments86

Error handling90

Methods91

Variadic functions92

Closures93

Defer94

Interfaces95

Network automation example96

Standard library example98

Interfaces as contracts100

Input and output operations101

The io.Reader interface101

The io.Writer interface102

The io.Copy function104

Composition106

Decoding and encoding107

Decoding107

Encoding114

Concurrency115

Goroutines116

Channels122

Channels and Timers124

Shared data access126

Concurrency caveats130

Summary130

Further reading131

4

Networking (TCP/IP) with Go135

Technical requirements136

The link layer136

Network interfaces136

Ethernet139

The internet layer143

The net package143

The New netip package146

Working with IP addresses149

Route lookups150

The transport layer155

UDP ping application156

The application layer161

Working with an HTTP client162

Working with an HTTP (server)165

Summary169

Further reading169

Part 2: Common Tools and Frameworks

5

Network Automation173

Technical requirements173

What is network automation?174

Why network automation exists174

Bottom-up view175

Top-down view176

Automating network operation tasks177

Configuration management177

Network state analysis179

Network audits and reporting181

Systems approach182

Closed-loop automation182

Demo application183

Summary188

Further reading188

6

Configuration Management189

Technical requirements190

Environment setup190

Creating the topology190

Interacting with network devices via SSH192

Describing the network device configurations193

Using Go’s SSH package to access network devices195

Automating routine SSH tasks200

Interacting with network devices via HTTP202

Using Go’s HTTP package to access network devices202

Getting config inputs from other systems via HTTP206

State validation210

Checking routing information210

Validating end-to-end reachability217

Summary218

Further reading218

7

Automation Frameworks221

Technical requirements221

Ansible222

Overview of Ansible components222

Working with Ansible modules224

Developing an Ansible module226

Running the playbook230

Terraform230

Overview of Terraform components231

Working with Terraform232

Developing a Terraform provider235

Networking providers242

Other automation frameworks242

Gornir243

Consul-Terraform-Sync243

mgmt243

Looking into the future244

Summary244

Further reading245

Part 3: Interacting with APIs

8

Network APIs249

Technical requirements250

API data modeling250

OpenAPI252

Data modeling253

Data input255

Device configuration256

JSON-RPC259

Code generation260

Building configuration261

Device configuration264

RESTCONF266

Code generation266

Building configuration268

Device configuration270

State validation271

Operational state modeling272

Operational state processing272

gRPC275

Protobuf275

gRPC transport278

Defining gRPC services278

Configuring network devices with gRPC280

Streaming telemetry from a network device with gRPC284

Summary290

Further reading290

9

OpenConfig293

Technical requirements294

Device provisioning294

Set RPC295

Using gNMI to configure network interfaces297

Streaming telemetry302

Subscribe RPC304

Streaming telemetry processing pipelines with gNMI306

Event-manager sample program306

Visualizing the data311

Network operations314

Traceroute RPC316

Path verifier application317

Summary325

Further reading325

10

Network Monitoring327

Technical requirements328

Data plane telemetry processing328

Packet capturing328

Packet filtering330

Packet processing331

Generating traffic333

Debugging Go programs334

Debugging from an IDE336

Data plane telemetry aggregation338

Top talkers339

Testing Go programs344

Measuring control plane performance348

Measuring BGP Update propagation time350

Event-driven BGP state machine351

Encoding and decoding BGP messages356

Collecting and exposing metrics357

Storing and visualizing metrics358

Developing distributed applications359

Summary360

Further reading360

11

Expert Insights363

David Barroso363

Stuart Clark364

Claudia de Luna365

Network automation truths... so far...365

Alexis de Talhouët368

John Doak370

Roman Dodin371

David Gee373

My journey toward a decade of Go373

Go’s type system374

A note on version control376

Growing your code376

Comments379

Being blindsided380

Daniel Hertzberg381

Marcus Hines382

In a nutshell382

Reasoning about automation383

Background383

Ability to impact the industry384

Sneha Inguva385

Antonio Ojea386

Carl Montanari387

Brent Salisbury389

Maximilian Wilhelm390

A little bit of history390

Mental shift to holistic automation391

This is the way392

Matt Oswalt392

Further reading394

12

Appendix : Building a Testing Environment395

What is a testing environment?395

Step 1 – building a testing environment396

Step 2 – uploading container images401

Step 3 – iInteracting with the testing environment402

Launching a virtual network topology404

Connecting to the devices405

Destroying the network topology406

Step 4 – cleaning up of the cloud-hosted environment406

Further reading407

Index409

Other Books You May Enjoy420

Share Your Thoughts

Once you’ve read Network Automation with Go, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don't worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don't stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/978-1-80056-092-5

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: The Go Programming Language

This part provides an introduction to the book, the topics we will cover, and how to run the code examples provided throughout the course of the book. You can choose to use your personal computer or a virtual machine for this purpose.

Also, it provides a solid foundation for Go. By the end, you will be able to install and run Go programs. You will also learn how to manipulate network data with Go, such as IP addresses and XML/YAML/JSON documents, and use Go to run network transactions/protocols.

This part of the book comprises the following chapters:

Chapter 1, Introduction Chapter 2, Go BasicsChapter 3, Getting Started with Go Chapter 4, Networking (TCP/IP) with Go

1

Introduction

Go has emerged as one of the top three most wanted programming languages according to the Stack Overflow Developer Survey 2021 (Further reading), and it has become the preferred choice for writing cloud-native applications such as Kubernetes, Docker, Istio, Prometheus, and Grafana.

Despite this, we still don't see this trend manifest in the network engineering community, where fewer than 20% of network engineers saying they currently use Go for their network automation projects, according to the NetDevOps 2020 survey (Further reading), even though 41% of Go developers say they use Go for network programming in Go Developer Survey 2020 Results (Further reading).

This book strives to address this disparity by offering a practical introduction to Go and network automation for network engineers who want to evolve network management and operation using Go, and software engineers wanting to get into network infrastructure automation. We also hope that this book may be useful to network automation engineers who know and use Python today but want to expand their skill set with a different programming language.

We start by discussing the benefits of Go from different angles and how they apply to the networking field. By the end of this chapter, you should have a good understanding of the main aspects of Go and how to get Go installed on your computer to follow along with the code examples.

In this first chapter, we cover the following topics:

Networking and GoWhy Go? The future of GoGo versus Python Installing Go on your computer

Technical requirements

We assume basic familiarity with the command line, Git, and GitHub. You can find the code examples for this chapter in the book's GitHub repository (https://github.com/PacktPublishing/Network-Automation-with-Go), under the ch01 folder.

To run the examples, proceed as follows:

Install Go 1.17 or later for your operating system. You can follow the instructions in the Installing Go on your computer section of this chapter or go to https://go.dev/doc/install.Clone the book's GitHub repository with git clone https://github.com/PacktPublishing/Network-Automation-with-Go.git.Change the directory to an example's folder with cd Network-Automation-with-Go/ch01/concurrency.Execute go run main.go.

Networking and Go

Go is widely used in generic infrastructure software—from workload orchestration (Docker and Kubernetes), through telemetry and monitoring (Prometheus and Grafana), all the way to automation tooling (Terraform and Vagrant).

Networking is not the exception—some notable networking projects using Go include Container Network Interface (CNI) plugins such as Cilium or Calico, routing protocol daemons such as GoBGP and Bio-RD, virtual private network (VPN) software such as Tailscale, and most of OpenConfig's ecosystem, including projects such as gRPC Network Management Interface (gNMI) and goyang.

Other use cases include cloud and network services, command-line interfaces (CLIs), web development, development-operations (DevOps), and site reliability.

Go is a programming language the Go founders created to address modern challenges such as multi-core processing, distributed systems, and large-scale software development from day one.

Go's built-in first-class concurrency mechanisms make it an ideal choice for long-lived low-bandwidth input/output (I/O) operations, which are typical requirements of network automation and network operations applications.

What makes the Go language so appealing to software developers? Why, out of all the programming languages out there, should you invest time in learning Go? This is what we address in the next section.

Why Go?

When choosing which programming language to learn next, most people focus mainly on technical reasons. We believe that the choice can be a bit more nuanced, so we try to approach this question from different angles. We start with non-technical arguments, something that's often overlooked but that we believe is important and can have a major impact on both the learning process and day-to-day use. Following that, we cover generic technical arguments that help Go stand out in the very competitive landscape of modern programming languages. We close out this section by exploring different facets of Go that can benefit people, specifically in the fields of networking and network automation.

Non-technical reasons

Whether you are new to the language or have some experience with it, you can access more experienced Go developers in the community who are willing to help you learn more about the language. We include some pointers to community resources and go through the adoption and popularity of Go.

Last but not least, we want to address the maturity of the language, whether it's still in development, and where Go is headed in the future.

Community

A healthy community is almost always an attribute of a successful project. The Go programming language is no exception, with its welcoming and growing community of Go developers—Gophers, with about 2 million of them in the world, according to Russ Cox's article, How Many Go Developers Are There? (Further reading). You can see Renée French's Go Gophermascot here:

Figure 1.1 – Go Gopher, by Renée French

The Go user community has several places where newcomers can ask questions and get help from more experienced Go developers, as listed here:

golang-nuts mailing list (Further reading)—Google Groups mailing list for any generic language discussionsGo Forum (Further reading)—a standalone forum for technical discussions, release announcements, and community updatesGo Language Collective (Further reading)—the official question-and-answer (Q&A) channel on Stack OverflowGophers Slack channel (Further reading)—a place for generic and topic-specific discussions, including a dedicated networking channel

If you want more live interactions, there are some options available as well, as outlined here:

A good deal of in-person meetups are available via the Go Developers Network (GDN) (Further reading).One of the principal events in the Go community is GopherCon, held regularly in different parts of the world.The official Go wiki page hosted on GitHub keeps track of all future and past Go conferences and major events (Further reading).

Popularity

Ever since its foundation in the late 2000s, Go has gained a lot of interest from the developer community, not least because of who was behind it. Developed by a group of some of the best computer scientists employed by Google to solve the problems of C/C++, Go is a language that's both simple to understand and nearly as efficient as its predecessors. It took a few years to mature, but it had soon become the new hot start up language, and many up-and-coming software companies such as Docker and HashiCorp adopted it.

Most recently, the Stack Overflow Developer Survey 2021 (Further reading) recognized Go as one of the top three most wanted programming languages by developers. Continuous support from its mothership, and the success of Kubernetes, have made it a de facto standard language to write cloud-native applications with such notable projects as Istio, CoreDNS, Prometheus, and Grafana. As more and more users adopt these applications, it's hard to imagine Go's popularity waning in the future.

Here are a few extra data points in support of Go's rising popularity that are worth mentioning:

225 out of 291 Cloud Native Computing Foundation (CNCF) projects use Go, as reported in the CNCF DevStats toolset (Further reading).Go ranks third as the language with the most stars on GitHub, according to GitHut 2.0 (Further reading).Go is behind three out of the four most popular development tools (Docker, Kubernetes, and Terraform) (Further reading).Go is in the top 10 of the Stack Overflow Developer Survey 2021's top-paying technologies ranking (Further reading).

Maturity

While the Go team released Go (version 1) not too long ago (March 2012), Go has been getting minor changes ever since. The language designers assume a strict position against adding unnecessary features that may result in feature creep. At GopherCon 2014's opening keynote, Rob Pike made this comment explicitly: "The language is done." Russ Cox also mentioned this in his article Go, Open Source, Community (Further reading), referring to Go 1 specifically.

This doesn't mean Go does not have its fair share of pain points. For example, dependency management has been a problem the Go team addressed fairly recently with the introduction of Go modules to better group Go packages you release together. There was also a lack of generics support, a feature that the Go team is now introducing in Go 1.18, probably the most significant change since the release of Go (version 1). Now, users can represent functions and data structures with generic types, which enables code reuse. This addresses one of the primary requests from the community, as Go Developer Survey 2020 Results shows (Further reading).

Despite that, these few changes are very selective and designed to dramatically improve developer productivity. It's safe to assume that we won't see a situation where you have to learn new language concepts and idioms every year and have to rewrite your code to maintain forward compatibility. The Go 1 compatibility guarantee in Go 1 and the Future of Go Programs (Further reading) states the following:

It is intended that programs written to the Go 1 specification will continue to compile and run correctly, unchanged, over the lifetime of that specification. ...code that runs under Go 1.2 should be compatible with Go 1.2.1, Go 1.3, Go 1.4, and so on.

Go benefits from the lessons learned from other programming languages. Pascal, Oberon, C, and Newsqueak are among the languages that influenced Go. We explore their impact in Chapter 2, Go Basics.

Go follows a 6-month release cycle (Further reading). In the Go release notes for each version (Further reading), there is a section at the top that describes changes to the language, which in general is very brief or empty. Over the last couple of years, they reported only four small enhancements to the language, which is a good sign of maturity.

How much Go will change in the future is something we discuss in the next section.

The future of Go

The success of Go version 1 has attracted a lot of developers, most of them with prior experience in other languages that helped shape their mindset and expectations of what a programming language should deliver. The Go team has defined a process to propose, document, and implement changes to Go (Further reading), to give a way for these new contributors to voice their opinions and influence the design of the language. They would label any proposals that break the language-compatibility guarantee, described in the preceding section, as Go 2.

The Go team announced the start of the process of developing Go version 2 at GopherCon 2017 and with the blog post Go 2, here we come! (Further reading). The intention is to ensure the language continues to enable programmers to develop large-scale systems, and to scale to a sizable code base that big teams work on simultaneously. In Toward Go 2 (Further reading), Russ Cox said the following:

Our goal for Go 2 is to fix the most significant ways Go fails to scale.

Any language change proposal needs to follow the Go 2 language change template (Further reading). They are shipping all Go 2 features that are backward-compatible incrementally in Go 1. After that is complete, they can introduce backward-incompatible changes (see Go 2 proposals: Further reading), in case they offer a significant benefit, into Go 2.0.

Support for generic data types is part of the Go 2 draft designs document (Further reading), along with improved error handling, and error-value semantics. The first implementation of generics has already made it into Go 1. The other items in the list are still under evaluation, pushing the release of 2.0 further into the future.

Technical reasons

Go's build speed is a top-of-the-chart aspect of Go that Go developers are more satisfied with, according to Go Developer Survey 2020 Results (Further reading). It's followed very closely by Go's reliability, in second place.

The list of technical aspects we could highlight is large, but aside from build speed and reliability, we cover performance, cross-compiling, readability, and Go's tooling.

Type safety

Most programming languages can be broadly categorized as either statically typed when variable types are checked at compile time or dynamically typed when this check happens during the program execution (runtime). Go belongs to the first category and requires programs to declare all variable types explicitly. Some beginners or people with a background in dynamically typed languages might see this as a detractor.

Type declarations increase the amount of code that you need to write, but in return, you not only get performance benefits but also protection from type errors occurring at runtime, which can be a source of many subtle and hard-to-troubleshoot bugs. For example, consider the program in the next code example at https://github.com/PacktPublishing/Network-Automation-with-Go/blob/main/ch01/type-safety/main.go:

func process(s string) string { return"Hello " + s } func main() { result := process(42) }

A process function takes a string data type as input and returns another string that concatenates Hello and the value of the input string. A dynamically typed program can crash if this function receives a value of a type different from string, such as an integer, for example.

These errors are very common, especially when dealing with complex data structures that can represent a network configuration or state. Go's static type checking prevents the compiler from producing a working binary generating the following error:

cannot use 42 (type untyped int) as type string in argument to process

Readability also improves with Go's static typing. A developer might be able to keep the entire data model in mind when writing code from scratch, but as new users come into a project, code readability becomes critical to help them understand the logic to make their required code changes. No longer do they need to guess which value type a variable stores—everything is explicitly defined by the program. This feature is so valuable that some dynamically typed languages forgo the benefit of their brevity to introduce the support for type annotations (such as Python typing: Further reading), with the only goal to help integrated development environments (IDEs) and static linters catch obvious type errors.

Go builds are fast

Go is a compiled language that creates small binary files in seconds or a couple of minutes tops. Initial build time may be a bit longer, mostly because of the time it takes to download dependencies, generate extra code, and do other household activities. Subsequent builds run in a fraction of that time. For example, the next capture shows that it takes no more than 10 seconds to rebuild a 120-megabytes (MB) Kubernetes application programming interface (API) server binary:

$ time make kube-apiserver +++ [0914 21:46:32] Building go targets for linux/amd64:   cmd/kube-apiserver > static build CGO_ENABLED=0: k8s.io/kubernetes/cmd/kube-apiserver make kube-apiserver  10.26s user 2.25s system 155% cpu 8.041 total

This allows you to iterate quickly through the development process and to keep focus, without spending minutes waiting for code to recompile. Some developer productivity tools, such as Tilt, take further actions to optimize the development workflow so that it takes seconds for changes to propagate from a developer's IDE to their local staging environment.

Reliability

Let's define this term as a set of properties of a programming language that help developers write programs that are less likely to fail because of bugs and other failure conditions, as Jiantao Pan from Carnegie Mellon University (CMU) describes in Software Reliability (Further reading). This is one of Go's core tenets, as its website (Further reading) highlights:

Build fast, reliable, and efficient software at scale.

Go developers also say reliability is the second aspect of Go they are most satisfied with, only behind build speed, based on Go Developer Survey 2020 Results (Further reading).

A more reliable software means less time spent chasing bugs and more time invested in the design and development of extra features. We've tried to put together a set of features that we think contribute to increased program reliability. This is not a definitive list, though, as interpretation and attribution of such features can be very subjective. Here are the features we've included:

Code complexity—Go is a minimalistic language by design. This translates into simpler and less error-prone code.Language stability—Go comes with strong compatibility guarantees, and the design team tries to limit the number and impact of newly added features.Memory safety—Go prevents unsafe memory access, which is a common source of bugs and exploits in languages with pointer arithmetic, such as C and C++.Static typing—Compile-time type-safety checks catch many common bugs that would otherwise go unnoticed in dynamically typed languages.Static analysis—An automatic way to analyze and report several errors, such as unused variables or unreachable code paths, comes built into the language tooling with go vet.

Performance

Go is a highly performant language. The Computer Language Benchmarks Game (Further reading) shows that its performance is in the vein of languages with manual memory management, such as C/C++ and Rust, and that it offers considerably better performance than dynamic type languages such as Python and Ruby.

It has native support for multi-core multithreaded central processing unit (CPU) architectures, allowing it to scale beyond a single thread and to optimize the use of CPU caches.

Go's built-in garbage collector helps you keep the memory footprint of your program low, and Go's explicit type declaration optimizes memory management and storage of values.

The Go runtime gives you profiling data, which you can visualize with pprof to help you hunt for memory leaks or spot bottlenecks in your program and fine-tune your code to achieve better performance and optimize resource utilization.

For more details on this subject, we recommend checking out Dave Cheney's Five things that make Go fast blog post (Further reading).

Cross-platform compiling

Go can natively produce binaries for different target architectures and operating systems. At the time of writing, the go tool dist list command returns 45 unique combinations with operating systems ranging from Android to Windows and instruction sets that go from PowerPC to ARM. You can change the default values inherited from the underlying operating system and architecture with GOOS and GOARCH environment variables.

You can build an operating system-native version of your favorite tool written in Go, regardless of which operating system you are currently on, as illustrated in the following code snippet:

ch01/hello-world$ GOOS=windows GOARCH=amd64 go build ch01/hello-world$ ls hello-world* hello-world.exe

The preceding output shows an example to create a Windows executable on a Linux machine.

Readability

This is, arguably, one of the best qualities of Go when compared to other high-performance languages such as C or C++. The Go programming language specification (Further reading) is relatively short, with around 90 pages (when other language specifications can span over 1,000 pages). It includes only 25 keywords, with only one for loop (for). The number of features is intentionally low to aid code clarity and to prevent people from developing too many language idioms or best practices.

Code formatting is an active battleground in other languages, while Go prevented this problem early on by shipping automatic opinionated formatting as part of the go command. A single run of go fmt on any unformatted (but syntactically correct) code updates the source file with the right amount of indentation and line breaks. This way, all Go programs have a similar look, which improves readability by reducing the number of personal style preferences in code.

Some might say that explicit type declarations alone improve code readability, but Go takes this a step further by making comments an integral part of the code documentation. All commented lines preceding any function, type, or variable declaration gets parsed by the go doc tool website (Further reading) or an IDE to autogenerate code documentation, as the following screenshot shows:

Figure 1.2 – Automatic code documentation

Most modern IDEs have plugins that support not only documentation but automatic code formatting with go fmt, code linting and autocompletion, debugging, and a language server—a tool that allows developers to navigate through the code by going back and forth between type, variable, and function declarations and their references (gopls, the Go language server: Further reading). This last feature not only allows you to navigate code bases of any complexity without having to resolve import statements manually or search for string patterns in text, but also highlights any type inconsistencies on the fly before you compile a program.

Tooling

When setting up a new environment, one of the first things a typical developer would do is download and install a set of their favorite language tools and libraries to help with testing, formatting, dependency management, and so on. Go comes with all these utilities included by default, which are part of the go command. The following table summarizes some Go built-in tools and their purpose:

Table 1.1 – Go tools

These are just a few of the most popular tools that get shipped together with the Go binary. This certainly reduces the room for creativity in the tooling ecosystem by giving developers a default choice that is good enough for most average use cases. Another benefit of this artificial scarcity is not having to reinstall and relearn a new set of tools every time you switch between different Go projects.

Go for networking

Some network automation processes can trigger hundreds—if not thousands—of simultaneous connections to network devices. Being able to orchestrate this at scale is one of the things that Go enables us to do.

You can see Egon Elbre's Network Gopher mascot in the following screenshot:

Figure 1.3 – Network Gopher, by Egon Elbre

Go comes with a strong networking package that offers you all the constructs to create network connections, packages to encode and decode data from popular formats, and primitives to work with bits and bytes.

Concurrency

Go has first-class support for concurrency with the help of lightweight threads managed by the Go runtime, called goroutines. This language construct makes it possible to embed asynchronous functions into an otherwise sequential program.

Any function call that you prepend with the go keyword runs in a separate goroutine—different from the main application goroutine—that does not block execution of the calling program.

Channels are another language feature that allows communication between goroutines. You can think of it as a first-in, first-out (FIFO) queue with sending and receiving ends existing in two different goroutines.

Together, these two powerful language constructs offer a way to write concurrent code in a safe and uniform way that allows you to connect to various networking devices simultaneously, without paying the tax of running an operating system thread for each one. For example, consider the following program in the next code example (https://github.com/PacktPublishing/Network-Automation-with-Go/blob/main/ch01/concurrency/main.go) that simulates interaction with remote network devices:

func main() { devices := []string{"leaf01", "leaf02", "spine01"} resultCh := make(chan string, len(devices)) go connect(devices, resultCh) fmt.Println("Continuing execution") for msg := range resultCh { fmt.Println(msg) } }

Connecting to remote devices can take a long time, and it would normally block the execution of the rest of the program. With the connect function running in a goroutine, as illustrated in the following code snippet, our program can continue its execution, and we can come back and collect the responses at any point in the future:

ch01/concurrency$ go run main.go Continuing execution Connected to device "leaf01" Connected to device "spine01" Connected to device "leaf02"

As the remote devices process the requests and return a response, our program starts printing the responses in the order it receives them.

Strong standard library

Go has a versatile standard library that covers different areas that may be applicable to networking—from cryptography to data encoding, from string manipulation to regular expressions (regexes) and templating. Standard library packages such as net and encoding offer interfaces for both client- and server-side network interactions, including the following:

Internet Protocol (IP) prefix parsing and comparison functionsClient and server implementations for IP, Transmission Control Protocol/User Datagram Protocol (TCP/UDP), and HyperText Transfer Protocol (HTTP) connectionsDomain Name System (DNS) lookup functionsUniform Resource Locator (URL) parsing and manipulationsSerializing data formats such as Extensible Markup Language (XML), binary, and JavaScript Object Notation (JSON) for storage or transmission

Unless you have unique performance requirements, for example, most Go developers recommend against using external libraries for logic that can otherwise be implemented natively with the standard library. All standard packages are thoroughly tested with each release and used extensively in several large-scale projects. All this creates a better learning experience for newcomers because most-often-used data structures and functions are there already.

Data streaming

Network services are I/O-bound in general—they read or write bytes from or to the network. This mode of operation is how data streaming works in Go, which makes it appealing to network engineers who are familiar with byte processing for network protocol parsing, for example.

I/O operations in Go follow a model where a Reader reads data from a source, which can stream as an array of bytes to a Writer that writes that data to a destination. The following diagram should give you a clearer picture of what this means:

Figure 1.4 – Streaming from a network connection to a file example

A Reader is an interface that can read from a file, a cipher, a shell command, or a network connection, for example. You can then stream the data you capture to a Writer interface, which could also be a file or most of the other Reader examples.

The Go standard library offers these streaming interfaces, such as net.Conn, that, in this case, allow you to read and write from a network connection, transfer data between interfaces, and transform this data if needed. We cover this topic in much more detail in Chapter 3, Getting Started with Go.

While there are other variables to consider when selecting a programming language to work with, such as which one your company is currently using or which one you feel more comfortable with, our goal is to equip you with all the resources to understand what makes Go so appealing to large-scale system developers. If you want to begin in familiar territory, we compare and contrast Go with Python next. Python is the most popular programming language used for network automation today.

Go versus Python

The topic of comparing programming languages can very quickly turn into a heated debate. We believe all languages have their merits and we don't want to advocate for one being better than the other. Still, we do acknowledge that most people with a network automation background would know and use Python, so it would make sense to present some form of comparison between the two languages and highlight some of their most salient points.

Code execution

One of the biggest differences that affect the developer experience is how you distribute and execute your code.

Python programs require an interpreter to run on a target machine and access to all library dependencies. While there are projects such as Nuitka to compile Python, you need commercial support to obfuscate your source code, for example. Having all source code available allows you to make changes and iterate quickly when developing a feature or troubleshooting a bug.

Go programs do not require an interpreter, as you distribute them as a compiled binary file. Compiling to machine code may seem like an unnecessary hurdle, but compilation takes only a few seconds, and the resulting binary has all its required dependencies, so it's the only file that needs to exist on the target system.

Type system

Go requires all variable types to be statically defined, with type inference allowed only during initial variable declaration.

Although generics are making their way into Go, they do not allow the same amount of freedom as a Python type system. A lack of explicit type declaration makes Python a more approachable language for beginners and for use cases where development speed is more important than code robustness. However, as Python projects become more mature, they must make up for these initial gains by putting more focus on testing.

Performance

Go programs perform better when compared to Python across a wide range of use cases (see The Computer Language Benchmarks Game: Further reading). This is, in part, an outcome of the points we already mentioned in this section, but it's also the result of the effort the Go team has put into optimizing the language.

While things such as goroutines and type definition give Go developers enough tools to write high-performance code, each Go release brings new improvements in memory management and compiler optimizations that make code execution faster in the background.

Ease of use

Python is a language designed to be used for teaching and prototyping. At the same time, it's versatile and powerful enough to write complex programs such as web servers (Flask, Django), machine learning (ML) frameworks (PyTorch, TensorFlow), and infrastructure software (RabbitMQ, Ansible).

As the number of Python projects you work on grows, maintaining different virtual environments for dependency and environment management might become a hassle. This is an area where Go shines, with its self-hosted dependency manager and statically linked binaries.

Despite that, Python continues to hold its dominant position as the most approachable language with a large open source community and is unlikely to relinquish it any time soon.

Memory management

Both languages use dynamic memory management with automatic garbage collection. Most of the time, you wouldn't need to and are not advised to change any of the default settings, although both languages expose a few threshold variables that can be fine-tuned if needed.

The biggest difference comes from the fact that Go allocates memory based on a more precise set of data types and that it does static memory allocation at compile time in the stack for goroutines and functions, and only a subset of variables escape to the heap. In contrast, Python treats everything as an object, so even the most primitive types, such as int or string, are considerably larger, and they are dynamically allocated memory at runtime (in the heap).

Access to memory in the heap is not only slower but also needs to be garbage-collected, which adds an overhead to the program execution.

Syntax

Python has a very lightweight syntax and uses indentation to separate different blocks of code. The lack of trailing semicolons and excessive curly braces make it comprehensible, but writing it without an IDE—which would automatically manage the indentation—can be a challenge.

Go never considered white space for indentation, as the language designers don't believe having your semantics depend on invisible characters is a good idea. This, of course, comes down to personal preferences; formats such as YAML Ain't Markup Language (YAML), for example, also use spaces to structure data.

Go benefits from its built-in formatting tool that auto-indents the code and makes it look neat by automatically inserting blank lines in certain places. Also, Go developers use blank lines to split logically separate a set of lines in a function that makes the final program less dense and easier to read.

Failure handling

Another big difference is in error handling. Python uses implicit error handling as a convention by relying on exceptions that can be carefully caught in parts of code where you expect them to happen. This keeps in line with Python's readability and ease-of-use nature. Go uses explicit error checks, and most functions have errors as the last positional return value. This often results in the code looking like this:

config, err : buildConfig(deviceName) if err !=nil { return err } d, err : connect(deviceName) if err !=nil { return err } if err : configure(d, config); err !=nil { return err }

Although this makes a program more robust by forcing the developers to always think about the returned error and act on it as soon as it happens, this does create a lot of visual noise that human brains quickly learn to ignore. This is a recurrent topic in the Go community and one of the areas that Go version 2 is putting a focus on. The Go 2 draft design document for error handling covers the problem and proposal in detail (Further reading).

Concurrency

Concurrency has not only been a feature of Go since day one but also one of the key drivers behind the creation of Go in the first place. Go has enough first-class language constructs to deal with most common concurrency challenges, such as communication between processes and access to shared resources.

By contrast, you cannot run more than two or more Python threads at the same time because the Global Interpreter Lock (GIL) prevents it, which the Python language designers made part of the language early on. This is unless you architect your program to use the threading library. The GIL has performance benefits for single-threaded programs, and removing it from the language has been a recurrent topic in the Python community.

To implement concurrency, Python makes you run multiple processes to leverage all the CPUs that you have at your disposal (multiprocessing or concurrency pools). Over time, different libraries have attempted to improve the performance and user experience (UX) of concurrency in Python, with the most popular one being asyncio.

Despite that, better concurrency and parallelism are in the top three most desired features to add to Python, according to Python Developers Survey 2020 Results (Further reading). Most Python developers don't like the current implementation, as writing concurrent code in Python can be challenging and requires the use of compatible libraries.

Community

Being the more popular language of the two, Python has a larger community with a huge number of open source libraries and frameworks. Although its major use cases are data analysis, web development, and ML (Python Developers Survey 2020 Results: Further reading), today you can find libraries that deal with anything from game development to desktop plugins.

Most importantly, Python is the most popular language for network automation and has amassed many libraries and frameworks to work with network devices. Go has been more systems- and performance-centric, so we don't see as many network libraries and tools. Still, one heavy user of Go in the network engineering community has been the OpenConfig ecosystem, which today includes almost a dozen different projects written in Go.

Go is being rapidly adopted by web-scale companies, which means we are likely to see more network-related projects appearing in the future.

We hope this gives you a perspective and appreciation of the Go language features. The next step is to install Go on your computer.

Installing Go on your computer

The Go download and install instructions (https://golang.org/doc/install#install) require you to download a file from https://go.dev/ and follow a couple of instructions. We include here the steps for Go version 17.7, which is the latest version available at the time of writing. Newer versions of Go 1 should continue to work.

Windows

To install Go on Windows, follow these steps:

Download https://golang.org/dl/go1.17.7.windows-amd64.msi.Execute the go1.17.7.windows-amd64.msi file and follow the instructions.Open the Command Prompt window (cmd) and run go version to verify the installation.

Mac

If you have Homebrew installed, you can run brew install go. Otherwise, you can follow these steps:

Download https://golang.org/dl/go1.17.7.darwin-amd64.pkg.Execute the go1.17.7.darwin-amd64.pkg file and follow the instructions.Open a Terminal and run go version to verify the installation.

Linux

Go is typically available as a system package in a Linux distribution, but is often an older version. Follow these steps to install a more recent release:

Download https://golang.org/dl/go1.17.7.linux-amd64.tar.gz.Remove any existing Go installation with rm -rf /usr/local/go.Extract the archive you downloaded into /usr/local with tar -C /usr/local -xzf go1.17.7.linux-amd64.tar.gz.Add /usr/local/go/bin to the PATH environment variable with export PATH=$PATH:/usr/local/go/bin. To make this persistent, add this line as well in $HOME/.bash_profile. This last part is valid for bash, but you might want to do something similar if you use a different shell.Run go version to verify the installation

There you go! You can now download and install Go in your system without any hassle. To install a different version, just replace 17.7 in the instructions with a target version of your choice.

Summary

In this chapter, we reviewed why Go is relevant for networking and network automation. We looked at the various aspects of Go that make it the preferred choice for millions of developers. We also explored how you can install it on your computer. In the next chapter, we will dive deeper into the Go programming language, its source files, and its tools.

Further reading

You can refer to these resources for further reading:

Stack Overflow Developer Survey 2021: https://insights.stackoverflow.com/survey/2021#most-loved-dreaded-and-wanted-language-wantNetDevOps 2020 survey: https://dgarros.github.io/netdevops-survey/reports/2020Go Developer Survey 2020 Results: https://go.dev/blog/survey2020-resultsHow Many Go Developers Are There?: https://research.swtch.com/gophercountgolang-nuts: https://groups.google.com/forum/#!forum/golang-nutsGo Forum: https://forum.golangbridge.org/Go Language Collective: https://stackoverflow.com/collectives/goGophers Slack channel: https://invite.slack.golangbridge.org/Go Developers Network (GDN): https://www.meetup.com/pro/goCNCF DevStats toolset: https://k8s.devstats.cncf.io/d/67/licenses-and-programming-languages?orgId=1https://madnight.github.io/githut/#/stars/2021/2Go 6-month release cycle: https://github.com/golang/go/wiki/Go-Release-CycleGo release notes: https://golang.org/doc/devel/releasehttps://github.com/golang/proposal#proposing-changes-to-goToward Go 2: https://go.dev/blog/toward-go2Go 2 language change template: https://github.com/golang/proposal/blob/master/go2-language-changes.mdGo 2 proposals: https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3AGo2+label%3AProposalGo 2 draft design document: https://go.googlesource.com/proposal/+/master/design/go2draft.mdPython typing: https://docs.python.org/3/library/typing.htmlgo doc tool website: https://pkg.go.dev/Go language server: https://go.googlesource.com/tools/+/refs/heads/master/gopls/README.md#editorsGo 2 draft design document: https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling-overview.mdGo Conferences and Major Events: https://github.com/golang/go/wiki/Conferences#go-conferences-and-major-eventsPopular development tools: https://insights.stackoverflow.com/survey/2021#most-loved-dreaded-and-wanted-tools-tech-love-dreadTop-paying technologies ranking: https://insights.stackoverflow.com/survey/2021#technology-top-paying-technologieshttps://insights.stackoverflow.com/survey/2021#technology-top-paying-technologiesGo version 1: https://go.dev/blog/go1Why does Go not have feature X?: https://golang.org/doc/faq#Why_doesnt_Go_have_feature_Xhttps://golang.org/doc/faq#Why_doesnt_Go_have_feature_XGo, Open Source, Community: https://go.dev/blog/open-sourceGo 1 and the Future of Go Programs: https://golang.org/doc/go1compatGo 2, here we come!: https://go.dev/blog/go2-here-we-comeSoftware Reliability: https://users.ece.cmu.edu/~koopman/des_s99/sw_reliability/The Computer Language Benchmarks Game: https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/go-gpp.htmlFive things that make Go fast: https://dave.cheney.net/2014/06/07/five-things-that-make-go-fastThe Go Programming Language Specification: https://golang.org/ref/specPython Developers Survey 2020 Results: https://www.jetbrains.com/lp/python-developers-survey-2020/

2

Go Basics

With so many programming languages out there, it’s fair to wonder why anyone would have to invent yet another one. What the background is of the people behind Go and what the problems are they are trying to solve with this new language are some of the items we will address in this chapter.

These topics give us some perspective on the challenges large-scale software development presents to software developers today and why modern technologies such as programming languages are constantly evolving.

By the end of this chapter, you should have a better understanding of where Go comes from and its role in developing distributed systems running on multi-core processors, as well as be familiar with Go’s source code structure as we go through the following areas:

What is Go? Go’s guiding principles Go source code file structure Go packages and modules Compiling Go programs Running Go programs online Exploring the Go tool to manage Go source code

Technical requirements

We assume that you have basic familiarity with the command line, Git, and GitHub. You can find the code examples for this chapter in the book’s GitHub repository, https://github.com/PacktPublishing/Network-Automation-with-Go, in the ch02 folder.

To run the examples, follow these steps:

Install Go 1.17 or later for your operating system. You can follow the instructions in Chapter 1, Introduction, in the Installing Go on your computer section, or go to https://go.dev/doc/install.Clone the book’s GitHub repository with git cloneat https://github.com/PacktPublishing/Network-Automation-with-Go.git.Change the directory to an example’s folder – cd Network-Automation-with-Go/ch02/pong.Execute go run main.go.

What is Go?

During the second half of 2007, Robert Griesemer, Rob Pike, and Ken Thompson started discussing the design of a new programming language that would solve some problems they were experiencing when writing software at Google, such as the increased complexity to use some languages, long code compilation times, and not being able to program efficiently on multiprocessor computers.

Rob Pike was trying to take some concurrency and communicating channels ideas into C++, based on his earlier work on the Newsqueak language in 1988, as he describes in Go: Ten years and climbing (Further reading) and Less is exponentially more (Further reading). This turned out to be too hard to implement. He would work out of the same office with Robert Griesemer and Ken Thompson. Ken had worked together with Rob Pike in the past to create the character-encoding UTF-8, while Ken Thompson had designed and implemented the Unix operating system and invented the B programming language (the predecessor to the C programming language).

They chose the name Go for this new programming language because it’s short, but the DNS entry for go.com wasn’t available, so Go’s website ended up at golang.org. And so, golang became a nickname for Go. While golang is convenient for search queries, it’s not the name of the language (which is Go):

Figure 2.1 – The initial Go discussion email thread

Though they initially thought of C/C++ to be the starting point, they ended up starting from scratch to define a more expressive language, despite a large number of simplifications when compared to its predecessors. Go inherits some things from C, such as, but not limited to, basic data types, expression syntax, pointers, and compilation to machine code, but it doesn’t have things such as the following:

Header filesExceptionsPointer arithmeticSubtype inheritance (no subclasses)thisin methodsPromotion to a superclass (it uses embedding instead)Circular dependencies

Pascal, Oberon, and Newsqueak are among the programming languages that have influenced Go. In particular, its concurrency model comes from Tony Hoare’s Communicating Sequential Processes (CSPs) (Further reading) white paper, and CSP’s implementations in Rob Pike’s interpreted language Newsqueak and, later, Phil Winterbottom’s C-like compiled version, Alef. The next figure shows Go’s family tree:

Figure 2.2 – The Go ancestors

The number of C++ programmers that come to Go is just a few compared to what the Go founders expected. Most Go programmers actually come from languages such as Python and Ruby.