The Linux DevOps Handbook E-Book

Damian Wojsław

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Veröffentlichungsjahr: 2023

Beschreibung

The Linux DevOps Handbook is a comprehensive resource that caters to both novice and experienced professionals, ensuring a strong foundation in Linux. This book will help you understand how Linux serves as a cornerstone of DevOps, offering the flexibility, stability, and scalability essential for modern software development and operations.
You’ll begin by covering Linux distributions, intermediate Linux concepts, and shell scripting to get to grips with automating tasks and streamlining workflows. You’ll then progress to mastering essential day-to-day tools for DevOps tasks. As you learn networking in Linux, you’ll be equipped with connection establishment and troubleshooting skills. You’ll also learn how to use Git for collaboration and efficient code management.
The book guides you through Docker concepts for optimizing your DevOps workflows and moves on to advanced DevOps practices, such as monitoring, tracing, and distributed logging. You’ll work with Terraform and GitHub to implement continuous integration (CI)/continuous deployment (CD) pipelines and employ Atlantis for automated software delivery. Additionally, you’ll identify common DevOps pitfalls and strategies to avoid them.
By the end of this book, you’ll have built a solid foundation in Linux fundamentals, practical tools, and advanced practices, all contributing to your enhanced Linux skills and successful DevOps implementation.

Leseprobe

The Linux DevOps Handbook

Customize and scale your Linux distributions to accelerate your DevOps workflow

Damian Wojsław

Grzegorz Adamowicz

BIRMINGHAM—MUMBAI

The Linux DevOps Handbook

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Preet Ahuja

Publishing Product Manager: Preet Ahuja

Senior Editor: Runcil Rebello

Technical Editor: Nithik Cheruvakodan

Copy Editor: Safis Editing

Project Coordinator: Ashwin Kharwa

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Ponraj Dhandapani

Marketing Coordinator: Rohan Dobhal

First published: November 2023

Production reference: 1171023

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-80324-566-9

www.packtpub.com

To my wife, my kids, and all the people who have helped me on my journey to become the best version of me.

– Damian Wojsław

To my mum and dad, who supported me with all my interests, including computers. To my wife and daughter, who are my biggest supporters and motivation in all endeavors.

– Grzegorz Adamowicz

Contributors

About the authors

Damian Wojsław has been working in the IT industry since 2001. He specializes in the administration and troubleshooting of Linux servers. Being a system operator and support engineer, he has found the DevOps philosophy to be a natural evolution of the way SysOps work with developers and other members of the software team.

I would like to thank my wife and kids for their patience and support.

Grzegorz Adamowicz has been working in the IT industry since 2006 in a number of positions, including systems administrator, backend developer (PHP and Python), systems architect, and site reliability engineer. Professionally, he is focused on building tools and automations for the projects he is involved in. He also engages with the professional community by organizing events such as conferences and workshops. Grzegorz has worked in many industries, including oil and gas, hotel, Fintech, DeFI, automotive, and space.

I want to thank my wife, Agnieszka, for her never-ending support, and everyone I’ve had the pleasure of working with throughout my career – most of what I know professionally, I’ve learned from you.

About the reviewer

Marcin Juszkiewicz is a seasoned software engineer on a journey since 2004. Over the years, he has worked on building and optimizing code in various projects. His career has spanned from embedded Linux systems to ARM servers. In his spare time, Marcin dons the hat of a system admin, a role he embraces with the same enthusiasm as his daily job. Through his extensive journey as a software engineer and part-time system administrator, Marcin has garnered a wealth of knowledge and a reputation for his meticulous approach to coding and system optimization.

Table of Contents

Preface

Part 1: Linux Basics

Choosing the Right Linux Distribution

What makes distributions different?

Debian

Ubuntu Linux

Red Hat Enterprise Linux (RHEL)

Fedora Linux

CentOS

Rocky Linux

Alpine

Command-Line Basics

Getting to know your shell

I call to thee

The filesystem structure

Running a program

The command to teach you all commands

Know your environment

Where in the PATH am I?

Process this

Looking for something?

Let’s be manipulative

Intermediate Linux

Cron jobs

Systemd timer

Debian and Ubuntu

CentOS, RHEL, and Fedora

Alpine Linux

Adding users

Modifying users

Removing users

Managing groups

Configuring OpenSSH

Creating and managing SSH keys

Automating with Shell Scripts

Variables

Parameters

Loops

Conditional execution – if statement

Part 2: Your Day-to-Day DevOps Tools

Managing Services in Linux

The history of Linux service management

systemd

OpenRC

SysV init

Networking in Linux

Physical layer

Data link layer – MAC, VLAN

Network layer – IPv4 and IPv6

Transport layer – TCP and UDP

Session layer

Presentation layer – SSL and TLS

Application layer – HTTP and FTP

iptables

nftables

ufw

firewalld

NAT

Port forwarding

Interface bonding

TCP multipath

BGP

Git, Your Doorway to DevOps

Configuring the local Git environment

Setting up a local Git repository

Interacting with remote repositories

What’s the git diff command?

Viewing the commit history

Branching

Squashing commits using an interactive rebase

Solving Git conflicts

Global git configuration – .gitconfig

Ignoring some files using a .gitignore configuration file

Docker Basics

Virtualization

Containerization

OverlayFS

What is an image?

What is a container runtime?

cgroups

docker build

docker run

docker start

docker stop

docker ps

docker login

docker pull

docker push

docker image

docker exec

docker logs

docker rm

docker rmi

docker network

docker volume

FROM

COPY and ADD

EXPOSE

ENTRYPOINT and CMD

RUN

LABEL

ENV and ARG

VOLUME

USER

WORKDIR

None network

Bridge mode

Host mode

Overlay

A Deep Dive into Docker

Running public images

Running a debugging container

Cleaning up unused containers

Docker volumes and bind mounts

Docker networking advanced use cases

Security features of Docker

Docker for CI/CD pipeline integration

Multi-stage build

ADD command use cases

Secrets management

Docker Swarm

Kubernetes and OpenShift

Part 3: DevOps Cloud Toolkit

Monitoring, Tracing, and Distributed Logging

CloudWatch Logs and metrics

AWS X-Ray

Prometheus

Grafana

SigNoz

New Relic Pixie

Graylog

Sentry

Datadog

New Relic

Ruxit

Splunk

Full retention

Time-based retention

Event-based retention

Selective retention

Tiered retention

Using Ansible for Configuration as Code

SaltStack

Chef

Puppet

CFEngine

Basics of using Ansible

Tasks

Roles

Plays and playbooks

Ansible Vault

SOPS

Other solutions

Debugging

Linting Ansible playbooks

Speeding up SSH connections

Dynamic inventory

Leveraging Infrastructure as Code

AWS CloudFormation

AWS Cloud Development Kit

Terraform

Cloud Development Kit for Terraform

Pulumi

Terraform modules

Terraform state

Terraform CLI

Variables

Comments

Terraform meta-arguments

EC2 instance module

CI/CD with Terraform, GitHub, and Atlantis

An example of CI/CD pipelines

Integration testing

Deployment

Deploying Atlantis to AWS

Running Terraform using Atlantis

Avoiding Pitfalls in DevOps

Repetitive tasks

Time-consuming tasks

Manual and error-prone tasks

Tasks with version control integration

Tasks with repeatable patterns

Tasks with well-defined APIs or interfaces

Tasks with clear and well-defined requirements

Stay curious and proactive

Start with the basics

Hands-on learning

Collaborate and share knowledge

Stay updated

Lack of leadership

Siloed organizational structure

Lack of trust and communication

Tool-centric approach

Increased software defects

Deployment failures

Security vulnerabilities

Lack of documentation

Inadequate test coverage

Lack of continuous improvement

Real-time

Comprehensive

Scalable

Actionable

Continuously improving

What are security measures?

Characteristics of good security measures

DevOps is only for small teams or projects

Inability to scale infrastructure

Flexibility compromises stability

Lack of flexibility in release management

Technical documentation

API documentation

User documentation

Process documentation

Operational documentation

Release notes and changelogs

Index

Other Books You May Enjoy

Preface

DevOps has become a critical component of modern software development and delivery. It has revolutionized the way we build, test, deploy, and operate software systems. DevOps is not just a set of tools and practices but also a culture and mindset that focuses on collaboration, communication, and automation.

This book is designed to be a comprehensive guide to DevOps, covering everything from choosing the right Linux distribution to avoiding pitfalls in DevOps. Each chapter in this book provides detailed information and practical examples to help you understand the concepts and apply them to real-world scenarios.

Who this book is for

This book is designed for individuals who have already gained some knowledge and experience in the field of software development and IT operations and are now seeking to further expand their knowledge of DevOps and Linux systems.

If you are not well versed in Linux systems, this book will provide you with the necessary guidance and tools to quickly learn and become proficient in managing Linux-based infrastructures. You will gain an understanding of the Linux operating system, its architecture, and its fundamental concepts.

Furthermore, this book emphasizes learning about public cloud technologies with a focus on AWS. If you are interested in learning how to use AWS to build and manage scalable and reliable systems, this book will provide you with the necessary knowledge and tools to get started.

Whether you are new to DevOps or have already gained some experience, this book provides a solid foundation for learning more complex concepts. It covers a wide range of topics, from the basics of Linux systems to more advanced DevOps practices such as configuration and infrastructure as code and CI/CD.

What this book covers

Chapter 1, Choosing the Right Linux Distribution, discusses the GNU/Linux history, and the differences between popular distributions.

Chapter 2, Command-Line Basics, guides you through the usage of a command line and common tools we’ll be using throughout the book.

Chapter 3, Intermediate Linux, describes more advanced features of GNU/Linux that you will find useful.

Chapter 4, Automating with Shell Scripts explains how to start writing your own scripts using the Bash shell.

Chapter 5, Managing Services in Linux, discusses different ways of managing services in Linux and shows you how to define your own services using systemd.

Chapter 6, Networking in Linux, describes how networking works, how to control different aspects of network configurations, and how to use command-line tools.

Chapter 7, Git, Your Doorway to DevOps, discusses what Git is and how to use Git’s version control system, including less commonly known Git features.

Chapter 8, Docker Basics, explores the containerization of your services or tools, and running and managing containers.

Chapter 9, A Deep Dive into Docker, discusses the more advanced features of Docker, including Docker Compose and Docker Swarm.

Chapter 10, Monitoring, Tracing, and Distributed Logging, discusses how to monitor your services, what tools you can use in the cloud, and how to do a basic setup.

Chapter 11, Using Ansible for Configuration as Code, looks at Configuration as Code with the use of Ansible; it’ll guide you through the basic setup and more advanced features of Ansible.

Chapter 12, Leveraging Infrastructure as Code, discusses what Infrastructure as Code (IaC) is, what the popular tools are, and how to manage your infrastructure using Terraform.

Chapter 13, CI/CD with Terraform, GitHub, and Atlantis, takes IaC one step further with the use of the Continuous Integration (CI) and Continuous Deployment (CD) of infrastructure with the use of Terraform and Atlantis.

Chapter 14, Avoiding Pitfalls in DevOps, discusses challenges you may encounter with your work in DevOps.

To get the most out of this book

You will need Debian Linux or Ubuntu Linux installed on a virtual machine or as your main operating system on your computer. Other software we use is either already installed as a default toolset, or we will show you where to get it to install it on your system.

It is assumed that you have some basic knowledge of Linux and its command-line interface. Familiarity with shell scripting and basic programming concepts would also be helpful. Additionally, some understanding of IT infrastructure and how it is managed is recommended, as well as some exposure to software development practices.

This book is aimed at beginners in the DevOps world, and it assumes that you are eager to learn about the tools and concepts that are commonly used in this field. By the end of this book, you will have gained a solid understanding of how to manage infrastructure using IaC tools such as Terraform and Atlantis, as well as how to automate repetitive tasks using Ansible and Bash scripting. You will also learn how to set up logging and monitoring solutions to help you maintain and troubleshoot your infrastructure.

Software/hardware covered in the book

Operating system requirements

Bash

Linux OS has it preinstalled

Ansible

Python 3 or newer

Terraform

Linux OS

AWS CLI

Python 3 or newer

Docker

Linux OS

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/The-Linux-DevOps-Handbook. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “While logged in as root, your prompt will end with the # sign. When logged in as a normal user, it will present you with $.”

A block of code is set as follows:

docker build [OPTIONS] PATH | URL | -

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

docker build [OPTIONS] PATH | URL | -

Any command-line input or output is written as follows:

chmod ug=rx testfile

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Ansible Galaxy is a community-driven platform that hosts an extensive collection of Ansible roles and playbooks.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read The Linux DevOps Handbook, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781803245669

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Linux Basics

In this opening part of the book, we will focus on Linux distributions and the basic skills you will need to efficiently use Linux operating systems. You will also learn about the basics of shell scripting to automate day-to-day tasks.

This part has the following chapters:

Chapter 1, Choosing the Right Linux DistributionChapter 2, Command-Line BasicsChapter 3, Intermediate LinuxChapter 4, Automating with Shell Scripts

1

Choosing the Right Linux Distribution

In this chapter, we will dive into the Linux world from the very beginning. We will briefly touch on Linux history, explain what a distribution is, and explain what to take into account when choosing one for production use. You are not expected to know anything about Linux, its administration, or the cloud. If you don’t understand some words that we use, worry not. There shouldn’t be a lot of confusing terminology in this chapter, and if there is, we will explain it in later chapters. When you finish reading this chapter, you should be able to understand why there are so many Linuxes out there, how much you should expect to pay for it, and how to think about choosing the right Linux for yourself.

In this chapter, we will cover the following main topics:

What is Linux and what is a Linux distribution?What can you use to help you make the right decision?Several major Linux distributions that are quite popular today

Technical requirements

This chapter doesn’t have any technical requirements. We won’t run any commands or install any software yet. This will come in the later chapters.

The code we’re presenting in the book is available in the public GitHub repository for your consideration at this address: https://github.com/PacktPublishing/The-Linux-DevOps-Handbook.

What exactly is a Linux distribution?

Linux is the standard operating system for cloud workloads. However, there is not a single Linux operating system that goes by that name. Its ecosystem is quite complex. This comes from how it came to be originally.

Long before Linux was conceived by its creator, Linus Torvalds, there was Unix. Unix source code was – for legal reasons – licensed to anyone who bought it, thus making it very popular among many institutions. This included universities. The code, however, was not entirely free. This didn’t sit well with many people, who believed that software should be free – as in speech or beer – including source code. In the 1980s, a completely free and open implementation of Unix was born under the aegis of the GNU Project. Its goal was to develop an operating system that gave complete control over a computer to the user. The project was successful in that it was able to produce all the software required to run an operating system, except one thing – the kernel.

A kernel of the operating system is, in short, the core that operates hardware and manages the hardware and programs for the user.

In 1991, Finnish student Linus Torvalds famously announced his hobby kernel – Linux. He called it at that time “just a hobby – won’t be big and professional like GNU.” It wasn’t supposed to get big and popular. The rest is history. The Linux kernel became popular among developers and administrators and became the missing piece of the GNU Project. Linux is the kernel, the GNU tools are the so-called userland, and together they make the GNU/Linux operating system.

The preceding short story is important to us for two reasons:

While the GNU userland and the Linux kernel is the most popular combination, you’ll see it is not the only one.Linux delivers a kernel and the GNU Project delivers userland tools, but they have to be somehow prepared for installation. Many people and teams had separate ideas on how to do it best. I will expand on this thought next.

The way that a team or a company delivers a GNU/Linux operating system to end users is called a distribution. It facilitates operating system installation, the means to manage software later on, and general notions on how an operating system and the running processes have to be managed.

What makes distributions different?

The open nature of Linux and the GNU Project made it possible for almost anyone to create their own distribution. One of the things that made new users dizzy was the sheer amount of Operating System (OS) versions they could use. The surefire way to start a holy war between Linux users is by asking which distribution is the best.

One of the ways we can group Linux distributions is the format in which they deliver the software (packages) and additional software used to install and remove that software (package managers). There are a number of them, but the two most prevalent are RPM (RPM Package Manager) and DEB packages. Packages are more than just an archive with binaries. They contain scripts that set the software up for use – creating directories, users, permissions, log rules, and a number of other things that we will explain in later chapters.

The RPM family of distributions starts with Red Hat Enterprise Linux (RHEL), created and maintained by the Red Hat company. Closely related is Fedora (a free community distribution sponsored by Red Hat). It also includes CentOS Linux (a free version of RHEL) and Rocky Linux (another free version of RHEL).

The DEB distributions include Debian (where the DEB packages originate from) – a technocracy community project. From the Debian distribution arose a number of distributions based on it, using most of its core components. Most notable is Ubuntu, a server and desktop distribution sponsored by Canonical.

There are also distributions that use packages with minimum automation, most notably Slackware, one of the oldest existing Linux distributions.

There are distributions that give the user a set of scripts that compile actual software on the hardware it will be used on – most notably, Gentoo.

Finally, there is a distribution that is actually a book with a set of instructions that a user can follow to build the whole OS by hand – the Linux From Scratch project.

Another way of grouping distributions is by their acceptance of closed software – software that limits the distribution of source code, binaries, or both. This can mean hardware drivers, such as the ones for NVIDIA graphic cards, and user software, such as movie codecs that allow you to play streamed media and DVD and Blu-Ray discs. Some distributions make it easy to install and use them, while some make it more difficult, arguing that we should strive for all software to be open source and free (as in speech and as in beer).

Yet another way to differentiate them is the security framework a given distribution uses. Two of the most notable ones are AppArmor, used mainly by Ubuntu, and SELinux (from the USA’s National Security Agency), used by, among others, Red Hat Enterprise Linux (and its derivatives) and Fedora Linux.

It’s also worth noting that while most Linux distributions use the GNU Project as the userland, the popular one in the cloud Alpine Linux distribution uses its own set of software, written especially with minimum size in mind.

Looking at how the distribution is developed, it can be community-driven (without any commercial entity being an owner of any part of the process and software – Debian being one prime example), commercial (wholly owned by a company – RHEL being one example and SuSE another), and all the mixes in between (Ubuntu and Fedora being examples of a commercially owned distribution with a large body of independent contributors).

Finally, a way we can group distributions is by how well they facilitate the cloud workload. Here, we can look at different aspects:

The server side: How well a given distribution works as an underlying OS for our infrastructure, virtual machines, and containersThe service side: How well a given distribution is suited to run our software as a container or a virtual machine

To make things even more confusing and amusing for new adopters, each distribution can have many variants (called flavors, variants, or spins, depending on the distribution lingo) that offer different sets of software or default configurations.

And to finally confuse you, dear reader, for use on a desktop or laptop, Linux offers the best it can give you – a choice. The number of graphical interfaces for the Linux OS can spin the head of even the most experienced user – KDE Plasma, GNOME, Cinnamon Desktop, MATE, Unity Desktop (not related to the Unity 3D game engine), and Xfce. The list is non-exhaustive, subjective, and very limited. They all differ in the ease of use, configurability, the amount of memory and CPU they use, and many other aspects.

The number of distributions is staggering – the go-to site that tracks Linux distributions (https://distrowatch.com/dwres.php?resource=popularity) lists 265 various Linux distributions on its distribution popularity page at the time of writing. The sheer number of them makes it necessary to limit the book to three of our choosing. For the most part, it doesn’t make a difference which one you choose for yourself, except maybe in licensing and subscription if you choose a commercial one. Each time the choice of distribution makes a difference, especially a technical one, we will point it out.

Choosing a distribution is more than just a pragmatic choice. The Linux community is deeply driven by ideals. For some people, they are the most important ideals on which they build their lives. Harsh words have been traded countless times over which text editor is better, based on their user interface, the license they are published with, or the quality of the source code. The same level of emotion is displayed toward the choice of software to run the WWW server or how to accept new contributions. This will inevitably lead to the way the Linux distribution is installed, what tools there are for configuration and maintenance, and how big the selection of software installed on it out of the box is.

Having said that, we have to mention that even though they have strong beliefs, the open-source community, the Linux community included, is a friendly bunch. In most cases, you’ll be able to find help or advice on online forums, and the chances are quite high that you will be able to meet them in person.

To choose your distribution, you need to pay attention to several factors:

Is the software you wish to run supported on the distribution? Some commercial software limits the number of distributions it publishes packages for. It may be possible to run them on unsupported versions of Linux, but this may be tricky and prone to disruptions.Which versions of the software you intend to run are available? Sometimes, the distribution of choice doesn’t update your desired packages often enough. In the world of the cloud, software a few months old may already be outdated and lack important features or security updates.What is the licensing for the distribution? Is it free to use or does it require a subscription plan?What are your support options? For community-driven free distributions, your options are limited to friendly Linux gurus online and in the vicinity. For commercial offerings, you can pay for various support offerings. Depending on your needs and budget, you can find a mix of support options that will suit your requirements and financial reserves.What is your level of comfort with editing configuration files and running long and complex commands? Some distributions offer tools (both command-line and graphical) that make the configuration tasks easier and less error-prone. However, those tools are mostly distribution-specific, and you won’t find them anywhere else.How well are cloud-related tools supported on a given distribution? This can be the ease of installation, the recency of the software itself, or the number of steps to configure for use.How well is this distribution supported by the cloud of your choosing? This will mean how many cloud operators offer virtual machines with this distribution. How easy is it to obtain a container image with this distribution to run your software in it? How easy do we suspect it to be to build for this distribution and deploy on it?How well is it documented on the internet? This will not only include the documentation written by distribution maintainers but also various blog posts and documentation (mainly tutorials and so-called how-to documents) written by its users.

So far, you’ve learned what a Linux distribution is, how distributions differentiate from one another, and what criteria you can use to actually choose one as the core of the system you will manage.

In the next section, we will look deeper into each distribution to get to know the most popular ones better, giving you a first glimpse of how each one works and what to expect.

Introducing the distributions

After that bit of a lengthy but condensed history of the Linux OS, it is time to finally explore the few we have chosen to cover in this book. In this section, we will cover the factors we just listed, as we believe they are important in making a decision. Please remember though that while we strive to present you with objective facts and valuations, we cannot escape our own subjective views. Always evaluate on your own before you choose, as it’s highly possible that you will stick with this distribution for many years to come.

A point to note is that we won’t be covering distributions comprehensively. We will only try to create a foundation on which you, dear reader, must build through research.

Also, while you are learning, do not be afraid to hop from distribution to distribution. Only through real-life experiences will you fully understand which one covers your needs best.

Debian

Debian (https://www.debian.org/) is one of the oldest active Linux distributions. Its development is led by the community-supported Debian Project. It is known for two things – the sheer number of packages that the distribution provides and the slow release of stable versions. The latter has improved in recent years and stable releases are published every two years. Software is delivered in archives called packages. Debian packages’ names have a .deb file extension and are colloquially called debs. They are kept online in repositories and repositories are broken down into pools. Repositories offer almost 60,000 packages with software in the latest stable release.

Debian always has three versions available (so-called branches) – stable, testing, and unstable. The releases are named after characters from the Toy Story movie franchise. The latest stable release – version 11 – is called Bullseye.

The unstable branch is the rolling branch for developers, people who like living on the edge, or those who require the newest software more than they require stability. Software is accepted into the unstable branch with minimal testing.

The testing branch is where, as the name implies, the testing happens. A lot of testing happens here, thanks to the end users. Packages come here from the unstable branch. The software here is still newer than in the stable branch but not as fresh as in the unstable branch. A few months before the new stable release, the testing branch is frozen. It means that no new software will be accepted, and new versions of the already accepted packages are allowed only if they fix bugs.

After a few months, testing becomes the stable branch. The software is updated only for security fixes.

This distribution is available for many hardware platforms – Intel, ARM, PowerPC, and so on. Along with unofficial ports, there are a multitude of hardware platforms on which you can install it.

Debian is viewed as the most stable distribution there is, and it is used as a platform for various compute clusters, so it is generally installed on bare-metal servers somewhere in a rack in a data center and intended for use consistently over many years.

According to W3Techs (https://w3techs.com/technologies/details/os-linux), Debian makes up for 16% of all servers running on the internet. Its derivative, Ubuntu, runs 33% of them. Together, they account for 49% of all servers. This makes administration skills related to Debian highly marketable.

Ubuntu Linux

The Ubuntu Linux distribution (https://ubuntu.com/) is widely credited for making Linux popular on personal computers, and rightly so. Sponsored by Canonical, its mission was to make Linux easily usable for most people. It was one of the first, if not the first, Linux versions to distribute non-free and non-open binary drivers and libraries that made desktop use simpler and more comfortable.

Famously, the first bug report opened for Ubuntu distribution by Mark Shuttleworth (Canonical and Ubuntu founder) was, “Microsoft has majority market share.”

The distribution itself is based on Debian Linux, and in the beginning, being fully binary-compatible was one of the major objectives. As the development has progressed, this has lost some of its importance.

This distribution is developed by the community and Canonical. The main source of income for the company is premium services related to Ubuntu Linux – support, training, and consultations.

Due to the very close-knit relationship between Debian Linux and Ubuntu Linux, many developers and maintainers for one distribution serve the same roles in the other one. This results in a lot of software being packaged for both distributions in parallel.

Ubuntu has three major flavors – Desktop, Server, and Core (for the internet of things). Desktop and Server may differ slightly in how services are configured out of the box, and Core differs a lot

The software is distributed in .deb packages, the same as with Debian, and the sources are actually imported from the Debian unstable branch. However, this doesn’t mean you can install Debian packages on Ubuntu or vice versa, as they are not necessarily binary-compatible. It should be possible to rebuild and install your own version.

There are four package repositories per release – the free and non-free software supported officially by Canonical is called main and restricted, respectively. Free and non-free software delivered and maintained by the community is called universe and multiverse, respectively.

Important note

A word of advice – a widely accepted practice of system upgrades between major versions is to wait for the first sub-release. So, if the currently installed version of the distribution is 2.5 and the new version 3.0 is released, it is wise to wait until 3.1 or even 3.2 is released and upgrade then. This is applicable to all the distributions we list here.

The Long-Term Support (LTS) versions are supported for five years. A new LTS version is released every two years. It is possible to negotiate extended support. This gives a very good timeline to plan major upgrades. A new Ubuntu version is released every six months.

Ubuntu Linux is widely adopted in education and government projects. Famously, the city of Munich, between 2004 and 2013, migrated over 14,000 municipal desktop computers to a variant of Ubuntu with the KDE desktop environment. While the migration saw disturbances politically – other operating system vendors lobbied strongly against this migration – it was considered a success technically.

Ubuntu is the Linux of choice for personal computers. Canonical works very closely with hardware vendors, notably Lenovo and Dell, but lately also with HP, to ensure full compatibility between the distribution and the computers. Dell sells its flagship laptops with Ubuntu preinstalled.

Several sources cite Ubuntu Linux as the most installed Linux distribution on servers and personal computers. The actual number can only be estimated, as Ubuntu doesn’t require any subscription or registration.

As a byproduct of Ubuntu Linux’s popularity, software vendors, more often than not, offer .deb packages of their software, if they release a Linux version. This is especially true for desktop software.

The amount of unofficial versions, clones, or modified distributions based on Ubuntu is staggering.

Ubuntu has a very active community, both organized and unorganized. It’s quite easy to get a hold of a group of users near your city. This also directly translates to the amount of tutorials and documentation on the internet.

Ubuntu Linux, especially under a support plan, is installed as a foundation for many cloud computing infrastructure deployments. Many telecoms, banking, and insurance companies have chosen Ubuntu Server as their foundation.

Red Hat Enterprise Linux (RHEL)

RHEL (https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux) is a spiritual successor of Red Hat Linux and is developed and maintained by Red Hat Inc. (https://www.redhat.com/). Its main target is the commercial entities market. It is possible to use RHEL for free for development or in production with up to 16 servers (at the time of writing). However, the main advantage of this distribution is the enormous pool of articles that help solve issues and the assistance of support engineers, although the latter can only be acquired through a paid support plan.

RHEL is considered a very stable and solid distribution. It is one of the main choices for banks, insurance companies, and financial markets. It lacks many popular desktop software packages, but on the server side of things, especially as an OS to run other commercial applications, it is a first-class citizen.

The software is distributed in online repository packages that end with .rpm, hence the name RPMs. The main tool to administer the packages is RPM, with more sophisticated tools – yum, and lately its successor, dnf – also available.

In the true spirit of an open source-based company, Red Hat makes sources for its distribution available. This has led to the creation of a famous free and open clone of RHEL – CentOS. Until fairly recently, it had been quite a popular choice for people who wanted to use RHEL but didn’t want to, or couldn’t, pay a subscription. In 2014, CentOS joined the Red Hat company, and in 2020, Red Hat announced that the versioned releases of CentOS would no longer be available; there would only be the so-called rolling release, which constantly updates packages and does not mirror the RHEL releases. This resulted in a very heated reaction from CentOS users. The original CentOS founder, Gregory Kurtzer, started another clone of RHEL called Rocky Linux. Its main objective is the same as the original CentOS – to deliver a free, open, and community-driven distribution, fully binary-compatible with RHEL.

The RHEL distribution delivers stable versions every few years and supports them for 10 years, starting from release 5. The full support, however, is offered only for a few years. For the rest of the time, Red Hat provides only security fixes and critical updates for your systems, with no new package versions being introduced. Still, this life cycle is what users with large installations or mission-critical systems came to like.

As with Ubuntu, it is possible to negotiate extended support time.

The Red Hat company has a turbulent relationship with the open source community. While the company mostly plays fair, there have been some decisions that the community didn’t like. Lately, it was Red Hat’s decision to change the CentOS release model to a rolling release (https://lists.centos.org/pipermail/centos-announce/2020-December/048208.html).

RHEL, like Ubuntu, is the chosen foundation for commercially supported deployments of cloud infrastructure.

Fedora Linux

Fedora (https://fedoraproject.org/wiki/Fedora_Project_Wiki) is a distribution associated with the Red Hat company. While more than 50% of its developers and maintainers are community members not affiliated with Red Hat, the company holds full stewardship over the development. It is a RHEL upstream, which means that this is the real development frontend for the actual RHEL. It doesn’t mean that everything from Fedora is included in the release of RHEL. However, following Fedora closely will yield insight into the current direction of the RHEL distribution.

Contrary to RHEL, for which Fedora is the foundation, the new releases happen every six months. It uses the same package type as RHEL, RPM.

Fedora is considered a fast-paced distribution. It quickly adopts the newest and bleeding-edge versions of packages.

CentOS

CentOS (https://centos.org) used to be the go-to free version of RHEL. The name is an acronym for Community Enterprise Operating System. Its main goal was to be fully binary-compatible with RHEL and adhere to the same releases and numbering scheme. In 2014, CentOS joined Red Hat, but it was promised that the distribution would keep its independence from the company while benefiting from development and testing resources. Unfortunately, in 2020, Red Hat announced that CentOS 8 would be the last numbered release, and from then on, CentOS Stream would be the only variant. CentOS Stream is a midstream version. This means it sits in the middle between bleeding-edge and fast-paced Fedora and stable and production-ready RHEL. The difference between CentOS Stream and CentOS is that Stream is a development variant, while CentOS was simply a rebuilt and repackaged mirror of the actual final product, RHEL.

All the knowledge, skills, and experience gained when working with RHEL are 100% applicable to CentOS. Since CentOS is the third most-deployed Linux distribution on servers, according to W3Techs (https://w3techs.com/technologies/details/os-linux), the skills are very marketable.

Rocky Linux

As a response to the situation with the CentOS distribution, its founder announced the creation of Rocky Linux (https://rockylinux.org/). The goals are the same as the original CentOS. The release scheme and numbering follow RHEL. Shortly after the announcement, the GitHub repository of Rocky Linux became top trending (https://web.archive.org/web/20201212220049/https://github.com/trending). Rocky Linux is 100% binary-compatible with CentOS. The project has released a set of tools that easily migrate from CentOS to Rocky Linux without reinstalling the system.

The distribution is quite young, having been founded in 2020, and its popularity is still to be determined. It has made a lot of noise in the community, and it seems that a steady stream of CentOS users have moved to Rocky Linux as their preferred choice.

A very important contribution to the open source world from the Rocky Linux project is the build system. It ensures that even if Rocky Linux shuts down, the community will be able to easily start up a new RHEL clone.

All the skills, knowledge, and articles for RHEL and CentOS are 100% applicable to Rocky Linux. All the software that runs on RHEL and CentOS should run without any modifications on Rocky Linux too.

Alpine

Alpine Linux (https://alpinelinux.org/) is an interesting one. The main programming library and most basic command-line tools are not from the GNU Project. Also, the services management system, currently systemd in most distributions, is uncommon. This makes some of the administration skills from other major distributions non-applicable. The strength of Alpine lies in its size (which is rather small), its security-first mindset, and one of the fastest boot times among existing Linux distributions. Those characteristics, with the boot time being admittedly more important, make it the most popular choice for containers. If you run containerized software or build your own container images, it is very likely that it is on Alpine Linux.

Alpine has its roots in the LEAF (Linux Embedded Appliance Framework; see: https://bering-uclibc.zetam.org/wiki/Main_Page) project – a Linux distribution that fits on a single floppy disk. LEAF is currently a popular choice for embedded markets, routers, and firewalls. Alpine is a bigger distribution, but that sacrifice had to be made, since developers wanted to include several useful but rather large software packages.

The package manager is called apk. The build system is borrowed from another distribution called Gentoo Linux. As Gentoo builds software as it installs it, the portage obviously contains a lot of logic around building software that is used as a part of an OS.

Alpine can run from RAM entirely. There’s even a special mechanism that allows you to initially only load a few required packages from the boot device, and it can be achieved using Alpine’s Local Backup Utility (LBU).

As mentioned before, this is a preferred distribution for container images. You won’t see it running on a large server installation often, if at all. When we cross over to the cloud world, chances are you’ll see a lot of Alpine Linux.

Having said that, every single one of those distributions has a variant for the cloud as a container base image – a way to run your software in the true cloud way.

In this chapter, you learned the basics of popular Linux distributions and how they are different from one another. You should now have some understanding of what you can choose from and what consequences you will need to face – good and bad. To give you an even better idea of how to interact with some cherry-picked Linux distributions, we will look at how to interact with a system using your keyboard in Chapter 2.

Summary

The short list in this chapter is just a tiny portion of the Linux distributions available. The list is largely based on the popularity and marketability of skills, as well as our own experience and knowledge that we acquired over the years. They are not, by any means, the best or only choices that you have.

We tried to point out where the main strengths lie and what a user’s relationship is with respective distributions.

It’s not likely we were able to answer all your questions. Each of the Linux distributions from our list has its own book out there, and there is even more knowledge on blogs, wikis, and YouTube tutorials.

In the next chapter, we will dive into the magical world of the command line.

2

Command-Line Basics

In this chapter, we’re going to dive right into the Linux command line. We will explain what makes it so powerful and, by extension, important to every system administrator and DevOps person. But more importantly, we will start teaching you the most useful commands and a way to use them efficiently. Along the way, we will be adding other core Linux concepts, as they will be required to understand the chapter.

In this chapter, we’re going to cover the following main topics:

What a command line is and how it worksWhy it is so important to feel comfortable working with the command lineBasic commands for Linux system administration

It is not possible to introduce all the commands and tools in a single chapter. What follows is our choice of the most basic tools you will need to know. Managing the Linux system is a separate book topic on its own. It so happens that Packt does have several publications on that.

Technical requirements

It is highly recommended to have a Linux system installed and ready for use. We recommend it to be a virtual machine or a laptop that you can safely reinstall from scratch in case something goes horribly wrong. It will let you follow the examples in the book and perform any kind of exercise that we give you.

We are not going to cover an installation. Every distribution may use its own installer, either graphical or text (depending on the distribution and which variant you’ve picked). You’ll need to note down or remember the name of your user (conveniently called username or login) and password. There are ways to get into the system if you have physical access and you don’t know either the login or password, or both, but they are way outside of the scope of this book.

Our main distribution in the book is Debian. However, you should be alright with any of the major ones we covered in the previous chapter, Choosing the Right Linux Distribution, as long as it isn’t Alpine.

The Linux command line – shell

The natural environment for a Linux system administrator is the command line. You’ll never hear anyone call it that, however. The correct name is the shell, and from now on, that’s how we’re going to address it in the book.

The shell is a program that accepts input from a user (mostly keyboard strokes, but there are other ways, and you can even use a mouse pointer), interprets it, and, if it’s a valid command, executes it, providing the user with the result or with error information if they’ve made a mistake or if the commands couldn’t complete their execution properly.

There are a few ways to access the shell:

Log in to the Terminal (screenshot in Figure 2.1)

Note

You’ll also see the term console. There is a difference between the Terminal and a console. A console is a physical device that lets users interact with the computer. It is the physical input (nowadays, mostly keyboard) and output (nowadays, a monitor, but in the beginning, the output was printed out). Terminal is a console emulator, a program that lets users perform the same tasks.

Open a Terminal window in a graphical interface, if you have oneLog in remotely over a secure connection from another device (phone, tablet, or your computer)

The shell is a very powerful environment. It may seem cumbersome at first to do everything by typing commands, but soon you’ll learn that the more complex the task, the more you can do with the shell, more easily than with graphical interfaces. Every Linux system administrator worth their salt will know how to do tasks in the shell and how to manage the system through it, and I wouldn’t risk much by betting that they will prefer the shell to any GUI.

Getting to know your shell

The shell is a program, and as such, there is not a single shell. Instead, there are a number of more or less popular shells that bring forth their own view on how things should be done.

By far the most popular and default in most Linux distributions is Bash (Bourne again shell). There are other shells you may want to be aware of:

sh: The original Steve Bourne shell. It is the shell, the very first one ever written that we know of. While it lacks many interactive features that users came to appreciate from other, more modern shells, sh is known for its speed of script execution and small size.ksh: Developed as an evolution of the Bourne shell, it is a superset of its predecessor. Thus, all scripts written for sh will run in ksh.csh: The C shell. The name comes from its syntax, which closely follows the C programming language.zsh: The Z shell. It should be well known to macOS users, as it is a default on this operating system. It is a completely modern shell, providing a lot of features that you’d expect from it: command history, arithmetic operations, command completion, and so on.

We won’t trouble you much with shell variants and history. If you are interested in how Bash came to be, refer to this Wikipedia article: https://en.wikipedia.org/wiki/Bash_(Unix_shell).

In our book, we are working with Bash. As mentioned earlier, it is the default shell for most Linux distributions, it offers sh compatibility mode, it has all the features you’d expect from a modern shell, and the amount of books, articles, how-tos, and other material for extending your knowledge about it is staggering.

The first task that we will be performing is logging in to your shell. Depending on your chosen installation method, you may need to power up the local virtual machine, your physical machine, or a cloud-based Virtual Private Server (VPS). If you have picked a server installation without a graphical user interface, you should see something similar to the following screenshot:

Figure 2.1 – Login screen

You are presented with a login screen, where you can provide your username and password. Once successfully logged in, you are presented with a command prompt, which confirms that you’ve just successfully started your shell.

The way the prompt looks is configurable and may be different depending on your chosen distribution. There is, however, one thing that will stay the same, and we advise you to never change it. In the world of Linux, there are two types of users: normal users and the superuser. While the login for normal users can be anything as long as it adheres to the Linux user naming conventions, the superuser is called root. To log in to the root account, you’d type in root as the username/login and its password next.

The superuser account is named that way for a reason. In most Linux distributions, it’s the omnipotent user. Once logged in as root, you can do anything you wish, even deleting all files, including the operating system itself.

To help you distinguish between being a normal user and the root one, the prompt will present you with a clue. While logged in as root, your prompt will end with the # sign. When logged in as a normal user, it will present you with $.

While we’re at it, the # (hash) sign is also a so-called comment sign. If you happen to paste a command or type it from the internet, if it starts with # or $, it is your clue as to which type of user should run this command. You should omit this sign, especially as # in front will prevent the command from running.

In addition to the sign that ends the prompt, many distributions will prepend the username to it, making sure you know which user you are. Given a user admin on a Linux system called myhome, the default prompt for Debian 11 will look like this:

$admin@myhome:~$

For the root user, it would look like this:

root@myhome:~#

For the record, there are more ways to check your identity, but we will leave that forChapter 4.

I call to thee

So far, so good. You have logged in, you know who you are, you can type, and you can read. But how do you actually run programs?

In the Linux lingo, running a program is executing it or calling it. Actually, calling is mostly used when referring to system commands or shell built-in commands, and executing is used when talking about something that is not a part of the distribution—so-called third-party programs or binary.

Before I can tell you how to execute a program or call a command, I’ll have to explain a little bit about filesystem structure and a very important system variablecalled PATH.

The filesystem structure

Since it may be your first time, we are going to step back a little bit and explain how the filesystem is structured (in other words, how directories are organized in typical Linux).

Linux follows the Unix philosophy that states that everything is a file. (There are exceptions, but not many.) The consequence is that almost every aspect of the operating system is reflected either as a file or a directory. Memory states, processes' (running programs) states, logs, binaries, and device drivers all live within this structure. This also means that almost every aspect of your Linux system can be edited or inspected using just normal text editing tools.

In the directory tree, its structure always starts with a / folder, called the root directory. Every drive, network share, and system directory lives in a hierarchy that starts from the root.

The process of making a network share or local drive available to the system or user is called mounting and the resource that is made available is mounted. Remember that word, as we’re going to be using it from now on. Different from Microsoft Windows, where your drives appear as a separate letter with each being its own root directory, in Linux, your hard drive will be mounted somewhere within the directories hierarchy. As an example, the user home directory can be stored on a separate hard drive, mounted under the /home/ directory. You would never be able to tell it when browsing the filesystem structure. The only way to tell would be to inspect mounted drives and partitions using the following commands:

$ mount

or

$ df

We are going to elaborate on them in Chapter 3, so for now, just know they exist.

The name of the uppermost directory is /. We already covered that. The separator between folders nested in another folder is also /. So /usr/bin means a bin directory that exists in the usr directory, and the usr directory exists in the /directory. Simple.

There is a very nice command that lets us inspect the directory structure, conveniently called tree. It may not be present in your system. If so, don’t be worried; it’s not so important that you run it rather than go through our explanation. In Chapter 3, when we introduce the installation of packages, you can revisit and play around. By default, the tree command will flood your screen with the full hierarchy, making it difficult to read and follow. There is an option, however, that lets us limit the depth we will be inspecting:

admin@myhome:~$ tree -L 1 / / |-- bin -> usr/bin |-- boot |-- dev |-- etc |-- home |-- lib -> usr/lib |-- lib32 -> usr/lib32 |-- lib64 -> usr/lib64 |-- libx32 -> usr/libx32 |-- lost+found |-- media |-- mnt |-- opt |-- proc |-- root |-- run |-- sbin -> usr/sbin |-- srv |-- sys |-- tmp |-- usr `-- var 22 directories, 0 files

There are several important concepts to cover here; we won’t be explaining all the directories at this time, however. The first time a given folder becomes important, in Chapter 3 onward, we will briefly touch upon it.

First, the calling of tree. You saw my prompt, which tells me I am running as user admin, on a system named myhome and that I am not a root user (the dollar sign at the end). If you want to run the tree command, you will skip the prompt. Next is the actual call: tree with the -L option and number 1; this instructs the program to only print one level of depth. In other words, it will not go any deeper into the directories. Finally, the / symbol tells the program to start printing at the very beginning of the filesystem—the root folder.

Next, you’ll notice that some rows have this mysterious arrow pointing from one name to another. This denotes a shortcut. There are two types of shortcuts, hard and symbolic. Directories can only have a symbolic link. In the preceding output, the /bin directory is a link to a /usr/bin directory. For all intents and purposes, they can be treated as one. There are technical and historical reasons for the existence of this link. In days past, tools living in /bin and /sbin were used to mount a /usr partition and then allow access to /usr/bin and /usr/sbin. Nowadays, this task is handled earlier in the boot process by other tools and the requirement is no longer necessary. The structure is kept for backward compatibility with tools that may require the existence of both /bin and /sbin directories. More details can be found at https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.htmlor https://www.linuxfoundation.org/blog/blog/classic-sysadmin-the-linux-filesystem-explained.

Since we already touched on the /bin and /sbin directory, let’s explain the difference. The /usr/bin directory contains binaries—in other words, commands that can be of interest to every user in the system. The /usr/sbin directory contains so-called system binaries. Those are commands that should concern only the root user. It will also contain binaries for system processes (called daemons)—programs that run in the background and do important work for the running system.

The /root directory is the home of the superuser. This is where all its configuration files lay.

The interesting one is the /home directory. This is where all the user home directories exist. When I created my admin user for my home machine, it was placed in the /home/admin folder.

Of importance to us at this moment will also be the /etc/