The Software Developer's Guide to Linux E-Book

David Cohen

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 394

Veröffentlichungsjahr: 2024

Beschreibung

Developers are always looking to raise their game to the next level, yet most are completely lost when it comes to the Linux command line.
This book is the bridge that will take you to the next level in your software development career. Most of the skills in the book can be immediately put to work to make you a more efficient developer. It’s written specifically for software engineers, not Linux system administrators, so each chapter will equip you with just enough theory to understand what you’re doing before diving into practical commands that you can use in your day-to-day work as a software developer.
As you work through the book, you’ll quickly absorb the basics of how Linux works while you get comfortable moving around the command line. Once you’ve got the core skills, you’ll see how to apply them in different contexts that you’ll come across as a software developer: building and working with Docker images, automating boring build tasks with shell scripts, and troubleshooting issues in production environments.
By the end of the book, you’ll be able to use Linux and the command line comfortably and apply your newfound skills in your day-to-day work to save time, troubleshoot issues, and be the command-line wizard that your team turns to.

Leseprobe

The Software Developer’s Guide to Linux

A practical, no-nonsense guide to using the Linux command line and utilities as a software developer

David Cohen

Christian Sturm

BIRMINGHAM—MUMBAI

The Software Developer’s Guide to Linux

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Senior Publishing Product Manager: Aaron Tanna

Acquisition Editor – Peer Reviews: Gaurav Gavas

Project Editor: Parvathy Nair

Content Development Editor: Matthew Davies

Copy Editor: Safis Editing

Technical Editor: Karan Sonawane

Proofreader: Safis Editing

Indexer: Tejal Soni

Presentation Designer: Ganesh Bhadwalkar

Developer Relations Marketing Executive: Meghal Patel

First published: January 2024 Production reference: 1230124

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-80461-692-5

www.packt.com

Contributors

About the authors

David Cohen has, for the past 15 years, worked as a Linux system administrator, software engineer, infrastructure engineer, platform engineer, site reliability engineer, security engineer, web developer, and a few other things besides. In his free time, he runs the tutorialinux YouTube channel where he’s taught hundreds of thousands of people the basics of Linux, programming, and DevOps. David has been at Hashicorp since 2019—first as an SRE, then as a reference architect, and now as a software engineer.

Thank you, Aleyna, for your unwavering support over the past few years as I’ve been developing and writing this book. Without you, this would just be another of my promising-but-unfinished projects languishing in some forgotten “Archive” directory. Thanks to Christian, who has stuck with me for over a decade as a friend and a partner on practically every wild tech project idea I’ve come up with since we met. Finally, a big “thank you” is also due to my friends and colleagues at Hashicorp and everywhere else I’ve been over the past 15 years, who have made me a better engineer and encouraged projects like this.

Christian Sturm is a consultant on software and systems architecture, having worked in various technical positions for well over a decade. He has worked as an application developer for the frontend and backend at companies large and small, such as zoomsquare and Plutonium Labs. On top of that, he is also an active contributor to various open source projects and has a deep understanding of fields including operating systems, networking protocols, security, and database management systems.

About the reviewers

Mario Splivalo works as a consultant dealing with databases extended into modern cloud-based architectures. He also helps companies design their infrastructure using IaaC tools such as Terraform and AWS Cloudformation. For five years, Mario worked with Canonical as an OpenStack engineer.

Mario’s fascination with computers started back when Commodore 64 dominated the user space. He took his first steps using BASIC on his dad’s C64, quickly shifting to Assembler. He gradually moved to PCs, finding a great love for programming, systems design, and database administration. He switched to Linux (Knoppix, then Ubuntu, and never looked back) in the early 2000s, continuing as a database administrator, programmer, and system administrator.

Nathan Chancellor is an independent contractor working on the Linux kernel, based in Arizona, US. As a developer, his focus is on improving the compatibility between the Linux kernel and the LLVM toolchain. He has used Linux since 2016 and it has been his primary development operating system since 2018. His distributions of choice are Arch Linux and Fedora.

Learn more on Discord

To join the Discord community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:

https://packt.link/SecNet

Contents

Preface

Who this book is for

What this book is not

What this book covers

To get the most out of this book

Get in touch

How the Command Line Works

In the beginning…was the REPL

Command-line syntax (read)

Command line vs. shell

How does the shell know what to run? (evaluate)

A quick definition of POSIX

Basic command-line skills

Unix filesystem basics

Absolute vs. relative file paths

Absolute vs. relative pathname review

Opening a terminal

Looking around – command-line navigation

pwd - print working directory

ls - list

Moving around

cd – change directory

find – find files

Reading files

less – page through a file

Making changes

touch – create an empty file, or update modification time for an existing one

mkdir – create a directory

rmdir – remove empty directories

rm – remove files and directories

mv – move or rename files and directories

Getting help

Shell autocompletion

Conclusion

Working with Processes

Process basics

What is a Linux process made of?

Process ID (PID)

Effective User ID (EUID) and Effective Group ID (EGID)

Environment variables

Working directory

Practical commands for working with Linux processes

Advanced process concepts and tools

Signals

Practical uses of signals

Trapping

The kill command

lsof – show file handles that a process has open

Inheritance

Review – example troubleshooting session

Conclusion

Service Management with systemd

The basics

init

Processes and services

systemctl commands

Checking the status of a service

Starting a service

Stopping a service

Restarting a service

Reloading a service

Enable and disable

A note on Docker

Conclusion

Using Shell History

Shell history

Shell configuration files

History files

Searching through shell history

Exceptions

Executing previous commands with !

Re-running a command with the same arguments

Prepending a command to something in your history

Jumping to the beginning or end of the current line

Conclusion

Introducing Files

Files on Linux: the absolute basics

Plaintext files

What is a binary file?

Line endings

The filesystem tree

Basic filesystem operations

ls

pwd

cd

touch

less

tail

mv

Moving

Renaming

cp

mkdir

rm

Editing files

File types

Symbolic links

Hard links

The file command

Advanced file operations

Searching file content with grep

Finding files with find

Copying files between local and remote hosts with rsync

Combining find, grep, and rsync

Advanced filesystem knowledge for the real world

FUSE: Even more fun with Unix filesystems

Conclusion

Editing Files on the Command Line

Nano

Installing nano

Nano cheat sheet

File handling

Editing

Search and replace

Vi(m)

Vi/vim commands

Modes

Command mode

Normal mode

Tips for learning vi(m)

Use vimtutor

Think in terms of mnemonics

Avoid using arrow keys

Avoid using the mouse

Don’t use gvim

Avoid starting with extensive configuration or plugins

Vim bindings in other software

Editing a file you don’t have permissions for

Setting your preferred editor

Conclusion

Users and Groups

What is a user?

Root versus everybody else

sudo

What is a group?

Mini project: user and group management

Creating a user

Create a group

Modifying a Linux user

Adding a Linux user to a group

Removing a user from a group

Removing a Linux user

Remove a Linux group

Advanced: what is a user, really?

User metadata / attributes

A note on scriptability

Conclusion

Ownership and Permissions

Deciphering a long listing

File attributes

File type

Permissions

Number of hardlinks

User ownership

Group ownership

File size

Modification time

Filename

Ownership

Permissions

Numeric/octal

Common permissions

Changing ownership (chown) and permissions (chmod)

Chown

Change owner

Change owner and group

Recursively change owner and group

Chmod

Using a reference

Conclusion

Managing Installed Software

Working with software packages

Update your local cache of repository state

Search for a package

Install a package

Upgrade all packages that have available updates

Remove a package (and any dependencies, provided other packages don’t need them)

Query installed packages

Caution required – curl | bash

Compiling third-party software from source

Example: compiling and installing htop

Install prerequisites

Download, verify, and unarchive the source code

Configure and compile htop

Conclusion

Configuring Software

Configuration hierarchy

Command-line arguments

Environment variables

Configuration files

System-level configuration in /etc/

User-level configuration in ~/.config

systemd units

Create your own service

Quick note: configuration in Docker

Conclusion

Pipes and Redirection

File descriptors

What do these file descriptors reference?

Input and output redirection (or, playing with file descriptors for fun and profit)

Input redirection: <

Output redirection: >

Use >> to append output without overwriting

Error redirection with 2>

Connecting commands together with pipes (|)

Multi-pipe commands

Reading (and building) complex multi-pipe commands

The CLI tools you need to know

cut

sort

uniq

Counting

wc

head

tail

tee

awk

sed

Practical pipe patterns

“Top X”, with count

curl | bash

Security considerations for curl | sudo | bash

Filtering and searching with grep

grep and tail for log monitoring

find and xargs for bulk file operations

sort, uniq, and reverse numerical sort for data analysis

awk and sort for reformatting data and field-based processing

sed and tee for editing and backup

ps, grep, awk, xargs, and kill for process management

tar and gzip for backup and compression

Advanced: inspecting file descriptors

Conclusion

Automating Tasks with Shell Scripts

Why you need Bash scripting basics

Basics

Variables

Setting

Getting

Bash versus other shells

Shebangs and executable text files, aka scripts

Common Bash settings (options/arguments)

/usr/bin/env

Special characters and escaping

Command substitution

Testing

Testing operators

[[ file and string testing ]]

Useful operators for string testing

Useful operators for file testing

(( arithmetic testing ))

Conditionals: if/then/else

ifelse

Loops

C-style loops

for…in

While

Variable exporting

Functions

Prefer local variables

Input and output redirection

<: input redirection

> and >>: output redirection

Use 2>&1 to redirect STDERR and STDOUT

Variable interpolation syntax – ${}

Limitations of shell scripts

Conclusion

Citations

Secure Remote Access with SSH

Public key cryptography primer

Message encryption

Message signing

SSH keys

Exceptions to these rules

Logging in and authenticating

Practical project: Set up a key-based login to a remote server

Step 1: Open your terminal on the SSH client (not the server)

Step 2: Generate the key pair

Step 3: Copy the public key to your server

Step 4: Test it out!

Converting SSH2 keys to the OpenSSH format

What we are trying to achieve

How to convert the SSH2-formatted key to OpenSSH

The other direction: Converting SSH2 keys to the OpenSSH format

SSH-agent

Common SSH errors and the -v (verbose) argument

File transfer

SFTP

SCP

Clever examples

Without SFTP or SCP

Directory upload and .tar.gz compression

Tunnels

Local forwarding

Proxying

The configuration file

Conclusion

Version Control with Git

Some background on Git

What is a distributed version control system?

Git basics

First-time setup

Initialize a new Git repository

Make and see changes

Stage and commit changes

Optional: add a remote Git repository

Pushing and pulling

Cloning a repository

Terms you might come across

Repository

Bare repository

Branch

Main/master branch

HEAD

Tag

Shallow

Merging

Merge commit

Merge conflict

Stash

Pull request

Cherry-picking

Bisecting

Rebasing

Best practices for commit messages

Good commit messages

GUIs

Useful shell aliases

Poor man’s GitHub

Considerations

1. Connect to your server

2. Install Git

3. Initialize a repository

4. Clone the repository

5. Edit the project and push your changes

Conclusion

Containerizing Applications with Docker

How containers work as packages

Prerequisite: Docker install

Docker crash course

Creating images with a Dockerfile

Container commands

docker run

docker image list

docker ps

docker exec

docker stop

Docker project: Python/Flask application container

1. Set up the application

2. Create the Docker image

3. Start a container from your image

Containers vs. virtual machines

A quick note on Docker image repositories

Painfully learned container lessons

Image size

C standard library

Production is not your laptop: outside dependencies

Container theory: namespacing

How do we do Ops with containers?

Conclusion

Monitoring Application Logs

Introduction to logging

Logging on Linux can get... weird

Sending log messages

The systemd journal

Example journalctl commands

Following active logs for a unit

Filtering by time

Filtering for a specific log level

Inspecting logs from a previous boot

Kernel messages

Logging in Docker containers

Syslog basics

Facilities

Severity levels

Configuration and implementations

Tips for logging

Keywords when using structured logging

Severity

Centralized logging

Conclusion

Load Balancing and HTTP

Basic terminology

Gateway

Upstream

Common misunderstandings about HTTP

HTTP statuses

Don’t just check for 200 OK

404 Not Found

502 Bad Gateway

503 Service Unavailable

504 Gateway Timeout

Introduction to curl: checking HTTP response status

HTTP headers

Case-insensitive headers

Custom headers

Viewing HTTP headers with curl

HTTP versions

HTTP/0.9

HTTP/1.0 and HTTP/1.1

HTTP/2

HTTP/3 and QUIC

Load balancing

Sticky sessions, cookies, and deterministic hashing

Round-robin load balancing

Other mechanisms

High availability

Troubleshooting redirects with curl

Using curl as an API testing tool

Accepting and displaying bad TLS certificates with curl

CORS

Conclusion

Other Books You May Enjoy

Index

Landmarks

Cover

Index

Preface

Many software engineers are new to Unix-like systems, even though these systems are everywhere in the software engineering world. Whether developers know it or not, they’re expected to work with Unix-like systems running in their work environment (macOS), their software development process (Docker containers), their build and automation tooling (CI and GitHub), their production environments (Linux servers and containers), and more.

Being skilled with the Linux command line can help software developers go beyond what’s expected of them, allowing them to:

Save time by knowing when to use built-in Unix tools, instead of writing thousand-line scripts or helper programsHelp debug complex production outages, often involving Linux servers and their interface to the applicationMentor junior engineersHave a more complete understanding of how the software they write fits into the larger ecosystem and tech stack

We hope that the theory, examples, and projects included in this book can take your Linux development skills to the next level.

Who this book is for

This book is for software developers who are new to Linux and the command line, or who are out of practice and want to quickly dust off their skills. If you still feel a bit insecure about your abilities when you’re staring at a Linux command-line prompt on a production server at 2:00 in the morning, this book is for you. If you want to quickly fill a Linux skills gap to advance your career, this book is for you. If you’re just curious, and you want to see what kind of efficiency gains you can make in your day-to-day development setup and routines by adding some command-line magic, this book will serve you as well.

What this book is not

One of the ways we have tried to fulfill our vision for this kind of uniquely useful book is by being extremely careful about what’s included. We’ve tried to cut out everything that isn’t essential to your life as a developer, or to a basic understanding of Linux and its core abstractions. In other words, the reason this book is useful is because of all the things we left out.

This book is not a full Linux course. It’s not for people working as Linux system engineers or kernel developers. Because of this, it’s not 750+ pages long, and you should be able to work through it in a few days, perhaps during a quiet sprint at work.

What this book covers

Chapter 1, How the Command Line Works, explains how a command-line interface works, what a shell is, and then immediately gives you some basic Linux skills. You’ll get a bit of theory and then begin moving around on the command line, finding and working with files and learning where to look for help when you get stuck. This chapter caters to new developers by teaching the most important command-line skills. If you read nothing else, you’ll still be better off than when you started.

Chapter 2, Working with Processes, will take you on a guided tour of Linux processes. You’ll then dive into useful, practical command-line skills for working with processes. We’ll add detail to a few aspects that are a common source of process-related problems that you’ll encounter as a software developer, like permissions, and give you some heuristics for troubleshooting them. You’ll also get a quick tour of some advanced topics that will come up again later in the book.

Chapter 3, Service Management with systemd, builds on the knowledge about processes learned in the previous chapter by introducing an additional layer of abstraction, the systemd service. You’ll learn about what an init system does for an operating system, and why you should care. Then, we cover all the practical commands you’ll need for working with services on a Linux system.

Chapter 4, Using Shell History, is a short chapter covering some tricks that you can learn to improve your speed and efficiency on the command line. These tricks revolve around using shortcuts and leveraging shell history to avoid repeated keystrokes.

Chapter 5, Introducing Files, introduces files as the essential abstraction through which to understand Linux. You’ll be introduced to the Filesystem Hierarchy Standard (FHS), which is like a map that you can use to orient yourself on any Unix system. Then it’s time for practical commands for working with files and directories in Linux, including some special filetypes you probably haven’t heard of. You’ll also get a taste of searching for files and file content, which is one of the most powerful bits of knowledge to have at your fingertips as a developer.

Chapter 6, Editing Files on the Command Line, introduces two text editors – nano and vim. You will learn the basics of using these text editors for command-line editing while also becoming aware of common editing mistakes and how to avoid them.

Chapter 7, Users and Groups, will introduce you to how the concepts of users and groups form the basis for the Unix security model, controlling access for resources like files and processes. We’ll then teach you the practical commands you’ll need to create and modify users and groups.

Chapter 8, Ownership and Permissions, builds on the previous chapter’s explanation of users and groups to show you how access control works for resources in Linux. This chapter teaches you about ownership and permissions by walking you through file information from a long listing. From there, we’ll look at the common file and directory permissions that you’ll encounter on production Linux systems, before engaging with the Linux commands for modifying file ownership and permissions.

Chapter 9, Managing Installed Software, shows you how to install software on various Linux distributions (and even macOS). First, we introduce package managers, which are the preferred way of getting software onto a machine: you’ll learn the important theory and practical commands for the package management operations you’ll need as a software developer. Then we’ll introduce a few other methods, like downloading install scripts and the time-honored, artisanal Unix tradition of compiling your own software locally, from source (it’s not as scary as it sounds!).

Chapter 10, Configuring Software, piggybacks off the previous chapter’s focus on installing software by helping you with configuring software on a Linux system. You will learn about the places that most software will look for configuration (“the configuration hierarchy”). Not only will this knowledge come in handy during late-night troubleshooting sessions, but it can actually help you to write better software. We’ll cover command-line arguments, environment variables, configuration files, and how all of this works on non-standard Linux environments like Docker containers. There’s even a little bonus project: you’ll see how to take a custom program and turn it into its own systemd service.

Chapter 11, Pipes and Redirection, will give you an introduction to what is possibly the “killer feature” of Unix: the ability to connect existing programs into a custom solution using pipes. We’ll move through the prerequisite theory and practical skills you need to understand: file descriptors and input/output redirection. Then you’ll jump into creating complex commands using pipes. You’ll be introduced to some essential CLI tools and practical pipe patterns, which you’ll still find yourself using long after you finish this book.

Chapter 12, Automating Tasks with Shell Scripts, serves as a Bash scripting crash course, teaching you how to go from typing individual commands in an interactive shell to writing scripts. We assume you’re already a software developer, so this will be a quick introduction that shows you the core language features and doesn’t spend a lot of time re-explaining the basics of programming. You’ll learn about Bash syntax, best practices for script writing, and some important pitfalls to avoid.

Chapter 13, Secure Remote Access with SSH, explores the Secure Shell Protocol and the related command-line tools available to you. You’ll learn the basics of public-key cryptography (PKI), which is always useful for a developer to know, before diving into creating SSH keys and securely logging into remote systems over the network. You’ll build on this knowledge and get some experience copying files over the network, using SSH to create ad-hoc proxies or VPNs, and see examples of various other tasks that involve moving data over an encrypted SSH tunnel.

Chapter 14, Version Control with Git, shows you how to use a tool you probably already know well – git – from the command line, instead of through your IDE or a graphical client. We quickly go through the basic theory behind git and then jump into the commands you’ll need to use in a command-line environment. We’ll cover two powerful features that it pays to understand – bisecting and rebasing – and then give you our take on best practices and useful shell aliases. Finally, the Poor man’s GitHub section presents a small but legitimately useful project that you can do to practice and integrate the Linux skills you’ve learned up to this point.

Chapter 15, Containerizing Applications with Docker, gives you the basic theory and practical skills that will make it easy to work with Docker as a developer. We’ll explore the problems that Docker solves, explain the most important Docker concepts, and take you through the core workflow and commands you’ll use. You’ll also see how to build your own images by containerizing a real application. And because we’re approaching this from a software development and Linux perspective, you’ll also develop a good intuition for how containerization works under the hood, and how it’s different from virtual machines.

Chapter 16, Monitoring Application Logs, gives an overview of logging on Unix and Linux. We’ll show you how (and where) logs are collected on most modern Linux systems using systemd, and how more traditional approaches work (you’ll come across both in the real world). You’ll build practical command-line skills finding and viewing logs and learn a bit about how logging is being done in larger infrastructures.

Chapter 17, Load Balancing and HTTP, covers the basics of HTTP for developers, with a special focus on the complexities that you’ll come across when working with HTTP services in larger infrastructures. We’ll correct some common misunderstandings about HTTP statuses, HTTP headers, and HTTP versions and how applications should handle them. We’ll also introduce how load balancers and proxies work in the real world, and how they make the experience of troubleshooting a live application quite different from troubleshooting a development version on your laptop. Many of the Linux skills that you will have learned up to this point will come in handy here, and we’ll introduce a new tool – curl – to help you troubleshoot a wide variety of HTTP-related issues.

To get the most out of this book

If you can get yourself to a Linux shell prompt – by installing Ubuntu in a virtual machine or running it as a Docker container, for example – you can follow along with everything in this book.

You can get away with even less – on Windows, there’s WSL, and macOS is a bona-fide Unix operating system, so almost all of the practical commands you learn in this book (except those called out as Linux-only) will work out of the box. That said, for the best experience, follow along on a Linux operating system.

The skills required to get the most out of this book are only the basic computer skills that you already have as a software developer – editing text, working with files and folders, having some notion of what “operating systems” are, installing software, and using a development environment. Everything beyond that, we’ll teach you.

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781804616925.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. For example: “The -f flag stands for ‘follow,’ and the -u flag stands for ‘unit.’”

A block of command line is set as follows:

/home/steve/Desktop# ls anotherfile documents somefile.txt stuff /home/steve/Desktop# cd documents/ /home/steve/Desktop/documents# ls contract.txt

Bold: Indicates a new term, an important word, or words that you see on the screen For instance, words in menus or dialog boxes appear in the text like this. For example: “When a file is set to be executable, Unix will do its best to execute it, either succeeding in the case of ELF (Executable and Linkable Format, probably the most widely used executable format today) or failing.”

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email [email protected], and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book we would be grateful if you would report this to us. Please visit, http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.

Share your thoughts

Once you’ve read The Software Developer’s Guide to Linux, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781804616925

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

1

How the Command Line Works

Before we dive into practical Linux commands, you need to have a basic understanding of how the command line works. This chapter will give you that understanding.

For new developers, we’ll explore the initial skills that you need to get started on the Linux command line. For those with a little more experience, there are still some nuances to discover, such as the difference between “shell” and “command line.” It pays to know the difference!

In this chapter, we will cover the following topics:

The basic idea of a command-line interface, or CLIThe form that commands takeHow command arguments work and how they look when you’re typing commands and when you’re looking up documentationAn introduction to “the shell,” and how it differs from the “command line”The core rules that the shell uses to look up commands

To begin with, we’ll start off with the basic idea of a command-line interface. We will get ourselves up to speed with how a CLI works and run through a quick example.

In the beginning…was the REPL

What is a command-line interface (CLI)? It’s a text-based environment for interacting with your computer that:

Reads some input from you,Evaluates (or processes) that input,Prints some output to the screen in response, and thenLoops back to the beginning to repeat that process.

Let’s look at what happens at each step, on a practical level, with the ls (list) command, which you’ll see in a few pages. For now, it’s enough to know that the ls command lists the contents of a directory.

Step

What it means

1. Read input

You type the ls command and press Enter.

2.Evaluate command

The shell looks up the ls binary, finds it, and tells the machine to execute it.

3. Print output

The ls command emits some text – the names of all files and directories it found – and the shell prints that output to your terminal window.

4. Loop back to 1 (repeat the process)

Once the programs called by the command have exited, repeat the process by accepting more user input.

If you read steps 1-4 again, you’ll notice that the first letter of each step spells “REPL”, which is a common way of referring to this kind of Read-Eval-Print Loop in the languages that invented and refined this workflow, such as Lisp.

To put this into programming terms, you can translate the REPL instructions above into code:

while (true) { // the loop print(eval(read())) }

Indeed, you can create a REPL capable of doing basic calculations with just a few lines of code in most programming languages. Here’s a one-liner “shell” program written in Perl:

perl -e 'while (<>){print eval, "\n"}'1+23

Here, we write the code as a parameter, printing the output of the evaluation as long as there is input to read from. At the end, we append a new line and exit.

This program is tiny, but it’s enough to implement an interactive Read-Eval-Print Loop in a command-line environment – a shell. The shells you’ll use in Linux and Unix are significantly more complex than this Perl mini-shell, but the principles are the same.

The point is simple: as a developer, you might already be using REPLs without realizing it, because almost all modern scripting languages come with one. In essence, the Linux (or macOS, or other Unix) command line functions like the “interactive shells” that interpreted languages give you. So even if you’re not familiar with the Lisp REPL, the Perl snippet above should remind you of a very basic Ruby or Python shell.

Now that you understand the basic mechanics of the command-line interfaces you’ll be using in Linux, you’re ready to try out your first commands. To do that, you’ll need to know the correct command-line syntax to use.

Command-line syntax (read)

All REPLs start by reading some input. On the Linux command line, commands that the shell reads in need to have the correct syntax. Commands take this basic form:

commandname options

In programming terms, you can think of the command name as a function name, and the options as any number of arguments that will be passed to that function. This is important, because there is no single fixed syntax for all the options – each command defines which parameters it will accept. Because of this, the shell can do very little to validate a command’s correctness beyond checking that the command maps to an executable.

Note

The terms “program” and “command” are used interchangeably in this chapter. There’s a very slight difference because some shell builtins are defined in the shell’s code and are therefore not technically separate programs of their own, but you don’t need to worry about it – leave that distinction to the Unix greybeards.

Let’s dive into more complex variations on this “command [options]” syntax, which you’ll see frequently:

command [-flags,] [--example=foobar] [even_more_options ...]

This is the conventional format you’ll see used in help documentation such as the program manual pages (manpages) included in most Linux environments, and it’s fairly simple:

command is the program you’re runningItems in brackets are optional, and brackets with ellipses ([xyz ...]) tell you that you can pass zero or more arguments here-flags means any valid option (“flag,” in Unix-speak) for that program, e.g. -debug or -foobar

Some programs will also accept short and long versions of a parameter, usually denoted by single- vs. double-hyphenation: so -l and --long might do the same thing. It’s not consistent across commands, though; this kind of behavior requires that the command’s creator implemented short and long arguments that set the same parameter.

Not all commands will implement all these ways of passing configuration when invoking them, but these represent the most common forms you’ll see.

By default, a space denotes the end of an argument, so just like in most programming languages, an argument string that includes spaces must be single- or double-quoted. You’ll read more about this in Chapter 12, Automating Tasks with Shell Scripts.

In just a moment, we’ll follow the process of how the shell interprets a command that you issue using this syntax, but first we want to clearly define the difference between two sometimes-interchangeable terms we’ve been using in this chapter: “command line” and “shell.”

Command line vs. shell

In this book, we refer to a “command-line environment.” We define this as any text-based environment that acts as a kind of REPL, specifically for interacting with the operating system, programming language interpreter, database, etc. A “command-line” environment or interface describes the general idea of how you’re interacting with a system.

But there’s a more specific term which we’ll use here: shell.

A shell is a specific program that implements this command-line environment and lets you give it text commands. Technically, there are lots of different shells which provide the same kind of REPL-based command-line environment, often for wildly different things:

Bash is a common shell environment for interacting with Linux and Unix operating systems.Popular databases like Postgres, MySQL, and Redis all provide a shell for developers to interact with and run commands in.Most interpreted languages provide a shell environment to speed up development. In these, valid commands are simply programming language statements. See irb for Ruby, the interactive Python shell, etc.Zsh (the Z shell) is an alternative operating system shell (like Bash), which you might see on some developers’ laptops if they’ve customized their environments.

When we talk about a shell in this book, we’re referring to a Unix shell (generally Bash), which is a command-line interface specifically designed to let you interact with the underlying Linux or Unix operating system.

How does the shell know what to run? (evaluate)

After reading in a command, the shell needs to evaluate it, by executing a program, fetching some information, or doing something else that’s actually useful to you.

Note

Such a detailed description of how shells work may seem tedious at first, but we promise that this knowledge will come in handy when you have to troubleshoot an issue with a missing or incorrectly permissioned program.

When you type a command like foobar -option1 test.txt in a shell like Bash and press Enter, a few things happen:

If the command has a path specified, it will be used. This can take various forms:A full path, like /usr/bin/foobar in the command /usr/bin/foobar -option1 test.txt.A relative path, like the current working directory in the command ./foobar-option1 test.txt (the . denotes the current directory, which we’ll cover in the Absolute vs. Relative Filepaths section below; this command essentially says “please execute the “foobar” file that’s in my current directory”).The path may be based on variables and symbols either in:The shell’s environment (env vars) like $HOME/foobar, orProvided by the shell, like ~/foobar (the ~ character means “this user’s home directory”)If not, the shell checks to see whether it knows what foobar means:It could be a built-in shell command.It could be an alias, which is a way to set up macros or shortcuts for commands.If not, the shell generally looks at the $PATH environment variable, which contains a few different locations to check for commands: /bin, /usr/bin, /sbin, etc. Users can add locations to this $PATH list, and various software will modify your $PATH: version managers for scripting languages, Python’s virtual environments, and many other programs make heavy use of this mechanism. The shell tries those places specified in your $PATH, in the order it finds them in the $PATH variable, to see if any of them contain an executable with the name foobar.

If the shell still hasn’t found anything, it’ll return an error like bash: foobar: command not found:.

On the other hand, if at any point the shell indeed finds an executable file named foobar, it executes that file and passes -option1 and test.txt (in that order) as arguments.

At this point, the shell knows what program to use to evaluate the command, and it does so. As the command is evaluated, any output is printed to the user, completing the third step of the REPL process. Now all that’s left to do is to loop back to the beginning and start the process over again, accepting another command as input from the user.

The shell tries its best to guess which program the user wants to run, using the general process we outlined above to resolve ambiguity. However, ambiguity can be a bad thing and lead to misunderstandings or bugs. During troubleshooting, you’ll often want to find out which command is really being run. To accomplish this, you can use the command which <command>, which will print the full path (or the alias or script being run) and will let you know whether that command is a shell builtin. Depending on the system, which might not be available. In these situations, you can use command –v instead. This is the POSIX equivalent, which we’ll learn about next:

bash-3.2$ which ls /bin/ls bash-3.2$ command -v ls /bin/ls

A quick definition of POSIX

Wikipedia tells us that “the Portable Operating System Interface (POSIX) is a family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems.” Practically speaking, it’s an attempt at defining some common standards between Unix systems, which can otherwise have wildly different sets of basic commands available.

POSIX basically says things like, “every POSIX-compatible OS should have a list command called ls"; in this case, “every POSIX-compatible OS should have a way to check to see if a matching executable exists for a given command name.”

If your scripts need to be portable across Unix operating systems, restricting yourself to POSIX commands is a good thing to do. However, it’s still not a guarantee – many extremely popular Linux distributions divert from POSIX in numerous ways, most of which you won’t notice until they bite you.

Understanding POSIX is the last brick in the foundation you need before getting started with the practical job of working on the command line. We’ve covered a lot of ground so far:

You learned about REPLs and saw how this basic process maps to how all modern shells workWe explored the basic command syntax you’ll be using while working with Linux

You saw how your shell decides how to take your command input and “evaluate” it correctly. You learned important terminology that you’ll come across frequently: shell, command-line interface, POSIX, and a few more terms that will pay dividends if you learn them now. Armed with this knowledge, you’re ready to move from theory to practice. In the next section, we’ll talk about the Linux-specific context that you’ll be in while running commands. You’ll learn the absolute basics of the Linux filesystem and how different kinds of paths work. After that, the rest of the chapter is all about running Linux commands!

Basic command-line skills

To work effectively with Linux, you need to know the absolute basics: how the system is structured, how to look and move around on the system, and how to read and edit files. In this section, we’ll cover all of that, and get you comfortable with the very basics of navigating a Linux system.

Throughout the rest of this book, we’ll dive deeper into each of these topics and commands, but we want to make sure you have a minimal, functioning set of skills by the time you get to the end of this chapter.

Unix filesystem basics

In graphical user interfaces, directories (called folders in macOS) are represented by icons. Perhaps you’re used to seeing neat little rows of these in your home directory - Desktop, Documents, Videos, and so on. Double-clicking on a directory icon opens a new window with a new view from inside that directory.

When we use the term “filesystem,” we mean exactly this – a collection of directories and files that organizes all data on the system. The underlying concept is exactly the same in a command-line environment, it just looks a bit different.

Instead of seeing lots of windows and icons, everything is represented as text, and the contents of directories are only shown when you ask for them. However, files and directories still work exactly the way that you’re used to.

Keeping the filesystem in your head as you navigate seems difficult at first, but once you get used to it, it’s often a more efficient way of dealing with a computer. After a few days of working this way, most people have no problem holding a detailed view of the filesystem in their heads as they work on a system and verify this view only occasionally.

Absolute vs. relative file paths

When beginners work with Linux, they often get caught up on the difference between an absolute path and a relative path. This simple misunderstanding results in frustrating amounts of time wasted staring at errors like this one:

No such file or directory

Because you need to understand paths as a prerequisite for almost every Linux command you run, we’ll cover them first.

An absolute path is the full path to any file on the filesystem, starting from the root directory. You can recognize this because it starts with a /, which references the root directory (the very top, or beginning, of the filesystem, which contains all other files and directories).

Here are some examples of absolute paths:

/home/dave/Desktop/var/lib/floobkit//usr/bin/sudo

These absolute paths are like a full set of driving directions, giving turn-by-turn instructions from a known starting point (your apartment, or in the case of a Unix system, the root directory).

You can immediately recognize an absolute path by the fact that it starts with a “/" character. No matter where you are on the filesystem, absolute paths will work, since they are full, unique addresses for file objects.

A relative path is a partial path, and it’s assumed that it starts at the current location instead of at the root directory. You can recognize an absolute path by the fact that it doesn’t start with a / character.

Relative paths are like driving directions that use your current location as the starting point. If you’ve pulled off the road because you’re lost and you need new directions, you want directions that start from your current location, not your home address. Relative paths give you exactly this.

As a result, relative paths are often more convenient to type: if you’re already sitting in your /home/Desktop directory, it’s easier to reference a file as mydocument.txt than as /home/Desktop/mydocument.txt (even though both ways are valid, given your location on the filesystem). The real difference comes when you change directories. When you move up a directory from /home/Desktop to /home, the absolute path will still reference the same file, while the relative path reference won’t (now, typing mydocument.txt would reference /home/mydocument.txt).

Imagine a partial directory structure like this – in our example, we’ll say this is a directory tree listing of /home/dave/Desktop:

Desktop ├── anotherfile ├── documents │ └── contract.txt ├── somefile.txt └── stuff ├── nothing └── important