Linux Command Line and Shell Scripting Techniques E-Book

Vedran Dakic

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 629

Veröffentlichungsjahr: 2022

Beschreibung

Linux Command Line and Shell Scripting Techniques begins by taking you through the basics of the shell and command-line utilities. You’ll start by exploring shell commands for file, directory, service, package, and process management. Next, you’ll learn about networking - network, firewall and DNS client configuration, ssh, scp, rsync, and vsftpd, as well as some network troubleshooting tools.
You’ll also focus on using the command line to find and manipulate text content, via commands such as cut, egrep, and sed. As you progress, you'll learn how to use shell scripting. You’ll understand the basics - input and output, along with various programming concepts such as loops, variables, arguments, functions, and arrays. Later, you’ll learn about shell script interaction and troubleshooting, before covering a wide range of examples of complete shell scripts, varying from network and firewall configuration, through to backup and concepts for creating live environments. This includes examples of performing scripted virtual machine installation and administration, LAMP (Linux, Apache, MySQL, PHP) stack provisioning and bulk user creation for testing environments.
By the end of this Linux book, you’ll have gained the knowledge and confidence you need to use shell and command-line scripts.

Leseprobe

Linux Command Line and Shell Scripting Techniques

Master practical aspects of the Linux command line and then use it as a part of the shell scripting process

Vedran Dakic

Jasmin Redzepagic

BIRMINGHAM—MUMBAI

Linux Command Line and Shell Scripting Techniques

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Vijin Boricha

Publishing Product Manager: Shrilekha Malpani

Senior Editor: Shazeen Iqbal

Content Development Editor: Rafiaa Khan

Technical Editor: Shruthi Shetty

Copy Editor: Safis Editing

Project Coordinator: Shagun Saini

Proofreader: Safis Editing

Indexer: Subalakshmi Govindhan

Production Designer: Vijay Kamble

Marketing Coordinator: Hemangi Lotlikar

First published: April 2022

Production reference: 1150222

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80020-519-2

www.packt.com

Writing this book was quite a journey for me. It was a very busy year, compounded by a lot of weird situations – earthquakes, COVID-19, health issues in the family – basically, this year threw everything that it possibly could at me. Often, I found myself pondering the fact that writing a book is as much therapeutic as it is potentially useful. Writing a book is as much about talking to your inner self as it is about talking to your future audience, especially when knowledge sharing is involved. And, as an ex-IT journalist/editor who used to write IT magazine articles for a living, I was constantly reminded of that fact.

To my co-workers, for offering valuable insight and for being a whiteboard to bounce ideas off (Jasmin, Zlatan, and Andrej). Your coherent and random "in passing" thoughts helped a lot.

To my Packt crew, for putting up with the extreme, parallel universe-level randomness that was my year 2021 – thank you.

To my partner, Sanja, for pushing me forward and kicking my derrière, when it needs to be kicked.

– Vedran Dakic

My first Linux, Slackware, came on 50+ floppy disks. Installing it was a pain, and somewhere in the middle of the process, a floppy disk was faulty so I started my Linux journey on a machine that was unable to install a GUI. Being confined to a text-only terminal for a few weeks got me hooked.

The nice thing about scripting is that I rarely have any need to leave the terminal even now, 25+ years later, while still being able to do almost anything on any machine I encounter.

Having said that, running scripts would not be possible if it weren't for a few special people in my life, so here goes:

To Filip, my son, who is learning to spell while I try to teach him how to type.

To Dinka, my wife, who keeps us all together through the sun and the rain.

And to all the SysAdmins and SysOps I have encountered during this quarter of a century, exchanging ideas and solutions.

Remember, it's usually DNS.

– Jasmin Redzepagic

Contributors

About the authors

Vedran Dakic is a master of electrical engineering and computing and an IT trainer, covering system administration, cloud, automatization, and orchestration courses. He is a certified Red Hat, VMware, and Microsoft trainer. He's currently employed as head of department of operating systems at Algebra University College in Zagreb. As part of this job, he's a lecturer for 3- and 5-year study programs in system engineering, programming, and multimedia tracks. Also, he does a lot of consulting and systems integration for his clients' projects – something he has been doing for the past 25 years. His approach is simple – bring real-world experience to all the courses that he teaches as it brings added value to his students and customers.

Jasmin Redzepagic is a professional master in computer engineering, with a sub-specialization in system engineering, but is a person with many hats worn over the years. Having worked in IT as long as he has, he has been a sound technician, SysOps, DevOps, hardware tester, editor-in-chief of a major magazine, writer, IT support lead, and head of IT at a couple of companies. He is currently employed by Algebra University College in Zagreb. This enables him to have a very wide overview of different technologies while still maintaining his focus – implementing and teaching the implementation of open source technologies in the field.

Right now, his main interest is trying to teach his students and clients to understand the immense expanse of tools available for any conceivable task, with his goal being to get people to understand computers as something that is the most formidable tool we, as humanity, have ever had.

About the reviewers

Sergio Guidi Tabosa Pessoa is a software engineer with more than 30 years of experience with software development and maintenance, from complex enterprise software projects to modern mobile applications. In the early days, he was working primarily with the Microsoft stack, but soon discovered the power of the Unix and Linux operating systems. Even though he has worked with many languages over the years, C and C++ remain his favorite languages on account of their power and speed.

He has a bachelor's degree in computer science and an MBA in IT management, and is always hungry to learn new technologies, break code, and learn from his mistakes. He currently lives in Brazil with his wife, two Yorkshire terriers, and two cockatiels.

First and foremost, I would like to thank all the people involved in this project, including the author, for such a great piece of work, and those from Packt Publishing for giving me this opportunity. I also would like to thank my beautiful wife, Lucia, and my children, Touché and Lion, for their patience and for allowing me the time needed to help with this book.

Nicholas Cross was born in the UK and educated in New Zealand. He is a distinguished engineer at a tier-1 technology company and has 20 years' experience in Linux. He was working with Linux and Bash scripting a long time before the cloud, automation, and DRY were cool.

Nicholas is passionate about infrastructure automation, DevOps culture, SRE, automation, containerization, and security, and everything else associated with these broad topics.

When not at his computer hacking code for work or pleasure, he enjoys running, walking his dogs, and watching his sons play rugby.

Jason Willson has been working in IT for 17 years since his first job at the help desk at his alma mater, Grove City College. He was first introduced to Linux in 2007 at a start-up in Boston and has worked with it professionally and personally ever since. He has used command-line and shell scripting techniques for a variety of tasks relating to data analysis, systems administration, and DevOps. He currently works as a Linux systems administrator at the Software Engineering Institute of Carnegie Mellon University.

I'd like to thank the incredible LinkedIn community for making this connection possible with Packt Publishing. I'd also like to thank all my coworkers, classmates, and mentors (personal, professional, and academic) who have helped to shape me into the person I am today.

Table of Contents

Preface

Chapter 1: Basics of Shell and Text Terminal

Technical requirements

Accessing the shell

Getting ready

How to do it…

How it works…

See also

Setting up the user shell

Getting ready

How to do it…

How it works…

Setting up the Bash shell

Getting ready

How to do it…

How it works…

There's more…

Using the most common shell commands

Getting ready

How to do it…

How it works...

Using screen

Getting ready

How to do it…

How it works…

There's more…

Chapter 2: Using Text Editors

Technical requirements

Learning the basics of the Vi(m) Editor

Getting ready

How to do it…

How it works…

See also

Learning the basics of the nano editor

Getting ready

How to do it…

There's more…

Going through the advanced Vi(m) settings

Getting ready

How to do it…

How it works…

There's more…

Chapter 3: Using Commands and Services for Process Management

Technical requirements

Process management tools

Getting ready

How to do it…

How it works…

See also

Managing background jobs

Getting ready

How to do it…

How it works…

There's more…

Managing process priorities

Getting ready

How to do it…

How it works…

There's more…

Configuring crond

Getting ready

How to do it…

How it works…

There's more…

Chapter 4: Using Shell to Configure and Troubleshoot a Network

Technical requirements

Using nmcli and netplan

Getting ready

How to do it

How it works

There's more

Using firewall-cmd and ufw

Getting ready

How to do it

How it works

There's more

Working with open ports and connections

Getting ready

How to do it

How it works

There's more

Configuring /etc/hosts and DNS resolving

Getting ready

How to do it

How it works

There's more

Using network diagnostic tools

Getting ready

How to do it

How it works

There's more

Chapter 5: Using Commands for File, Directory, and Service Management

Technical requirements

Basic file and directory-based commands

Getting ready

How to do it…

How it works…

See also

Additional commands for manipulating file/directory security aspects

Getting ready

How to do it…

How it works…

See also

Manipulating text files by using commands

Getting ready

How to do it…

How it works…

There's more…

Finding files and folders

Getting ready

How to do it…

How it works…

There's more…

Archiving and compressing files and folders

Getting ready

How to do it…

How it works…

There's more…

Managing services and targets

Getting ready

How to do it…

How it works…

There's more…

Chapter 6: Shell-Based Software Management

Technical requirements

Using dnf and apt for package management

Getting ready

How to do it…

How it works…

There's more…

Using additional repositories, streams, and profiles

Getting ready

How to do it…

How it works…

Creating custom repositories

Getting ready

How to do it…

How it works…

There's more…

Compiling third-party software

Getting ready

How to do it…

How it works…

There's more…

Chapter 7: Network-Based File Synchronization

Technical requirements

Learning how to use SSH and SCP

Getting ready

How to do it…

How it works…

There's more…

Learning how to use rsync

Getting ready

How to do it…

How it works…

There's more…

Using vsftpd

Getting ready

How to do it…

How it works…

There's more

Chapter 8: Using the Command Line to Find, Extract, and Manipulate Text Content

Technical requirements

Using text commands to merge file content

Getting ready

How to do it…

How it works…

There's more…

Converting DOS text to Linux text and vice versa

Getting ready

How to do it…

How it works…

There's more…

Using cut

Getting ready

How to do it…

How it works…

There's more…

Using egrep

Getting ready

How to do it…

How it works…

There's more…

Using sed

Getting ready

How to do it…

How it works…

There's more…

Chapter 9: An Introduction to Shell Scripting

Technical requirements

Writing your first Bash shell script

Getting ready

How to do it…

How it works…

Serializing basic commands – from simple to complex

Getting ready

How to do it…

How it works…

There's more…

Manipulating shell script input, output, and errors

Getting ready

How to do it…

How it works…

Shell script hygiene

Getting ready

How to do it…

There's more…

Chapter 10: Using Loops

The for loop

Getting ready

How to do it…

How it works…

See also

break and continue

Getting ready

How to do it…

How it works…

See also

The while loop

Getting ready

How to do it…

How it works…

See also

The test-if loop

Getting ready

How to do it…

How it works…

See also

The case loop

Getting ready

How to do it…

How it works…

See also

Logical looping with and, or, and not

Getting ready

How to do it…

How it works…

See also

Chapter 11: Working with Variables

Technical requirements

Using shell variables

Getting ready

How to do it…

How it works…

See also

Using variables in shell scripting

Getting ready

How to do it…

How it works…

See also

Quoting in the shell

Getting ready

How to do it…

How it works…

See also

Performing operations on variables

Getting ready

How to do it…

How it works…

See also

Variables via external commands

Getting ready

How to do it…

How it works…

See also

Chapter 12: Using Arguments and Functions

Technical requirements

Using custom functions in shell script code

Getting ready

How to do it…

How it works…

See also

Passing arguments to a function

Getting ready

How to do it…

How it works…

See also

Local and global variables

Getting ready

How to do it…

How it works…

See also

Working with returns from a function

Getting ready

How to do it…

How it works…

See also

Loading an external function to a shell script

Getting ready

How to do it…

How it works…

See also

Implementing commonly used procedures via functions

Getting ready

How to do it…

How it works…

See also

Chapter 13: Using Arrays

Technical requirements

Basic array manipulation

Getting ready

How to do it…

How it works…

See also

Advanced array manipulation

Getting ready

How to do it…

How it works…

See also

Chapter 14: Interacting with Shell Scripts

Technical requirements

Creating text-based interactive scripts

Getting ready

How to do it…

How it works…

See also

Using expect to automate repetitive tasks based on text output

Getting ready

How to do it…

How it works…

See also

Using dialog for menu-driven interactive scripts

Getting ready

How to do it…

How it works…

See also

Chapter 15: Troubleshooting Shell Scripts

Technical requirements

Common scripting mistakes

Getting ready

How to do it…

How it works…

See also…

Simple debugging approach – echoing values during script execution

Getting ready

How to do it…

How it works…

See also

Using the bash -x and -v options

Getting ready

How to do it…

How it works…

See also…

Using set to debug a part of the script

Getting ready

How to do it…

How it works…

See also…

Chapter 16: Shell Script Examples for Server Management, Network Configuration, and Backups

Technical requirements

Creating a file and folder inventory

How to do it…

See also

Checking if you're running as root

Getting ready

How to do it…

See also

Displaying server stats

Getting ready

How to do it…

There's more…

Finding files by name, ownership, or content type and copying them to a specified directory

Getting ready

How to do it…

There's more…

Parsing date and time data

Getting ready

How to do it…

Configuring the most common firewall settings interactively

Getting ready

How to do it…

There's more…

Configuring network settings interactively

Getting ready

How to do it…

There's more…

Backing up the current directory with shell script arguments and variables

Getting ready

How to do it…

There's more…

Creating a current backup based on the user input for the backup source and destination

Getting ready

How to do it…

There's more…

Chapter 17: Advanced Shell Script Examples

Technical requirements

Implementing a web server service and security settings

Getting ready

How to do it…

See also

Creating users and groups and forcing users to change them on the next login

Getting ready

How to do it…

See also

Creating users and groups from a standardized input file and a random password for each user

Getting ready

How to do it…

See also

Scripted VM installation on KVM

Getting ready

How to do it…

See also

Using a shell script to provision SSH keys

Getting ready

How to do it…

See also

A shell script for VM administration

Getting ready

How to do it…

See also

Other Books You May Enjoy

Preface

Linux Command-Line and Shell Scripting Techniques is a book that will help you learn how to use the Command-Line Interface (CLI) and to further expand your CLI knowledge with the ability to do scripting. It looks at a big collection of CLI commands, shell scripting basics (loops, variables, and functions), and advanced scripting topics – such as troubleshooting. It also includes two chapters with script examples that should get you further ahead in your understanding of scripting while also offering good insight into how the shell scripting process works.

Who this book is for

This book is for beginners and professionals alike, as it doesn't necessarily need a lot of prior Linux knowledge. That's partially what this book is for – to get to grips with using the command line and to further that usage model to shell scripting. For more advanced users, there's a bulk of content about shell scripting and corresponding examples that will help you to organize and improve your knowledge about shell scripting.

What this book covers

Chapter 1, Basics of the Shell and Text Terminal, discusses the concept of the shell and text terminal, configuration of the Bash shell, using some basic shell commands, and using the screen to get access to multiple virtual terminals in text mode.

Chapter 2, Using Text Editors, takes us to the highly subjective world of text editors, where discussions have been happening for the past 30-40 years on the topic of best editor. As a part of this chapter, we're going to use vi(m), nano, and some more advanced vi(m) settings.

Chapter 3, Using Commands and Services for Process Management, is about using files, folders, and services, specifically, how to administer them, how to secure them (files and folders), and how to manage them (services). A big chunk of this chapter is related to ACLs and systemctl, essential tools for system administrators.

Chapter 4, Using Shell to Configure and Troubleshoot Network, is all about working with files, folders, and services – working with permissions, manipulating file content, archiving and compressing files, and managing services. Throughout this chapter, there will be a lot of simple commands that we will be using later when we go on to scripting, as well.

Chapter 5, Using Commands for File, Directory, and Service Management, is about making sure that we know the basics of fundamental networking configuration – nmcli and netplan, FirewallD and ufw, DNS resolving, and diagnostics. These are some of the settings that we most commonly re-configure post-deployment, so deep insight into them is a necessity.

Chapter 6, Shell-Based Software Management, takes us through two of the most commonly used packaging systems (dnf/yum and apt), as well as some more advanced concepts, such as using additional repositories, streams, and profiles, creating custom repositories, and third-party software. Every Linux deployment needs us to have knowledge about package management, so this chapter is all about that.

Chapter 7, Network-Based File Synchronization, teaches us about the most commonly used tools to send and receive files and connect to remote destinations via a network – ssh and scp, rsync, and vsftpd. For anything ranging from hosting a Linux distribution mirror all the way to synchronizing files and backups, this is mandatory knowledge.

Chapter 8, Using the Command Line to Find, Extract, and Manipulate Text Content, is all about using basic and more advanced ways of manipulating text files and content. We start off by doing simple things such as paste and dos2unix, and then move on to some of the most used commands in the IT world – cut, (e)grep, and sed.

Chapter 9, An Introduction to Shell Scripting, is the starting point for the second part of this book, which is all about shell scripting and using previously mentioned tools and commands to create shell scripts. This chapter is about the basics of shell scripting and working with general concepts, such as input, output, error, and shell script hygiene.

Chapter 10, Using Loops, goes deep into the concept of loops. We cover all the most used loops here – the for loop, break and continue, the while loop, the test-if loop, the case loop, and logical looping with conditions such as and, or, and not. This will further enhance our ability to do more things in shell scripts.

Chapter 11, Working with Variables, is about using variables in our shell script code – shell variables, quoting and special characters in variable values, assigning external variables via commands, as well as some logical operations on variables. Variables are the spine of shell scripting, and all permanent and temporary data gets stored in them, so, for whatever purpose we're developing a shell script, variables are a must-have.

Chapter 12, Using Arguments and Functions, is about further customizing and modularizing the shell script code, as we can use functions to do that. For that purpose, we are going to use external and shell arguments, to do away with the static nature of most of our previous shell script examples.

Chapter 13, Using Arrays, is about using arrays to store and manipulate data. Arrays are just one of those structures – we need them, we learn to not necessarily like them, but we can't live without them, especially as we venture into the world of working with their many different capabilities, such as indexing, adding and removing members, and working with files as a de facto array source.

Chapter 14, Interacting with Shell Scripts, is about moving from the idea of shell script code as a purely text-driven principle, and going in the opposite direction – to create a TUI-based interface to interact with a script. We are also going to have a play with expect script, which makes it easier for us to create a script that's waiting for specific output and then doing something based on that output, which can be useful for the configuration of third-party systems at times.

Chapter 15, Troubleshooting Shell Scripts, deals with shell script troubleshooting – common mistakes, debugging output via echoing values during script execution, Bash -xv, and other concepts. This is the last chapter before we start dealing with the many script examples that we prepared for you to use both as a learning tool and to work with them in production, if you so desire.

Chapter 16, Shell Script Examples for Server Management, Network Configuration, and Backups, sets us off in the direction of simple shell scripts – nine different examples to be exact. Topics vary from simple, modular code that can be implemented in any shell script (for example, how to check if we're executing a script as root), to more complex examples such as dealing with date and time, the interactive configuration of network settings and firewalls, as well as some backup script examples.

Chapter 17, Advanced Shell Script Examples, deals with more complex examples, such as a script to modify web server and security settings, bulk-creating users and groups with random passwords, scripted KVM virtual machine installation, and scripted KVM virtual machine administration (start, stop, getting info, manipulating snapshots, and so on). These are examples that we use in everyday life to drive the point of shell scripting home, which is all about automating boring, repetitive tasks and offloading them to a script that can do all of that for us.

To get the most out of this book

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781800205192_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: To configure the host side of the network, you need the tunctl command from the User Mode Linux (UML) project.

A block of code is set as follows:

#include <stdio.h>

#include <stdlib.h>

int main (int argc, char *argv[])

{

    printf ("Hello, world!\n");

    return 0;

}

Any command-line input or output is written as follows:

$ sudo tunctl -u $(whoami) -t tap0

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: ClickFlashfrom Etcher to write the image.

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Linux Command Line and Shell Scripting Cookbook, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.