33,59 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Create and maintain powerful Bash scripts for automation and administration.
Key Features
- Get up and running with Linux shell scripting using real-world examples
- Leverage command-line techniques and methodologies to automate common yet complex administration tasks
- A practical guide with exposure to scripting constructs and common scripting patterns
Book Description
Shell scripts allow us to program commands in chains and have the system execute them as a scripted event, just like batch files. This book will start with an overview of Linux and Bash shell scripting, and then quickly deep dive into helping you set up your local environment, before introducing you to tools that are used to write shell scripts. The next set of chapters will focus on helping you understand Linux under the hood and what Bash provides the user. Soon, you will have embarked on your journey along the command line. You will now begin writing actual scripts instead of commands, and will be introduced to practical applications for scripts. The final set of chapters will deep dive into the more advanced topics in shell scripting. These advanced topics will take you from simple scripts to reusable, valuable programs that exist in the real world. The final chapter will leave you with some handy tips and tricks and, as regards the most frequently used commands, a cheat sheet containing the most interesting flags and options will also be provided.
After completing this book, you should feel confident about starting your own shell scripting projects, no matter how simple or complex the task previously seemed. We aim to teach you how to script and what to consider, to complement the clear-cut patterns that you can use in your daily scripting challenges.
What you will learn
- Understand Linux and Bash basics as well as shell scripting fundamentals
- Learn to write simple shell scripts that interact with Linux operating system
- Build, maintain, and deploy scripts in a Linux environment
- Learn best practices for writing shell scripts
- Avoid common pitfalls associated with Bash scripting
- Gain experience and the right toolset to write your own complex shell scripts
Who this book is for
This book targets new and existing Linux system administrators, Windows system administrators or developers who are interested in automating administrative tasks. No prior shell scripting experience is needed but in case you do this book will make a pro quickly. Readers should have a basic understanding of the command line.
Sebastiaan Tammer is a Linux enthusiast from the Netherlands. After attaining his BSc in information sciences, he graduated with an MSc in business informatics, both from Utrecht University. His professional career started in Java development, before he pivoted into a Linux opportunity. Because of this dual background, he feels most at home in a DevOps environment. Besides working extensively with technologies such as Puppet, Chef, Docker, and Kubernetes, he has also attained the RHCE and OSCP certificates. He spends a lot of time in and around Bash. Whether it is creating complex scripting solutions or just automating simple tasks, there is hardly anything he hasn't done with Bash!Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 625
Veröffentlichungsjahr: 2018
Ähnliche
Learn Linux Shell Scripting – Fundamentals of Bash 4.4
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Vijin BorichaAcquisition Editor:Rohit RajkumarContent Development Editor:Abhishek JadhavTechnical Editor:Prachi SawantCopy Editor:Safis EditingProject Coordinator:Jagdish PrabhuProofreader: Safis EditingIndexer:Priyanka DhadkeGraphics:Tom ScariaProduction Coordinator: Tom Scaria
First published: December 2018
Production reference: 1281218
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78899-559-7
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the author
Sebastiaan Tammer is a Linux enthusiast from the Netherlands. After attaining his BSc in information sciences, he graduated with an MSc in business informatics, both from Utrecht University. His professional career started in Java development, before he pivoted into a Linux opportunity. Because of this dual background, he feels most at home in a DevOps environment.
Besides working extensively with technologies such as Puppet, Chef, Docker, and Kubernetes, he has also attained the RHCE and OSCP certificates. He spends a lot of time in and around Bash. Whether it is creating complex scripting solutions or just automating simple tasks, there is hardly anything he hasn't done with Bash!
About the reviewer
Heathe Kyle Yeakley holds degrees in technical communications and network management. He began his IT career in 1999 doing entry-level help desk support. In 2003, he began his first enterprise data center job performing tape backup and recovery for the United States Federal Aviation Administration. He worked in the aerospace sector for several years, during which time he worked on a wide range of products, including HP Tru64 Unix, Red Hat Enterprise Linux, Solaris 10, Legato Networker, Symantec NetBackup, HP and NetApp storage arrays, Spectra Logic and ADIC tape libraries, VMware, and HP Blade servers.
He currently works for Agio, where he and his coworkers deliver managed IT services to some of the world's most prestigious companies.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Learn Linux Shell Scripting – Fundamentals of Bash 4.4
About Packt
Why subscribe?
PacktPub.com
Contributors
About the author
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Disclaimer
Introduction
What is Linux?
What is Bash?
Summary
Setting Up Your Local Environment
Technical requirements
Choosing between a virtual machine and a physical installation
Setting up VirtualBox
Creating an Ubuntu virtual machine
Creating the virtual machine in VirtualBox
Installing Ubuntu on the virtual machine
Accessing the virtual machine via SSH
Summary
Questions
Further reading
Choosing the Right Tools
Technical requirements
Using graphical editors for shell scripting
Atom
Atom installation and configuration
Notepad++
Using command-line editors
Vim
Vim summary
.vimrc
Vim cheat sheet
nano
Combining graphical editors with command-line editors when writing shell scripts
Summary
Questions
Further reading
The Linux Filesystem
Technical requirements
The Linux filesystem explained
What is a filesystem?
What makes the Linux filesystem unique?
Structure of the Linux filesystem
Tree structure
Overview of top-level directories
What about multiple partitions?
/bin/, /sbin/, and /usr/
/etc/
/opt/, /tmp/, and /var/
Everything is a file
Different types of files
Summary
Questions
Further reading
Understanding the Linux Permissions Scheme
Technical requirements
Read, write, and execute
RWX
Users, groups, and others
Manipulating file permissions and ownership
chmod, umask
sudo, chown, and chgrp
sudo
chown, chgrp
Working with multiple users
Advanced permissions
File attributes
Special file permissions
Access Control Lists (ACLs)
Summary
Questions
Further reading
File Manipulation
Technical requirements
Common file operations
Copying
Removing
Renaming, moving, and linking
Archiving
Finding files
locate
find
Summary
Questions
Further reading
Hello World!
Technical requirements
First steps
The shebang
Running scripts
Readability
Comments
Script header
Verbosity
Verbosity in comments
Verbosity of commands
Verbosity of command output
Keep It Simple, Stupid (KISS)
Summary
Questions
Further reading
Variables and User Input
Technical requirements
What is a variable?
Why do we need variables?
Variables or constants?
Variable naming
Dealing with user input
Basic input
Parameters and arguments
Interactive versus non-interactive scripts
Combining positional arguments and read
Summary
Questions
Further reading
Error Checking and Handling
Technical requirements
Error checking
Exit status
Functional checks
Test shorthand
Variable refresher
Bash debugging
Error handling
if-then-exit
if-then-else
Shorthand syntax
Error prevention
Checking arguments
Managing absolute and relative paths
Dealing with y/n
Summary
Questions
Further reading
Regular Expressions
Technical requirements
Introducing regular expressions
What is a regular expression?
grep
Greediness
Character matching
Line anchors
Character classes
Globbing
What is globbing?
Similarities with regular expressions
More globbing
Advanced globbing
Disabling globbing, and other options
Using regular expressions with egrep and sed
Advanced grep
Introducing egrep
sed, the stream editor
Stream editing
In-place editing
Line manipulation
Final remarks
Summary
Questions
Further reading
Conditional Testing and Scripting Loops
Technical requirements
Advanced if-then-else
A recap on if-then-else 
Using regular expressions in tests
The elif condition
Nesting
Getting help
The while loop
The until loop
Creating an interactive while loop
The for loop
Globbing and the for loop
Loop control
Breaking the loop
The continue keyword
Loop control and nesting
Summary
Questions
Further reading
Using Pipes and Redirection in Scripts
Technical requirements
Input/output redirection
File descriptors
Redirecting output
stdout
stderr
Redirect all output
Special output redirection
/dev/null
/dev/zero
Input redirection
Generating a password
Advanced redirecting
Redirecting redirections
Command substitution
Process substitution
Pipes
Binding stdout to stdin
Practical examples
Yet another password generator
Setting passwords in a script
tee
Here documents
Heredocs and variables
Using heredocs for script input
Here strings
Summary
Questions
Further reading
Functions
Technical requirements
Functions explained
Hello world!
More complexity
Variable scopes
Practical examples
Error handling
Augmenting functions with parameters
Colorful
Returning values
Function libraries
Source
More practical examples
Current working directory
Type checking
Yes-no check
Summary
Questions
Further reading
Scheduling and Logging
Technical requirements
Scheduling with at and cron
at
Time syntax
The at queue
at output
cron
crontab
Syntax for the crontab
Logging script results
Crontab environment variables
PATH
SHELL
MAILTO
Logging with redirection
Final logging considerations
A note on verbosity
Summary
Questions
Further reading
Parsing Bash Script Arguments with getopts
Technical requirements
Positional parameters versus flags
Using flags on the command line
The getopts shell builtin
The getopts syntax
Multiple flags
Flags with arguments
Combining flags with positional arguments
Summary
Questions
Further reading
Bash Parameter Substitution and Expansion
Technical requirements
Parameter expansion
Parameter substitutions – recap
Default values
Input checking
Parameter length
Variable manipulation
Pattern substitution
Pattern removal
Case modification
Substring expansion
Summary
Questions
Further reading
Tips and Tricks with Cheat Sheet
Technical requirements
General tips and tricks
Arrays
The history command
Creating your own aliases
Command-line shortcuts
Fun with exclamation marks
Running commands from the history
Keyboard shortcuts
Copying and pasting from the terminal
Reverse search
Cheat sheet for interactive commands
Navigation
cd
ls
pwd
File manipulation
cat
less
touch
mkdir
cp
rm
mv
ln
head
tail
Permissions and ownership
chmod
umask
chown
chgrp
sudo
su
useradd
groupadd
usermod
Summary
Final words
Assessments
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Chapter 15
Chapter 16
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
Shell scripts allow us to program commands in chains and have the system execute them as a scripted event, just like batch files.This book will start with an overview of Linux and Bash shell scripting, and then quickly deep dive into helping you set up your local environment, before introducing you to tools that are used to write shell scripts. The next set of chapters will focus on helping you understand Linux under the hood and what Bash provides the user. Soon, you will have embarked on your journey along the command line. You will now begin writing actual scripts instead of commands, and will be introduced to practical applications for scripts. The final set of chapters will deep dive into the more advanced topics in shell scripting. These advanced topics will take you from simple scripts to reusable, valuable programs that exist in the real world. The final chapter will leave you with some handy tips and tricks and, as regards the most frequently used commands, a cheat sheet containing the most interesting flags and options will also be provided.
After completing this book, you should feel confident about starting your own shell scripting projects, no matter how simple or complex the task previously seemed. We aim to teach you how to script and what to consider, to complement the clear-cut patterns that you can use in your daily scripting challenges.
Who this book is for
This book targets new and existing Linux system administrators, as well as Windows system administrators or developers who are interested in automating administrative tasks. No prior shell scripting experience is required, but if you do possess some experience, this book will quickly turn you into a pro. Readers should have a (very) basic understanding of the command line.
What this book covers
Chapter 1, Introduction, primes you for the remainder of the book. Aided by some background in Linux and Bash, you should be better able to understand how and why shell scripting can provide clear benefits to you.
Chapter 2, Setting Up Your Local Environment, helps you to prepare your local machine for the examples and exercises throughout the rest of the book. You will be shown how to set up an Ubuntu 18.04 Linux virtual machine on your local machine, using VirtualBox. This virtual machine will be used to write, run, and debug commands and scripts in this book.
Chapter 3, Choosing the Right Tools, introduces you to the tools that will be used to write shell scripts. Two different kinds of tools will be described: IDE editors (Atom, Notepad++), and terminal-based editors (vim and nano). You will be encouraged to initially write scripts in an IDE, and troubleshoot scripts in a terminal-based editor, to most resemble real-world use.
Chapter 4, The Linux Filesystem, coves how the Linux filesystem is organized by exploring the virtual machine created in the previous chapters. You will achieve this by performing your first command-line actions, such as cd, pwd, and ls. Context regarding the different structures will be provided so that you can use this information when writing scripts. And, most importantly, the concept of everything is a file will be explained.
Chapter 5, Understanding the Linux Permissions Scheme, gets you acquainted with permissions under Linux, once again by exploring the virtual machine. Commands such as sudo, chmod, and chown will be used to interactively learn about file and directory privileges. The skills acquired in this chapter will be heavily used in shell scripting, so it is imperative that you gain exposure to both the successful execution of commands as well as failure messages.
Chapter 6, File Manipulation, introduces you to the most relevant file manipulation commands, including the most commonly used flags and modifiers for those commands. This will be achieved by means of commands inside the virtual machine.
Chapter 7, Hello World!, educates you in terms of thinking ahead and developing good habits when it comes to writing scripts. You will write your first actual shell script during this chapter.
Chapter 8, Variables and User Input, introduces you to variables and user input. You will see how parameters are used by Bash, and where the differences lie between parameters and arguments. User input will be handled and used to produce new functions in your scripts. Finally, the difference between interactive and non-interactive scripts will be clarified and discussed.
Chapter 9, Error Checking and Handling, gets you familiar with (user) input, and error checking and handling. Introducing user input into a script is bound to result in more errors, unless the script specifically deals with the possibility of users submitting incorrect or unexpected input. You will learn how to best deal with this.
Chapter 10, Regular Expressions, gets you familiar with regular expressions, which are often used in shell scripting. The most common patterns and uses for these regular expressions will be presented. Basic usage of sed will be covered in this chapter, complementing regular expression explanations.
Chapter 11, Conditional Testing and Scripting Loops, discusses the different kind of loops and the relevant control structures that are used in shell scripting with Bash.
Chapter 12, Using Pipes and Redirection in Scripts, introduces you to redirection on Linux. This chapter will start with the basic input/output redirection, before continuing to stream redirection and pipes.
Chapter 13, Functions, introduces you to functions. Functions will be presented as blocks of code that are grouped together in such a way that they can be reused, often with different arguments, to produce a slightly different end result. You will learn to understand the benefit to reusing code, and planning scripts accordingly.
Chapter 14, Scheduling and Logging, teaches you how to schedule scripts and how to make sure these scheduled scripts perform the task they were intended for, by using the crontab and the at command, coupled with proper logging.
Chapter 15, Parsing Bash Script Arguments withgetopts, helps you to improve your scripts by adding flags instead of positional parameters, thereby making the scripts much easier to use.
Chapter 16, Bash Parameter Substitution and Expansion, shows how previous patterns used in earlier scripts can be optimized by means of parameter expansion, substitution, and variable manipulation.
Chapter 17, Tips and Tricks with Cheat Sheet, provides you with some handy tips and tricks that are not necessarily used in Bash scripts, but that are very convenient when working on the terminal. For the most frequently used commands, a cheat sheet containing the most interesting flags and options will be provided so that you can use this chapter as reference while scripting.
To get the most out of this book
You will require an Ubuntu 18.04 Linux virtual machine to follow along with the book. We will guide you through setting this up during in the second chapter. You will only truly learn shell scripting if you follow along with all code examples. The entire book has been written with this in mind, so be sure to follow this advice!
Download the example code files
You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packtpub.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Learn-Linux-Shell-Scripting-Fundamentals-of-Bash-4.4. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://www.packtpub.com/sites/default/files/downloads/9781788995597_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Let's try to copy the /tmp/ directory into our home directory."
A block of code is set as follows:
#!/bin/bashecho "Hello World!"
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
reader@ubuntu:~/scripts/chapter_10$ grep '
use
' grep-file.txt We can
use
this regular file for testing grep.but in the USA they
use
color (and realize)!
Any command-line input or output is written as follows:
reader@ubuntu:~/scripts/chapter_10$ grep 'e.e' character-class.txt eeee2ee e
Bold: Indicates a new term, an important word, or words that you see on screen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Click the Install button and watch the installation."
Get in touch
Feedback from our readers is always welcome.
General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.
Disclaimer
The information within this book is intended to be used only in an ethical manner. Do not use any information from the book if you do not have written permission from the owner of the equipment. If you perform illegal actions, you are likely to be arrested and prosecuted to the full extent of the law. Packt Publishing does not take any responsibility if you misuse any of the information contained within the book. The information herein must only be used while testing environments with proper written authorizations from appropriate persons responsible.
Introduction
Before we start writing shell scripts, we need to have some context about our two most relevant components: Linux and Bash. We'll give an explanation of Linux and Bash, look into the history behind the two technologies, and discuss the current state of both.
The following topics will be covered in this chapter:
What is Linux?
What is Bash?
What is Linux?
Linux is a generic term that refers to different open source operating systems that are based on the Linux kernel. The Linux kernel was originally created by Linus Torvalds in 1991, and open sourced in 1996. A kernel is a piece of software that is designed to act as an intermediate layer between low-level hardware (such as the processor, memory, and input/output devices) and high-level software, such as an operating system. Apart from the Linux kernel, most Linux operating systems rely heavily on GNU project utilities; for example, the Bash shell is a GNU program. Because of this, Linux operating systems are referred to by some as GNU/Linux. The GNU project, where GNU stands for GNU's Not Unix! (a recursive acronym), is a collection of free software, a lot of which is found in most Linux distributions. This collection includes many tools, but also an alternative kernel called GNU HURD (which is not as widespread as the Linux kernel).
Why do we need a kernel? Since a kernel sits between hardware and the operating system, it provides an abstraction for interacting with hardware. This is why the Linux ecosystem has grown so large: the kernel can be used freely, and it handles a lot of low-level operations on many types of hardware. Creators of an operating system can therefore spend their time making an easy-to-use, beautiful experience for their users, instead of having to worry about how the users' pictures are going to be written to the physical disk(s) attached to the system.
The Linux kernel is so-called Unix-like software. As you might suspect, this implies that it is similar to the original Unix kernel, which was created between 1971 and 1973 at Bell Labs, by Ken Thompson and Dennis Ritchie. However, the Linux kernel is only based on Unix principles and does not share code with Unix systems. Famous Unix systems include the BSDs (FreeBSD, OpenBSD, and so on) and macOS.
Linux operating systems are broadly used for one of two purposes: as a desktop or as a server. As a desktop, Linux can serve as a replacement for the more commonly used Microsoft Windows or macOS. However, most Linux usage is accounted for the server landscape. At the time of writing, it is estimated that around 70% of all servers on the internet use a Unix or Unix-like operating system. The next time you're browsing the news, reading your mail, or are scrolling through your favorite social media website, remember that there's a big chance the pages you are being shown have been processed by one or more Linux servers.
There are many distributions, or flavors, of Linux. Most Linux operating systems fall within distribution families. A distribution family is based on a common ancestor, and often use the same package management tools. One of the more well-known Linux distributions, Ubuntu, is based on the Debian distribution family. Another prominent Linux distribution, Fedora, is based on the Red Hat family. Other notable distribution families include SUSE, Gentoo, and Arch.
Not many people realize how many devices run the Linux kernel. For example, the most common smartphone operating system in use today, Android (with a market share of around 85%), uses a modified version of the Linux kernel. The same goes for many smart TVs, routers, modems, and various other (embedded) devices. If we were to include Unix and other Unix-like software, we can safely say that most of the devices in the world run on these kernels!
What is Bash?
The most commonly used shell in Linux systems is the Bourne-again shell, or Bash. The Bash shell is based on the Bourne shell, known as sh. But what is a shell?
A shell is, in essence, a user interface. Most often, it is used to refer to a text-based interface, also called a command-line interface (CLI). However, it is called a shell because it can be seen as a shell around the kernel; this means that it applies not just to CLIs, but just as well to graphical user interfaces (GUIs). When we refer to a shell in this book, we are talking about a CLI, and unless stating differently, we're talking about the Bash shell.
The purpose of a shell, both CLI and GUI, is to allow the user to interact with the system. After all, a system that does not offer interaction would be hard to justify, not too mention hard to use! Interaction in this context means many things: typing on your keyboard will make letters appear on your screen, moving your mouse will change the location of the cursor on screen, giving the command to delete a file (either with a CLI or GUI) will remove the bytes from the disk, and so on.
In the earliest days of Unix and computers, no GUIs were available, so all work was performed via a CLI. To connect to the shell on a running machine, a video terminal was used: often this would be a very simple monitor combined with a keyboard, which was connected with, for example, a RS-232 serial cable. Commands entered on this video terminal were processed by the shell running on the Unix machine.
Luckily for us, things have changed quite a bit since the first computers. Today, we no longer use dedicated hardware to connect to shells. A piece of software running in a GUI, a terminal emulator, is used for interaction with the shell. Lets take a quick look at how connecting to a Bash shell with a terminal emulator can look:
In the preceding screenshot, we're connected to a Linux virtual machine (we'll be setting this up in the next chapter), using a terminal emulator (GNOME Terminal) via the Secure Shell (SSH) protocol. A few interesting things to note:
We're on a CLI interface; we do not have access to, nor do we need, a mouse
We're connected to an Ubuntu machine, but we're running this within another operating system (Arch Linux, in this case)
We're presented with a welcome message by Ubuntu 18.04, showing some general information about the system
Besides using the Bash shell for direct system interaction, it provides another important functionality: the ability to execute multiple commands sequentially, with or without user interaction, tailored to a specific goal. This might sound complicated, but it's actually pretty simple: we're talking about Bash scripts, the subject of this book!
Summary
In this chapter, you've read about the GNU/Linux operating systems and the Linux kernel, what a kernel really is, and how big an impact Linux distributions have on daily life. You've also learned what a shell is, and that the most common Linux shell, Bash, can be both used to interact with a Linux system, and is also utilized to write shell scripts.
In the next chapter, we'll set up a local environment which we will use throughout the rest of the book, in both the examples and exercises.
Setting Up Your Local Environment
In the previous chapter, we ventured into some context for the wonderful world of Linux and Bash. Since this is a practical, exercise-driven book, we're going to use this chapter to set up a machine where you can follow along with the examples and perform the exercises at the end of each chapter. This can either be a virtual machine or a physical installation; that is up to you. We will discuss this in the first part of this chapter, before continuing with the installation of VirtualBox and, finally, creating an Ubuntu virtual machine.
The following command will be introduced in this chapter: ssh and exit.
The following topics will be covered in this chapter:
Choosing between a virtual machine and a physical installation
Setting up VirtualBox
Creating an Ubuntu virtual machine
Technical requirements
To complete the exercises in this chapter (and the following chapters), you will need either a PC or laptop with at least 2 GHz of CPU power, 10 GB of hard disk space, and about 1 GB of RAM to spare. Pretty much all hardware created in the last 5 years should suffice.
Choosing between a virtual machine and a physical installation
A virtual machine is an emulation of a physical machine. This means it runs inside a physical machine, as opposed to directlyon the hardware. A physical machine has direct access to all hardware, such as the CPU, the RAM, and other devices such as the mouse, the keyboard, and the monitor. It is, however, impossible to share the CPU or the RAM between multiple physical machines. Virtual machines do not directly get access to hardware, but through an emulation layer, which means resources can be shared between multiple virtual machines.
Because we're discussing Bash shell scripting in general, in theory it does not matter what kind of an installation is performed. As long as that installation runs a compatible Linux operating system with Bash 4.4 or later, all exercises should work. There are, however, many advantages to using a virtual machine over a physical installation:
There is no need to remove your current preferred operating system, or set up a complicated dual-boot configuration
Virtual machines can be snapshotted, which allows recovery from critical failures
You are able to run (many) different operating systems on a single machine
Unfortunately, there are also drawbacks associated with virtual machine use:
Because you're running a virtual operating system within an already running operating system, there is some overhead from the virtualization (in comparison to running a bare-metal installation)
Since you're running multiple operating systems at the same time, you will need more resources than with a bare-metal installation
In our opinion, modern computers are fast enough to make the drawbacks almost trivial, while the advantages provided by running Linux in a virtual machine are very helpful. Because of this, we will only be explaining a virtual machine setup in the rest of this chapter. If you feel confident enough to run Linux as a physical installation (or perhaps you already have Linux running somewhere!), feel free to explore the rest of the book with that machine.
If you want to be sure all examples and exercises work as seen in this book, run an Ubuntu 18.04 LTS virtual machine in VirtualBox with the recommended specifications of 1 CPU, 1 GB RAM, and a 10 GB hard disk: this setup is described in the rest of this chapter. Even if many other types of deployment should work, you would not want to be banging your head against the wall for hours when an exercise is not working, before discovering it was caused by your setup.
Setting up VirtualBox
To use virtual machines, we need software called a hypervisor. A hypervisor manages resources between the host machine and the virtual machines, provides access to disks, and has an interface to manage it all. There are two different types of hypervisors: type-1 and type-2. Type-1 hypervisors are the so-called bare-metal hypervisors. These are installed instead of a regular operating system such as Linux, macOS, or Windows, directly on the hardware. These types of hypervisors are used for corporate servers, cloud services, and so on. For this book, we will use a type-2 hypervisor (also called hosted hypervisors): these are installed within another operating system, as a piece of software not much different than, for example, a browser.
There are many type-2 hypervisors. The most popular choices at the time of writing are VirtualBox, VMware workstation player, or OS-specific variants such as QEMU/KVM on Linux, Parallels Desktop on macOS, and Hyper-V on Windows. Because we are going to use a virtual machine throughout this book, we do not assume anything about the host machine: you should work comfortably with whatever operating system you prefer. Because of this, we've chosen to use VirtualBox as our hypervisor, since it runs on Linux, macOS, and Windows (and even others!). Furthermore, VirtualBox is free and open source software, which means you can just download and use it.
Presently, VirtualBox is owned by Oracle. You can download the installer for VirtualBox from https://www.virtualbox.org/. Installation should not be hard; follow the instructions by the installer.
Creating an Ubuntu virtual machine
In this book, we're scripting with Bash, which means we do not need a GUI for our Linux installation. We have chosen to use Ubuntu Server 18.04 LTS as the virtual machine operating system, for a number of reasons:
Ubuntu is considered a beginner-friendly Linux distribution
The 18.04 is a
Long-Term Support
(
LTS
) release, which means it will receive updates until April 2023
Because an Ubuntu server offers a CLI-only installation, it is easy on system resources and representative of real-life servers
At the time of writing, Ubuntu is maintained by Canonical. You can download the ISO image from https://www.ubuntu.com/download/server. Download the file now, and remember where you save this file, since you'll need it soon.
Creating the virtual machine in VirtualBox
First, we will start with creating the virtual machine to host our Ubuntu installation:
Open VirtualBox and choose
Machine
|
New
in the menu toolbar.
For reference, we have circled the
Machine
entry in the menu toolbar given in the following screenshot. Choose a name for the virtual machine (this can be a different name than the server name, but we like to keep it the same for simplicity), set the
Type
to
Linux
, and
Version
to
Ubuntu (64-bit)
. Click
Next
:
On this screen, we determine memory settings. For most servers, 1024 MB of RAM is a great start (and is recommended by VirtualBox for virtual machines as well). If you have beefy hardware, this can be set to 2048 MB, but 1024 MB should be fine. Make your selection and press
Next
:
Once again, the recommended values by VirtualBox are perfect for our needs. Press
Create
to start the creation of the virtual hard disk:
Virtual hard disks can be many different types. VirtualBox defaults to its own format,
VDI
, as opposed to
VMDK
, which is the format used by VMware (another popular virtualization provider). The last option is
VHD (Virtual Hard Disk)
, which is a more generic format usable by multiple virtualization providers. Since we'll be using VirtualBox exclusively in this book, keep the selection on
VDI
(VirtualBox Disk Image)
and press
Next
:
We have two options on this screen: we can allocate the full virtual hard disk on the physical hard disk right away, or we can use dynamic allocation, which does not reserve the full size of the virtual disk, but only what is used.
The difference between these options is often most relevant in situations where many virtual machines are running on a single host. Creating a total of disks larger than what is physically available, but assuming not all disks will be fully used, allows us to place more virtual machines on a single machine. This is called overprovisioning, and will only work if not all disks are filled up (because we can never have more virtual disk space than we have physical disk space). For us, this distinction does not matter since we'll be running a single virtual machine; we keep the default of Dynamically allocated and go to the next screen:
On this screen, we can do three things: name the virtual disk file, select the location, and specify the size. If you care about the location (it defaults to somewhere in your
home
/
user
directory), you can press the circled icon in the following screenshot. For the name, we like to keep it the same as the virtual machine name. Lastly, a size of 10 GB is sufficient for the exercises in this book. After you've set up the three values, press
Create
. Congratulations, you've just created your first virtual machine, as demonstrated in following screenshot:
However, before we can start the installation of Ubuntu on our virtual machine, we need to do two more things: point the virtual machine to the installation ISO, and set up the networking. Select the newly created virtual machine and click
Settings
. Navigate to the
Storage
section:
You should see a disk icon with the word Empty (in the location circled on the left in the preceding screenshot). Select it and mount an ISO file by clicking the select disk icon (circled on the right), choose virtual optical disk file, and then select the Ubuntu ISO that you downloaded earlier. If you do this correctly, your screen should resemble the preceding screenshot: you no longer see the word Empty next to the disk icon and the Information section should be filled in.
Once you have verified this, go to the
Network
section.
The configuration should default to the
NAT
type. If not, set it to
NAT
now.
NAT
stands for
Network Address Translation
. In this mode, the host machine acts as a router for the virtual machines. Finally, we're going to set up some port forwarding so we can use SSH tooling later on. Click on the
Port Forwarding
button:
Set up the SSH rule just as we have done. This means that port
22
on the guest (which is the virtual machine) is exposed as port
2222
on the host (which is, surprise, the host machine). We've chosen port
2222
for two reasons: ports lower than 1024 require root/administrator permissions, which we might not have. Secondly, there is a chance an SSH process is already listening on the host machine, which would mean VirtualBox would not be able to use that port:
With this step, we've finished setting up the virtual machine!
Installing Ubuntu on the virtual machine
Now you can start your virtual machine from the VirtualBox main screen. Right click on the machine, select Start followed by Normal Start. If all goes well, a new window will pop up, showing you the virtual machine console. After a while, you should see the Ubuntu server installation screen in that window:
On the screen shown in the following screenshot, select your favorite language using the arrow keys (we're using
English
, so if you're not sure, English is a good choice) and press
Enter
:
Select the keyboard layout that you're using. If you're unsure, you can use the interactive
Identify keyboard
option to determine which layout is best for you. Once the proper layout is set, move the focus to
Done
and press
Enter
:
We now choose the type of installation. Because we're using the server ISO, we do not see any options related to the GUI. In the preceding screenshot, select
Install Ubuntu
(both other options use Canonical's
Metal-As-A-Server
(
MAAS
) cloud offering, which is not relevant for us) and press
Enter
:
You will see the
Network connections
screen. The installer should default to using DHCP on the default network interface that was created with the virtual machine. Verify that an IP has been allocated to this interface, and press
Enter
:
The
Configure proxy
screen is not relevant to us (unless you're running with a proxy setup, but there's a good chance you do not need our assistance with the installation in that case!). Leave the
Proxy address
blank and press
Enter
:
Sometimes it's helpful to manually partition your Linux disk to fit specific needs. In our case, the default value of
Use An Entire Disk
is a great fit, so press
Enter
:
After having selected that we want to use an entire disk, we need to specify which disk to use. Since we only created one disk when we configured the virtual machine, select it and press
Enter
.
Now you will encounter a warning about performing a destructive action. Because we are using an entire (virtual!) disk, all information present on that disk will be erased. We created this disk when we created the virtual machine, so it does not contain any data. We can safely perform this action, so select
Continue
and press
Enter
:
For the
Filesystem setup
, once again the default values are perfect for our needs. Verify that we have at least 10 GB of hard disk space (it might be a little less, like 9.997 GB in the following example: this is fine) and press
Enter
:
The Ubuntu server should now start installing to the virtual disk. In this step, we'll be setting the server name and creating an administrative user. We've chosen the server name
ubuntu
, the username
reader
, and the password
password
. Note that this is a
very weak
password that we will only use on this server for simplicity. This is acceptable because the server will only ever be accessible from our host machine. When configuring a server that accepts incoming traffic from the internet, never use a password as weak as this! Choose anything you like, as long as you can remember it. If you're unsure, we'd advise using the same values of
ubuntu
,
reader
, and
password
:
Now that you've chosen a server name and configured an administrative user, press Enter to finalize the installation.
Depending on how long it took to complete the previous screen and how fast the host machine is, Ubuntu will either still be installing or will have finished already. If you still see text moving around the screen, the installation is still running. Once the installation is completely finished, you will see the
Reboot Now
button appear. Press
Enter
:
After a few seconds, a message stating Please remove the installation medium, then press Enter should appear. Follow the instructions and, if all goes well, you should be greeted with a terminal login prompt:
Now the moment of truth: try to log in with your created username and password. If successful, you should see a screen similar to the following:
Give yourself a pat on the back: you have just created a virtual machine, installed Ubuntu Server 18.04 LTS, and logged in via the Terminal console. Well done! To exit, type exit or logout and press Enter.
Accessing the virtual machine via SSH
We have successfully connected to the Terminal console provided to us by VirtualBox. However, this Terminal connection is really basic: for example, we can't scroll up, we can't paste copied text, and we do not have colored syntax highlighting. Fortunately for us, we have a nice alternative: the Secure Shell (SSH) protocol. SSH is used to connect to the running shell on the virtual machine. Normally, this would be done over the network: this is how enterprises maintain their Linux servers. In our setup, we can actually use SSH within our host machine, using the power forwarding we set up earlier.
If you followed the installation guide, port 2222 on the host machine should be redirected to port 22 on the virtual machine, the port where the SSH process is running. From a Linux or macOS host machine, we can connect using the following command (substitute the username or port number if necessary):
$ ssh reader@localhost -p 2222
However, there is a good chance you're running Windows. In that case, you will probably not have access to a native SSH client application within Command Prompt. Luckily, there are many good (and free!) SSH clients. The simplest and most well-known client is PuTTY. PuTTY was created in 1999 and, while it's definitely a very stable client, its age is starting to show. We would recommend some newer SSH client software, such as MobaXterm. This provides you with more session management, a better GUI, and even a local Command Prompt!
Whichever software you decide on, be sure to use the following values (again, change the port or username if you deviated from the installation guide):
Host Name
:
localhost
Port
:
2222
User Name
:
reader
Summary
In this chapter, we have started preparing our local machine for the rest of the book. We now know about the differences between virtual and physical machines, and why we prefer to use a virtual machine for the remainder of this book. We've learned about the two different types of hypervisors. We have installed and configured VirtualBox with a virtual machine, on which we have installed the Ubuntu 18.04 operating system. Finally, we have connected to our running virtual machine using SSH instead of the VirtualBox terminal, which affords better usability and options.
The following command was introduced in this chapter: ssh and exit.
In the next chapter, we will finish setting up our local machine by looking at some different tools we can use that will help us with bash scripting, both on the GUI and on the virtual machine CLI.
Questions
What are some of the reasons running a virtual machine would be preferable to a bare-metal installation?
What are some of the downsides of running a virtual machine as opposed to a bare-metal installation?
What is the difference between a type-1 and type-2 hypervisor?
In which two ways can we start a virtual machine on VirtualBox?
What makes an Ubuntu LTS version special?
What should we do if, after the Ubuntu installation, the virtual machine boots to the Ubuntu installation screens again?
What should we do if we accidentally reboot during installation, and we never end up at the Ubuntu installation (but instead see an error)?
Why did we set up NAT forwarding for the virtual machine?
Further reading
The following resources might be interesting if you'd like to go deeper into the subjects of this chapter:
Getting Started with Oracle VM VirtualBox
by Pradyumna Dash, Packt:
https://www.packtpub.com/virtualization-and-cloud/getting-started-oracle-vm-virtualbox
Mastering Ubuntu Server – Second Edition
by Jay LaCroix, Packt:
https://www.packtpub.com/networking-and-servers/mastering-ubuntu-server-second-edition
Choosing the Right Tools
This chapter will introduce some tools that will help us when we're writing Bash scripts. We will focus on two types of tools: GUI-based editors (Atom and Notepad++) and Terminal-based editors (Vim and nano). We will describe the tools and how to use them, their strengths and weaknesses, and how to use both GUI- and Terminal-based editors together for the best results.
The following commands will be introduced in this chapter: vim, nano, and ls.
The following topics will be covered in this chapter:
Using graphical editors for shell scripting
Using command-line editors for shell scripting
Combining graphical editors with command-line editors when writing shell scripts
Technical requirements
You will need the virtual machine we created in the previous chapter when using Vim or nano. If you want to use Notepad++, you will need a Windows host machine. For Atom, the host machine can run either Linux, macOS, or Windows.
Using graphical editors for shell scripting
Tooling has come a long way since the first Unix and Unix-like distributions. In the earliest days, writing shell scripts was significantly harder than today: the shells were less powerful, text editors were command-line only and things such as syntax highlighting and autocomplete were non-existent. Today, we have very powerful GUI editors that will help us in our scripting adventures. Why would we want to wait until we run a script to find an error, when a GUI editor could have already shown us the error in advance? Today, using an advanced editor for shell scripting is almost a necessity that we wouldn't want to live without.
We'll describe two text editors in the coming pages: Atom and Notepad++. Both are GUI-based, which we can use for efficient shell scripting. If you have a preference for either already, pick that one. If you're unsure, we would recommend using Atom.
Atom
The first graphical editor we will consider is Atom, made by GitHub. It's described as A hackable text editor for the 21st Century. Hackable, in this sense, means that while the default installation of Atom is as complete as any text editor, this application really shines because it is very configurable and extensible. Anything that has not been integrated by GitHub can be written as an extension package. By using these extensions, you can make your Atom installation fully your own; if you do not like something, change it. If it can't be changed out of the box, find a package that does it. And even if there's not a package that does what you're hoping, you still have the option to create your own package!
Another nice feature of Atom is the default integration with Git and GitHub. Git is currently the most popular version control system. Version control systems are used when writing code or scripts. They ensure that history of files is preserved, and make it possible for multiple, even many, contributors to work on the same files at the same time, without getting burdened down by conflict management. GitHub, as the name suggests, is currently the most prominent web-based Git provider for open source software.
The last great thing about Atom we want to mention is that, by default, it supports many scripting and programming languages. When we say supports, we mean that it can recognize file types by their extensions, and offer syntax highlighting (which makes for much easier scripting!). This functionality is provided through core packages, which work the same way as normal packages but are included from the start. For our purposes, the core package, language-shellscript, will help us in our shell scripting endeavors.
Atom installation and configuration
Let's go ahead and install Atom. As long as you're running Linux, macOS, or Windows, you can go to https://atom.io/ and grab the installer. Run the installer and, if necessary, follow along with the prompts until Atom is installed. Now, start Atom and you'll be greeted by the welcome screen, which at the time of writing looks like the following:
Be sure to look at all the screens Atom has to offer. When you feel like you've explored enough, let's add a package to Atom that will complement our shell scripting. If you have the Welcome Guide screen still open, select Install a Package from there. Otherwise, you can use the keyboard shortcut Ctrl + , to bring up the Settings screen. You will see an Install option there, which will take you to the Install Packages screen. Search for bash, and you should see the following package:
Click the Install button and watch the installation. You might be prompted to reboot Atom after the install; be sure to do so. If you're not prompted but see errors of any kind, a reboot of Atom is never a bad idea. After installing the package, you will now have autocomplete functionality when writing shell scripts. This means that you can start typing and Atom will try to predict what you want, in the following manner:
On the right-hand side, you can see we started typing the echo shell command, and after the first two letters, Atom presented us with two options that contain those two letters. Once it makes a suggestion, we can press Enter and the command is inserted fully. While it will not save much time in this instance, it can be great for two main reasons:
If you're unsure what the command is called exactly, you might be able to find it with autocomplete.
Once you start writing conditionals and loops (in the second part of this book), the autocomplete will span multiple lines, saving you from typing many words and remembering all the syntax.
Finally, let's look at how Atom looks when you've got a Git project open and are working on files:
When working in Atom, the screen will mostly look like this. On the left-hand side, you'll see the Tree View, which you can toggle on/off by pressing Ctrl + \. The Tree View contains all the files in your current project (which is the directory you've opened). All these files can be opened by double-clicking them, which causes them to appear in the middle: the Editor View. This is where you'll spend most of your time, working on the shell scripts. The Editor View will always be visible, even if there are currently no files open.
By default, there is one last view, the Git View, located on the right-hand side. This view can be toggled by pressing Ctrl +Shift + 9. The code for this book is hosted on GitHub, which you will download (or, as Git calls it, clone) once, without the need to edit it on the remote server. Because of this, the Git View is not needed in this book, but we mention it since you will probably use it for other projects.
Notepad++
Where Atom is closer to an integrated development environment (IDE) than a text editor, Notepad++ is pretty much what the name implies: good old Notepad with some added features. Some of these added features include being able to have multiple files open at the same time, syntax highlighting, and limited autocomplete. It was initially released in 2003 and only works on Windows.
