Linux Administration Best Practices E-Book

Scott Alan Miller

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 850

Veröffentlichungsjahr: 2022

Beschreibung

Linux is a well-known, open source Unix-family operating system that is the most widely used OS today. Linux looks set for a bright future for decades to come, but system administration is rarely studied beyond learning rote tasks or following vendor guidelines. To truly excel at Linux administration, you need to understand how these systems work and learn to make strategic decisions regarding them.
Linux Administration Best Practices helps you to explore best practices for efficiently administering Linux systems and servers. This Linux book covers a wide variety of topics from installation and deployment through to managing permissions, with each topic beginning with an overview of the key concepts followed by practical examples of best practices and solutions. You'll find out how to approach system administration, Linux, and IT in general, put technology into proper business context, and rethink your approach to technical decision making. Finally, the book concludes by helping you to understand best practices for troubleshooting Linux systems and servers that'll enable you to grow in your career as well as in any aspect of IT and business.
By the end of this Linux administration book, you'll have gained the knowledge needed to take your Linux administration skills to the next level.

Leseprobe

Linux Administration Best Practices

Practical solutions to approaching the design and management of Linux systems

Scott Alan Miller

BIRMINGHAM—MUMBAI

Linux Administration Best Practices

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Rahul Nair

Publishing Product Manager: Rahul Nair

Senior Editor: Shazeen Iqbal

Content Development Editor: Rafiaa Khan

Technical Editor: Arjun Varma

Copy Editor: Safis Editing

Project Coordinator: Shagun Saini

Proofreader: Safis Editing

Indexer: Manju Arasan

Production Designer: Aparna Bhagat

Marketing Coordinator: Hemangi Lotlikar

First published: February 2022

Production reference: 1120122

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80056-879-2

www.packt.com

To my father, who had the wherewithal and foresight to introduce me to programming and computers at a very young age and taught me to see technology as a business tool. And to my wife, Dominica, and my daughters, Liesl and Luciana, for suffering through the writing of a book on top of all of the normal craziness that life is always throwing at us. My team makes this all possible.

– Scott Alan Miller

Contributors

About the author

Scott Alan Miller is an information technology and software engineering industry veteran of 30+ years, with more than a quarter of a century on UNIX and Linux. His experience has included companies of every size, in every region of the world, in nearly every industry. Scott has been a technician, lead, manager, educator, consultant, writer, author, speaker, and mentor. Today, and for more than the last 20 years, Scott has led the IT consulting team at NTG. He now lives in Nicaragua.

About the reviewer

René Jensen has 21 years of professional experience with UNIX/Linux administration, both as an employed administrator and, for the last 9 years, as a consultant. His experience ranges from branches such as medical, banking, tax, and mobile business, to working in areas such as CI/CD, container deployment, architecting server clusters, daily operations, and many other areas.

I would like to thank my family for being patient, since my work started as a hobby and I spend a lot of time going in depth with new challenges.

Table of Contents

Preface

Section 1: Understanding the Role of Linux System Administrator

Chapter 1: What Is the Role of a System Administrator?

Where are system administrators in the real world?

Wearing the administrator and engineering hats

The difference between the role of an administrator and the role of an engineer

Hats

The wonderous variety of the role

Understanding systems in the business ecosystem

Learning system administration

Build a home lab

Getting family and friends involved

Start as a generalist and progress onto a specialist in the System Administrator field

Volunteer for non-profits or non-business organizations

Self-study

Age does not matter

Internships

Introducing the IT Professional

The fallacy of success at any cost

Summary

Chapter 2: Choosing Your Distribution and Release Model

Understanding Linux in production

Is Linux UNIX?

Linux licensing

Key vendors and products

What about BSD?

Debian

Ubuntu

IBM Red Hat Enterprise Linux (RHEL)

RHEL alternatives

Fedora

OpenSUSE and SLES

Digging into distribution history

Other Linux distributions

The myth of popularity

Using multiple distributions

Making the choice

Releases and support: LTS, current, and rolling

What does support mean?

Release model: rapid release

Release model: LTS

Release and support schedule interplay: The overlap

Release model: Rolling

Why not just update the packages manually

Choosing the release model for our workloads

Choosing your distribution

Do not fear risk

Summary

Section 2: Best Practices for Linux Technologies

Chapter 3: System Storage Best Practices

Exploring key factors in storage

Cost

Durability

Availability

Performance

Scalability

Capacity

Understanding block storage: Local and SAN

Locally attached block storage

Storage Area Networks (SAN)

The terrible terminology of SAN

Surveying filesystems and network filesystems

EXT4

XFS

ZFS

BtrFS

Clustered file systems

Network filesystems

Getting to know logical volume management (LVM)

Whatever happen to partitions

Utilizing RAID and RAIN

RAID

RAIN

Learning about replicated local storage

DRBD

Gluster and CEPH

Proprietary and third-party open-source solutions

Virtualization abstraction of storage

Analyzing storage architectures and risk

General storage architectures

Simple local storage: The brick

RLS: The ultra-high reliability solution

The lab environment: Remote shared standard storage

The giant scale: Remote replicated storage

Storage best practices

Storage example

Summary

Chapter 4: Designing System Deployment Architectures

Virtualization

Type 1 hypervisor

Type 2 hypervisor

Hypervisor types are confusing

VMware ESXi

Microsoft Hyper-V

Xen

KVM

Is virtualization only for consolidation?

Containerization

Cloud and VPS

Virtual Private Servers (VPS)

On premises, hosted, and hybrid hosting

Colocation

System Design Architecture

Standalone server, aka the snowflake

Simple does not necessarily mean simple

Many to many servers and storage

Viewing the world as a workload

Layered high availability

Reliability is relative

Hyperconvergence

Best practices in System Design Architecture

Risk assessment and availability needs

Workload interplay

Defining high availability

Summary

Chapter 5: Patch Management Strategies

Binary, source, and script software deployments

Compiled and interpreted software

Misleading use of source installation

Patching theory and strategies

The risk of delayed patching

Avoiding patches because of Windows

Testing patches is rarely feasible

Timeliness of patching

Compilations for the administrator

The compilation era

Compilation by engineering department

Linux deployment and redeployment

Rebooting servers

Finding your green zone

Avoiding planned downtime is planning for unplanned downtime

Summary

Chapter 6: Databases

Separating a Database from a DBMS

The Database

The Database engine

The Database management system

Comparing relational and NoSQL databases

Discovering common databases on Linux

Common relational databases on Linux

Drop In replacements

Common NoSQL Database Products on Linux

Document databases

Understanding database replication and data protection concepts

Summary

Section 3: Approaches to Effective System Administration

Chapter 7: Documentation, Monitoring, and Logging Techniques

Modern documentation: Wiki, live docs, repos

Repos

Ticketing systems

Approaching documentation

Tooling and impact

Netdata

Capacity planning

It Is already designed when purchased

Log management and security

Why central logging?

Alerts and troubleshooting

On-device and centralized alerting systems

Pushed and pulled alerts

In house and hosted monitoring

RMMs and monitoring

Summary

Chapter 8: Improving Administration Maturation with Automation through Scripting and DevOps

The GUI and the CLI: Administration best practices

Consolidation and the age of squeezing systems

Automation maturity

Local and remote automation

Command line

Scheduled tasks

Scripting

PowerShell on Linux

Scripting combined with task scheduling

State management

Infrastructure as code

Platforms and systems

Modern tools of automation

Configuration management systems

Version control systems

Summary

Chapter 9: Backup and Disaster Recovery Approaches

Agents and crash consistency

Locking mechanisms in Linux

MySQL example with mysqldump utility

Backup strategies & mechanisms

Types of backups

Snapshots, archives, backups, and disaster recovery

Snapshots

Archives

Backups

Disaster recovery

Backups in a DevOps world

Version control systems

IT provides solutions, vendors sell components

Triage concepts

Summary

Chapter 10: User and Access Management Strategies

Local and remote users

User management mechanisms

Using automation to turn local uses into remote users

The famous RDP exposure risk

Are operating system logins relevant in the modern world?

Remote access approaches

How do I approach remote access

SSH, key management, and jump boxes

Do you still need both a network edge firewall and an operating system firewall?

Does changing the default port of SSH work?

SSH key management

Jump boxes

Alternative remote access approaches

Terminal servers and virtual desktop infrastructure (VDI)

Understanding terminal services and VDI conceptually

Summary

Chapter 11: Troubleshooting

The high cost of disaster avoidance

Sources of solutions

There is no magic support

Visualizing what IT handles and what engineering handles

IT vendor managements

Triage skills and staff

I can give status, or I can fix things

Staffing for triage: The perceiver

Logical approaches to troubleshooting

Stories of troubleshooting

Technical social media in problem solving

Investigating versus fixing

Summary

The postmortem

Other Books You May Enjoy

Preface

Linux Administration Best Practices is a guide for understanding the context, decision making, and ideologies behind one of the most critical functions in business infrastructure: systems. Systems, that is, operating system level management, remains the cornerstone of communications and infrastructure. Linux remains the most popular operating system family of choice today and is only gaining more and more traction, making the need for well-trained, deeply knowledgable Linux administration teams that much more important.

Who this book is for

This book is intended for those IT professionals working with Linux or as system administrators who want to take their craft to the next level. This book is about best practices and so we approach the role and thinking of the system administrator rather than learning individual tasks. This book will challenge how you think and how you approach system administration. This book will not teach you about the tasks of system administration, but it will teach you how to think like a career administrator.

What this book covers

Chapter 1, What Is the Role of a System Administrator?, explains the actual role and mandate of the system administration function and how to apply this to your own role in your career and your organization.

Chapter 2, Choosing Your Distribution and Release Model, goes through how to choose the right Linux variation for your organization (Linux comes in many flavors and styles), understanding the importance of release models.

Chapter 3, System Storage Best Practices, attempts to take you from newbie to master regarding storage, which remains one of the least understood areas of system administration, taking a high-level approach.

Chapter 4, Designing System Deployment Architectures, breaks down assessing deployment approaches and when different models will work for you.

Systems do not exist in a vacuum; they are deployed in conjunction with other systems, often needing to interoperate for functionality or redundancy. Combining systems in meaningful ways is complex and can be counterintuitive.

Chapter 5, Patch Management Strategies, looks at patching and updates, which might sound mundane but are at the core of any system task list, and are often more complex than is realized. Good patch management will help protect you and your organization from disasters both accidental and malicious.

Chapter 6, Databases, digs into database concepts and how they apply to the systems realm so that you can provide better support and guidance to your database users. Technically not part of the operating system, database management has historically fallen to system administrators.

Chapter 7, Documentation, Monitoring, and Logging Techniques, looks at strategies for both manual and automated processes for tracking the desired state and current state of our systems.

Chapter 8, Improving Administration Maturation with Automation through Scripting and DevOps, looks at many different ways to approach automation considering practicality and real-world needs in addition to perfect, theoretical approaches. Everyone talks about the automation of system tasks, but many organizations fail to do it.

Chapter 9, Backup and Disaster Recovery Approaches, goes far beyond conventional wisdom and approaches disaster recovery with a fresh eye and modernism to provide ways to make backups easier and more effective than they normally are. The single most important task in system administration is protecting data.

Chapter 10, User and Access Management Strategies, looks at best practices for maintaining users, discusses decision making and user management approaches, and investigates architectures for remote access to the operating system. What good is a system if no one can access it?

Chapter 11, Troubleshooting, looks at how taking a planned, intentional approach to troubleshooting improves resolution speed, reduces stress, and might just find problems that would have been kept hidden otherwise. Nothing is harder than figuring out what to do when something is wrong and the pressure is on.

To get the most out of this book

You are expected to have general knowledge of Linux and operating systems. We assume experience in working with systems in a production environment and a business setting. This book covers high-level concepts rather than technical processes and so a working knowledge of Linux and operating systems is beneficial.

The software covered in this book are Linux-based operating systems such as Ubuntu, Fedora, Red Hat Enterprise Linux, and Suse.

No running system is necessary to use this book. This book focuses on high-level concepts and while knowledge of Linux and operating systems is very useful, there is no need to be hands-on with a running system while reading this book.

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781800568792_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "System is a reference to operating system and designates the scope of the role: managing the platform on which applications run."

Any command-line input or output is written as follows:

$ lvcreate -l 100%FREE -n lv_data vg1

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "As we progress through our exploration of Linux System Administration, the idea of hats and really digging into job roles and functions will become more and more clear."

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Linux Administration Best Practices, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Section 1: Understanding the Role of Linux System Administrator

The objective of Section 1 is to help you to comprehend the scope, responsibilities, role, and mandate of the System Administrator function. We take the reader past the concept of tasks and really attempt to dig into the purpose and value of the role at a much deeper level.

This part of the book comprises the following chapters:

Chapter 1, What Is the Role of a System Administrator?Chapter 2, Choosing Your Distribution and Release Model

Chapter 1: What Is the Role of a System Administrator?

Few things in our industry sound like they should be simpler to answer than this one, simple question: what is a system administrator? And yet, ask anyone and you'll get some widely differing opinions. Everyone seems to have their own take on what the title or role of System Administrator implies, including and possibly most varying in people who use this title for themselves or from the companies that hand it out!

Welcome to system administration and specifically Best Practices of Linux Administration. In this chapter we are going to dive into understanding the job, role, and functions of a real system administrator and try to understand how we, in that role, fit into an organization.

In tackling this book, it is necessary both for myself to have some semblance of a clear course in writing, but also for you to understand if this book is for you, or to grasp the scope that I am attempting to cover, for me to clearly define what a system administrator is to me.

Understanding exactly what is expected of a true system administrator will be the foundation for applying that definition of the role to the upcoming best practices that apply both to system administration generally and specifically to Linux administration.

In this chapter we are going to cover the following main topics:

Where are system administrators in the real worldWearing the administrator and engineering hatsUnderstanding systems in the business ecosystemLearning system administrationIntroducing the IT professional

Where are system administrators in the real world?

I think that one of the most challenging things about attempting to understand what a system administrator is comes from the fact that the title of system administrator is often given out, willy nilly, by companies with little to no understanding of Information Technology (IT), systems, or administration and treat it like a general filler for IT roles that they do not understand or know how to name. It also has a strong tendency to be given out in lieu of pay raises or promotions to entice junior staff to remain in an otherwise unrewarding job in the hopes that an impressive title will help them later in their career, so much so that in the end, the number of people working as system administrators is a very small number of people compared to the number of people with the title. In fact, it is no small stretch to guess that the average person with the title of system administrator has never thought about the meaning of the title and may have little inkling of what someone in that role would be expected to do.

If we look solely by title, system administrators are everywhere. But they exist mostly at companies too small to have plausibly employed a system administrator at all. Systems administration as a dedicated job is nearly exclusive to large companies. Most companies need someone to do the tasks of system administration, but only as a part of, and often only a small part of, their overall duties. It is the nature of IT that in small and medium sized companies you typically have generalists who wear many hats and do every needed IT role while having little to no time to focus on any one specific function. Whereas in large enterprises you generally get focused roles, often grouped into focused departments, that do just a single IT role: such as system administrator. But even in some enterprises you find departments organized like separate, small businesses and still having generalists doing many different tasks rather than separating out duties to lots of different people.

There is nothing wrong with this, of course. It is totally expected. It's much like how, as a homeowner, you will often do a lot of work on the house yourself, or you might hire a handyman who can do pretty much whatever is needed. You might need some plumbing, painting, carpentry, wiring, or whatever done. Whether you do it yourself, or your handyman does, you do not refer to either of yourselves as plumbers, painters, carpenters, and others. You are just a handy person, or the person that you hired is. You still recognize that a dedicated, full time, focused plumber, painter, carpenter, or electrician is a specialized role. You might do all those tasks occasionally, you might even be good at it, but it's not the same as if that was your full-time career. If you decided to claim to be these things to your friends, they would quickly call you out on the fact that you are quite obviously not those things.

System administrators are like plumbers. Everyone who owns a house does at least a little plumbing. A handyman who does home maintenance full time might do a fair amount of plumbing. But neither is a plumber. A very large housing development, or a construction crew might have a dedicated plumber on staff. Maybe even more than one. And nearly every homeowner must engage one from time to time. If you are me, regularly. Most plumbers either work for large companies that have need of continuous plumbing services or they work for plumbing contracting firms and have the benefits of peers and mentors to help them advance in their knowledge.

Nearly every business no matter what size we are talking about needs system administration tasks done. For very small businesses it is not uncommon for these tasks to amount to no more than a few hours per year, and when needed the scheduling is often unpredictable with many hours needed all at once and large gaps of time in which nothing is needed. In large businesses, you might need tens of thousands of hours of system administration tasks per week and require entire departments of dedicated specialists. So just like plumbers, you find small businesses either hiring IT generalists (akin to the homeowner's handyman) or outsourcing system administration tasks to an outside firm like an Managed Service Provider (commonly referred to as an MSP) or keeping a consultant on retainer; and you will find large companies typically hiring full time specialist system administrators that do nothing else and work only for that firm.

System administration tasks exist in every business, in every industry and create the foundation of what I feel is one of the most rewarding roles within the IT field. With system administration skills you can chart your own course to work in a large firm, be a consultant, join a service provider, or enhance other skills to make yourself a better and more advanced specialist. Without a firm foundation in system administration a generalist will lack one of the most core skills and have little ability to advance even in the generalist ranks. And at the top of the generalist field, true CIO roles primarily pull from those with extensive system administration comprehension.

At this point we know what a system administrator is, where you will find them in businesses, and why you might want to pursue system administration either as your career focus or as an enhancement to a career as a generalist. Now we can go into real detail about what a system administrator really does!

Wearing the administrator and engineering hats

In this section, we will explore two parts:

How does administration and engineering differ.How to identify the role you are performing.

The name system administrator itself should clue us in as to what the role should entail. The title is not meant to be confusing or obfuscated. Yet many people believe that it is some kind of trick. If you spend enough time working in the small business arena you might even find that many people, people who are full time IT professionals, may not even believe that true system administrators exist!

System is a reference to operating system and designates the scope of the role: managing the platform on which applications run. This differentiates the system administrator role from, say, a database administrator (commonly called a DBA) who manages a database itself (which runs on top of an operating system), or an application administrator (who manages specific applications on top of an operating system), or a platform administrator (who manages the hypervisor on which the operating system runs), or a network administrator (who manages the network itself.) Being called a system administrator should imply that the focus primarily or nearly entirely of the person or role is centered around the care and feeding of operating systems. If your day isn't all about an operating system, you aren't really a system administrator. Maybe system administration is part of your duties but being a system administrator is not the right title for you.

Administrator tells us that this role is one that manages something. The direct alternative to an administrator is an Engineer. An engineer plans and designs something; an administrator runs and maintains something. I often refer to these roles as the A&E roles and often the titles are used loosely and meaninglessly based on how the speaker thinks that it will sound. But, when used accurately, they have very definite meanings and in each area within IT (systems, platforms, network, databases, applications, and others.) you have both working in concert with one another. Of course, it is exceptionally common to have one human acting as both an engineer and an administrator within an organization, the roles have extensive overlap in skills and knowledge and necessarily must work in great cooperation to be able to do what they do effectively.

The difference between the role of an administrator and the role of an engineer

There is a key difference between the two roles, however, that impacts organizations and practitioners in a very meaningful way, which is very important to discuss because otherwise, we are tempted to feel that separating the two roles is nothing more than pedantic or a game of semantics. This difference is in how we measure performance or success.

An engineering role is measured by throughput or the total quantity of work done. It's all about productivity – how many systems the engineering team can design or build in a given period of time.

The administrator role would make no sense in this same context. Administrators manage systems that are already running rather than implementing new ones. An administrator is measured on availability, rather than productivity. This may sound odd to hear, but this is mostly because the average organization has little to no understanding of administration and has never contemplated how to measure the effectiveness of an administration department.

Hats

I spend a lot of time talking about hats. Understanding hats is important.

By hats, I mean in the sense of the different job roles that we take on and understanding when we are performing the tasks of one role or another – referred to as wearing a hat. For example, if I work in a restaurant I might act as a waiter one day and we would say that I was wearing a waiter hat. The next day I might be working in the kitchen making salads and then I would be wearing the short order cook hat.

This may sound a bit silly, but it is important to understand. The term hats tends to allow us to understand the difference between being a thing and performing the actions associated with that thing better. We all know what is required to be a plumber, but most plumbers drive trucks to their job sites. Does this make them truck drivers? Is the skill of driving a truck actually part of the skill of plumbing? No, it is not. But it is a generally useful ancillary skill. But a large plumbing company could easily consider hiring professional drivers that drive their plumbers from job to job to keep everyone focused on what they are trained to do, lower insurance costs, increase plumbing billable hours, and even allow for hiring excellent plumbers who maybe do not even have a driver's license!

In IT, it is so common to be asked to perform the duties of myriad different roles and functions that we often forget that we are doing it. IT professionals are often seen as a jack of all trades, able to do a little of everything and often this ends up being true just from the nature of what makes people get into IT in the first place. But it really helps when we separate our intended roles, our specialties, our training, and what earns us our salaries from when we are just helping out by performing other duties outside of our strict arena because we happen to be skilled, flexible, or just otherwise helpful.

As we progress through our exploration of Linux System Administration, the idea of hats and really digging into job roles and functions will become more and more clear. Thinking of our functions as wearing a specific hat at a specific time is a powerful tool for understanding, and possibly more importantly, for communicating our roles, requirements, capabilities, expectations, and responsibilities to others and ultimately, to ourselves. In the real world, it is very rare to find a company or role where the engineering and administration aspects of systems can be truly separated. This has benefits, and it has consequences. But it is worth mentioning that most many larger companies do, in fact, keep these two roles apart – sometimes even to the extent of having them in different departments with unique management teams. There are a lot of benefits to these being separate, primarily because the soft skills needed for the two aspects of systems are generally opposing.

Engineers benefit from the strong soft skill of being good planners. Engineering is all about planning, almost exclusively. You get to take your time, think about how a system will be used, and design around that future need. Your job is to prevent emergencies rather than reacting to them.

Administrators benefit from the strong soft skill of being good perceivers – that is, responding to events in real time, rather than planning ahead. Administrators are tasked with managing live, running systems and that means that their primary challenges are presented to them in real time, and being able to triage, prioritize, and work under pressure is paramount.

Those who make good engineers rarely make good administrators and vice versa. While the technical skills are almost a total overlap, the human skills are highly opposing and very few people manage to be truly adept at both planning and triage. Those highly skilled at administration will also naturally deprioritize engineering work because they know that they can react to it effectively later, even if poorly planned for.

There is other language that can often guide us here as well. Engineers tend to work in a world of projects. There is a goal, an end point when their work is handed off to the administrators. Administration works in a world of steady, ongoing support. There is rarely a starting or ending point and the term project would make no sense to them. They run the systems that the company needs until those systems are replaced, generally with a new workload that the administrator runs in the same way. An administrator might need to give input to a project manager as a project might need to report what its long-term administration impact is going to be, or to get the blessings of administration that they are willing to take on the results of the project when it is handed over to them. But the project itself would always be done, by definition, by an engineering role.

Knowing when we are wearing an engineer's hat or when we are wearing an administrator's hat gives us the power to understand how we can function effectively as well as how to communicate our needs and capabilities to the rest of the organization. Most organizations are blind to these kinds of psychological and performance ramifications of the job roles and require that we provide this understanding. The better that we are at identifying our role and our needs means the better prepared we are to attempt to articulate those to management.

A common problem arising from requiring IT staff to function as both an engineer and an administrator at the same time is burnout. Being an administrator naturally leads to being very busy with requests and always being the first called when things go wrong: systems performing slowly, a computer crashing, a new bug discovered, and patches needing to be applied. Administrators tend to not only work long hours but also be on-call extensively. Getting downtime when they can get it is critical to their ability to remain in the role long term.

Engineers have no need to be on-call and are not the responsible role during an emergency or problem. Engineers, however, would not be expected to need special downtime to compensate for the extreme demands on their time and schedule. They do not get interrupted, they do not have to carry a beeper (although no one literally carries a beeper anymore), they do not miss their kids' school play or have to answer questions seconds before boarding a plane, or have to run out of a family dinner to walk someone through fixing a problem remotely. They get to live scheduled lives where they can rest like normal jobs do. The image of the IT worker running ragged and never getting a break, always being expected to drop everything for the company, and living off of promises that often never materialize of a chance to rest sometime in the future is one of system administrators in nearly all cases. No other role carries so much responsibility at the times when it is most critical.

Combining these roles together means that not only would a person wearing both hats risk being on-call and having to drop everything to respond to issues all day, every day, but then needing to fill every potential remaining moment with project work for their engineering role that expects them to have available capacity to do. Trying to stay productive on projects with deadlines while also trying to react to every question, ticket, or outage is a recipe for burnout or worse.

If we can convey to management the difference between the administration function and the engineering function, we can begin a dialogue on how to make our jobs tenable, which ultimately makes things better for both parties. Being pushed too hard leads to a loss of productivity, mistakes, oversights, and staff turnover.

The fifteen-minute rule

In the software development world, which is all engineering with no administration, the standard rule of thumb is that any interruption that requires an engineer's focus will cost that engineer the time of the interruption plus fifteen minutes additional time to get back to the point of productivity where they were before the interruption. It does not take much to see that a handful of interruptions throughout a day would reduce the effective time of an engineer to essentially zero. They might be attempting to work all day, sitting at their desk trying to focus on the task, burning through their available time, but if they keep getting interruption, they will just spin their wheels and feel more and more exhausted and disheartened as they do not get a chance to rest, but neither do they get to be productive and show something for their efforts.

This fifteen-minute rule is known as task switching overhead.

The administrator's role is one of nearly all interruptions. Monitoring systems to see what might be going wrong, being alert for new patches requiring their attention, responding to tickets or questions from other teams, and so forth. Administrators are, of course, impacted by the fifteen-minute rule just the same as an engineer is. But unlike the engineer, an administrator is normally resolving a problem, or a request and the worst is a small, atomic chunk that is completed before the next interruption takes focus. When a task is completed, or an outage resolved the next task to be tackled is a different one whether it is already lined up and ready to go or doesn't arrive for some time. The administrator must go through task switching between each issue regardless, so the overhead that this incurs is a given that cannot be avoided.

Not only does task switching overhead help us to explain and understand why administrators and engineers need to either be different people or be given extreme accommodation, but it also helps to explain the much broader need for quiet, effective work environments for all staff. Interruptions don't just come from system emergencies but from anywhere like meetings, water cooler banter, office workers who just stop by, general office noise, fire drills, you name it.

Tools to measure skills for an administrator or engineer

If we are in a larger organization where these roles can be kept separate, we can show how measuring each by their unique benefit can make each department better. If we are in a smaller organization and are forced to transition from wearing one hat to another, we need to be able to effectively work with the organization to demonstrate our needs and work with management to produce working schedules or extra resources can make us more effective.

One of the best tools that I've seen used to understand the soft skills for administrator versus engineer is the Myers-Briggs Type Indicator of Judging and Perceiving. Myers-Briggs is an industry standard psychological exam that, when handled well, I feel can be highly beneficial for helping us to understand our own natural strengths and weaknesses.

The following shows an overview of the different personality types and their respective cognitive functions in the Myers Briggs model:

Figure 1.1 – Myers Briggs Cognitive functions of the personality types

Engineers typically require a strong leaning towards the judging profile, while administrators need a strong leaning towards the perceiving profile. Judging effectively equates to planning, in this case, and perceiving equates roughly to the ability to react or perform triage. Almost no one is good at both planning and responding. Knowing your strengths lets you focus on what will make you happy and what will let you excel. While knowing your weaknesses allows you (and your organization) to adapt accordingly.

The wonderous variety of the role

One of the more challenging aspects of a book like this is that the role of system administrator, even when we agree on the definition of what a system administrator is, varies so wildly that when performing the role at one company to performing the same role at another company our position can look like a completely different career path! It is easy to find system administrators who spend nearly their entire day manually deploying software and never, ever do performance tuning; another system administrator might do little other than performance tuning; another might focus on managing system applications or databases; one will manage lots of printers, but most admins will never manage even a single printer; and another managing users and userland storage concerns, while again, most, will never manage users at all. All of these system administrators may never meet another system administrator who does a similar workload to what they do, and few will ever get a second job within their field that has them do the same tasks again. Each system administration position is, effectively, unique. Almost absurdly so.

Even more extreme is that so many people think of system administration purely in terms of servers, and certainly the title tends to only be applied there, but there are careers in desktop, appliance, and even IoT realms as well. Linux, in all its forms, may remain a rather small player in the desktop and laptop space, but companies of all sizes do deploy it in production and need system administrators who understand the aspects of the operating system as they pertain to end user devices. Do not casually dismiss the desktop administration subset of systems as being all that different. While desktop administration is often less stressful or critical than server administration it is often more varied and complex and often presents some unique and extreme technical challenges.

In this book I am going to do my best to look at system administration from a high level and provide insight and guidance that applies to essentially everyone - whether you are a full-time working system administrator, a student hoping to become one someday, or an IT generalist for whom system administration is part of your regular tasks. I feel that too many guides to system administration take a myopic approach and try to look at the field as being only one or two of the myriads of aspects out there and totally forget that most of the field will never encounter the need for most of what they are taught.

Here's an image to entertain you with the jack of all trades and the master of one:

Figure 1.2 – Generalist Vs Specialist

There are three essential types of books on system administration. They are as follows:

The first type takes a high level and attempts to present the field of administration. The second type digs in to detailed and highly specific tasks. And the third focuses on teaching skills required by a certification exam.

All three of these styles of books or resources are very valuable. In this book, we are tackling the first and we will leave detailed commands, syntax, tools, and other in the weeds implementation details up to other excellent books that already focus on those things today. I will also attempt to keep my advice as distribution agnostic as is reasonably possible. Linux is all but unique in that it is an enormous and varied ecosystem rather than a single product. When talking about best practices we are given a natural pass on needing to dig in too deeply to a specific operating system built off Linux, but it is still tempting to lean overly heavily on a certain one or two popular implementations.

It is also important to make a distinction between what is intrinsic to Linux, to an operating system built from Linux (more on that later), or intrinsic to system administration versus what is simply convention or assumption. In approaching a book like this, we can go in so many potential directions and I will, whenever possible, attempt to clarify when something is simply convention versus when something is truly how it must be done.

Great, so now we know what aptitude and psychological skills we will need to succeed at system administration, and we know about how awesome this role can be. We are getting a clear picture of what the purpose of the system administrator is. Next, we need to see how that role is going to fit into the IT department and the business itself.

Understanding systems in the business ecosystem

Systems, that is the operating system layer of IT infrastructure, plays one of the most critical roles within IT and the business. In most businesses, most of the most critical aspects of IT functionality tend to fall to the systems roles to oversee. The system administrator is often saddled with tackling security, performance, data integrity, storage, planning, access control, backups, innovation, design, consulting, and so much more. No other role commonly must address all, if even any, of those functions.

For many reasons, system administrators often form the backbone of an IT departmental infrastructure. The operating system, because of its deep roots into storage and networking, and its close association with data and applications, sits in the position with the greatest control and visibility throughout the IT organization. System administrators may have little direct contact with end users but tend to be the nexus around which nearly all the infrastructure and support departments focus and rely.

System administration is generally seen as forming the largest group of focused IT professionals, as well, at least within the scope of purely technical roles. End user touching positions such as helpdesk or deskside support may involve larger headcount, but much of those roles often involves customer service or end user training that accounts for much of their time spent during a day. Within the technical realms of networking, storage, applications, databases, and so forth, it's most likely that systems will represent the largest portion of your team and cover the broadest range of skills.

Being so central, core, and often large, you can think of systems as often functioning as the glue that holds the IT department together; or you can think of systems as being the hub onto which all other IT disciplines will tend to attach. The following diagram demonstrates this:

Figure 1.3 - Systems at the Core of the IT Ecosystem

Unlike networking who might know little or nothing of systems, for example, systems professionals cannot be unfamiliar with networking concepts, even extremely detailed ones. Or unlike a database administrator who can generally just assume that someone else will handle backups properly, the system administrator is often the comprehensive point of communications that must understand how the database is talking to storage, how storage is quiescing data, and how a backup is being decoupled from the system. Just as the operating system is essentially the communications hub of a workload, the system administrator naturally becomes the point of holistic understanding and management of a workload.

In my experience, systems administration often is expected to work as a full consulting peer to other teams. I've seen applications, database, and networking teams all turn to systems departments when they are looking for broader experience within their own realms. The nature of system administrators to have to deeply understand not only their own realm, but all adjacent ones, leads to a department that can often act as a consultant to others, much as how IT often does so to the business at large.

Naturally, system administration gains the greatest overall knowledge of the overall workings of a business from the IT perspective which because it touches nearly every aspect of a business, is often one of the most important views that there can be of an organization. This will lend itself to the system administrator also being in a key role to report on and potentially advise business leaders within the organization as to issues and opportunities. Of all specialized roles within IT the system administrator is the most likely to interact heavily with the business on a large scale.

Now we have a good feel for how our role is going to fit into the IT department and the business. Next, we will learn about to tackle learning this role in the first place.

Learning system administration

If you are not already working as a system administrator, you might feel that getting the necessary experience and training to become one will prove to be difficult. This is not true. In fact, moreso than nearly any other area of IT you can experiment with and practice nearly every aspect of system administration for very low cost in a home lab. System administration may have a lot of facets and favor those with deep knowledge, but it is also ultimately accessible to those willing to put in the effort.

In the following sections, I will describe some ways that you can get started on learning to become a System Administrator.

Build a home lab

Few people are as big of a proponent of building a home lab as I am. My own home lab experience has been among the most valuable of my career. And in my role as a hiring manager, no matter how much office experience someone has, I always want to know what they do at home on their own time. Home labs show dedication, interest, passion, and provide unique opportunities to really learn a process from end to end. In my decades in system administration, I have found that it is actually a rare administrator who has gained any real experience running a system from cradle to grave and understanding what a system and workload lifecycle are really like. Having that experience, even if only at home, can be a major point of differentiation in an interview or on the job. It can easily be the stand out factor that makes all of the difference.

Virtualization has brought the cost and complexity of a home lab from something that was onerous in the late 1990s to something that is almost an afterthought in the 2020s. The resource needs of a lab are often incredibly small, while the spare resources of most computers, by comparison, are large. And this is before we consider cloud computing and the ability to do lab work using on demand cloud resources today. While not free, cloud computing is often extremely affordable as another resource that a home lab can deploy expanding the student's experience even further. Leveraging both gives you even more to put on a resume and to discuss at the interview table.

Of course, to make this experience valuable, you must really dedicate a lot of effort - every bit as much, or more, than you would if it was a production system at a paying job. Take the time to automate, secure, monitor, update, and maintain real workloads even if you just use them around the house. A web server, a house VoIP phone PBX, email, instant messaging, a database, a media server, a remote access server, a DNS filter, you name it. There are many things that you can run in your own lab that are similar, or even identical, or heck even superior, to workloads that you would see in a common business environment. Do not be afraid to use your lab to make interesting and fun projects for family and friends, too.

Getting family and friends involved

Building a home lab that is only for yourself can be challenging because you never get to see your environment being used. When possible, I highly recommend recruiting friends and family to get involved as well. This lets you get end user feedback and experience systems more like they are meant to be used. Running systems only for yourself is far more difficult to get the experience that would match what happens in a business.

When I was first running my own home labs for system administration, I ran a family website, an email server, and probably the most interesting a full PBX. A VoIP PBX (or Voice over IP Private Branch Exchange) for home can be great because it tends to be complex compared to most home style workloads, can be used completely realistically as a valuable part of your family's communications platform and few experienced IT professionals have worked with them themselves.

In the PBX example, you can approach it with having extensions in multiple rooms and offering extensions to family members outside of your physical home. Using a PBX to build a free calling mechanism for family to stay in contact with each other easily can be truly utilitarian beyond the IT experience that it provides. Building something that provides real value and gets really used will not only make your projects more enjoyable but also makes the educational experience more like the real world.

Part of the secret to leveraging the home lab experience is to not only do everything right as you would in the best of businesses, but also to be prepared to discuss and explain in detail what you have done and how that meets and possibly exceeds the experience that someone is going to have gained from having worked in more typical work environments. Highlighting best practices, holistic management, experimentation, and the chance to have worked on the latest technologies can make all the difference between an interviewer dismissing home experience as inconsequential to them feeling that their own staff may be ill prepared to interview you due to your broader experience base and more up to date knowledgebase!

Start as a generalist and progress onto a specialist in the System Administrator field

Beyond your home lab there are other ways to gain experience before attempting to jump straight into the big business world where system administrators are most likely to be found. Of course, a common path taken is to work as a generalist in a small company where systems administration is part of the job. This approach does work but does not work often or as well as is generally expected. Making the leap from small business generalist to big business specialist can be almost as challenging as going directly from student to big business specialist in a single leap. So, we need to look at some ways to either get there directly or ways that we can assist in the transition.

Volunteer for non-profits or non-business organizations

Another option is volunteering. Non-profits and other non-business organizations often need resources that they have no means to find, attract, or afford. Volunteering in these types of organizations can be a great way to gain production experience and to demonstrate how dedicated you are to your craft. They can rarely provide any sort of mentoring or guidance, which is a significant negative, but they also generally provide little oversight or structure and will appreciate the effort that you put in, giving you more latitude to choose approaches that benefit your experience as much as benefiting them. As a volunteer, you will easily find that you can spend more time focusing on real solutions for the organization rather than needing to play politics or attempt to please a manager since you are working for free and have no job to defend or promotion to angle toward.

One of my better career decisions early on in my system administration years was to use my free time for a couple of years to volunteer as the sole system administrator of an all-Linux environment at a K-12 school. The school had almost no existing computers at all, and certainly no network or Internet access, and I was able to put on my engineering hat and design their entire environment including desktops, servers, telephones, networking, printing, and others. And once implemented, I was able to switch to my administrator's hat and manage the entire desktop, server, telephone, and storage environment from cradle to grave built entirely from Linux. I was able to do everything and do it all myself. I paid the price as an administrator for my oversights as an engineer. I saw the initial cost and design considerations and how they played out with end users. I deal with a workload from procuring the initial hardware, installing the software, configuring, deploying, training, and supporting. That experience didn't just catapult me forward in my career in the short term but is so significant that it is a valid talking point even in senior enterprise CIO level interviews much later in my career.

Presenting this experience in a Fortune 100 interview was very easy. Showing that I understood the processes and ramifications of decisions was great. Being able to demonstrate the ability to research, plan, and action every step from racking a server to production support of a system was unique. In the enterprise those skills rarely exist because typically different teams often handle each different portion of the process. Being able to show holistic oversight of the entire process brough much to the table. This range and type of experience was something other candidates did not have and made a huge different in my career trajectory.

Do not overlook doing small time support for friends and family. Almost everyone knows someone who could use some technical support at home and mostly what people need at home is system administration support! It goes without saying that this will be desktop support rather than support of servers. Real world support of desktops is certainly better than no experience at all. Putting together many different types of experience is best to show the most well-rounded total portfolio.

Self-study

Learning system administration has the benefit of there being so many great resources available on the market from books and certifications to online communities and videos. Both structured and unstructured materials abound, and thousands of passionate professionals are always online in communities just waiting to answer questions and help others in the field.

It is tempting to seek out classes or the newer trend of boot camps to learn IT skills in general, and system administration in specific, but in most cases I will advise against that approach. Boot camps especially are designed to teach you only very specific skills necessary to get up and running and through an interview process on a specific process and cannot take the necessary time to deeply teach topics. This is dangerous and often leads to long term career failure. More formal traditional class style education can be better, and some people learn especially well that way, but there are caveats.

Typically, classroom learning happens at a very slow pace and is not flexible to your schedule. This is only so much of a problem, but you lose the opportunity to move as fast as possible while focusing on what matters to you most. Very rarely can you find classroom education that is current and relevant. While possible, it is almost unheard of and there are seldom checks and balances to verify that what is being taught is useful or even accurate. Most importantly is that classroom learning is not a sustainable approach. Once you are working in the field you will not have the time to continuously take classes to keep your knowledge up to date, nor will you be able to find classes that provide for the immediate needs of the next project or challenge. Classroom learning is essentially a special case that is only available to people before they enter their careers. It is not a useful approach in an ongoing way. Because of this, learning through a classroom setting does not demonstrate a practical skill for the workplace.

Instead, teaching yourself through books, articles, experimentation, online communities, and other self-starting means that show that you are able to learn without someone else's assistance are crucial both because you will almost never have someone available to know everything that you need to know before you do and be prepared to teach it to you, and because you will be spending your entire career constantly needing to stay current and learn new technologies and techniques. Any serious profession involves lifelong learning, but very few demands it to the degree that is required in IT. Showing the ability to teach yourself whatever is needed is a very big deal to a potential employer.

Learning to teach yourself gives you a broader range of potential learning opportunities, more flexibility in how and when you learn, the potential to start learning at any time, and the ability to move at the fastest possible pace. Of course, as a reader of this book, you have already taken at least this step in seeking out formal, self-paced learning.

Age does not matter

You might be wondering when you should consider tackling the beginning of studies into system administration. Is this something you begin at university? Do you need several years in the field working in networking or perhaps in helpdesk before you can move over to systems? Do you start running desktop support before you can switch to servers?

Great questions, I am glad that you asked. The answer is actually: You can start at any age! For real, I mean it. While landing a big corporate job might be out of reach for almost any high schooler getting hands on experience and education in systems is anything but. Between books, online videos, articles, home labs, volunteer opportunities, and sometimes even actual paying work with smaller companies a high school student has so many potential avenues to build a resume, get an interview, and get their foot in the proverbial door that the best time for anyone to start studying to be a system administrator is always today.

A big advantage of IT compared to most other fields is that we have a lot of industry-backed certifications that can do quite a lot for building a resume early on in your career and almost none of them require previous industry experience or that you be of any specific age or have a degree. In fact, many high school trade programs specifically target assisting students get their first industry certifications during high school. A motivated student could go far beyond what any normal school program could provide and quite quickly building a resume of both experience and certifications.

Getting a great job might require turning eighteen before getting a chance to really apply, but there is every possibility that a high school student could use their school years to build up an impressive level of study and walk almost immediately into a great starting role and move up very quickly. If a student started studying towards an IT career in middle school and stuck with it for four to six years, it would not be unreasonable to have gained more hands-on experience than many mid-career working adults. Not every business will be keen to hire someone so young regardless of what they can bring to