Architecture and Design of the Linux Storage Stack E-Book

Muhammad Umer

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 381

Veröffentlichungsjahr: 2023

Beschreibung

The Linux storage stack serves as a prime example of meticulously coordinated layers. Embark on a journey through the kernel code with Architecture and Design of the Linux Storage Stack, crafted for anyone seeking in-depth knowledge about the layered design of Linux storage and its landscape.
You’ll explore the Linux storage stack and its various concepts. You’ll unlock the secrets of the virtual filesystem and the actual filesystem and the differences in their implementation, the role of the block layer, the Multi-Queue and Device Mapper frameworks, I/O schedulers, physical storage layout, and how to analyze all the layers in the storage stack.
By the end of this book, you’ll be acquainted with how a simple I/O request from a process travels down through all the layers and ends up in physical storage.

Leseprobe

Architecture and Design of the Linux Storage Stack

Gain a deep understanding of the Linux storage landscape and its well-coordinated layers

Muhammad Umer

BIRMINGHAM—MUMBAI

Architecture and Design of the Linux Storage Stack

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Pavan Ramchandani

Publishing Product Manager: Prachi Sawant

Senior Editor: Runcil Rebello

Technical Editor: Arjun Varma

Copy Editor: Safis Editing

Project Coordinator: Ashwin Kharwa

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Alishon Mendonca

Marketing Coordinator: Marylou De Mello

First published: July 2023

Production reference: 1280623

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-83763-996-0

www.packtpub.com

To Ammi, Abu, Hammad, Sharjeel, and Amna, who faced some arduous times because of me and taught me the meaning of sacrifice, patience, hard work, and perseverance. You have always stood by my side on good days and stood even closer on my bad days.

And to Ahad, who brings joy and wonder into our lives.

I dedicate this book to you with all my love.

Contributors

About the author

Muhammad Umer is a systems engineer and a trainer, with more than six years of experience working with Linux-based systems, high availability (HA) design and architecture, tuning operating systems, and underlying hardware for optimal performance and root cause analysis. He likes to keep himself updated with the latest happenings in the world of technology and is a Red Hat Certified Architect (RHCA). Umer is an avid Linux enthusiast, with a strong focus on storage-related technologies. This passion has driven him to author a book on the topic, drawing on his extensive experience in the field.

I want to thank the people with whom I’ve worked over the years. Each one of them has taught me something in one way or another. I’m also grateful to the wonderful team at Packt for their support throughout this journey.

About the reviewer

Pradeep Roy Kandru is a Linux kernel programmer and has extensive experience working in different layers of the Kernel. Having worked in both technical and development roles, his mainstay has been a broad base of skills and the ability to work on diverse projects. Academically, he has secured All India Rank 43 in the Graduate Aptitude Test in Engineering and holds a master’s degree in internet science engineering, gained at the Department of Computer Science at the Indian Institute of Science (IISc), Bangalore, India. Along with reviewing other technical books, he also works on his own forthcoming technical book as an author. He worked for the companies IBM, India Software Labs, Veritas, and Depik Technologies. His current interest includes eBPF observability.

Table of Contents

Preface

Part 1: Diving into the Virtual Filesystem

1

Where It All Starts From – The Virtual Filesystem

Technical requirements

Understanding storage in a modern-day data center

Interacting with storage in Linux

Understanding system calls

Explaining the need for a Virtual Filesystem

Describing the VFS

Implementing a common filesystem interface through the VFS

Peeking at the source code

Tracing VFS functions

Explaining the Everything is a file philosophy

Summary

2

Explaining the Data Structures in a VFS

Technical requirements

Data structures in VFS

Inodes – indexing files and directories

Defining an inode in the kernel

Defining inode operations

Tracking file data on disk through inodes

Can a filesystem run out of inodes?

Directory entries – mapping inodes to filenames

Dentry cache

Dentry states

Dentry operations

File objects – representing open files

Defining file operations

Can a process run out of file descriptors?

Superblocks – describing filesystem metadata

Superblock operations in the kernel

Linking

Summarizing the four structures

Page cache

Reading

Summary

3

Exploring the Actual Filesystems Under the VFS

Technical requirements

The Linux filesystem gallery

The diary of a filesystem – the concept of journaling

The burning question – doesn’t journaling adversely affect performance?

The curious case of CoW filesystems

Extended filesystem

Blocks – the lingua franca of filesystems

Is there a more efficient approach to organizing data?

The structural layout of an Ext4 filesystem

Network filesystem

NFS architecture

Comparing NFS with regular block filesystems

FUSE – a unique method for creating filesystems

Summary

Part 2: Navigating Through the Block Layer

4

Understanding the Block Layer, Block Devices, and Data Structures

Technical requirements

Explaining the role of the block layer

Defining block devices

The defining characteristics of a block device

Looking at the representation of block devices

Looking at data structures in the block layer

The register_blkdev function (block device registration)

The block_device structure (representing block devices)

The gendisk structure (representing physical disks)

The buffer_head structure (representing blocks in memory)

The bio structure (representing active block I/Os)

The bio_vec structure (representing vector I/O)

Requests and request queues (representing pending I/O requests)

The journey of an I/O request in the block layer

Summary

5

Understanding the Block Layer, Multi-Queue, and Device Mapper

Technical requirements

Looking at problems with single-request queues

Understanding the multi-queue block I/O framework

Looking at data structures

Looking at the device mapper framework

Looking at multi-tier caching mechanisms in the block layer

Summary

6

Understanding I/O Handling and Scheduling in the Block Layer

Technical requirements

Understanding the I/O handling techniques in block layer

Sorting

Merging

Coalescing

Plugging

Explaining the Linux I/O schedulers

The MQ-deadline scheduler – guaranteeing a start service time

Budget fair queuing – providing proportional disk share

Kyber – prioritizing throughput

None – minimal scheduling overhead

Discussing the scheduling conundrum

Summary

Part 3: Descending into the Physical Layer

7

The SCSI Subsystem

Technical requirements

The device driver model

Explaining the SCSI subsystem

SCSI architecture

The client and server model

Summary

8

Illustrating the Layout of Physical Media

Technical requirements

Understanding mechanical hard drives

Looking at the physical layout and addressing

Looking at bad sectors

Looking at hard drive performance

Understanding where mechanical drives lag

Explaining the architecture of solid-state drives

Looking at the physical layout and addressing

Looking at reads and writes

Erasing, garbage collection, and the illusion of available space

Looking at wear leveling

Looking at bad block management

Looking at SSD performance

Understanding where SSDs lag

Understanding drive endurance

Reinventing SSDs with NVMe

Summary

Part 4: Analyzing and Troubleshooting Storage Performance

9

Analyzing Physical Storage Performance

Technical requirements

How do we gauge performance?

Understanding storage topology

Analyzing physical storage

Understanding disk service time

Disk access patterns

Determining reads/writes ratio and I/O size

Disk cache

IOPS and throughput

Queue depth

Determining disk busyness

I/O wait

Using disk I/O analysis tools

Establish a baseline using top

The iotop utility

The iostat utility

Performance Co-Pilot

The vmstat command

Pressure Stall Index

Summary

10

Analyzing Filesystems and the Block Layer

Technical requirements

Investigating filesystems and the block layer

The different types of filesystem I/O

What causes filesystem latency?

Identifying the target layers

Finding the right tools

Tracing application calls

Tracing VFS calls

Analyzing cache usage

Analyzing filesystems

Analyzing block I/O

Summarizing the tools

Summary

11

Tuning the I/O Stack

Technical requirements

How memory usage affects I/O

Tuning the memory subsystem

Using direct I/O

Controlling the write-back frequency

Tuning the filesystem

Block size

Filesystem I/O alignment

LVM I/O alignment

Journaling

Barriers

Timestamps

Read-ahead

Discarding unused blocks

Choosing the right scheduler

Summary

Index

Other Books You May Enjoy

Preface

The major benefit of open source operating systems such as Linux is that anyone can dig deep and uncover how they work under the hood. Even with the astronomical advances in software development, the Linux kernel remains one of the most complex pieces of code. Developers, programmers, and would-be kernel hackers are constantly diving into the kernel code and are pushing for new features. Hobbyists and enthusiasts like me are trying to understand and unravel mysteries.

As one of those enthusiasts, I have spent quite some time exploring the intricacies of the Linux storage stack. From simple hard drives to complex networked storage systems, Linux is at the heart of many of the world’s most complex storage technologies. Over the last few years, I’ve had a chance to work on Linux and several storage technologies, which piqued my interest in this area. This book is the result of that exploration. I’ve tried peeling off all the layers of the Linux storage stack to show how they all work in unison. My goal is to share what I have learned with others who share my fascination with this topic.

This book provides an in-depth and conceptual overview of the Linux storage stack. It covers all of the major components of the stack and includes an in-depth and detailed analysis of the storage subsystem and its architecture, the virtual filesystem layer, different filesystems and the differences in their implementations, the block layer, multi-queue and device mapper frameworks, scheduling, and physical layers. It also covers a variety of topics relating to storage performance analysis, tuning, and troubleshooting.

I’m sure anyone who wants to widen their understanding of Linux and its storage landscape will find this book informative and useful.

Who this book is for

The primary target audience for this book is Linux system and storage administrators, engineers, Linux professionals, the Linux community in general, and anyone who wants to widen their understanding of Linux. When working in any environment, it is imperative to have a deep understanding of the technology that you’re working on. I believe this book will give you a better understanding of Linux internals, equip you with the necessary knowledge, and increase your general interest in the subject.

What this book covers

Chapter 1, Where It All Starts From – The Virtual Filesystem, provides an introduction to the virtual filesystem (VFS) in the Linux kernel. The chapter will explain the pivotal role of the VFS in the I/O stack and provide a strong conceptual understanding of VFS as it is the starting point of an I/O request in Linux.

Chapter 2, Explaining the Data Structures in a VFS, introduces the various data structures utilized by the VFS in the kernel. The usage of structures by the kernel such as inodes, directory entries, and file objects to store file metadata, directories, and open files will be explained. Furthermore, the method by which the superblock structure enables the kernel to record filesystem characteristics will also be covered. Finally, the page caching mechanism in the kernel will be explained.

Chapter 3, Exploring the Actual Filesystems Under the VFS, introduces the concept of filesystems in Linux. One of the most popular block-based filesystems in Linux, the extended filesystem, will be explained. Additionally, important filesystem concepts such as journaling and copy-on-write will be discussed in detail. The chapter will also delve into the differences between file and block I/O and explore the network filesystem. Finally, the concept of userspace filesystems will be introduced.

Chapter 4, Understanding the Block Layer, Block Devices, and Data Structures, introduces the block layer in the kernel. The chapter will explain the concept of block devices and how they differ from character devices, and covers the major data structures in the block layer.

Chapter 5, Understanding the Block Layer, Multi-Queue, and Device Mapper, introduces the device mapper framework and the multi-queue block I/O queuing mechanism in the kernel. The chapter will explain how the multi-queue framework improves the performance of I/O operations on modern storage devices. Further, the role of the device mapper in laying the foundation for features such as LVM will be explained.

Chapter 6, Understanding I/O Handling and Scheduling in the Block Layer, discusses the different I/O handling mechanisms and I/O schedulers in the kernel. Operations such as merging, coalescing, and plugging are discussed in terms of how they are handled in the block layer. The different I/O schedulers supported in the Linux kernel are also explained in detail, along with the differences in their operational logic.

Chapter 7, The SCSI Subsystem, focuses on the SCSI subsystem in Linux and its multi-layered architecture. The chapter will explain the multilayered SCSI architecture, the SCSI device addressing mechanism, and the main data structures in the SCSI layer.

Chapter 8, Illustrating the Layout of Physical Media, discusses the different storage media available today, and the differences in their architecture. The traditional mechanical drives, solid-state drives, and the new NVMe interface are compared.

Chapter 9, Analyzing Physical Storage Performance, covers the performance analysis and characteristics of the storage subsystem. This chapter will present the different metrics that can be used to assess the performance of physical storage. Later on, the different tools and mechanisms that can be used to gauge storage performance are discussed.

Chapter 10, Analyzing Filesystems and the Block Layer, focuses on the techniques that can be used to analyze the performance of the block layer and filesystems. This chapter will explain the different types of filesystem I/O and the factors that can affect the I/O request of an application. Later on, the different tools and tracing mechanisms, such as the Berkeley Packet Filter compiler collection, are looked at to identify potential bottlenecks at each layer.

Chapter 11, Tuning the I/O Stack, discusses some recommended practices for tuning the underlying storage layer for application needs. The different options that can be used to tune the performance at each layer are discussed.

To get the most out of this book

The main objective of the book is to comprehend the inner workings of the Linux kernel and its major subsystems. Therefore, to get the most out of this book you’ll need to have a decent understanding of operating system concepts in general and Linux in particular. Above all, it is important to approach these topics with patience, curiosity, and a willingness to learn.

Software/hardware covered in the book

Operating system requirements

Linux

The commands for installing specific packages are included in the technical requirements section of each chapter.

Download the example code files

We have code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/Uz9ge

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “If we look at the sd* devices in /dev, notice that the file type is being shown as b, for block devices.”

A block of code is set as follows:

struct block_device {         sector_t                bd_start_sect;         sector_t                bd_nr_sectors;         struct disk_stats __percpu *bd_stats;         unsigned long           bd_stamp;         bool                    bd_read_only;         dev_t                   bd_dev;         atomic_t                bd_openers;         struct inode *          bd_inode; [……..]

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

struct block_device {         sector_t                bd_start_sect;         sector_t                bd_nr_sectors;         struct disk_stats __percpu *bd_stats;         unsigned long           bd_stamp;         bool                    bd_read_only;         dev_t                   bd_dev;         atomic_t                bd_openers;         struct inode *          bd_inode; [……..]

Any command-line input or output is written as follows:

[root@linuxbox ~]# find / -inum 67118958 -exec ls -l {} \; -rw-r--r-- 1 root root 220 Jun 15 22:30 /etc/hosts [root@linuxbox ~]#

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “In the case of a directory, the type field in an inode is a directory.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Architecture and Design of Linux Storage Stack, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781837639960

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Diving into the Virtual Filesystem

This part offers a detailed introduction to the virtual filesystem (VFS) layer and the actual filesystems underneath it. You will learn about VFS, its major data structures, the extended filesystem family, and the major concepts associated with the different filesystems in Linux.

This part contains the following chapters:

Chapter 1, Where It All Starts From – The Virtual FilesystemChapter 2, Explaining the Data Structures in a VFSChapter 3, Exploring the Actual Filesystems Under the VFS