Mastering Embedded Linux Development E-Book

Frank Vasquez

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 932

Veröffentlichungsjahr: 2025

Beschreibung

Mastering Embedded Linux Development is designed to be both a learning resource and a reference for your embedded Linux projects.
In this fourth edition, you'll learn the fundamental elements that underpin all embedded Linux projects: the toolchain, the bootloader, the kernel, and the root filesystem. First, you will download and install a pre-built toolchain. After that, you will cross-compile each of the remaining three elements from scratch and learn to automate the process using Buildroot and the Yocto Project. The book progresses with coverage of over-the-air software updates and rapid prototyping with add-on boards. Two new chapters tackle modern development practices, including Python packaging and deploying containerized applications. These are followed by a chapter on writing multithreaded code and another on techniques to manage memory efficiently. The final chapters demonstrate how to debug your code, whether it resides in user space or in the Linux kernel itself. In addition to GNU debugger (GDB), the book also covers the different tracers and profilers that are available for Linux so that you can quickly pinpoint any performance bottlenecks in your system.
By the end of this book, you will be able to create efficient and secure embedded devices with Linux that will delight your users.

Leseprobe

Mastering Embedded Linux Development

Fourth Edition

Craft fast and reliable embedded solutions with Linux 6.6 and The Yocto Project 5.0 (Scarthgap)

Frank Vasquez | Chris Simmonds

Mastering Embedded Linux Development

Fourth Edition

Copyright © 2025 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Portfolio Director: Rohit Rajkumar

Relationship Lead: Kaustubh Manglurkar

Content Engineer: Anuradha Joglekar

Program Manager: Sandip Tadge

Technical Editors: Kushal Sharma and Gaurav Gavas

Copy Editor: Safis Editing

Indexer: Rekha Nair

Proofreader: Safis Editing

Production Designers: Ganesh Bhadwalkar, Gokul Raj ST, and Shantanu Zagade

Marketing Owner: Nivedita Pande

Growth Lead: Namita Velgekar

First edition: December 2015 Second edition: June 2017 Third edition: May 2021 Fourth edition: May 2025

Production reference: 1220525

Published by Packt Publishing Ltd.Grosvenor House 11 St Paul’s Square Birmingham B3 1RB, UK.

ISBN 978-1-80323-259-1

www.packt.com

To Chris Simmonds, whose words stand the test of time. And to my wife, Deborah, for encouraging me to do this again. The world still runs on Linux.

– Frank Vasquez

Contributors

About the authors

Frank Vasquez is an independent software consultant specializing in consumer electronics. He has more than a decade of experience designing and building embedded Linux systems. During that time, he has shipped numerous products, including a rackmount DSP audio server, a diver-held sonar camcorder, an IoT hotspot, a home battery, and a grid-scale energy storage system. Since the third edition of this book was published, Frank has also become a frequent speaker at open-source software conferences including The Yocto Project Summit, Embedded Linux Conference, FOSDEM, and All Systems Go! He is passionate about learning new technologies and teaching them to others.

Chris Simmonds is a software consultant and trainer living in southern England. He has almost two decades of experience in designing and building open-source embedded systems. He is the founder and chief consultant at 2net Ltd, which provides professional training and mentoring services in embedded Linux, Linux device drivers, and Android platform development. He has trained engineers at many of the biggest companies in the embedded world, including ARM, Qualcomm, Intel, Ericsson, and General Dynamics. He is a frequent presenter at open-source and embedded conferences, including the Embedded Linux Conference and Embedded World.

About the reviewers

Alex Trifonov is a senior software engineer at Cisco, specializing in embedded systems, Linux, and networking. With a strong background in firmware development, real-time systems, and hardware-software integration, he has contributed to building reliable and scalable embedded solutions for a variety of applications.

Passionate about the intersection of software and hardware, Alex has extensive experience in networking, home automation, and industrial embedded systems. His expertise spans low-level programming, system optimization, and security, making him a valuable contributor to the embedded development community.

As a technical reviewer for Mastering Embedded Linux Development, Alex ensures that the book provides accurate, practical, and industry-relevant insights, making it an essential resource for engineers looking to advance their skills.

Khem Raj is an embedded Linux architect and a well-known open-source developer and thought leader. Presently, he works at Comcast on CPE devices and has helped create the RDK stack, which is used on device platforms in millions of devices today. He is a member of The Yocto Project Advisory Committee and serves on the project Technical Steering Council, shaping the project’s technical direction. He is a frequent speaker at open-source conferences and has spoken on varied subjects ranging from embedded Linux systems to open-source processes. Along with core toolchains in The Yocto Project, he is also the maintainer of several important Yocto Project layers, notably meta-openembedded, meta-clang, meta-riscv, and meta-raspberrypi, to name a few. He has also been nominated as an RISC-V architecture ambassador. He has reviewed other books, including Mastering Linux Device Driver Development: Write custom device drivers to support computer peripherals in Linux operating systems. In his free time, he reads non-fiction, writes programs for fun, likes hiking, and co-hosts the TMPDIR podcast.

I would like to thank my wife, Sweta, and children, Himangi and Vihaan, for always supporting me in my endeavors.

Join our community on Discord

Join our community’s Discord space for discussions with the authors and other readers: https://packt.link/embeddedsystems

Contents

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Download a Free PDF Copy of This Book

Part 1: Elements of Embedded Linux

Starting Out

Choosing Linux

When not to choose Linux

Meeting the players

Moving through the project life cycle

The four elements of embedded Linux

Navigating open source

Licenses

Selecting hardware for embedded Linux

Obtaining the hardware for this book

The Raspberry Pi 4

The BeaglePlay

QEMU

Provisioning your development environment

Summary

Learning about Toolchains

Technical requirements

Introducing toolchains

Types of toolchains

CPU architectures

Choosing the C library

Finding a toolchain

Anatomy of a toolchain

Finding out about your cross compiler

sysroot, library, and header files

Other tools in the toolchain

Looking at the components of the C library

Linking with libraries – static and dynamic linking

Static libraries

Shared libraries

Understanding shared library version numbers

Art of cross-compiling

Simple makefiles

Autotools

An example: SQLite

Package configuration

Problems with cross-compiling

CMake

Summary

Further study

All about Bootloaders

Technical requirements

What does a bootloader do?

Boot sequence

Phase 1 – ROM code

Phase 2 – Secondary Program Loader

Phase 3 – Tertiary Program Loader

Moving from the bootloader to the kernel

Introducing device trees

Device tree basics

reg property

Labels and interrupts

Device tree include files

Compiling a device tree

U-Boot

Building U-Boot

Obtaining a 32-bit toolchain

Building U-Boot SPL for R5

Generating an image for the R5

Building TF-A for the main A53 CPU

Building U-Boot for the main A53 CPU

Installing U-Boot

Using U-Boot

Environment variables

Boot image format

Loading images

Booting Linux

Automating the boot with U-Boot scripts

Porting U-Boot to a new board

Board-specific files

Configuring header files

Building and testing

Falcon mode

Summary

Configuring and Building the Kernel

Technical requirements

What does the kernel do?

Choosing a kernel

Kernel development cycle

Stable and long-term support releases

Vendor support

Licensing

Best practices

Configuring the kernel

Getting the source

Understanding kernel configuration– Kconfig

Using LOCALVERSION to identify your kernel

When to use kernel modules

Compiling with Kbuild

Finding out which kernel target to build

Build artifacts

Compiling device trees

Compiling modules

Cleaning kernel sources

Building and booting the kernel

Building a kernel for the Raspberry Pi 4

Booting the Raspberry Pi 4

Building a kernel for the BeaglePlay

Booting the BeaglePlay

Building a kernel for QEMU

Booting QEMU

Observing the kernel boot process

Kernel panic

Early user space

Kernel messages

Kernel command line

Porting Linux to a new board

A new device tree

Summary

Further study

Building a Root Filesystem

Technical requirements

What should be in the root filesystem?

Directory layout

Staging directory

POSIX file access permissions

File ownership permissions in the staging directory

Programs for the root filesystem

init program

Shell

Utilities

BusyBox to the rescue!

Building BusyBox

ToyBox – an alternative to BusyBox

Libraries for the root filesystem

Reducing size by stripping

Device nodes

Proc and sysfs filesystems

Mounting filesystems

Kernel modules

Transferring the root filesystem to the target

Creating a boot initramfs

Standalone initramfs

Booting an initramfs

Booting with QEMU

Booting the BeaglePlay

Mounting proc

Building an initramfs into the kernel image

Building an initramfs using a device table

Old initrd format

init program

Starting a daemon process

Configuring user accounts

Adding user accounts to the root filesystem

A better way of managing device nodes

Using devtmpfs

Using mdev

Are static device nodes so bad after all?

Configuring the network

Network components for glibc

Creating filesystem images with device tables

Booting the BeaglePlay

Mounting the root filesystem using NFS

Testing with the BeaglePlay

Problems with file permissions

Using TFTP to load the kernel

Summary

Further study

Part 2: Building Embedded Linux Images

Selecting a Build System

Technical requirements

Comparing build systems

Distributing binaries

Introducing Buildroot

Background

Stable releases and long-term support

Installing

Configuring

Running

Targeting real hardware

Creating a custom BSP

U-Boot

Linux

Build

Adding your own code

Overlays

Adding a package

License compliance

Introducing The Yocto Project

Background

Stable releases and supports

Installing The Yocto Project

Configuring

Building

Running the QEMU target

Layers

BitBake and recipes

Customizing images via local.conf

Writing an image recipe

Creating an SDK

The license audit

Summary

Further study

Developing with Yocto

Technical requirements

Building on top of an existing BSP

Building an existing BSP

Controlling Wi-Fi

Controlling Bluetooth

Adding a custom layer

Capturing changes with devtool

Development workflows

Creating a new recipe

Modifying the source built by a recipe

Upgrading a recipe to a newer version

Building your own distro

When and when not to

Creating a new distro layer

Configuring your distro

Adding more recipes to your distro

Runtime package management

Provisioning a remote package server

Summary

Further study

Yocto under the Hood

Technical requirements

Decomposing Yocto’s architecture and workflow

Metadata

Build tasks

Image generation

Separating metadata into layers

Troubleshooting build failures

Isolating errors

Dumping the environment

Reading the task log

Adding more logging

Running commands from devshell

Graphing dependencies

Understanding BitBake syntax and semantics

Tasks

Dependencies

Inter-task dependencies

Build-time dependencies

Runtime dependencies

Variables

Assignment and expansion

Appending and prepending

Overrides

Inline Python

Functions

Shell

Python

RDEPENDS revisited

Summary

Further study

Part 3: System Architecture and Design Decisions

Creating a Storage Strategy

Technical requirements

Storage options

NOR flash

NAND flash

Managed flash

The MultiMediaCard and Secure Digital cards

eMMC

Other types of managed flash

Accessing flash memory from the bootloader

U-Boot and NOR flash

U-Boot and NAND flash

U-Boot and MMC, SD, and eMMC

Accessing flash memory from Linux

Memory technology devices

MTD partitions

MTD device drivers

The MTD character device, mtd

The MTD block device, mtdblock

Logging kernel oops to MTD

Simulating NAND memory

The MMC block driver

Filesystems for flash memory

Flash translation layers

Filesystems for NOR and NAND flash memory

JFFS2

Summary nodes

Clean markers

Creating a JFFS2 filesystem

YAFFS2

Creating a YAFFS2 filesystem

UBI and UBIFS

UBI

UBIFS

Filesystems for managed flash

Flashbench

Discard and TRIM

Ext4

F2FS

FAT16/32

Read-only compressed filesystems

SquashFS

Temporary filesystems

Making the root filesystem read-only

Filesystem choices

Summary

Further study

Updating Software in the Field

Technical requirements

From where do updates originate?

What to update

Bootloader

Kernel

Root filesystem

System applications

Device-specific data

Components that need to be updated

Basics of software updates

Making updates robust

Making updates fail-safe

Making updates secure

Types of update mechanism

Symmetric image update

Asymmetric image update

Atomic file updates

OTA updates

Using Mender for local updates

Building the Mender client

Installing an update with Mender

Using Mender for OTA updates

Using SWUpdate for local updates

Summary

Interfacing with Device Drivers

Technical requirements

Role of device drivers

Character devices

Block devices

Network devices

Finding out about drivers at runtime

Getting information from sysfs

Devices – /sys/devices

Drivers – /sys/class

Block drivers – /sys/block

Finding the right device driver

Device drivers in user space

GPIO

Handling interrupts from GPIO

LEDs

I2C

SPI

Writing a kernel device driver

Designing a character driver interface

Anatomy of a device driver

Compiling kernel modules

Loading kernel modules

Discovering the hardware configuration

Device trees

Platform data

Linking hardware with device drivers

Summary

Further study

Prototyping with Add-On Boards

Technical requirements

Mapping schematics to pins

Reading schematics

Installing Debian on the BeaglePlay

Prototyping with add-on boards

Testing hardware peripherals

Attaching the GNSS Click 7 add-on board

Receiving NMEA messages

Attaching the Environment Click add-on board

Reading sensor values

Attaching the OLED C Click add-on board

Displaying an animation

Summary

Further study

Starting Up – The init Program

Technical requirements

After the kernel has booted

Introducing the init programs

BusyBox init

Buildroot init scripts

System V init

inittab

The init.d scripts

Adding a new daemon

Starting and stopping services

systemd

Building systemd with The Yocto Project and Buildroot

Introducing targets, services, and units

Units

Services

Targets

How systemd boots the system

Adding your own service

Adding a watchdog

Implications for embedded Linux

Summary

Further study

Managing Power

Technical requirements

Measuring power usage

Scaling the clock frequency

CPUFreq driver

Using CPUFreq

Selecting the best idle state

CPUIdle driver

Tickless operation

Powering down peripherals

Putting the system to sleep

Power states

Wakeup events

Timed wakeups from the real-time clock

Summary

Further study

Part 4: Developing Applications

Packaging Python

Technical requirements

Retracing the origins of Python packaging

distutils

setuptools

setup.py

Installing Python packages with pip

requirements.txt

Managing Python virtual environments with venv

Installing precompiled binaries with conda

Environment management

Package management

Summary

Further study

Deploying Container Images

Technical requirements

Getting Docker

What is DevOps?

Continuous integration and continuous deployment

Infrastructure as code

Security is a shared responsibility

Monitoring and observability

Continuous improvement

Transparency

DevOps and Embedded Linux

Continuous integration and cross-compilation

Automated testing on real hardware

Continuous delivery and OTA updates

Infrastructure as code and build systems

Securing edge devices

Monitoring and observability of edge devices

Deploying Python applications with Docker

Anatomy of a Dockerfile

Building a Docker image

Running a Docker image

Fetching a Docker image

Publishing a Docker image

Cleaning up

Setting up a CI/CD pipeline for a Python application

Creating a Dockerfile

Creating a GitHub Actions workflow

Pulling and running the latest image

Adding Docker to a Yocto image

Adding the meta-virtualization layer

Installing Docker

Verifying the Docker daemon is running

Updating software with Docker

Creating an account

Creating an application

Adding a device

Installing the CLI

Pushing a project

Summary

Further study

Learning about Processes and Threads

Technical requirements

Process or thread?

Processes

Creating a new process

Terminating a process

Running a different program

Daemons

Inter-process communication

Message-based IPC

Unix (or local) sockets

FIFOs and named pipes

POSIX message queues

Summary of message-based IPC

Shared memory-based IPC

POSIX shared memory

Threads

Creating a new thread

Terminating a thread

Compiling a program with threads

Inter-thread communication

Mutual exclusion

Changing conditions

Partitioning the problem

ZeroMQ

Getting pyzmq

Messaging between processes

Messaging within processes

Scheduling

Fairness versus determinism

Time-shared policies

Niceness

Real-time policies

Choosing a policy

Choosing a real-time priority

Summary

Further study

Managing Memory

Technical requirements

Virtual memory basics

Kernel-space memory layout

How much memory does the kernel use?

User-space memory layout

Process memory map

Swapping

Swapping to compressed memory (zram)

Mapping memory with mmap

Using mmap to allocate private memory

Using mmap to share memory

Using mmap to access device memory

How much memory does my application use?

Per-process memory usage

Using top and ps

Using smem

Other tools to consider

Identifying memory leaks

mtrace

Valgrind

Running out of memory

Summary

Further study

Part 5: Debugging and Optimizing Performance

Debugging with GDB

Technical requirements

GNU debugger

Preparing to debug

Debugging applications

Remote debugging using gdbserver

Setting up The Yocto Project for remote debugging

Setting up Buildroot for remote debugging

Starting to debug

Connecting GDB and gdbserver

Setting the sysroot

GDB command files

Overview of GDB commands

Running to a breakpoint

Extending GDB with Python

Native debugging

The Yocto Project

Buildroot

Just-in-time debugging

Debugging forks and threads

Core files

Using GDB to look at core files

GDB user interfaces

Terminal User Interface

Data Display Debugger

Visual Studio Code

Debugging kernel code

Debugging kernel code with kgdb

A sample debug session

Debugging early code

Debugging modules

Debugging kernel code with kdb

Looking at an Oops

Preserving the Oops

Summary

Further study

Profiling and Tracing

Technical requirements

Observer effect

Beginning to profile

Profiling with top

Profiling with GDB

Introducing perf

Configuring the kernel for perf

Building perf with The Yocto Project

Building perf with Buildroot

Profiling with perf

Call graphs

perf annotate

Tracing events

Introducing Ftrace

Preparing to use Ftrace

Using Ftrace

Dynamic Ftrace and trace filters

Trace events

Using LTTng

LTTng and The Yocto Project

LTTng and Buildroot

Using LTTng for kernel tracing

Using eBPF

Configuring the kernel for eBPF

Building ply with Buildroot

Using ply

Using Valgrind

Callgrind

Helgrind

Using strace

Summary

Further study

Real-Time Programming

Technical requirements

What is real time?

Identifying sources of non-determinism

Understanding scheduling latency

Kernel preemption

Real-time Linux kernel (PREEMPT_RT)

Threaded interrupt handlers

Preemptible kernel locks

Getting the PREEMPT_RT patches

The Yocto Project and PREEMPT_RT

High-resolution timers

Avoiding page faults

Interrupt shielding

Measuring scheduling latencies

cyclictest

Using Ftrace

Combining cyclictest and Ftrace

Summary

Further study

Index

Download a Free PDF Copy of This Book

Landmarks

Cover

Index

Share your thoughts

Once you’ve read Mastering Embedded Linux Development, Fourth Edition, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a Free PDF Copy of This Book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily.

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below:

https://packt.link/free-ebook/9781803232591

Submit your proof of purchase.That’s it! We’ll send your free PDF and other benefits to your email directly.

Part 1

Elements of Embedded Linux

In this part, you will explore the four key elements of any embedded Linux project. You will learn how to select a toolchain, build the bootloader, and build the kernel for your target device. Chapter 5 requires you to build a root filesystem step by step from scratch. These manual exercises are difficult, but by the end of this section, you with have a deeper understanding of how embedded Linux works and a greater appreciation for tools that can automate this board bring-up phase.

This part has the following chapters:

Chapter 1, Starting OutChapter 2, Learning about ToolchainsChapter 3, All about BootloadersChapter 4, Configuring and Building the KernelChapter 5, Building a Root Filesystem

1

Starting Out

You are about to begin working on your next project, and this time, it is going to run Linux. What should you think about before you put finger to keyboard? Let’s begin with a high-level look at embedded Linux and see why it is popular, what the implications of open source licenses are, and what kind of hardware you need to run it.

Linux first became a viable choice for embedded devices around 1999. That was when AXIS released the 2100 Network Camera and TiVo released their first Digital Video Recorder (DVR). Both were the first Linux-powered devices in their category. Since 1999, Linux has become increasingly popular to the point that today it is the Operating System (OS) of choice for many classes of product. In 2024, there were over three billion devices running Linux. That includes all the smartphones running Android, which uses a Linux kernel, and hundreds of millions of set-top boxes, smart TVs, and Wi-Fi routers. We must not forget other devices, such as vehicle diagnostics, industrial equipment, and medical monitoring units, that ship in smaller volumes.

In this chapter, we will cover the following topics:

Choosing LinuxWhen not to choose LinuxMeeting the playersMoving through the project life cycleNavigating open sourceSelecting hardware for embedded LinuxObtaining the hardware for this bookProvisioning your development environment

Choosing Linux

Why is Linux so pervasive? And why does something as simple as a TV need to run something as complex as Linux just to display streaming video on a screen?

The simple answer is Moore’s law. Gordon Moore, cofounder of Intel, observed in 1965 that the density of components on a chip doubles approximately every two years. That applies to the devices that we design and use in our everyday lives just as much as it does to desktops, laptops, and servers. At the heart of most embedded devices is a highly integrated chip that contains one or more processor cores and interfaces with main memory, mass storage, and peripherals of many types. This is referred to as a System on Chip (SoC). SoCs are increasing in complexity in accordance with Moore’s law. A typical SoC has a technical reference manual that stretches to thousands of pages.

Your TV isn’t simply displaying a video stream like the analog sets of old. The stream is digital, possibly encrypted, and needs processing to produce an image. Your TV is (or soon will be) connected to the internet. It can receive content from smartphones, tablets, laptops, desktops, and home media servers. It can be used to play games, stream video, and display live feeds from security cameras. You need a full OS to manage this degree of complexity.

Here are some points that drive the adoption of Linux:

Linux has the necessary functionality. It has a good scheduler, a good network stack, support for USB, Wi-Fi, Bluetooth, many kinds of storage media, multimedia devices, and so on. It ticks all the boxes.Linux has been ported to a wide range of processor architectures, including some that are very commonly found in SoC designs – Arm, RISC-V, x86, PowerPC, and MIPS.Linux is open source, so you have the freedom to get the source code and modify it to meet your needs. You or someone working on your behalf can create a board support package for your device. You can add protocols, features, and technologies that may be missing from the mainline source code. You can remove features that you don’t need to reduce memory and storage requirements. Linux is flexible.Linux has an active community (in the case of the Linux kernel, very active). There is a new release of the kernel every 8 to 10 weeks, and each release contains code from more than 1,000 developers. An active community means that Linux is up to date and supports current hardware, protocols, and standards. The Linux Foundation is a non-profit organization with backing from big tech.

The foundation acts as a steward for several major open source projects besides Linux, including Kubernetes and PyTorch. It also hosts yearly events around the world like the Open Source Summit and Linux Plumbers Conference.

Open source licenses guarantee that you have access to the source code. There is no vendor lock-in.

For these reasons, Linux is an ideal choice for complex devices. But there are a few caveats I should mention here. Complexity makes it harder to understand. Coupled with the fast-moving development process and the decentralized structures of open source, you need to put some effort into learning how to use it and to keep on re-learning as it changes. I hope that this book helps in the process.

When not to choose Linux

Is Linux suitable for your project? Linux works well where the problem being solved justifies the complexity. It is especially good where connectivity, robustness, and complex user interfaces are required. However, it cannot solve every problem, so here are some things to consider before you jump in:

Is your hardware up to the job? Compared to a traditional Real-Time Operating System (RTOS) such as VxWorks or QNX, Linux requires a lot more resources. It needs at least a 32-bit processor and lots more memory. I will go into more detail in the Selecting hardware for embedded Linux section.Do you have the right skill set? The early parts of a project, the board bring-up, require detailed knowledge of Linux and how it relates to your hardware. Likewise, when debugging and tuning your application you will need to be able to interpret the results. If you don’t have the skills in-house you may want to outsource some of the work. Of course, reading this book helps!Is your system real time? Linux can handle many real-time activities as long as you pay attention to certain details, which I cover in depth in Chapter 21.Will your code require regulatory approval (medical, automotive, aerospace, and so on)? The burden of regulatory verification and validation might make another OS a better choice. Even if you do choose Linux for use in these environments, it may make sense to purchase a commercially available distribution from a company that has supplied Linux for existing products like the one you are building. These commercial Linux vendors include Siemens, Timesys, and Wind River.

Consider these points carefully. Probably the best indicator of success is to look around for similar products that run Linux and see how they did it, and follow best practices.

Meeting the players

Where does open source software come from? Who writes it? In particular, how does it relate to the key components of embedded development – the toolchain, bootloader, kernel, and basic utilities found in the root filesystem?

Open source community: This, after all, is the engine that generates the software you are going to be using. The community is a loose alliance of developers many of whom are funded in some way by a non-profit organization, an academic institution, or a commercial company. They work together to further the aims of the various projects. There are many of them – some small, some large. Some that we will make use of are Linux itself, U-Boot, BusyBox, Buildroot, The Yocto Project, and the many projects under the GNU umbrella.CPU architects: These are the organizations that design the CPUs we use. The important ones here are Arm/Linaro (Arm Cortex-A), Intel (x86 and x86-64), SiFive (RISC-V), and IBM (PowerPC). They implement or at the very least influence support for the basic CPU architecture.SoC vendors: These include Broadcom, Intel, Microchip, NXP, Qualcomm, TI, and many others. They take the kernel and toolchain from the CPU architects and modify them to support their chips. They also create reference boards: designs that are used by the next level down to create development boards and working products.Board vendors and OEMs: These people take the reference designs from SoC vendors and build them into specific products like set-top boxes or cameras. They also create more general-purpose development boards such as those from Advantech and Kontron. An important category is the cheap Single-Board Computers (SBCs) such as BeagleBoard and Raspberry Pi, which have created their own ecosystems of software and hardware add-ons.Commercial Linux vendors: Companies such as Siemens, Timesys, and Wind River offer commercial Linux distributions that have undergone strict regulatory verification and validation across multiple industries (medical, automotive, aerospace, and so on).

These form a chain, with your project usually at the end, which means that you do not have a free choice of components. You cannot simply take the latest kernel from kernel.org, except in rare cases, because it does not have support for the chip or board that you are using.

This is an ongoing problem with embedded development. Ideally, the developers at each link in the chain would push their changes upstream but they don’t. Developers are under constant time pressure and getting patches accepted into the Linux kernel takes major effort. It is not uncommon to find a kernel that has many thousands of patches that are not merged. In addition, SoC vendors tend to actively develop open source components only for their latest chips, meaning that support for any chip more than a couple of years old will be frozen and not receive any updates.

The consequence is that most embedded designs are based on old versions of software. They do not receive security fixes, performance enhancements, or features that are in newer versions. Problems such as Heartbleed (a bug in the OpenSSL library) and Shellshock (a bug in the Bash shell) go unfixed.

What can you do about it? First, ask questions of your vendors (NXP, TI, and Xilinx to name just a few): what is their update policy, how often do they revise kernel versions, what is the current kernel version, what was the one before that, and what is their policy for merging changes upstream? Some vendors are making great strides in this direction. You should prefer their chips.

Secondly, you can take steps to make yourself more self-sufficient. The chapters in Part 1 explain the dependencies in more detail and show you where you can help yourself. Don’t just take the package offered to you by the SoC or board vendor and use it blindly without considering the alternatives.

Moving through the project life cycle

This book is divided into five sections that reflect the phases of a project. The phases are not necessarily sequential. Usually, they overlap, and you will need to jump back to revisit things that were done previously. However, they are representative of a developer’s preoccupations as the project progresses:

Elements of Embedded Linux (Chapters 1 to 5) will help you set up the development environment and create a working platform for the later phases. It is often referred to as the board bring-up phase.Building Embedded Linux Images (Chapters 6 to 8) shows you how to automate the process of building an embedded Linux image by leveraging a build system like Buildroot or The Yocto Project. Automating complex build tasks accelerates the project life cycle so that teams can deliver higher-quality products in less time.System Architecture and Design Choices (Chapters 9 to 14) will inform some of the design decisions you will have to make concerning the storage of programs and data, how to divide work between kernel device drivers and applications, and how to initialize the system.Developing Applications (Chapters 15 to 18) shows you how to package and deploy Python applications, make effective use of the Linux process and thread model, and manage memory in a resource-constrained device. What do packaging and deploying Python applications have to do with embedded Linux? The answer is “not much”, but bear in mind that the word “development” also happens to be in the title of this book. And Chapters 15 and 16 have everything to do with modern-day software development.Debugging and Optimizing Performance (Chapters 19 to 21) describes how to trace, profile, and debug your code in both the application and the kernel. The last chapter explains how to design for real-time behavior when required.

Now, let’s focus on the four basic elements of embedded Linux that comprise the first section of the book.

The four elements of embedded Linux

Every project begins by obtaining, customizing, and deploying these four elements: the toolchain, the bootloader, the kernel, and the root filesystem. This is the topic of the first section of this book.

Toolchain: This is the cross compiler and other tools needed to create code for your target device. A cross compiler generates machine code for a target CPU architecture while running on a different host CPU architecture.Bootloader: This is a bare metal program that initializes the board and the Linux kernel. The term “bare metal” means the program runs directly on the CPU, not on top of an OS.Kernel: This is the heart of the system, managing system resources and interfacing with the hardware.Root filesystem: This contains the libraries and programs that are run once the kernel has completed its initialization.

There is also a fifth element not mentioned here. That is the collection of programs specific to your embedded application that make the device do whatever it is supposed to do, be it weighing groceries, displaying movies, controlling a robot, or flying a drone.

Typically, you will be offered some or all of these elements as a package when you buy your SoC or board. But for the reasons mentioned earlier, they may not be the best choices for you. In the first eight chapters, I will give you the background to make the right selection and introduce two tools that automate the whole process for you: Buildroot and The Yocto Project.

Navigating open source

The components of embedded Linux are open source so now is a good time to consider what that means, why open source licenses work the way they do, and how this affects the often proprietary embedded device you will be creating from it.

Licenses

When talking about open source the word free is often used. People new to the subject often take it to mean nothing to pay and open source software licenses do indeed guarantee that you can use the software to develop and deploy systems for no charge. However, the more important meaning here is freedom since you are free to obtain the source code, modify it in any way you see fit, and redeploy it in other systems. Open source licenses give you this right, but some also require you to share these changes with the public.

Compare that with freeware licenses, which allow you to copy the binaries for no cost but do not give you the source code. Other licenses allow you to use the software for free under certain circumstances, for example, for personal use, but not commercial. These are not open source.

I will provide the following comments in the interest of helping you understand the implications of working with open source licenses, but I would like to point out that I am an engineer and not a lawyer. What follows is my understanding of the licenses and how they are interpreted.

Open source licenses fall broadly into two categories:

Copyleft licenses such as the GNU General Public License (GPL)Permissive licenses such as the BSD and MIT licenses

The permissive licenses say, in essence, that you may modify the source code and use it in systems of your own choosing as long as you do not modify the terms of the license in any way. In other words, apart from that one restriction, you can do with it what you want, including building it into possibly proprietary systems.

The GPL licenses are similar but have clauses that compel you to pass the rights to obtain and modify the software on to your end users. In other words, you share your source code. One option is to make it completely public by putting it onto a public server. Another is to offer it only to your end users by means of a written offer to provide the code when requested.

The GPL goes further to say that you cannot incorporate GPL code into proprietary programs. Any attempt to do so would make the GPL apply to the whole. In other words, you cannot combine GPL and proprietary code in the same program. Aside from the Linux kernel, the GNU Compiler Collection and GNU Debugger, as well as many other freely available tools associated with the GNU project, fall under the umbrella of the GPL.

So, what about libraries? If they are licensed with the GPL, any program linked with them becomes GPL also. However, most libraries are licensed under the GNU Lesser General Public License (LGPL). If this is the case, you are allowed to link with them from a proprietary program.

IMPORTANT NOTE

All of the preceding descriptions relate specifically to the GPL v2 and LGPL v2. I should mention the latest versions of the GPL v3 and LGPL v3. These are controversial and I will admit that I don’t fully understand the implications. However, the intention is to ensure that the GPL v3 and LGPL v3 components in any system can be replaced by the end user, which is in the spirit of open source software for everyone.

The GPL v3 and LGPL v3 have their problems though. There are security issues. If the owner of a device has access to the system code, then so might an unwelcome intruder. Often the defense is to have kernel images signed by an authority such as the vendor so that unauthorized updates are not possible. Is that an infringement of my right to modify my device? Opinions differ.

IMPORTANT NOTE

The TiVo set-top box is an important part of this debate. It uses a Linux kernel, which is licensed under the GPL v2. TiVo has released the source code of their version of the kernel in compliance with the license. TiVo also has a bootloader that will only load a kernel binary that is signed by them. Consequently, you can build a modified kernel for a TiVo box, but you cannot load it on the hardware.

The Free Software Foundation (FSF) takes the position that this is not in the spirit of open source software and refers to this procedure as tivoization. The GPL v3 and LGPL v3 were written explicitly to prevent this from happening. Some projects, the Linux kernel in particular, have been reluctant to adopt the GPL version 3 licenses because of the restrictions they place on device manufacturers.

Selecting hardware for embedded Linux

If you are designing or selecting hardware for an embedded Linux project, what do you look out for?

First, a CPU architecture that is supported by the kernel – unless you plan to add a new architecture yourself of course! Looking at the source code for Linux 5.15 there are 23 architectures each represented by a subdirectory in the arch/ directory. They are all 32-or 64-bit architectures, most with an MMU, but some without. The ones most often found in embedded devices are Arm, RISC-V, PowerPC, MIPS, and x86 each in 32-and 64-bit variants all of which have Memory Management Units (MMUs).Most of this book is written with this class of processor in mind. There is another group that doesn’t have an MMU and that runs a subset of Linux known as a microcontroller Linux or uClinux. These processor architectures include ARC (Argonaut RISC Core), Blackfin, MicroBlaze, and Nios. I will mention uClinux from time to time, but I will not go into detail because it is a rather specialized type.Second, you will need a reasonable amount of RAM. 16 MB is a good minimum, although it is quite possible to run Linux using half of that. It is even possible to run Linux with 4 MB if you are prepared to go to the trouble of optimizing every part of the system. It may even be possible to get lower, but there comes a point at which it is no longer Linux.Third, there is non-volatile storage, usually flash memory. 8 MB is enough for a simple device such as a webcam or basic router. As with RAM, you can create a workable Linux system with less storage if you really want to, but the lower you go the harder it becomes. Linux has extensive support for flash storage devices, including raw NOR and NAND flash chips, and managed flash in the form of SD cards, eMMC chips, USB flash memory, and so on.Fourth, a serial port is very useful, preferably a UART-based serial port. It does not have to be fitted on production boards but makes board bring-up, debugging, and development much easier.Fifth, you need some means of loading software when starting from scratch. Many microcontroller boards are fitted with a Joint Test Action Group (JTAG) interface for this purpose. Modern SoCs can also load boot code directly from the removable media, especially SD and microSD cards, or serial interfaces such as QSPI or USB.

In addition to these basics, there are interfaces to the specific bits of hardware your device needs to get its job done. Mainline Linux comes with open source drivers for many thousands of different devices, and there are drivers available (of variable quality) from the SoC manufacturer and from the OEMs of third-party chips that may be included in the design.

Remember my comments on the commitment and ability of some manufacturers. As a developer of embedded systems, you will find that you spend quite a lot of time evaluating and adapting third-party code, if you have it, or liaising with the manufacturer if you don’t. Finally, you will have to write the device support for the interfaces that are unique to the device or find someone to do it for you.

Obtaining the hardware for this book

The examples in this book are intended to be generic. To make them relevant and easy to follow I have had to choose specific hardware. I have chosen three exemplary devices: the Raspberry Pi 4, BeaglePlay, and QEMU. The first is by far the most popular Arm-based SBC on the market. The second is a widely available SBC that can also be used in serious embedded hardware. The third is a machine emulator that can be used to create a range of systems that are typical of embedded hardware.

It was tempting to use QEMU exclusively, but like all emulations, it is not quite the same as the real thing. Using the Raspberry Pi 4 and BeaglePlay, you have the satisfaction of interacting with real hardware and seeing real LEDs flash. The BeaglePlay, like the BeagleBone Black before it, is open source hardware, unlike the Raspberry Pi 4. This means that the board design materials are freely available for anyone to build the BeaglePlay or a derivative into their products.

In any case, I encourage you to try out as many of the examples as you can, using either of these three platforms or any embedded hardware you may have on hand.

The Raspberry Pi 4

From June 2019 until October 2023, the Raspberry Pi 4 Model B was the flagship SBC produced by the Raspberry Pi Foundation. The Raspberry Pi 4’s technical specs include the following:

A Broadcom BCM2711 1.5 GHz quad-core Cortex-A72 (Arm v8) 64-bit SoC2, 4, or 8 GB DDR4 RAM2.4 GHz and 5 GHz 802.11ac wireless, Bluetooth 5.0, BLEA serial port for debugging and developmentA microSD slot, which can be used as a boot deviceA USB-C connector to power the boardTwo full-size USB 3.0 and two full-size USB 2.0 host portsA Gigabit Ethernet portTwo micro HDMI ports for video and audio output

In addition, there is a 40-pin expansion header for which there are a great variety of daughter boards known as Hardware Attached on Top (HATs) that allow you to adapt the board to do many different things. However, you will not need any HATs for the examples in this book.

In addition to the board itself you will require the following:

A microSD card and a means of writing to it from your development PC or laptopA USB-to-TTL serial cable with a 3.3 V logic levelA 5 V USB-C power supply capable of delivering 3 AAn Ethernet cable and a router to plug it into as some of the examples require network connectivity

The BeaglePlay

The BeaglePlay is an open source hardware design for an SBC produced by the BeagleBoard.org Foundation. The main points of the specification are:

A TI AM6254 1.4 GHz Arm quad-core Cortex-A53 (Arm v8) 64-bit Sitara SoC2 GB DDR4 RAM16 GB eMMC on-board flash2.4 GHz and 5 GHz MIMO Wi-Fi, BLE, ZigbeeA serial port for debugging and developmentA microSD slot, which can be used as a boot deviceA USB-C connector to power the boardA full-size USB 2.0 host portA Gigabit Ethernet portA full-size HDMI port for video and audio output

Instead of a large expansion header, the BeaglePlay has mikroBUS, Grove, and Qwiic interfaces for connecting add-on boards.

In addition to the board itself, you will require the following:

A microSD card and a means of writing to it from your development PC or laptopA USB-to-TTL serial cable with a 3.3 V logic levelA 5 V USB-C power supply capable of delivering 3 AAn Ethernet cable and a router to plug it into as some of the examples require network connectivity

In addition to the above,Chapter 12 also requires the following:

A MikroE-5764 GNSS 7 Click add-on board An external active GNSS antenna with an SMA connectorA MikroE-5546 Environment Click add-on boardA MikroE-5545 OLED C Click add-on board

QEMU

QEMU is a machine emulator. It comes in different flavors, each of which can emulate a processor architecture and various boards built using that architecture. For example, we have the following:

qemu-system-arm: 32-bit Armqemu-system-aarch64: 64-bit Armqemu-system-mips: MIPSqemu-system-ppc: Power PCqemu-system-x86: x86 and x86-64

For each architecture, QEMU emulates a range of hardware that you can see by using the -machine help option. Each architecture emulates most of the hardware that would normally be found on that board. There are options to link hardware to local resources, such as using a local file for the emulated disk drive. Here is a concrete example:

$ qemu-system-arm -machine vexpress-a9 -m 256M -drive file=rootfs.ext4,sd -net nic -net use -kernel zImage -dtb vexpress-v2p-ca9.dtb -append "console=ttyAMA0,115200 root=/dev/mmcblk0" -serial stdio -net nic,model=lan9118 -net tap,ifname=tap0

IMPORTANT NOTE

The preceding command is not meant to be executed and will fail since qemu-system-arm is not installed and the rootfs.ext4.sd, zImage, and vexpress-v2p-ca9.dtb files do not exist on your host system. It is just an example for us to expand on.

The options used in the preceding command line are as follows:

-machine vexpress -a9: Creates an emulation of an Arm Versatile Express development board with a Cortex-A9 processor.-m 256M: Populates it with 256 MB of RAM.-drive file=rootfs.ext4,sd: Connects the SD interface to the local rootfs.ext4 file, which contains a filesystem image.-kernel zImage: Loads the Linux kernel from the local file named zImage.-dtb vexpress-v2p-ca9.dtb