The Embedded Linux Security Handbook E-Book

Matt St. Onge

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 384

Veröffentlichungsjahr: 2025

Beschreibung

As embedded Linux systems power countless devices in our daily lives, they’ve become prime targets for cyberattacks. In this in-depth guide to safeguarding your Linux devices, the author leverages his 30+ years of technology experience to help you mitigate the risks associated with hardware and software vulnerabilities.
This book introduces you to the world of embedded systems, the brains behind your everyday appliances. It takes you through the different types of embedded systems, their uses, and the platforms they run on, while addressing their unique security challenges and support considerations. You’ll learn how to build a successful, secure, and user-friendly solution by exploring the critical hardware and software components that form the foundation of a secure appliance. We won't forget the human element either; you'll find out how to configure your system to prevent user errors and maintain its integrity. The book lets you put your newfound knowledge into action, guiding you through designing a robust build chain that supports the entire life cycle of your appliance solution, enabling seamless updates without your direct involvement.
By the end of this book, you’ll be able to adapt your appliance to the ever-evolving threat landscape, ensuring its continued security and functionality in real-world conditions.

Leseprobe

The Embedded Linux Security Handbook

Fortify your embedded Linux systems from design to deployment

Matt St. Onge

The Embedded Linux Security Handbook

Copyright © 2025 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author nor Packt Publishing or its dealers and distributors will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Portfolio Director: Rohit Rajkumar

Relationship Lead: Kaustubh Manglurkar

Content Engineer: Shreya Sarkar

Program Manager: Sandip Tadge

Growth Lead: Namita Velgekar

Copy Editor: Safis Editing

Technical Editor: Nithik Cheruvakodan

Proofreader: Shreya Sarkar

Indexer: Hemangini Bari

Production Designer: Gokul Raj S.T

Marketing Owner: Nivedita Pandey

First published: April 2025

Production reference: 1210325

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83588-564-2

www.packtpub.com

To my loving partner, Nicolle, for pushing me when I needed it and for joining me on this amazing life journey.

- Matt

Foreword

Security matters.

In today’s world with the technology landscape evolving rapidly and the ever-increasing interdependence of the systems within the digital infrastructure, security is no longer a luxury but an essential bedrock of every system, from the smallest of embedded devices to the largest data center supercomputers.

Recent reports, such as the Verizon Data Breach Investigations Report (DBIR) and IBM’s Cost of a Data Breach Report, underscore a surge in threats ranging from sophisticated malware to simpler insider threats and social engineering attacks. The Verizon 2023 DBIR states that over 74% of breaches involved human elements, underscoring the need for robust security practices at both the hardware and software levels. Additionally, IBM’s 2023 report found that the global average cost of each data breach had risen to $4.45 million, with critical industries like healthcare becoming especially vulnerable. According to Cybersecurity Ventures (https://cybersecurityventures.com/), the global annual cost of cybercrime was predicted to reach $9.5 trillion in 2024 and will reach $10.5 trillion in 2025. Further, numerous authors and reports have featured how cyber attackers don’t even have to hack into computer systems anymore. They simply log in.

Today, no system administrator or security professional can afford to overlook the importance of a well-secured Linux environment.

In the face of growing cybersecurity threats, this book fills a critical gap by providing a comprehensive guide to Linux security tailored for those who build and maintain embedded Linux systems or appliances.

Matt has over 30 years of experience in the world of Linux systems, both appliance and other server systems, helping Linux customers, partners, and the community build secure embedded Linux servers. He has acquired deep insights into the strengths and vulnerabilities of these systems, and his knowledge spans from design stages to challenges of safeguarding hardware and software environments.

This makes Matt uniquely qualified to write this book.

In this book, Matt focuses on topics that every system and security professional needs to understand to prevent security incidents, beginning with an introduction to the cybersecurity landscape, underlining the importance of starting from the requirements and design stage, guiding you through securing the hardware and OS, and exploring concepts and components like Trusted Platform Modules, disk encryption, and OS immutability.

Matt doesn’t stop at merely securing hardware and software sides. He also explores the frequently overlooked areas like the BIOS, firmware security, and the boot process ensuring that the system is protected even before the software comes into play. His approach includes safeguarding the system from potential threats posed by both the end-users and the environment, emphasizing a holistic approach to Linux security for appliances.

What makes this book stand out is Matt’s ability to draw on his experience to offer actionable advice and best practices to continuously improve appliance security through their entire lifecycle and navigate the complex economic and regulatory environment. This book goes beyond immediate threats, addressing the longer-term challenges of designing and keeping systems safe and secure, and provides the knowledge and tools you need to protect your systems in today’s complex security threat environment.

As you embark on this journey through Linux security, you are in the capable hands of someone who has spent his career securing open-source systems. Matt’s experience, coupled with his strong commitment to the Linux community, ensures that this book will serve as both a foundational text and a practical guide to securing Linux environments at every level.

It is with great pleasure that I introduce you to The Embedded Linux Security Handbook by Matt St. Onge. I am confident that you will find this book not only informative but also essential to your work in securing Linux systems.

Rama Krishnan,

Senior Director of Engineering, Veritas Technologies

Author’s Note

Before we dive into things, I’d like to tell you how I initially got involved with embedded Linux systems appliances and why it’s been my passion.

My Linux journey began in the mid-1990s; however, my journey with embedded Linux systems began about a decade ago, when I took on the role of senior solutions architect at Red Hat. This is where I began working with Mike Zitomer, who soon became not just a leader to me but also a great mentor and friend. Mike built Red Hat’s Embedded Partner Program and was building a team to expand it.

I remember this like it was just yesterday. Mike brought me into a meeting with my first embedded systems partner, a telecommunications systems provider (name withheld due to NDA) way back in 2015. By that time, I had already been working with Linux and open source software for about twenty years. The concepts of appliance building fascinated me. I was hooked. It wasn’t much longer before I joined Mike’s team as the lead technical resource. The wild ride begins. There’s never been a dull moment.

In our careers, it’s rare but sometimes we have those who help shape the direction we travel and guide us toward success; for me that mentor was Mike. I cannot thank Mike enough for all he has taught me over the years. I am forever grateful.

Over the past nine-plus years, I’ve spent my days working alongside the product managers, architects, developers, and product support staff of these Red Hat partners. My focus was never selling anything. If I helped my partners, good things would surely come. Enablement, empowerment, consultation, and education are all my focus. I’ve created countless lab exercises, presentations, reference materials, and so on all in support of building better, secure Linux appliances.

Through these activities, I have assisted my partners in prototyping, designing, building, and supporting such a vast array of solutions. Some save lives. Many improve creature comforts. Some are purely for fun. Others help defend our nation. This role has left me feeling like I have truly helped make the world a better place… somewhat at least.

So here we are. I hope the lessons I share with you here can ultimately assist you (metaphorically, of course) to “build a better mousetrap.” I am humbled and happy you’ve chosen to join me on this journey into applying security into a system.

Also, let me clearly state now that Red Hat® has no involvement in this book. None whatsoever. This work is of my mind and opinion, not that of any employer, past or present. This project is for the benefit of anyone building an appliance. I’m not focused on any specific distribution of Linux; however, I may have my own opinions. I intend to share them. The good, the bad, and the ugly. This is all in the pursuit of enabling you not to feel the pains that many others have already endured.

Contributors

About the author

Matt St. Onge is an Associate Principal Solutions Architect at Red Hat, focused on providing enablement through their Embedded Systems partner program. Since 2015, his activities at Red Hat have ranged from best practices sessions to design reviews and even leading rapid prototyping workshops. Matt has assisted hundreds of product teams over the past decade in building quality, secure Linux appliances.

Matt has been active in the open source and Linux communities for over 30 years by contributing to solutions via GitHub, creating new projects for the benefit of the embedded systems community, and hosting webinars and user group meetings.

When not building solutions in the lab, Matt can be found hiking with his dog in the mountains, golfing, or cooking elaborate meals on the grill for his extended family.

I want to sincerely thank the people who have been close to me and supported me, especially my partner, Nicolle Langone, for her patience and love. I also want to express my gratitude for the guidance, mentorship, and advice from some amazing leaders in my life: Mike Zitomer, Ron Pacheco, Kevin Miller, Chuck Mattern, Matt Davis, and John Tietjen.

Finally, I’d like to thank many others who have helped me with research, data, review, or other tasks related to this work: OnLogic, Derek Fanton, Adam Rael, Herve Lemaitre, Mark Thacker, Mark Russell, Matt Micene, Dan Pacek, Rama Krishnan, Dawn Marini, Prashant Divate, Richard Rios, Paul Armstrong, Tony Mazon, Kevin Holmes, Phil Avery, Rick Stahovec, Kamlesh Gurudasani, Oreoluwa Oluwafemi, Ahmed Elbanna, and to all those who helped and didn’t know it.

Thanks to all of you!

About the reviewers

Prashant Divate is a passionate technologist, open-source evangelist, and embedded systems expert with over 6 years of experience. He specializes in embedded Linux systems, the Yocto Project, hardware bring-up, and crafting cutting-edge solutions for real-world challenges.

A maintainer of the meta-sirius Yocto Board Support Package (BSP) layer, Prashant excels at optimizing Linux-based systems and pushing the boundaries of embedded development. Beyond tech, he is a drawing artist and is a general-class ham radio operator (VU2OWC), licensed for worldwide communication.

With a keen eye for detail and a love for knowledge sharing, Prashant is dedicated to refining technical content to ensure clarity and impact.

Dawn Marini has been in the IT field for over two decades in a variety of roles, from help desk to systems administration. She is currently a Specialist Solutions Architect for Virtualization Infrastructure at Red Hat, Inc. Largely self-taught, she started writing programs in Basic in fourth grade on her grandfather’s Commodore 64. Dawn is also active in Toastmasters, has a second-degree black belt in Isshin-ryu karate, and is learning Italian.

Kamlesh Gurudasani is the security lead at Texas Instruments and a member of the Trusted Firmware Technical Steering Committee and CIP Security working group, representing Texas Instruments (TI). He specializes in embedded security, cryptography, Linux Direct Rendering Manager (DRM), and power management for Arm architectures.

He contributes to Trusted Firmware-A (TF-A), OP-TEE, and Linux kernel cryptography, focusing on runtime security, secure boot, and vulnerability management. Kamlesh was also a speaker at EOSS24, presenting on Arm System Control and Management Interface (SCMI).

A big fan of open source, he actively supports its growth and enhances security in long-term industrial systems.

Oreoluwa Oluwafemi is a seasoned embedded systems engineer with a wealth of experience designing and developing IoT-based systems from the ground up. He earned a bachelor’s degree in Computer Engineering from Covenant University, which laid the foundation for his career. He has expertly led the design and development of Industrial Internet of Things (IIoT) systems utilizing wireless standards such as Zigbee, Thread, Wi-Fi, Bluetooth Low Energy (BLE), and Long Range Wide Area Network (LoRaWAN), and has deployed these solutions in conjunction with SCADA systems on offshore assets for process monitoring. Oreoluwa has also led IoT-based micro-mobility product teams that developed software solutions enabling EV-based transportation via mobile apps—built on Linux-based server systems and other major cloud computing solutions. His expertise is further enhanced by his work with Linux using RHEL at Huawei Technologies, where he focuses on process, network, and disk management. Oreoluwa currently applies his deep technical knowledge at Cors System Technologies.

Ahmed Elbanna is an Embedded Software Engineer with extensive experience in Embedded Linux, Automotive Cybersecurity, and various automotive domains, including C++, Model-Based Design, and Automotive Connectivity. Passionate about technology and literature, Ahmed enjoys delving into diverse genres and sharing his thoughtful, well-rounded book reviews.

Table of Contents

Preface

Part 1: Introduction to Embedded Systems and Secure Design

1

Welcome to the Cyber Security Landscape

What is a Linux-embedded system?

How are Linux-embedded systems used?

Why is securing Linux-embedded systems so important?

Examples where embedded Linux systems had a security breach

Summary

2

Security Starts at the Design Table

What are the business needs that the solution caters to?

Who is my target buyer and my target user?

The target buyer

The target user

Will any specific government compliance standards drive the decision tree?

Healthcare systems (and data privacy)

Financial services systems

Retail and online marketplace systems

Government and military systems

How will we support this appliance solution?

Managed service

Online support

Offline support

No support/self-support

Replacements

Other product-impacting needs and concerns

Hardware life cycle

Linux distribution life cycle

Supply chain issues

Summary

3

Applying Design Requirements Criteria – Hardware Selection

What are the targeted performance requirements?

Virtual appliances

T-shirt sizing

CPU/VCPU

Memory

Disk input/output (I/O)

Networking

GPU

Custom hardware and peripherals

Are there any environmental limitations?

Power

Offline/air-gapped

Climate control

COTS versus custom-built hardware

Dell

OnLogic™

What mainstream CPU/hardware platforms are available?

Xeon™

Core™

Atom™

Ryzen™

Advanced RISC Machine (ARM™)

RISC-V®

Power® and IBM Z®

Summary

4

Applying Design Requirements Criteria – the Operating System

Matching an operating system to your base hardware platform

IBM Power

IBM System z

RISC-V

ARM

Driver support, vendor support, and stability

Enterprise versus community distributions of Linux

Lifecycle of operating systems versus your solution

Hard costs versus soft costs

Hardware costs

Software costs

Soft costs

Summary

Part 2: Design Components

5

Basic Needs in My Build Chain

Technical requirements

Software supply chain control

Source code control

Automation and tool integration – a brief overview

Security scanning, testing, and remediation

Exercise – executing a network port scan

Manifest and configuration tracking

Exercise – tracking changes in your product

Update control mechanisms

Exercise – building custom software packages

Exercise – signing your custom RPM package

Exercise – creating a custom DNF repository

Exercise – configuring your solution to use your custom repository

Summary

6

Disk Encryption

Technical requirements

Introduction to LUKS

Basic implementation review

Implementing LUKS on an appliance with automated keys

Exercise – implementing LUKS with stored keys and leveraging the crypttab file

Is recovery possible?

Summary

7

The Trusted Platform Module

What is TPM?

The history of TPM

Configuring TPM by example

Exercise – enabling TPM 2 in conjunction with LUKS encryption

Summary

Join our community on Discord

8

Boot, BIOS, and Firmware Security

Deep dive into various booting system components

Understanding boot-level security using examples

Accessing the UEFI configuration

What is Secure Boot?

Possible threats in firmware

Summary

9

Image-Based Deployments

Technical requirements

Introducing image-based Linux deployments

rpm-ostree and atomic images

bootc and bootable container images

Special tooling and support infrastructure differences

Limitations of image-based deployments

rpm-ostree image limitations

bootc bootable container image limitations

Updating and rolling back changes

Upgrade of operating system version in place

Practical exercises

Exercise 1 – preparing the environment

Exercise 2 – creating a container file

Exercise 3 – creating an installer

Exercise 4 – initial installation

Exercise 5 – creating an updated container

Exercise 6 – updating your system

Summary

10

Childproofing the Solution: Protection from the End-User and Their Environment

Introduction to child-proofing (i.e., protecting the appliance from the end-user)

Ensuring hardware-level protections

Tamper-proofing with BIOS security

USB disablement

Case tamper-proofing

Operating-system-level and application protections

Minimizing access to root

SUDO and restricting console access

Non-interactive LUKS encryption

Keeping users in the application space

Application auto-launch at boot

Building a UI to simplify configuration while providing a great User Experience (UX)

Initial config – text UI

Initial config – web UI

Update controls – text UI

Factory reset controls

Summary

Part 3: The Build Chain, Appliance Lifecycle, and Continuous Improvement

11

Knowing the Threat Landscape – Staying Informed

Navigating the information and disinformation online

Government resources

Commercial resources

Community resources

Knowing what vulnerabilities can impact your builds

Running smart searches based on your components

Being part of the solution

Contribute to the development process

Join a user group

Summary

12

Are My Devices’ Communications and Interactions Secure?

Technical requirements

Bus types and issues

USB

Serial port

The CAN bus

Enhancing security with certificates

Exercise 1: Creating a self-signed certificate

Exercise 2: Adding a certificate to your custom repository server

Confirming that your networking is secure

Firewalls

The command line

Web console

Graphical UI-based tools

Limitations of legacy hardware and software

Validating your solution before shipping

Summary

13

Applying Government Security Standards – System Hardening

Technical requirements

Adherence to key US government standards

How do I implement this?

Implementation of security standards

Validation as part of the QA process

Exercise: Installing the OpenSCAP tools and running a scan

Example: Using the OpenSCAP Workbench

Implementation as part of your continuous integration/continuous deployment (CI/CD) process

How do I certify my solution?

FIPS certification re-branding by vendors

Summary

14

Customer and Community Feedback Loops

Use case development

User groups

Executive roundtables

Community feedback loops

Summary

Closing the loop

Putting all the pieces together

Staying engaged

Join our community on Discord

Index

Other Books You May Enjoy

Part 1:Introduction to Embedded Systems and Secure Design

In this part, we will introduce you to the threats that impact your decision processes, hardware selection, operating system selection, and all the strengths and vulnerabilities that may alter your design criteria.

This part has the following chapters:

Chapter 1, Welcome to the Cyber Security LandscapeChapter 2, Security Starts at the Design TableChapter 3, Applying Design Requirements Criteria – Hardware SelectionChapter 4, Applying Design Requirements Criteria – the Operating System

1

Welcome to the Cyber Security Landscape

Cue up the music. Welcome to the Jungle. The cyber security landscape is ever-changing and evolving. One could easily compare it to the mercurial New England weather patterns in the Northeast of the United States. Such weather can be dangerous for a hiker; similarly, such changes in cyber threats can be devastating for a technology solution.

In the coming chapters, we will dive deeper into the realm of Linux-embedded systems (or embedded Linux systems, also known as Linux appliances, or simply appliances) and exactly how you can apply this new knowledge to secure your solutions. We’ll make this journey together, and at the end of the tunnel, you will be best positioned to build a better appliance.

This chapter is here just to set the foundation of Linux-embedded systems before we dive deeper. In this chapter, we will cover the following topics:

What is a Linux-embedded system?How are Linux-embedded systems used?Why is securing Linux-embedded systems so important?

Let’s get started.

What is a Linux-embedded system?

It’s probably safe to assume that if you’re reading this book, you may have some knowledge of what your company defines as a Linux-embedded system. To truly have a rich understanding of this subject, let’s review the appliance model for creating sellable solutions.

The appliance model provides solutions and services that an organization (or individual) cannot easily build for themselves. This also addresses another limitation; oftentimes, users of these appliances have little to no access to any technologists who would take on ownership of the care of said solution. The appliance model is most prevalent in home consumer electronics but touches every known vertical in the industry.

Most appliances leverage purpose-built hardware and have unique form factors that best enable their usage. These solutions are all around us every day. Some easily identifiable consumer appliance examples would be your home’s Wi-Fi router, cable TV box, or smart laundry washing machine. A resounding example would be the smartphones that we all carry and cannot imagine living without.

Let’s affectionately pick a generic home Wi-Fi router to dive deeper into the appliance model concepts. Virtually anyone can use its services without understanding how the underlying technology works or is implemented.

The router has a custom form factor for far more than aesthetics. The housing contains all the electronic components needed to successfully provide the services for which it’s intended. Inside, there’s a small computer running a Linux operating system. I bet you can see where I am going with this. This is how we came to call these offerings Linux-embedded systems, as they contain a custom computer and a Linux operating system, along with custom software to provide a unique set of services to the intended end-user.

Defining an appliance doesn’t stop there. It must provide a service, or services, which can be easily consumed by its end-users. In an appliance, this is achieved by the implementation of complex software and the appropriate hardware. In that home Wi-Fi router example, the manufacturer has built it to provide a simple-to-use web-based interface in order to manage the services provided. In this case, this is how the end-users allow people to access wireless internet.

Another key facet of the appliance model that must be built in is the ability to accept and process updates to its configuration or software, which absolutely includes its Linux operating system. It has to be simple for the end-user; one must assume the end-user is not technical. If the end-users actually possessed the technical capabilities, they could, in theory, build this themselves and you’d be out of business. Staying with the Wi-Fi router example, it has built into its web interface the simple function for self-updating. The end-user clicks on a button in the interface and the magic happens behind the scenes. Programmatically, the appliance gets an update package from a secure repository provided by the solutions builder; it downloads the package; and then finally processes all those application and Linux updates without any intervention by the end-user. The process notifies the end-user of its completion status (and sometimes requires a restart of the appliance).

Not every embedded Linux system (appliance) is on dedicated hardware. The advent of virtualization and, more recently, the great push to move things into the cloud have unleashed a whole new delivery method: the virtual appliance.

Virtual appliances are rapidly gaining traction in multiple cloud marketplaces. Why? It’s simple. Companies that, for many years, have produced on-premises appliance solutions have had to evolve in order to retain their customer base. Because of this, many solutions builders now offer an on-premises hardware offering, a virtual appliance for an on-premises offering, and a cloud provider-based virtual appliance offering.

Generally, each of these solutions can be built from the same code base, however, depending on what exactly the appliance provides the end-user, it may need to be tailored and refactored differently for each virtualization platform’s networking and security requirements. This is because the cloud providers (often referred to as hyperscalers) each have their own proprietary methods and APIs for networking, security, and end-user access. What I mean by this specifically is the application stack may be unified upon all delivery methods; however, the virtual machine image format, networking, and other facets will be different. For example, Azure and Google handle their network stack quite differently. Storage is also quite different. Hence, the user interface (UI), especially for the appliance’s setup, needs to have these differences in virtualization platforms or hyperscalers’ requirements taken into consideration.

Next, we’ll review how embedded Linux systems are utilized.

How are Linux-embedded systems used?

Linux-embedded systems are everywhere. Most people don’t give it a second thought. I venture the guess that you reach for one of these things when you first wake up in the morning and engage with it countless times every day. Your smartphone is a Linux-embedded system and, in my opinion, is an awesome example of one.

Embedded Linux systems surround us; some with stealth, others with pure elegance and entertainment, with all the beauty and grace their builders have instilled in their design. These appliance solutions run a massive range of use cases. Some are a complete solution and others are but a simple component to a greater ecosystem or solution.

Most, when successfully designed and implemented, provide services to their user community seamlessly without the user even having to think about what is happening under the covers. Much of this must be credited to how resilient, lightweight, and flexible Linux solutions can be. A Microsoft Windows-embedded system is far more reliant on its GUI components than Linux. It’s far easier to deliver a complete solution on Linux with a GUI than its Windows-based competition.

Here, I will try to provide a generalized list of where Linux is leveraged today (regardless of distribution or hardware platform) in providing services. I’ll throw in my disclaimer as well – this list is general at best and may not include all use cases currently deployed. I present this in no particular order and will try not to leave anything out but it would be impossible for me to list everywhere Linux-embedded systems are in use. So, let’s look at the following list:

Artificial intelligence (AI): Linux is at the heart of all AI advancements in recent years. It is the preferred operating system of choice due to its flexibility and vast developer base. As this field is rapidly evolving, the possibilities are limitless and often classified, such as the following:Facial recognition systemsArtistic graphical rendering systemsLanguage processingAgriculture: As agriculture around the world has begun to embrace operations at ever-growing scales, technology has been needed to assist with the following operations:Heavy equipment monitoring solutionsGPS-guided equipment navigationProduce grading and sorting systemsProduction tracking systemsAutomotive: Embedded Linux reigns supreme in this rapidly evolving transportation space. Countless embedded Linux systems can be found today, stealthily providing many critical services, such as the following:Autonomous vehicle operating systems and componentsSensors and safety systemsGPS navigationElectric vehicle charging infrastructureAviation: Civilian aviation and transportation is a booming business for Linux-embedded systems solutions. These complex appliances can be found in the following areas (assuming you have the proper security clearance):Baggage scanning systemsPassenger screening solutionsBaggage shipping and tracking systemsRadar and air traffic control systemsTraveler identification and verification systemsBuilding management and construction: Building management controls have long been solved by Linux-based solutions providers. The following is but a mere example of some of the high-tech solutions:Card key access systemsEnvironmental control systemsHVAC management systemsFire detection and suppression systemsAlarm systemsTelecommunications: In my opinion, this vertical has seen the most growth, which is second only to those related to military projects. Virtualization has empowered the proliferation of a whole new breed of network functions virtualization (NFV) solutions. Both physical and virtual appliances in this space dwarf deployments virtually anywhere else. These virtual appliances can be found in server rooms, telco closets, data centers, and cloud providers’ marketplaces in order to make it easier to provide access to the following services:Radio access networks (RANs)Load balancersRouting solutionsPush-to-talk (PTT) phone systemsGeo-locational-tracking solutionsDomain Name System (DNS) solutionsIP address management solutionsFirewall solutionsEmail filtering solutionsOperations management systems that control fiber-based connectivity tothe internetGeology: Mostly driven by energy companies and government agency needs, this vertical has seen exceptional growth in the past five years. Although they may not be mainstream, these appliances provide many valuable services to academia and the energy industry, such as the following:Near real-time geological scanning and assessment solutionsModeling systemsSeismic tracking solutionsHealthcare: Linux systems are vastly deployed in almost every facet of healthcare, whose focus on security and patient safety is paramount. Hospital efficiency has been greatly improved as well. Many of the new appliance solutions can be found in virtually any healthcare practice, providing some of the following life-saving services:Patient safety solutionsElectronic records solutionsPatient monitoring solutionsVentilatorsLab testing solutionsImaging systems (PET scan, MRI scan, CAT scan, etc.)Industrial and manufacturing: Industrial automation has been prolific for over two decades, however, some key players in this space have revolutionized what used to be manual processes. Embedded Linux systems reduce costs and accelerate production rates by providing some of the following services:Industrial manufacturing automation systemsEnvironmental and safety monitoring systemsMilitary: So, in order to keep those guys in black suits from coming to visit me in an unfriendly manner, I’ll generalize, especially in this vertical, to preserve my life (and add a little intrigue to the mix). Linux appliances can stealthily be found providing critical security and defense-related services, such as the following:Facial recognition systemsData gathering solutionsComponents to countless vehicles and aircraftRadar systemsGuidance systemsNavigation systemsSpace: Without Linux and open source technologies, this vertical would not be achieving what it is today. Embedded Linux systems are reaching for the heavens. A few examples are as follows:Space exploration vehicles (example: Cassini)SatellitesRoversSensor systemsGPS systemsGround-based management systemsSpace telescopesWeather: Government agencies, aviation, highway safety, oceanic shipping, and other transportation agencies all rely upon data provided by Linux-embedded systems solutions for continuity of travel and public safety. These appliances, which can be found globally, provide the following services:Weather modeling and prediction solutionsGround and aerial radar systems

In summary, we’ve highlighted just some of the critical and interesting services embedded Linux systems provide today that enrich our lives, protect us silently, and help us travel to places that were once beyond our reach.

Why is securing Linux-embedded systems so important?

Why is securing Linux-embedded solutions systems (appliances) so important? Well, this answer truly has multiple aspects depending on who you ask.

The most obvious answer would simply be to prevent liability and loss of credibility in the marketplace. Lately, it seems that there’s an ever-growing list of companies whose offerings have become compromised and, as a result, their customers are grievously impacted.

The impact of such a breach can be the loss of personal information for the users (insert gasp here!) or even worse, unauthorized access to systems or data without the knowledge of those running the systems. Each year, these breaches result in billions of dollars in correlated losses and open these companies to legal action by their user base. The image impact on one’s brand can last far longer in people’s minds than you’d think.

It’s highly documented that data leaks, breaches, and other security lapses are often done by internal staff and contractors. This does not imply malfeasance. Often, it can be a lapse of judgment, forgetting to remove sensitive data from development environments, or accidentally taking home sensitive data on a device that itself gets lost, stolen, or compromised.

Then, there are the situations caused by intentional misuse of a system. Sadly, these events, in my opinion, can be the easiest to prevent. We will definitely dive deep into this matter in later chapters. Misuse comes in many forms. It starts with unauthorized access to a system or exceeding one’s intended limits. Then, there are those events where someone intentionally uses an application or system in a manner that it was not intended for. The worst-case scenario is when software on a system is modified, installed, or disabled so that an individual can perform tasks or monitor others. These are the situations where the wheels fall off the wagon.

Examples where embedded Linux systems had a security breach

In 2019, hackers compromised Ring security cameras and gained access to their video and audio streams, wreaking havoc with those devices’ owners’ lives. Several incidents were reported in several states. Software updates and better password requirements keep those bad actors out now.

Another example of a breach of security is when, in 2017, it was found that St. Jude’s cardiac implants could be hacked. When I heard of this, I was amazed and scared for those patients. Thankfully, St. Jude has patched the vulnerability and no patients were actually harmed.

A research team at IBM was able to compromise the firmware update mechanism of a Jeep in mid-2015. They demonstrated how a bad actor could actually take control of the vehicle by speeding it up, steering the wheel, or applying the brakes. It’s kind of frightening to think that a bad actor could wreck your family drive. These gaps have quickly been closed.

Hopefully, these few public examples can help reinforce just how important it is to design an embedded Linux appliance with security in mind.

Detecting these unauthorized changes or breaches can be rather difficult, especially if you no longer have access to your product once the customer takes delivery and begins consuming its services. Preventing them altogether is so much easier.

You may have heard the old saying, an ounce of prevention is worth a ton of cure. If you apply even some or all of the lessons this book seeks to impart upon you, it’s like having ten pounds of prevention and all the cure. We’ll dive deep into hardware security, operating systems security, secure connectivity, and how to know what threats you must prepare for.

Not every method or technology in this book may apply to your solution or even be feasible, but the more you apply these methodologies, the more likely you will create a rock-solid product. That said, may the only headlines your company creates be those of success and accolades.

Summary

In this first chapter, you will have gained a solid understanding of how an embedded Linux system is defined. We’ve shown you some wide-ranging examples of where embedded Linux systems are leveraged today. Most importantly, we’ve reviewed some critical reasons why having these devices delivered to their customers as securely as possible is mandatory for those building such solutions. In the next chapter, we’ll dive deep into why security starts with a great design.

2

Security Starts at the Design Table

Security is more than a process. Security cannot be an afterthought. Security is a mindset. When you are considering offering a new or upgraded product to the market, security starts at the design table. The choices you make here will define outcomes you may not have even pondered in the past.

In the previous chapter, we did a brief review about what embedded Linux systems (AKA appliances) are and often how they solve real-world problems. The next few chapters will focus on the design and build phase. Each chapter will build upon the concepts of the previous, and as you’ll soon readily see, it is all considerations to be aware of.

While in the design phase, a product team has the highest chance of success in mitigating future risks. Through proper scoping, planning, and execution, your team can achieve great success in creating a highly productive appliance solution. Each of the factors that we are about to review, if accounted for at this stage, will become a factor toward success, not a risk factor.

Security auditors, systems architects, and product managers, all by virtue of their expected job descriptions, are risk mitigators. That’s why we are here doing this right now. Thank you for taking this journey with me.

Measure twice and cut once is a saying many of us are quite familiar with. Here in the design phase, I’d say that’s radically insufficient. I say it’s Measure twice, change rulers, compare the new ruler to the old one, measure two more times with the new ruler, and then consider the cut carefully. If still in doubt, you should get a third ruler (and start again!). I would also recommend a peer review before the initial cut. A peer review is something I cannot recommend highly enough. A second set of eyes (or several more) can often find and point out something you may have overlooked.