34,79 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Fachliteratur
- Sprache: Englisch
- Veröffentlichungsjahr: 2017
Android System Programming involves both hardware and software knowledge to work on system level programming. The developers need to use various techniques to debug the different components in the target devices. With all the challenges, you usually have a deep learning curve to master relevant knowledge in this area. This book will not only give you the key knowledge you need to understand Android System Programming, but will also prepare you as you get hands-on with projects and gain debugging skills that you can use in your future projects.
You will start by exploring the basic setup of AOSP, and building and testing an emulator image. In the first project, you will learn how to customize and extend the Android emulator. Then you’ll move on to the real challenge—building your own Android system on VirtualBox. You’ll see how to debug the init process, resolve the bootloader issue, and enable various hardware interfaces. When you have a complete system, you will learn how to patch and upgrade it through recovery. Throughout the book, you will get to know useful tips on how to integrate and reuse existing open source projects such as LineageOS (CyanogenMod), Android-x86, Xposed, and GApps in your own system.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 466
Ähnliche
Title Page
BIRMINGHAM - MUMBAI
Copyright
Android System Programming
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: May 2017
Production reference: 1290517
ISBN 978-1-78712-536-0
www.packtpub.com
Credits
Author
Roger Ye
Copy Editor
Safis Editing
Reviewers
Bin Chen
Chih-Wei Huang
Shen Liu
Nanik Tolaram
Project Coordinator
Ritika Manoj
Commissioning Editor
Amarabha Banerjee
Proofreader
Safis Editing
Acquisition Editor
Shweta Pant
Indexer
Mariammal Chettiyar
Content Development Editor
Arun Nadar
Production Coordinator
Nilesh Mohite
Technical Editor
Prajakta Mhatre
About the Author
Roger Ye has worked in the area of embedded system programming for more than 10 years. He has worked on system programming for mobile devices, home gateways, and telecommunication systems for various companies, such as Motorola, Emerson, and Intersil.
Most recently, he has worked as an engineering manager, leading a team of Android engineers to develop mobile security applications at Intel Security. With extensive knowledge and experience in the areas of embedded systems and mobile device development, he published a book called Embedded Programming with Android, Addison-Wesley, in 2015.
About the Reviewers
Bin Chen is a senior engineer from Linaro. He has worked on various Android-based products since 2010: TV, STB, Galaxy Tab, Nexus Player, and Google Project Ara, in that order, and now AOSP 96Boards. He occasionally blogs and speaks about all things Android. He lives in Sydney, Australia.
Chih-Wei Huang is a developer and promoter of free software who lives in Taiwan. He is famous for his work in the VoIP and internationalization and localization fields in Greater China.
Huang graduated from National Taiwan University (NTU) in 1993 with a bachelor's degree in physics, and attained a master's degree in the electrical engineering department of NTU in 2000. Huang currently works as a chief engineer of Tsinghua Tongfang Co., Ltd.for the OPENTHOS project. He is one of the founding members of the Software Liberty Association of Taiwan (SLAT).
Chih-Wei Huang is the founder and coordinator of the Chinese Linux Documentation Project (CLDP). He is also the second coordinator of the Chinese Linux Extensions (CLE)and a core developer of GNU Gatekeeper (from 2001 to 2003).
He is a contributor to pyDict, OpenH323, Asterisk, GStreamer, and more. He is working on a way to leverage the ASUS Eee PC with the power of the free software community and aims to provide a complete solution for Android on the x86 platform. The Eee PC, VirtualBox, and QEMU have been tested and are OK.
Chih-Wei Huang and Yi Sun started the Android-x86 open source project in 2009. The project aims to bring Android to the x86 platform.
About the Reviewers
Shen Liu is a senior engineer, working at Intel China. He used to work at McAfee LLC, Broadcom Corporation, and Huawei Technologies. He has over 10 years of work experience on Linux/Android and embedded systems, in different roles. He had taken manager, architect, and engineer roles during his career. He is mainly responsible for the Android framework, but is not limited to it, and he has a lot of passion for software design. On top of that, he loves reading technical books.
Nanik Tolaram works as a senior Android platform engineer for BlocksGlobal in Australia, where he is responsible for developing Screener (screener.digital) and Lumin (mylumin.org). He is passionate about Android and is very active within both the local and international Android developer communities--from talks and teaching to writing articles for ODROID open source magazine (magazine.odroid.com). In his spare time, he loves to tinker with electronics and study human psychology and behavior. He lives in Sydney, Australia, with his lovely wife and two beautiful boys.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at www.amazon.com/dp/178712536X.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Preface
Virtual hardware platforms
Android version used in this book
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
Introduction to Android System Programming
What is system programming?
What is the scope of this book?
Overview of the Android system
Kernel
HAL
Android system services
Binder IPC
Application framework
Recovery
The third-party open source projects derived from AOSP
LineageOS (CyanogenMod)
Android-x86
CWM/CMR/TWRP
Strategy of integration
Virtual hardware reference platforms
Introduction to the x86-based Android emulator
Introduction to ranchu
VirtualBox-based Android emulators
Summary
Setting Up the Development Environment
Summary of Android versions
Installing Android SDK and setting up an Android Virtual Device
Creating AVD in an older version of SDK
Creating AVD in the latest version of SDK
Testing the goldfish emulator
Testing ranchu emulator
The AOSP build environment and the Android emulator build
The AOSP build environment
Installing the required packages
Installing Open JDK 7 and 8
Downloading the AOSP source
Installing repo
Initializing a repo client and downloading the AOSP source tree
Building AOSP Android emulator images
Testing AOSP images
Creating your own repository mirror
Repo and manifest
Using a local mirror for AOSP
Creating your own mirror of GitHub
Fetching Git repositories outside GitHub
Creating your own manifest for client download
Summary
Discovering Kernel, HAL, and Virtual Hardware
What is inside the AOSP?
Android emulator HAL
Calling sequence
Goldfish lights HAL
The system service and hardware manager
Android emulator kernel and hardware
Android emulator hardware
Goldfish platform bus
QEMU pipe device
Goldfish audio device
Goldfish serial port
Goldfish kernel
QEMU pipe
Summary
Customizing the Android Emulator
Why customize the Android emulator
Understanding build layers
Build variants
Creating a new x86emu device
Checking out from the AOSP
Checking out from a local mirror
Creating x86emu device
AndroidProducts.mk
BoardConfig.mk
device.mk
Building and testing x86emu
Building x86emu
Testing x86emu
Integrating with Eclipse
Summary
Enabling the ARM Translator and Introducing Native Bridge
Introducing Native Bridge
Setting up Native Bridge as part of the ART initialization
Pre-initializing Native Bridge
Initializing Native Bridge
Loading a native library
Integrating Houdini to the x86emu device
Changing the configuration of the x86emu build
Extending the x86emu device
Changes to BoardConfig.mk
Changes to x86emu_x86.mk
Changes to device.mk
Using the Android-x86 implementation
Analyzing libnb.so
Using binfmt_misc
Building and testing
Testing the command-line application
Testing the Android JNI application
Summary
Debugging the Boot Up Process Using a Customized ramdisk
Analyzing the Android start up process
Bootloader and the kernel
Analyzing the init process and ramdisk
Actions
Services
Device-specific actions and services
Source code and manifest changes
The Android-x86 start up process
The first-stage boot using initrd.img
Inside initrd.img
Inside install.img
Building x86emu with initrd.img
Creating a filesystem image
Kernel changes
Booting a disk image on the Android emulator
Summary
Enabling Wi-Fi on the Android Emulator
Wi-Fi on Android
The Wi-Fi architecture
QEMU networking and wpa_supplicant in Android
Adding Wi-Fi to the emulator
Enabling wpa_supplicant in BoardConfig.mk
Providing a proper wpa_supplicant configuration
Creating services in init scripts
Initializing network interface eth1
Starting up wpa_supplicant
Building the source code
Getting the source code
Enabling boot with initrd.img
Testing Wi-Fi on an emulator
Booting an Android emulator using initrd.img
Booting an Android emulator using ramdisk.img
Debugging Wi-Fi start up processes
Summary
Creating Your Own Device on VirtualBox
HAL of x86vbox
The manifest for x86vbox
Creating a new x86vbox device
Product definition Makefile of x86vbox
Board configuration of x86vbox
Common x86 devices
Getting the source code and building the x86vbox device
Boot up process and device initialization
Device initialization before Android start-up
HAL initialization during the Android start-up
Summary
Booting Up x86vbox Using PXE/NFS
Setting up a PXE boot environment
Preparing PXE Boot ROM
Downloading and building the LAN Boot ROM
Fixing up the ROM image
Configuring the virtual machine to use the LAN Boot ROM
Setting up the PXE boot environment
Configuring and testing the PXE boot
Setting up the virtual machine
Using VirtualBox internal PXE booting with NAT
Configuring pxelinux.cfg
pxelinux.cfg/default
Setting up a serial port for debugging
NFS filesystem
Preparing the kernel
Setting up the NFS server
Configuring the PXE boot menu
Booting to NFS installation
Booting from a hard disk
Booting to recovery
Summary
Enabling Graphics
Introduction to the Android graphics architecture
Delving into graphics HAL
Loading the Gralloc module
Initializing GPU
Initializing framebuffer
Allocating and releasing the graphic buffer
Allocating from framebuffer
Allocating from system memory
Releasing graphic buffers
Rendering framebuffer
Graphics HAL of the Android emulator
Overview of hardware GLES emulation
Initializing GPU0 and FB0 in GLES emulation
GPU0 device implementation
FB0 device implementation
Summary
Enabling VirtualBox-Specific Hardware Interfaces
OpenGL ES and graphics hardware initialization
Loading OpenGL ES libraries
Analyzing the loading process
Loading the driver
Creating the rendering engine
The uvesafb framebuffer driver
What is uvesafb?
Testing the uvesafb framebuffer driver
Initializing uvesafb in x86vbox
Integrating VirtualBox Guest Additions
Building VirtualBox Guest Additions
Integrating vboxsf
Integrating vboxvideo
Building and testing images with VirtualBox Guest Additions
Summary
Introducing Recovery
Recovery introduction
Android device partitions
Analyzing recovery
BCB
Cache partition
Main flow of recovery
Retrieving arguments from BCB and cache files
Factory data reset
OTA update
Building recovery for x86vbox
Building configuration
Changes to x86vbox
Changes to recovery
Changes to newinstaller
Testing recovery
Summary
Creating OTA Packages
What is inside an OTA package
Updater
The updater script
Edify functions
Built-in functions
Installation functions
Block image functions
Device extensions
Preparing an OTA package for x86vbox
Removing dependencies on /system
Hardware initialization in recovery
Minimum execution environment in recovery
Building and testing
Summary
Customizing and Debugging Recovery
Debugging and testing native Android applications
Debugging with GDB
Integration with Eclipse
Extending recovery and the updater
Extending recovery
Extending the updater
Extending the Python module
Building and testing the extended recovery and updater
Supporting the third-party recovery packages
Xposed recovery package
Opening GApps
Summary
Preface
Android is the most popular mobile operating system in the world. Since 2013, Android has around 80% market share worldwide, while the second largest mobile operating system, iOS, has less than 20% market share. Due to the popularity of Android, there are many books about Android programming in the market. Most of them are targeted at Android application developers, which are the largest community in the world of Android development.
There is also another group of people working on the layer beneath the Android framework. Many people call them Android system developers. Comparing to Android application developers, Android system developers use the C/C++ languages, or even assembly language, to develop system services or device drivers. The scope and the definition of Android system development is much more vague than comparing to Android application development. For Android application development, the development environment and tools are very clear: the Android SDK and Android Studio from Google should be used and the programming language is Java.
For Android system development, we may use the Android NDK to develop Android system services or native applications. Many people refer to development based on the Android Open Source Project (AOSP) as Android system development. Nevertheless, Android system development encompasses the activities that produces native applications, services, or device drivers for a particular hardware platform. It closer to the hardware and the operating system, whereas Android application development is more general and hardware-independent.
Due to the hardware and operating system dependencies, it is more difficult to teach Android system programming than Android application programming. From the number of books in the market, we can see this. It is much easier to teach Android application development using specific examples. The readers of application programming books can follow the examples and can test them on most available Android devices. However, most Android system programming book can only talk about general concepts or ideas. When the authors want to use examples, they must pertain to a particular hardware platform and Android version. This makes it difficult for readers to repeat the same process.
Virtual hardware platforms
To make the discussion more general and overcome the issue of specific hardware platforms, I use virtual hardware platforms to demonstrate the work at the Android system level.
Before this book, I made an attempt to use a virtual hardware platform to explain how we can learn embedded system programming using an Android emulator in my previous book, Embedded Programming with Android. It seems many readers liked the idea, because they can explore the code examples much more easily on a virtual hardware platform that is available for everyone.
Android version used in this book
Android is still changing at a very fast pace. When I completed the book Embedded Programming with Android, we were still using Android 5 (Lollipop), and Android 6 (Marshmallow) was on the way to market with preview releases. Now while I am working on this book, Android 7 devices are available on the market and the next release of Android 8 has been announced with preview releases. We will use Android 7 (Nougat) to build all source code used in this book.
What this book covers
In this book, we discuss the Android system programming practices. We will use two projects (x86emu and x86vbox) to teach essential knowledge of Android system programming. The book is split into includes two parts.
The first part of this book talks about how to customize, extend, and port an Android system. We will use an Android emulator as the virtual hardware platform to demonstrate how to customize and extend an Android system. You will learn how to integrate an ARM translator (Houdini) into the Intel x86-based emulator and how to add Wi-Fi support to an Android emulator. We will use an x86emu device to learn these topics. After that, we will learn how to port an Android system to a new platform using VirtualBox. You will learn how to boot Android in the PXE/NFS environment, how to enable the graphics system, and how to integrate VirtualBox Guest Additions into the Android system. We will use x86vbox device to learn these topics.
In the second part of this book, we will learn how to update or patch a released system using recovery. In this part, we will provide a general introduction to recovery first. After that, we will explore how to build recovery for x86vbox device. With recovery for x86vbox device, we will demonstrate how to flash an update package to change the system image. We will use examples such as the Gapps and xposed recovery packages to demonstrate how to update an Android system image using third-party recovery packages.
Chapter 1, Introduction to Android System Programming, covers a general introduction of Android system programming. It also explains the scope of this book.
Chapter 2, Setting Up the Development Environment, provides details of the development environment setup for AOSP programming. After we set up the development environment, we will build an Android emulator image to test our setup. Other than the environment setup, we specifically discuss how to create your own source code mirror of AOSP from GitHub to help your quickly switch between different configurations.
Chapter 3, Discovering Kernel, HAL, and Virtual Hardware, covers an introduction to the Linux kernel, Hardware Abstraction Layer, and virtual hardware. In this chapter, we look at all the layers in the Android system software stack related to porting. We also take a in-depth look at the internals of the virtual hardware that we are going to use in this book.
Chapter 4, Customizing the Android Emulator, covers the development of a new device, x86emu. We will learn how to customize and extend this device in the next few chapters.
Chapter 5, Enabling the ARM Translator and Introducing Native Bridge, explores a new feature introduced in Android 5--Native Bridge. Since we created an x86-based device, x86emu, we have to integrate the ARM translator module (Houdini) into our device so that most ARM-native applications can run on it.
Chapter 6, Debugging the Boot Up Process Using a Customized ramdisk, introduces an advanced debugging skill to troubleshoot issues during the boot up stage. The famous Android-x86 project uses a special ramdisk to start the boot up process. It helps to troubleshoot device driver and init process issues very easily.
Chapter 7, Enabling Wi-Fi on the Android Emulator, presents details of how to enable Wi-Fi on our Android emulator. The Android emulator only supports an emulated 3G data connection, but many applications are aware of data and the Wi-Fi connection. We demonstrate how to enable Wi-Fi in the Android emulator in this chapter.
Chapter 8, Creating Your Own Device on VirtualBox, explores how to port Android on VirtualBox by introducing a new device x86vbox. The x86emu device is used to demonstrate how to customize an existing implementation, while x86vbox is used to demonstrate how to port Android to a new hardware platform.
Chapter 9, Booting Up x86vbox Using PXE/NFS, explains how to boot up Android on VirtualBox using PXE/NFS. Since VirtualBox is a general virtual hardware, the first problem that we meet is we need a bootloader to boot the system. We will use the PXE/NFS boot to solve this issue. This is an advanced debugging skills which can be used in your own project.
To discuss a more advanced case about the PXE/NFS setup using an external DHCP/TFTP server running in the host-only network environment, I have written an article, which you can find at https://www.packtpub.com/books/content/booting-android-system-using-pxenfs.
Chapter 10, Enabling Graphics, covers the Android graphic system. We introduce the Android graphics architecture and how to enable it on the x86vbox device.
Chapter 11, Enabling VirtualBox-Specific Hardware Interfaces, explains how to integrate the device drivers in VirtualBox Guest Additions into the Android system.
Chapter 12, Introducing Recovery, provides an introduction to recovery. We will learn how to customize and port recovery to a new hardware platform by building a recovery for the x86vbox device.
Chapter 13, Creating OTA Packages, covers the scripting language used by recovery: Edify. We will learn how to build and test OTA updates.
Chapter 14, Customizing and Debugging Recovery, expands on the concepts we learned about recovery and OTA packages. We will customize both recovery and updater for x86vbox device. We will test third-party OTA packages from Gapps and Xposed using our own recovery.
What you need for this book
To read this book, you should have essential knowledge of embedded operating systems and C/C++ programming language.
Who this book is for
Before we talk about who should read this book, we should ask who are the people that usually do Android system programming in the real world? There are potentially quite a lot. Here, I can give a few general categories. Firstly, there are a large number of engineers at Google working on the Android system itself, since Android is a product from Google. Google usually work with silicon vendors to enable Android on various hardware platforms.
There are many engineers at silicon chip companies, such as Qualcomm, MTK, or Intel to enable Android on their platform. They develop HAL layer components or device drivers to enable hardware platforms. The hardware platforms are usually called reference platforms, which are provided to OEM/ODM to build the actual products. Then, the engineers at OEM/ODM companies usually customize the reference platform hardware and software to add unique features to their products. All these engineers form the major groups working on system-level programming. Thus, if you are working in any of these areas, you may want to read this book.
Besides the previously mentioned categories, it is also possible that you are a developer working for an embedded system company. You may work on projects such as embedded system for automobile, video surveillance, or smart home. Many of these systems use Android nowadays. One of the fastest growing areas in embedded systems is Internet of Things (IoT) devices. Google announced Brillo as the operating system for IoT devices. Brillo is a simplified embedded operating system based on Android. The source code of Brillo is also included in the AOSP. This book is also relevant to people who use Brillo.
For Android application developers, system-level knowledge can help you to resolve complex issues as well. If you are working on projects that involve new hardware features, you may want to extend your knowledge to the system level.
This book is also useful for people teaching Android system programming or embedded system programming. There is plenty of source code in this book that can be used to form your own lesson plans.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the
SUPPORT
tab at the top.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on
Code Download
.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Android-System-Programming. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at [email protected] with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.
Introduction to Android System Programming
This book is about Android system programming. In this chapter, we will start with a discussion on system programming and the scope of Android system programming (to give a high-level view of this book). After that, we will look at the Android system architecture. From the architecture, we can see the layers that we will focus on in this book. We will also talk about the virtual hardware platforms and third-party open source projects that we will use in this book. In summary, we will cover the following topics in this chapter:
Introduction to Android system programming
Overview of the Android system architecture
Introduction to the third-party projects used in this book
Introduction to virtual hardware platforms
What is system programming?
When we talk about what system programming is, we can start with the definition of system programming in Wikipedia:
From the preceding definition, we can see that when we talk about system programming we actually deal with the building blocks of the computer system itself. We may depict the system architecture and how it looks like inside the system. As an example, we can refer to system programming books for Windows or Linux. The book Linux System Programming published by O'Reilly Media, Inc. includes topics about file I/O, process management, memory management, interrupt handling, and so on. There is another book called Windows System Programming published by Addison-Wesley Professional that includes very similar topics to its Linux counterpart.
You may expect similar content in this book for Android, but you will find that the topics in this book are quite different from the classic system programming book. First of all, it doesn't really make sense to have a system programming book for Android talk about file I/O, process management, or memory management, because the Linux System Programming book can cover almost the same topics for Android (Android uses Linux kernels and device driver models).
When you want to explore kernel space system programming, you can read books such as Linux Device Drivers by O'Reilly or Essential Linux Device Drivers from Prentice Hall. When you want to explore user space system programming, you can read the book that I mentioned before, Linux System Programming by O'Reilly. Then you may wonder, Do we need an Android System Programming book in this case? To answer this question, it depends on how we look at system programming for Android. Or in other words, it depends on which angle we look at Android System Programming from. We can tell people different things about the same world from different perspectives. In that sense, we may need more than one book to talk about Android system programming.
To talk about Android system programming, we can talk about it theoretically or practically. In this book, we will do it practically with a few actual projects and hands-on examples. Our focus will be how to customize the Android system and how to port it to a new platform.
What is the scope of this book?
As we know, there are two kinds of programming for Android: application programming and system programming.
Usually, it is hard to draw a line between system programming and application programming, especially for C language-based operating systems, such as Linux and all kinds of Unix system. With the Android framework, the application layer is separated nicely from the rest of the system. You may know that Android application programming uses the Java language and Android system programming using Java, C/C++, or assembly languages. To make it simple, we can treat everything other than application programming in the Android world as the scope of system programming. In this sense, the Android framework is also in the scope of system programming.
From the perspective of the audiences of this book, they may want to learn more about the layers they may touch on in their project work. The Android framework is a layer that will be changed by Google only in most cases. From this point of view, we won't spend too much time talking about the framework itself. Instead, we will focus on how to port the system including the Android framework from the standard platforms in Android Open Source Project (AOSP) to other platforms. We will focus on the layers that need to be changed during the porting process in this book.
After we have done the porting work, a new Android system will be available. One thing that we need to do for the new system is deliver the changes for the new system to the end users from time to time. It could be a major system update or bug fixing. This is supported by over-the-air (OTA) updates in Android. This is also one of the topics in this book.
Traditionally, all Unix programming was system-level programming. Unix or Linux programming is built around three cornerstones, which are system calls, the C library, and the C compiler. This is true for Android system programming as well. On top of the system calls and C library, Android has an additional layer of abstraction for the Android application level. This is the Android framework and Java virtual machine.
In that sense, most Android applications are built using Android SDK and Java language. You may be wondering whether it is possible to do Android application development using C/C++ or even do system level programming using Java. Yes, all these are possible. Besides Android SDK, we can also develop native applications using Android NDK. There are also a lot of Android framework components developed using the Java language. We can even develop Android applications using C# with Visual Studio (Xamarin). However, we won't go to that kind of complexity in this book. We will focus on the layers below the application framework. Again, the focus will be on customizing and extending the existing system or porting the entire system to a new hardware platform.
The reason why we will focus on the porting of Android systems and the customization of Android systems is because these are what most people working on the Android system level will do. After Google releases a new version of Android, silicon vendors need to port the new version to their reference platform. When OEM/ODM companies get the reference platform, they have to customize the reference platform to their products. The customization includes the build of the initial system itself and the deployment of the updates to the deployed system. In the first part of this book, we will discuss the porting of Android systems. In the second part of this book, we will discuss how to update the existing system.
If we consider the architecture of Android in the right-hand side of the following figure, we can see that most porting work will focus on the Kernel and Hardware Abstraction Layer (HAL) in the Android system architecture. This is true for other Android derivatives as well. The knowledge and concepts in this book can apply to Android wearables and Brillo as well. The left-hand side of the following figure, it shows the architecture diagram of Brillo. Brillo is the IoT operating system from Google for IoT devices. It is a simpler and smaller version of Android for IoT devices. However, the porting layer is still the same as Android.
Overview of the Android system
As we can see from the architecture diagram, the architecture layers of Android include Application Framework, Android System Services, HAL, and Kernel. Binder IPC is used as a mechanism for inter-process communication. We will cover each of them in this section. Since recovery is also part of the system programming scope, we will also give an overview of recovery in this section.
Kernel
As we know, Android uses the Linux kernel. Linux was developed by Linus Torvalds in 1991. The current Linux kernel is maintained by the Linux Kernel Organization Inc. The latest mainline kernel releases can be found at https://www.kernel.org.
Android uses a slightly customized Linux kernel. The following is a concise list of the changes to the Linux kernel:
ashmem (Android Shared Memory)
: A file-based shared memory system to user space
Binder
: An
interprocess communication
(
IPC
) and
remote procedure call
(
RPC
) system
logger
: A high-speed in-kernel logging mechanism optimized for writes
paranoid networking
: A mechanism to restrict network I/O to certain processes
pmem (physical memory)
: A driver for mapping large chunks of physical memory into user space
Viking Killer
: A replacement OOM killer that implements Android's "kill least recently used process" logic under low-memory conditions
wakelocks
: Android's unique power management solution, in which the default state of the device is sleep and explicit action is required (via a wakelock) to prevent that
Most of the changes were implemented as device drivers with little or no changes necessary to the core kernel code. The only significant subsystem-spanning change is wakelocks.
There are many Android patches accepted by the mainline Linux kernel today. The mainline kernel can even boot up Android directly. There is a blog from Linaro about how to boot Nexus 7 running a mainline kernel. If you want to try it, you can find the instructions athttps://wiki.linaro.org/LMG/Kernel/FormFactorEnablement.
If a Linux device driver is available for a hardware device, it usually can work on Android as well. The development of device drivers is the same as the development of a typical Linux device driver. If you want to find out the merges on the mainline kernel related to Android, you can check the kernel release notes athttps://kernelnewbies.org/LinuxVersions.
The Android kernel source code is usually provided by SoC vendors, such as Qualcomm or MTK. The kernel source code for Google devices can be found athttps://android.googlesource.com/kernel/.
Google devices use SoC from various vendors so that you can find kernel source code from different vendors here. For example, the kernel source of QualComm SoC is under kernel/msm and the kernel source of Mediatek is under kernel/mediatek. The general Android kernel source code is in the folder kernel/common, which looks much like the Vanilla kernel.
The default build of AOSP is for various devices from Google, such as Nexus or Pixel. It started to include some reference boards from silicon vendors as well recently, such as hikey-linaro, and so on. If you need a vendor-specific Android kernel for your reference platform, you should get the kernel source code from your platform vendors.
There are also open source communities maintaining Android kernels. For example, the kernel for the ARM architecture can be found at Linaro for many reference boards. For Intel x86 architecture, you can find various versions of kernels in the Android-x86 project. As you can see from the following Linaro Linux Kernel status website, the linaro-android tree is a forward port of the out-of-tree AOSP patches. It provides a preview of what Google's next AOSP kernel/common.git tree "might" look like.
HAL
HAL defines a standard interface for hardware vendors to implement and allows Android to be agnostic about lower-level driver implementations. HAL allows you to implement functionality without affecting or modifying the higher level system. HAL implementations are packaged into module (.so) files and loaded by the Android system at the appropriate time. This is one of the focuses for porting Android systems to a new platform. We will discover more about HAL in Chapter 3, Discovering Kernel, HAL, and Virtual Hardware. Throughout this book, I will give a very detailed analysis of the HAL layer for various hardware interfaces.
Android system services
Functionality exposed by application framework APIs communicates with system services to access the underlying hardware. There are two groups of services that application developers may interact mostly with. They are system (services such as window manager and notification manager) and media (services involved in playing and recording media). These are the services that provide application interfaces as part of the Android framework.
Besides these services, there are also native services supporting these system services, such as SurfaceFlinger, netd, logcatd, rild, and so on. Many of them are very similar to Linux daemons that you may find in a Linux distribution. In a complicated hardware module, such as graphic, both system services and native services need to access HAL in order to provide the framework API to the application layer. We will talk about system services when we debug the init process in Chapter 6, Enabling Wi-Fi on the Android Emulator to Chapter 9, Booting Up x86vbox Using PXE/NFS.
Binder IPC
The Binder IPC mechanism allows the application framework to cross process boundaries and call into the Android system services code. This enables high-level framework APIs to interact with Android system services. An Android application usually runs in its own process space. It doesn't have the ability to access system resources or the underlying hardware directly. It has to talk to system services through Binder IPC to access the system resource. Since applications and system services run in different processes, the Binder IPC provides a mechanism for this purpose.
The Binder IPC proxies are the channel by which the application framework can access system services in different process spaces. It does not mean it is a layer between the application framework and system services. Binder IPC is the inter-process communication mechanism that can be used by any process that wants to talk to another process. For example, system services can use Binder IPC to talk to each other as well.
Application framework
The application framework provides APIs to the applications. It is used most often by application developers. After an interface is invoked by the applications, application frameworks talk to the system services through the Binder IPC mechanism. An Android application framework is not just a set of libraries for the application developers to use. It provides much more than that.
The break-through technology that the Android application framework brought to the developer community is a very nice separation between application layers and system layers. As we know Android application development uses the Java language and Android applications run in an environment similar to the Java virtual machine. In this kind of setup, the application layer is separated from the system layer very clearly.
The Android application framework also provides a unique programming model together with a tight integration with the integrated development environment (IDE) from Google. With this programming model and related tools, Android developers can work on application development with great efficiency and productivity. All these are key reasons why Android has gained so much traction in the mobile device world.
I have given an overall introduction to all the layers in the previous Android system architecture diagram. As I mentioned about the scope of Android system programming before, we can consider all programming in Android systems as within the scope of system programming other than application programming. With this concept in mind, we actually missed one piece in the previous architecture diagram, which is recovery.
Recovery
In this chapter, we want to have a brief look at recovery as well, since we have three chapters about it in the second part of this book.
Recovery is a tool that can be used to upgrade or reinstall Android systems. It is part of the AOSP source code. The source code for recovery can be found at $AOSP/bootable/recovery.
The unique point about recovery compared to the other parts of Android is that it is a self-contained system by itself. We can look at recovery using the following diagram, and compare it to the Android and Brillo architectures that we talked about before:
Recovery is a separate system from Android that shares the same kernel with the Android system that it supports. We can treat it as a mini operating system or an embedded application that we can find in many embedded devices. It is a dedicated application running on top of the same Linux kernel as Android and it performs a single task, which is to update the current Android system.
When the system boots to recovery mode, it boots from a dedicated partition in the flash. This partition includes the recovery image that includes a Linux kernel and a special ramdisk image. If we look at Nexus 5 partitions, we will see the following list:
# parted /dev/block/mmcblk0
parted /dev/block/mmcblk0
GNU Parted 1.8.8.1.179-aef3
Using /dev/block/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: MMC SEM32G (sd/mmc)
Disk /dev/block/mmcblk0: 31.3GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 524kB 67.6MB 67.1MB fat16 modem
2 67.6MB 68.7MB 1049kB sbl1
3 68.7MB 69.2MB 524kB rpm
4 69.2MB 69.7MB 524kB tz
5 69.7MB 70.3MB 524kB sdi
6 70.3MB 70.8MB 524kB aboot
7 70.8MB 72.9MB 2097kB pad
8 72.9MB 73.9MB 1049kB sbl1b
9 73.9MB 74.4MB 524kB tzb
10 74.4MB 75.0MB 524kB rpmb
11 75.0MB 75.5MB 524kB abootb
12 75.5MB 78.6MB 3146kB modemst1
13 78.6MB 81.8MB 3146kB modemst2
14 81.8MB 82.3MB 524kB metadata
15 82.3MB 99.1MB 16.8MB misc
16 99.1MB 116MB 16.8MB ext4 persist
17 116MB 119MB 3146kB imgdata
18 119MB 142MB 23.1MB laf
19 142MB 165MB 23.1MB boot
20 165MB 188MB 23.1MB recovery
21 188MB 191MB 3146kB fsg
22 191MB 192MB 524kB fsc
23 192MB 192MB 524kB ssd
24 192MB 193MB 524kB DDR
25 193MB 1267MB 1074MB ext4 system
26 1267MB 1298MB 31.5MB crypto
27 1298MB 2032MB 734MB ext4 cache
28 2032MB 31.3GB 29.2GB ext4 userdata
29 31.3GB 31.3GB 5632B grow
The list includes 29 partitions and recovery partition is one of them. The recovery ramdisk of recovery, it has a similar directory structure to the normal ramdisk. In the init script of recovery ramdisk, init starts the recovery program and it is the main process of the recovery mode. The recovery itself is the same as other native daemons in the Android system. The programming for recovery is part of the scope of Android system programming. The programming language and debug method for recovery is also the same as native Android applications. We will discuss this in more depth in the second part of this book.
The third-party open source projects derived from AOSP
As we know, AOSP source code is the major source that we can start to work with in system-level programming. Various silicon vendors usually work with Google to enable their reference platforms. This is a huge effort and they won't publish everything to the world except for their customers. This brings a limitation to the open source world. Since the AOSP source code is mainly for Google devices, such as emulator, Nexus, or Pixel series, there is no problem for developers who use Nexus devices as hardware reference platforms. How about other devices? Manufacturers may release the kernel source code for their devices, but nothing else. In the open source world, several third-party organizations provide solutions for this situation. We will have a brief look at the ones that we used in this book in the following sections.
LineageOS (CyanogenMod)
LineageOS is a community providing aftermarket firmware distribution for many popular Android devices. It is the successor to the highly popular CyanogenMod. If you cannot build the ROM for your devices from AOSP source code, you may look at LineageOS source code. Because there are many devices supported by LineageOS, many major third-party ROM images are built on top of its predecessor CyanogenMod. From the famous MIUI in China to the latest OnePlus device, they all use CyanogenMod source code as the base start from. The major contributions of LineageOS/CyanogenMod to the open source world are the adaptation of the Linux kernel and HAL to various Android devices.
To build LineageOS source code for your device, the overall build process is similar to the AOSP build. The key difference is the large number of devices supported by LineageOS. For each device, there is a web page to give information about how to build for a device. We use Nexus 5 as an example. You can go to the following page for detailed information:
https://wiki.lineageos.org/devices/hammerhead
In the information page, you can find information about how to download the ROM image, how to install the image, and how to build the image. There is a build guide for devices and we can find the build guide for Nexus 5 at https://wiki.lineageos.org/devices/hammerhead/build.
To build LineageOS for Nexus 5, the two key elements are Kernel and Device. The Kernel includes the Linux kernel and Nexus 5-specific device drivers, while the Device includes the major part of the device-specific HAL code. The naming convention for both the Kernel and Device folder is android_kernel/device_{manufacturer}_{code name}.
The code name for Nexus 5 is hammerhead and the manufacturer is lge, which is LG.
Other than the Kernel and Device, other important information is the LineageOS version. You may find it on the same device information page. For Nexus 5, the versions that can be used are 11, 12, 12.1, 13, and 14.1. You may be wondering how to match LineageOS versions to AOSP versions.
The LineageOS/CyanogenMod and AOSP versions supported for Nexus 5 are CM11 (Android 4.4), CM 12 (Android 5.0), CM 12.1 (Android 5.1), CM 13 (Android 6.0), and CM 14.1 (Android 7.1.1).
Nevertheless, the idea from the preceding configuration is that the key pieces of code to differentiate one device from another are the Kernel and Device. It is possible to share the rest of the code across devices. This is one of the goals for the projects in this book. We try to keep the changes for different hardware platforms within the Kernel and the Device, while keeping the rest of the AOSP source code untouched. This is not 100% possible, but we can try to do it as much as possible. The benefit is that we can keep our code separated from AOSP code and it is much easier to update to a new AOSP version.
Android-x86
While LineageOS/CyanogenMod provides excellent support for a large number of Android devices, many of these devices are ARM-based devices from various silicon vendors, such as Qualcomm, Samsung, MTK, and so on. Similarly, there is an open source community for Intel-based Android devices as well. This is another famous open source project, Android-x86. Even though the number of Intel x86-based Android devices on the market cannot compare to the number of ARM-based devices, there is another market using the Intel x86 Android build extensively. This is the Android emulator market. For commercial Android emulator products, you can find AMI DuOS, Genymotion, Andy, and so on.
The project Android-x86 uses a very different approach to support various Intel x86-based devices compared to LineageOS/CyanogenMod. Its goal is to provide Board Support Package (BSP) for any Intel x86 devices. It is similar to how you install Microsoft Windows or Linux on your PC. You have only one copy of the release and you can install it on any Intel PCs. There is no special build of Windows or Linux for each different PC or laptop.
To achieve this goal on Android, Android-x86 customized the Android boot up process significantly. There are two stages of boot up process in Android-x86. The first stage is booting up a minimal Linux environment using a special ramdisk--initrd.img. After the system can boot up to this Linux environment, it starts the second stage through the chroot or switch_root command. In this stage, it will boot up the actual Android system.
This is a very smart way to resolve the new challenge using existing technology. Essentially, we try to resolve the problem in two steps. In the first stage of the boot up process, since both Windows and Linux can boot on Intel x86 PCs without a dedicated build, you should be able to boot Linux on an Intel device without too much effort. This is exactly what the first stage of Android-x86 boot up does. After the minimal Linux system can run properly, this means the minimum set of hardware devices is initialized and you are able to debug or boot the rest of the system using this minimal Linux environment. In the second stage, a common Android image for Intel x86 can be started with limited hardware initialization. This approach can be used in the debugging of hardware devices as well. We will show how we can do the same thing on the Android emulator in this book.
The official website of the Android-x86 project is http://www.android-x86.org/. You can find the information about the Android-x86 project there. To build Android-x86, it is a little tricky to get the source code. The original source code was hosted at http://git.android-x86.org and it was maintained by volunteers from Taiwan Linux User Group (TLUG). It was valid for several years. However, it ceased to work from April 2015.
You can always find the latest status from the Google discussion group at https://groups.google.com/forum/#!forum/android-x86. There is an official announcement about the issue of git.android-x86.org
