Building an API Product E-Book

Bruno Pedro

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 532

Veröffentlichungsjahr: 2024

Beschreibung

The exponential increase in the number of APIs is evidence of their widespread adoption by companies seeking to deliver value to users across diverse industries, making the art of building successful APIs an invaluable skill for anyone involved in product development. With this comprehensive guide, you’ll walk through the entire process of planning, designing, implementing, releasing, and maintaining successful API products.
You’ll start by exploring all aspects of APIs, including their types, technologies, protocols, and lifecycle stages. Next, you’ll learn how to define an API strategy and identify business objectives, user personas, and jobs-to-be-done (JTBD). With these skills, you’ll delve into designing and validating API capabilities to create a machine-readable API definition. As you advance, the book helps you understand how to choose the right language and framework for securely releasing an API server and offers insights into analyzing API usage metrics, improving performance, and creating compelling documentation that users love. Finally, you’ll discover ways to support users, manage versions, and communicate changes or the retirement of an API.
By the end of this API development book, you’ll have the confidence and skills to create API products that truly stand out in the market.

Leseprobe

Building an API Product

Design, implement, and release API products that meet user needs

Bruno Pedro

Building an API Product

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Kunal Sawant

Senior Editor: Kinnari Chohan

Technical Editor: Vidhisha Patidar

Copy Editor: Safis Editing

Project Coordinator: Prajakta Naik

Indexer: Subalakshmi Govindhan

Production Designer: Prashant Ghare

Marketing DevRel Coordinator: Sonia Chauhan

First published: January 2024

Production reference: 1110124

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK

ISBN 978-1-83763-044-8

www.packtpub.com

To my wife, Vânia, and my two sons, Bernat and Enric. Without their ongoing love and support, I wouldn’t have been able to write this book.

Contributors

About the author

Bruno Pedro is a computer science professional with over 25 years of experience in the industry. Throughout his career, he has worked on a variety of projects, including internet traffic analysis, API backends and integrations, and web applications. He has also managed teams of developers and founded several companies, including Tarpipe, an iPaaS, in 2008, and the API Changelog in 2015. In addition to his work experience, Bruno has also made contributions to the API industry through his written work, including two published books on API-related topics and numerous technical magazine and web articles. He has also been a speaker at numerous API industry conferences and events since 2013.

About the reviewers

David Roldán Martínez is currently the Head of APIs at Shaper by atmira. He also holds the position of Associate Professor at the Universitat Politècnica de València (Spain) and is actively involved as a researcher at VRAIN (Valencian Research of Artificial Intelligence Network).

Professionally, David is a Business and Solutions architect with over 25 years of experience in software systems architecture. He holds a Ph.D. in Telecommunications Engineering and has a strong educational background, including master’s degrees and extensive technological training. Additionally, he is a scientific-technical evangelist, having authored more than thirty books.

David’s areas of expertise encompass APIs and their applications in various markets such as Banking, Insurance, and Retail. He is well-versed in Artificial Intelligence, Digital Transformation, and the API and Open Economy.

Christos Gkoros is a seasoned software engineer and architect with over 13 years of experience in the industry. He has worked on a variety of projects in different technologies and industries, always striving to find the best possible solution. With a focus on APIs, he is currently exploring ways to help engineers in areas like API Design, API Management, and Strategy.

Christos has a proven track record of transforming complex challenges into streamlined, secure systems, having spearheaded API design at Postman and microservice architecture at Vodafone. He is passionate about mentorship and education and is committed to helping future talent grow and succeed in the field.

Table of Contents

Preface

Part 1: The API Product

1

What Are APIs?

The different types of APIs

Local APIs

Remote APIs

The history of APIs

Unix

Network APIs

The web

Available technologies and protocols

Communication protocols

Implementation technologies

Tools

Summary

2

API User Experience

Who uses APIs?

Industries

Personas

Developer experience

Second-degree user experience

API friction

Summary

3

API-as-a-Product

Business value

Monetization models

The freemium model

Tiered model

PAYG model

Support and documentation

Security

Logging and monitoring

Rate-limiting

Authentication and authorization

Summary

4

API Life Cycle

Design

Implementation

Release

Maintenance

Summary

Part 2: Designing an API Product

5

Elements of API Product Design

Technical requirements

Ideation

Strategy

Definition

Validation

Specification

Summary

6

Identifying an API Strategy

The meaning of strategy

Stakeholders

Business objectives

Personas

Behaviors

Summary

7

Defining and Validating an API Design

Technical requirements

API capabilities

Use case analysis

Functional requirements

Integration needs

Security and access control

Compliance with laws and regulations

Documentation

API mock

Prototyping an API integration with a UI

Design iterations

Summary

8

Specifying an API

Technical requirements

Choosing the type of API to build

Different types of APIs

API specification formats

OpenAPI

IDL (protocol buffers)

GraphQL

WSDL

AsyncAPI

Creating a machine-readable API definition

Following API governance rules

API design

API life cycle management

Summary

Part 3: Implementing an API Product

9

Development Techniques

Technical requirements

Prototyping an API

Choosing a programming language and framework

Factors to consider

Popular languages for building APIs

Node.js

Python

Ruby

Java

Go

Rust

Comparing programming languages

Generating server code from a specification

Generating server code using Postman

Generating server code using OpenAPI Generator

Summary

10

API Security

What is API security?

Security testing

Authentication

API key management

Token management

Authorization

RBAC

OAuth scopes

Summary

11

API Testing

Contract testing

Performance testing

Acceptance testing

Summary

12

API Quality Assurance

Defining QA

Test planning and execution

Behavioral testing

Regression testing

API monitoring

Summary

Part 4: Releasing an API Product

13

Deploying the API

Continuous integration

API versioning

Incremental API versioning

Semantic API versioning

Calendar-based API versioning

Choosing an API gateway

Summary

14

Observing API Behavior

API usage analytics

Application performance monitoring

User feedback

Aggregating and categorizing feedback

Acting on feedback

Scaling user feedback

Summary

15

Distribution Channels

What is API distribution?

Pricing strategy

API portal

Community

Marketplaces

Documentation

Summary

Part 5: Maintaining an API Product

16

User Support

Helping users get their jobs done

Support channels

Prioritizing bugs and feature requests

Summary

17

API Versioning

Technical requirements

Managing multiple API versions

Breaking changes

Communicating changes

Summary

18

Planning for API Retirement

When should you retire an API?

Communicating API retirement

API product retrospective

Summary

Index

Other Books You May Enjoy

Preface

Building an API Product is a comprehensive guide that ranges from the fundamentals of APIs and their inner workings to mastering the steps involved in building successful API products. With this book, you will be able to confidently and effectively create cutting-edge API products that excel in today’s competitive market.

Who this book is for

This is a book that helps product managers and software developers navigate the world of APIs to build programmable products. You don’t have to be an experienced professional to learn from this book, as long as you have a basic knowledge of internet technologies and an understanding of how users interact with a product.

What this book covers

Chapter 1, What Are APIs?, introduces you to API fundamentals, origins, and types such as REST, gRPC, AMQP, and MQTT.

Chapter 2, API User Experience, explores how the API user experience is vital, second-degree experience, and the impact of friction on success.

Chapter 3, API as a Product, outlines an API as a standalone product, emphasizing business value, monetization options, support, documentation, and crucial security.

Chapter 4, API Life Cycle, provides an overview of the API life cycle stages, covering design, implementation, release, and maintenance, offering an opinionated approach to API product management.

Chapter 5, Elements of API Product Design, introduces you to the key API product design stages, connecting ideation, strategy, definition, validation, and specification, paving the way for an in-depth exploration.

Chapter 6, Identifying an API Strategy, analyses the strategy stage of API design, emphasizing identifying stakeholders, determining business objectives, and understanding user personas and behaviors.

Chapter 7, Defining and Validating an API Design, covers the techniques for defining and validating API design, starting with strategy-derived information and exploring API mocks, UI integration, and stakeholder iteration.

Chapter 8, Specifying an API, guides you on how to select an API architectural type based on behaviors and capabilities, refining the definition with constraints and industry practices and creating a machine-readable representation with governance rules.

Chapter 9, Development Techniques, offers a beginner-friendly guide to API development, covering language and framework selection, code generation from specifications, prototyping, and extending with business logic,

Chapter 10, API Security, explores API security, emphasizing its importance, distinguishing between authentication and authorization, and introducing a security testing technique called fuzzing.

Chapter 11, API Testing, introduces you to API testing methods, covering contract testing to ensure specification compliance, performance testing execution, and the connection of acceptance testing to user personas.

Chapter 12, API Quality Assurance, covers API quality assurance, introducing behavioral testing to validate against identified behaviors and setting up API monitors for periodic testing.

Chapter 13, Deploying the API, provides an overview of the API deployment process, covering continuous integration, agility, automated testing, deployment, and API gateway trade-offs.

Chapter 14, Observing API Behavior, introduces you to API usage analytics, APM, and user feedback analysis to identify and measure important metrics, usage patterns, and behavior.

Chapter 15, Distribution Channels, covers API distribution strategies, including pricing, API portals, marketplace listing, and documentation options to enhance user activation.

Chapter 16, User Support, delves into ways to ensure user success with an API, covering support channels, forums, and prioritizing bug fixes and feature requests from user feedback.

Chapter 17, API Versioning, explores techniques for managing multiple API versions, handling breaking changes effectively, and communicating changes to users using machine-readable methods.

Chapter 18, Planning API Retirement, discusses API retirement, covering its definition, considerations, and communication to users and conducting a retrospective to document what you have learned from the process.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Building-an-API-Product. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system.”

A block of code is set as follows:

html, body, #map { height: 100%; margin: 0; padding: 0 }

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

[default] exten => s,1,Dial(Zap/1|30) exten => s,2,Voicemail(u100) exten => s,102,Voicemail(b100) exten => i,1,Voicemail(s0)

Any command-line input or output is written as follows:

$ mkdir css $ cd css

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Select System info from the Administration panel.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Building an API Product, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781837630448

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1:The API Product

This part provides a comprehensive guide to API development and management, beginning with fundamental concepts, types, and origins, followed by a focus on user experience and the significance of the API as a standalone product. It then delves into the API life cycle stages, covering design, implementation, release, and maintenance, with an opinionated approach for effective API product management.

In this part, you’ll find the following chapters:

Chapter 1, What Are APIs?Chapter 2, API User ExperienceChapter 3, API-as-a-ProductChapter 4, API Life Cycle

1

What Are APIs?

APIs are the most powerful technology available today. While the API acronym can be deceitfully simple, the concept it describes offers infinite possibilities. Welcome to the world of APIs, where you’ll learn how to build an API product. Your first step in this expedition is to first learn what an API is. In this chapter, you will understand the nature of APIs, looking back to their origins. You’ll also get to know which technologies and tools are available for you to use.

The chapter begins by exploring different types of networks, such as the internet, and how APIs work on them. You will then be guided through the history of APIs. You’ll see how APIs came to life and understand how certain concepts in use today were born. Finally, you’ll see that there are different technologies and tools that you can use to build an API product from scratch.

By the end of this chapter, you will know that APIs can exist on different types of networks. You will understand what those networks are and what the most appropriate one for your API product is. You will also know that there are synchronous and asynchronous APIs and what those terms mean. Most importantly, you will know how to pick the right type of API and tools to build your API.

In this chapter, we’re going to cover the following main topics:

The different types of APIsThe history of APIsAvailable technologies and protocols

The different types of APIs

This section gives you an overview of the different types of APIs that exist. APIs are split between local and remote, and then by the protocols that they adhere to. You’ll start by understanding what an API is at a high level and why it’s so important. Then, you’ll dive into the different types of APIs. Let’s get started.

Application programming interfaces, or APIs, allow applications to be used programmatically. They create an interface—a layer of abstraction—that opens applications to interactions from the outside. The interface has the goal of standardizing any connection to the application. Suppose you think about an interface as a common boundary between two entities. In that case, an API is a way to let an application communicate with other entities in a programmatic fashion.

This type of interaction is what you can call an integration. Integrating different applications, or different parts of the same application, lets you build products by putting together pieces that are ready to be used. Instead of creating all the features of a product from scratch, you can utilize functionality that is already available in the form of an API. That alone is a powerful tool to use. Creating a product by using APIs can be done in a fraction of the time it would take if you develop all the features yourself. That happens because you can reuse pieces of functionality that are standardized and well understood. Those pieces of functionality can be a whole product, a single feature, or a subset of a product represented by a selection of features. Different types of APIs provide different types of interactions, as you’re about to learn.

Local APIs

Local interfaces are the most used type of API, even if they’re often seen as invisible. All the applications that run on a device need to communicate with the hardware. Applications interact with the device via local APIs. They offer the advantage of providing a standard method of programming the device to behave according to what users want. POSIX is one such standard created by the IEEE Computer Society. It stands for Portable Operating System Interface, and its goal is to establish a layer of communication that is standard across different operating systems. Another similar standard is the Single UNIX Specification (SUS). macOS, a popular operating system developed by Apple Inc., is considered partly compliant with POSIX and fully compliant with the SUS. This means that anyone that interacts with macOS knows that it follows certain rules and conventions that have been standardized. In theory, an application that is built to run on macOS could also run on other systems that are compliant with the same standards.

Another way of introducing a standardized local layer of communication is by using common software libraries. Even if a system doesn’t follow a full standard, some of its parts can use standardized libraries. Java offers a popular set of libraries and APIs that can be used across systems. The programming language was created by Sun Microsystems—acquired by Oracle Corporation—and has been used on almost all operating systems. Java fully embodies the goal of standardizing how applications interact. Its slogan is Write once, run anywhere, and it symbolizes the importance that its creators give to standardized interfaces. Java’s versatility is enormous. You can use Java to create mobile applications that run on Android devices, desktop applications, and everything in between.

By now, it’s clear that operating systems’ standards and libraries offer a way of interacting with the lowest layers of computing devices. Another form of abstraction that encapsulates reusability at a higher level is available through software modules. Most modern scripting programming languages have the ability to create and use modules. Modular software development has become a popular way of building applications. Modules provide functionality that is ready to be used and increase the speed at which applications are built.

A widespread module system exists for the JavaScript programming language. Its name is npm, the Node Package Manager. Its authors claim that npm is the world’s largest software registry, with over one million modules available to be used by anyone. According to GitHub, JavaScript is the number one used programming language at the time of writing. In fact, JavaScript has been in the first position for the last eight years at least. Because npm is used by applications written in JavaScript, it’s the most used module system.

Other module systems exist for different programming languages, and they all share that they want to facilitate the reuse of functionality and increase the speed of developing software. Python, the second most popular programming language, has PyPI, the Python Package Index. The third most popular programming language, Java, also has its package system, Maven. There are all kinds of modules ready for anyone to use on their applications. The point is that anyone is free to create and publish modules and also to reuse modules that other people have published. Hence, a vast ecosystem of modular software development keeps growing.

While local interfaces deal with the interaction between different parts of a local system, some of those parts let you communicate with the outside world. Communication with remote systems is also abstracted and standardized in the form of APIs. Read on to learn more about the different remote APIs and how they can enhance the features of an application.

Remote APIs

Most people think of APIs as a way to interact with software that is running remotely. They tend to ignore all the local interfaces that you’ve read about before—not because local APIs aren’t important, but because they feel invisible. The opposite happens with remote APIs. Instead of being invisible, remote APIs feel like they’re the most critical part of an application. The act of starting a connection to a system that is running on a different part of a computer network feels like something worth paying attention to. Remote connectivity can be split according to the type of network being used. Let’s focus on local area networks (LANs) and wide area networks (WANs) because that’s where most APIs operate.

LANs connect devices that are physically in the same location. The types of applications that exist on a LAN are meant to be accessed exclusively by devices that are connected to the same network. APIs that operate on LANs are typically focused on supporting specific classes of applications and not on providing generic services to consumers. In other words, LAN APIs offer a way for devices to connect to applications running on the same network. As with local APIs, here, the goal is to standardize how the same type of applications communicate on LANs.

Databases are one type of application that is widely used in local networks. The ODBC standard was created to standardize communication between applications and databases. ODBC stands for Open Database Connectivity and is a standard API for accessing databases. Applications that use ODBC can be ported across different database systems without having to be rewritten. You can, for instance, develop a warehouse stock application that uses the MySQL database system. Suppose that, at some point, you decide to switch to Oracle or some other database system. You don’t have to rewrite your application as long as the database system supports ODBC. In the same way, if you decide to change the implementation of your application to a different programming language, you don’t have to change the database system. As long as the programming language supports ODBC, you know that you’ll be able to interact with your database.

Printing is another popular activity on local networks. As you would expect, there are APIs that standardize the communication between printers and other devices in the same LAN. One such API is the Line Printer Remote protocol, or LPR. This protocol lets you interact with a printer, programming it to print documents and even changing the configuration options of the target printer. Even though printing happens primarily in LANs, it’s a type of application that can also be carried out across the internet. To make communication with printers work easily outside LANs, there is a remote API called Internet Printing Protocol, or IPP. According to Michael Sweet from Apple Inc., “at least 98% of all printers sold since 2010 support IPP." It became so popular because it offers features such as authentication, access control, and encryption of data transmitted to the printer. And it’s not the only API that operates on the internet, as you’ll see if you keep reading.

When you hear the term API, you immediately think about services that run on the internet. That’s because wide access to networked services helped popularize the creation of APIs. Externalizing features of an application feels natural in an environment where all services are connected to the same network. Many times, you can even confuse internet APIs with the services that they expose. We often talk about the API as the offering rather than the interface. That indicates that the internet has contributed to the fragmentation of the types of APIs that are available. There are API types that are best suited for reading data while other types are better used for synchronously storing information, and there are types that work well to asynchronously share information about events.

Let’s start by exploring API types that let you easily read data from a remote server on the internet. You can say that the simplest way to read data remotely is to directly access a document. However, you would only call that an API if there were some degree of programmability involved. In other words, when you’re directly retrieving a document, you’re not sending any parameters to an API. To make it programmable, there has to be something on the server that interprets the request parameters and changes the returned output based on what is being requested. A remote procedure call, or RPC, is an example of a type of API that lets the requester send parameters to a server and, in return, receive information. In the same way that you can read information with it, you can also use it to store information on a server. In that case, you’re sending parameters along with the information that you want to store. Depending on the size of the data—what you call an API payload—you can choose what type of API and which technology to use.

As a rule of thumb, anything that happens on the internet works better with short-lived connections. The internet is an open network. Connections between different points on the internet can change without notice, and that can affect the quality of communication. Communicating asynchronously is also an option. There are types of APIs that focus specifically on letting you share information in an asynchronous way. Usually, these are used for sharing information about events, but also for receiving the result of long-running operations. You make a request that you know is going to take a long time to finish. When the request is completed by the server, it will share the result with you. If availability is what matters the most, then you decide the type of API based on what is reliable most of the time. Many APIs end up running on top of the web because it’s the most widely used protocol, and you accept it as having a high resilience. In fact, web APIs are what you’ll be working on most of the time when you’re building API products.

Web APIs are a type of API that uses the internet and web-specific protocols to communicate. In the same way that the remote APIs that you’ve read about before make remote resources appear as local, web APIs offer the same functionality for resources available on the web. On the web, it’s a common approach to identify the supported media types that can be transferred from the server to the client. That is also the case for web APIs. Two of the most used media types are the Extensible Markup Language, or XML, and the JavaScript Object Notation, or JSON. These media types can be easily used and interpreted by API client software. The big advantage of web APIs over other types of remote APIs is the features that the web offers. Just by using the web, you have access to content caching, or the ability to temporarily store responses that can be reused between requests. You also have access to authentication mechanisms that don’t require any specific implementation. Finally, you become part of a vast ecosystem of server and client tools that are widely available for anyone to use.

While there are different types of APIs, you’re reading this book most probably because you’re interested in web APIs. As you’ve seen before, to most people, APIs are a synonym for something such as a programmable interface running on the web. The API that you’ll build will probably run on the web as well, so let’s use that as a guide throughout the rest of the book. Keeping in mind that there are several types of APIs, let’s focus on how to build web APIs. To get there, let’s look now at how APIs came to exist and how they have been evolving.

The history of APIs

By now, you already know that there are different types of APIs that you can use depending on what you’re trying to achieve. It’s important to know how APIs were created and which events were the key contributors to their evolution. Learning how we got to where we are now is the first step to understanding how to build successful products on top of APIs.

To understand how APIs were invented, let’s go back in time to circa 1950. At that time, computing was just getting started, and the first known mention of a software module was made by Herman Goldstine and John von Neumann. The authors were referring to local APIs in the form of modules or, as they called it, “subroutines.” In their Report on the Mathematical and Logical Aspects of an Electronic Computing Instrument from 1947, they explain what a subroutine is. To the authors, a subroutine is a special kind of routine—a set of program instructions that can be reused. They describe that a subroutine has the purpose of being substituted inside any existing routine. It’s clear that the authors were focusing on reusability, as you’ve read before in this chapter. In fact, from then on, APIs have been used as a way to reduce the amount of programming required to build an application.

However, the work of Goldstine and von Neumann was purely theoretical and couldn’t be put into practice at that time. It was only when the Electronic Delay Storage Automatic Calculator, or EDSAC, was created that software modules were actually used. In 1951, Maurice Wilkes, the creator of EDSAC, was inspired by the work of Goldstine and von Neumann. Wilkes introduced the use of modules as a way to reuse functionality and make it easier to write programs from scratch. Wilkes, jointly with Wheeler and Gill, describes how subroutines could be used in their book The Preparation of Programs for an Electronic Digital Computer. In this, they explain that with a library of subroutines, it should be simple to program sophisticated calculations. You’d only have to write a master routine that, in turn, calls the various subroutines where the different calculations are executed.

Still, the first time the term “application program interface” (notice that the word used isn’t “programming”) appeared was probably in the Data Structures and Techniques for Remote Computer Graphics paper published in 1968. In this, its authors, Ira Walter Cotton and Frank S. Greatorex, notice the importance of the reusability of code in the context of hardware replacement. The paper mentions that even if you change the hardware, a “consistent application program interface could be maintained.”

Since then, the concept of reusability through the isolation of code to create APIs has been evolving. While initially, the focus was on APIs that could be used locally within an operating system, at a later stage, remote APIs were explored, most notably web APIs. In 2000, Roy Fielding published an intriguing Ph.D. dissertation titled Architectural Styles and the Design of Network-based Software Architecture. In it, Fielding analyzes the differences between local APIs—which are based on libraries or modules—and remote APIs. The author calls local APIs library-based and remote APIs network-based. While local APIs have the goal of offering entry points to code that can be reused, remote APIs aim to enable application interactions. According to Fielding, the only restriction that remote APIs have is the ability to read and write to the network where they operate. Let’s continue our exploration through the history of APIs. Keep reading to understand how one of the most popular operating systems is in the origins of web APIs.

Unix

Web APIs originated with the Unix operating system and its way of letting different applications—or processes—communicate with each other. In reality, Unix is not a single operating system. It’s a group of operating systems with the same root: AT&T Unix. The first version was created in the 1970s by Ken Thompson and Dennis Ritchie at the Bell Labs research center. AT&T decided to license Unix to other companies after the first version was released to the public in 1973. The licensing of Unix made it—and all its variants—one of the most used types of operating systems of all time. One of those variants, the Sun Microsystems Solaris operating system, has contributed the most to the history of web APIs.

From its inception, Unix has been recognized for its modular structure, where individual processes are created with simplicity in mind and aimed at seamless collaboration. This approach, referred to as the Unix philosophy, has become one of the primary reasons for its immense success and a crucial factor in the evolution of web APIs. Brian Kernighan, one of the developers of the Unix operating system, described interoperability during a demo performed in 1982:

“(...) you can take a bunch of programs and stick them together end-to-end so the data simply flows from the one on the left to the one on the right.”

Inter-process communication, or IPC, is a system integrated into Unix that enables the transfer of messages between various processes. IPC is a collection of APIs that allows developers to coordinate the execution of concurrent processes. The IPC framework offers multiple forms of communication, including pipes, message queues, semaphores, shared memory, and sockets, to accommodate the needs of diverse applications. However, it’s worth noting that, except for sockets, all other communication methods are confined to processes running on the same server, limiting their scope and functionality. It’s precisely with sockets that network APIs gained traction and different use cases emerged.

Network APIs

Sun Microsystems leveraged the functionality of network sockets to introduce a method of communicating with remote processes, known as RPC. The concept of RPC was first introduced in the 1980s as part of Sun’s Network File System (NFS) project and adhered to the calling conventions used in Unix and the C programming language. It rapidly gained popularity due to its ability to enable any running application to send a request over a network to another application, which would then respond with the result of the requested operation. The messages and responses are encoded using the External Data Representation format, or XDR, which provides a standard format understood by both the producer and consumer. The RPC protocol offers the capability to deliver messages with XDR payloads through either the User Datagram Protocol (UDP) or the Transmission Control Protocol (TCP), thereby providing compatibility with different network types. While UDP is a protocol more oriented toward performance, TCP offers more reliability in situations where the quality of the network is questionable.

The journey from the first implementations of RPC to its official publication as an Internet Engineering Task Force (IETF) Request for Comments, or RFC, took about a decade. The RPC protocol was first published as RFC 1831 in 1995 and underwent various transformations through subsequent versions until it reached its latest form in 2009, as described in RFC 5531. That year, Sun Microsystems changed the RPC license to the widely used three-clause BSD, which made it available for free to anyone. Today, most variations of the Unix operating system provide some form of native RPC support. At the same time, Microsoft Windows also offers official support for the protocol through its Services for UNIX (SFU) software package. Other operating systems offer RPC compatibility with various implementations for programming languages such as C, C++, Java, and .NET.

Despite RPC’s widespread popularity and reputation as a lean and straightforward protocol to implement and use, there may be better choices for heterogeneous network environments. One of the primary issues with RPC has to do with the passing of parameters and the interpretation of data between clients and servers that are written in different programming languages. Although RPC relies on the XDR format to describe payloads, the definition of data types can vary between operating systems and programming languages, resulting in the misinterpretation of information. This has led to the rise of other protocols that provide an abstraction layer for messages and parameters.

The concept of service-oriented architecture, or SOA, emerged as the prevailing standard for facilitating collaboration among applications operating in heterogeneous environments. Around the same period, various internet-based public services were gaining popularity, with the World Wide Web (WWW) particularly attracting the attention of a wider audience outside the academic community.

The web

In 1989, Tim Berners-Lee, an English scientist, created the WWW. Since then, it has become the primary means of accessing information and communicating online. During its early years, the web comprised simple interconnected blocks of information, known as web pages, which could be accessed to view information. These web pages were manually updated by webmasters, who were responsible for maintaining the content. With the rise of commercial web initiatives, various services were developed to allow individuals to upload and share personal information such as photos, blogs, and other forms of multimedia. This led to the creation of desktop applications that enabled users to interact with online services more efficiently. Initially, these applications were used to download information, but eventually, they allowed users to upload content to the web.

The way desktop content-creation applications communicated with the newly launched web services gave rise to what we now refer to as web APIs. One such early example was Flickr, a widely used photo-sharing service that allowed developers to interact with it via a web API. The API enabled developers to perform a range of tasks such as uploading, downloading, listing, and searching photos from a single user account or across the entire service. Business applications such as Salesforce also benefited from what web APIs had to offer. In fact, Salesforce was probably the first modern web API to be launched and used.

On the more closed side of the software industry, other protocols started to emerge, with the goal of simplifying the life of developers and integration designers. One such protocol gained significant popularity because of its natural integration with existing Microsoft tools. It was the Simple Object Access Protocol, or, in short, SOAP. Microsoft promoted SOAP, and it became the number one way to integrate its different products. SOAP became popular outside of the Microsoft world with support from several programming languages and operating systems. For some time, SOAP was seen as the successor of RPC and the way to connect all kinds of business applications. You can see a simplified illustration of a SOAP request here:

Figure 1.1 – A simplified illustration of a SOAP request

Around the same time, another protocol was being developed to utilize as many features of HTTP as possible and better meet the needs of web services. This resulted in the creation of the Representational State Transfer Protocol, commonly known as REST. Its creator was Roy Fielding, who you already know from before in this chapter. In his Ph.D. dissertation, Fielding not only described how remote APIs should operate but also invented REST:

REST is a hybrid style derived from several of the network-based architectural styles (...) and combined with additional constraints that define a uniform connector interface.

The network-based architectural styles that Fielding refers to include cache and client-server, two things that are familiar in web interactions. Compared to SOAP, REST is much easier to understand and process and a natural winner on the open web because it doesn’t need as many rules to operate as SOAP does. You can see a simple illustration of this protocol here:

Figure 1.2 – A simplified illustration of a REST request where data is sent from a client to a server to create (POST), replace (PUT), or modify (PATCH) a resource

However, because it’s more open, it doesn’t offer the level of control and security that SOAP offers. While that might not be an issue with non-critical applications, it is a must for business-related APIs. Because of that, another protocol was created. This time, the goal was to increase control over what was transmitted so that the information could be validated in a reproducible way.

Google’s Remote Procedure Call, or, in short, gRPC, was born in 2015. It started to be used by almost all of Google’s open web APIs. gRPC works on top of HTTP/2, the second version of HTTP, offering, among other things, low latency and the ability to stream data between servers and clients. However, the big advantage of gRPC over other protocols is that it uses a strict interface description language. Protocol Buffers, or Protobuf, is the format used by gRPC to describe the interface and messages shared between clients and servers. Unlike other languages, Protobuf is binary and offers high security and performance. However, Protobuf is oriented toward providing a way to remotely execute code that is available on the API server.

At around the same time, the need for openly querying large amounts of data started to grow. Facebook, a popular social network, pioneered the use of graph databases. It was clear that REST wasn’t the best way to access data that was always changing, and gRPC wasn’t the answer either. So, GraphQL, a data query and manipulation language, was created. Compared to other API architectures, its main difference is that it allows clients to define the shape of the data that they want to query—that, too, at runtime, in a dynamic way. Even though it doesn’t sound like much, this was a big deal because of two factors. On the one hand, it allowed bandwidth-conscious clients—such as mobile phones—to retrieve just the data they needed, saving precious bandwidth. On the other hand, it opened the available data graph to clients, allowing them to openly query all existing data through the API.

Available technologies and protocols

Read on to learn more about the most relevant technologies and protocols that you can use to build API products. It would help if you didn’t restrict yourself to using web APIs. Instead, you should build the API product that best reflects the needs of your users, and to do that, the more knowledge you have about what’s available, the better. This section offers detailed information about the different API technologies, the communication protocols, and the tools that are considered important to someone building an API product. Let’s start by splitting the knowledge into the areas of communication protocols, implementation technologies, and tools.

Communication protocols

Among the available communication protocols, the ones that are most overlooked are the ones that run on local networks. Usually, local network protocols help Internet of Things (IoT) devices and applications communicate. These protocols use a low amount of power to preserve the batteries of the devices that they support. Among IoT protocols, you have one called Zigbee. This protocol is an IEEE 802.15.4-based specification and is used by popular home automation companies. Philips Hue, for instance, uses Zigbee to power communication between lamps and other devices. Indesit, a house appliance manufacturer, uses Zigbee to adjust its washing machines’ cycle starting time according to the price of electricity. Yale, one of the oldest locks companies, uses Zigbee to control its smart door locks. Samsung, a technology manufacturer, uses Zigbee to let you control and monitor its smart refrigerators. If you’re thinking about building an API product that interacts with a local device, then a protocol similar to Zigbee might be a good choice.

Even though Zigbee has been gaining popularity, the fragmentation of local connectivity protocols is something to pay attention to. For that reason, a group of organizations created a new standard called Matter. Among the organizations behind Matter is, in fact, Zigbee. Matter’s goal is to help new product builders adopt a communications standard that they know will work with a vast array of products.

Let’s now focus on communication protocols that operate on the internet. While local network protocols solve challenges related to power consumption and interoperability, internet protocols are more focused on the reliability of communication. Reliability, in this case, means the ability to consistently transport information between a user and a server. Users unknowingly engage with servers while they’re performing their online activities. The protocol behind most online activities is the Hypertext Transport Protocol, or HTTP. From a user perspective, it’s as if the information is right in front of you, being displayed on the screen of the device that you’re using. From a communication perspective, information is traveling across the world using the internet to arrive at your device and then be displayed. HTTP is behind that communication and translates what users request into commands that are sent to servers. The web is powered mostly by HTTP, and when you refer to APIs, most of the time what you’re referring to are web APIs.

In summary, HTTP is the protocol behind most of the available APIs. HTTP is a protocol that works in a synchronous way. In other words, when users request something, the information is sent to a server, and the client waits until a response is available. The response might become available almost immediately, in which case the interaction feels like it’s happening immediately. Or, in some situations, it might take longer for a server to produce a response. Other protocols have been created to handle situations where the user doesn’t need a response immediately or the user doesn’t want to wait for a response to become available. Those protocols handle what is called asynchronous communication.

Among the available asynchronous ways of communicating, you have the Advanced Message Queuing Protocol, or AMQP. This is a protocol that is primarily focused on handling queues of messages. A queue of messages is a group of pieces of information that are stored in a specific order. Each message is picked from the queue and processed one after another. Messages can contain any information and can be used in a variety of patterns that enable multiple kinds of products. You can have messaging patterns that let users perform a command asynchronously. Other patterns are used to let users receive notifications on their devices. There are even patterns that let a server broadcast messages to a group of users without having to connect to each user individually. The important thing to retain about AMQP is that it lets you create interactions that don’t require an immediate response from the server.

Another asynchronous protocol that is popular among IoT products is Message Queuing Telemetry Transport, or MQTT. This protocol focuses on being lightweight in the information it needs to let messages flow from a server to a client. MQTT was built to be as simple as possible and enable low-powered devices to subscribe to information that servers make available. Before, you saw how IoT devices can communicate synchronously inside a local network using Zigbee. In this case, MQTT enables those devices to send and receive information to other devices in an asynchronous way.

Implementation technologies

Now that you know what the available communication protocols are, let’s see which technologies you can use to build API products. Starting with local networks, the technologies that you can use either work on top of protocols such as Zigbee or something at a higher level such as HTTP or MQTT.

Let’s start with Zigbee. The Zigbee protocol describes three types of devices that can operate on the network. You can build a Zigbee coordinator, a router, or an end device. The Zigbee coordinator manages communication between other types of devices. It can also serve as a bridge between Zigbee and other types of networks, such as the internet. The Zigbee router is responsible for making sure that the information flow reaches all the devices present in the network. Finally, Zigbee end devices are the final nodes in the network. You can build on top of the Zigbee protocol by using—or asking your engineering team to use—one of the available frameworks. The Connectivity Standards Alliance (CSA), formerly known as the Zigbee Alliance, offers documentation and pointers to implement solutions for Zigbee and also for the Matter protocol. Operating systems such as Tizen offer direct support for Zigbee to applications built to run on it.

Moving into internet technologies, let’s look at what you can build on top of HTTP. Anything that runs on top of HTTP is understood as “the web.” Fortunately, there are plenty of technologies and approaches to building web APIs. From frameworks to API specifications, you have a lot to choose from. There may be many solutions because the web itself is the most used communication platform. To begin with, most programming languages offer a way of building a web API from scratch. For instance, Node.js, a popular programming language, has the Express.js framework. Python, another language, offers Flask and FastAPI. And there are other options for languages such as Java or PHP. You can pick the language and framework that best fits your needs and where you or the engineers that work with you feel more comfortable.

To specify the API that you’re building, you also have different options. In this case, depending on the type of web API that you’re building, you have different specification standards. OpenAPI is the preferred specification for REST APIs. While REST restricts your API consumers to the resources and operations that you make available, you can offer more generic access through GraphQL. Essentially, GraphQL lets your API consumers access the data that you are exposing as a graph. In other words, you don’t have to provide all the queries and operations beforehand because consumers can navigate the data itself. If you’re concerned about performance and data validation, you can use the gRPC specification. With this approach, you have a specific format for the information that is shared between consumers and servers, making the communication stricter than with the REST approach. All these technologies provide a synchronous communication solution. Read on to learn how you can build asynchronous APIs.

If the API that you’re building doesn’t require an immediate response, or if the operation that you offer takes too long to process, then you can think of offering an asynchronous solution. Asynchronous APIs usually make use of two communication channels. One of the channels is used by the API consumer to send information to the server. The API consumer uses this channel to execute an operation on the server or to request certain information. The second channel is used by the server to communicate the result of the request back to the consumer. Those two channels can coexist on the same type of network or can use totally different approaches. One example of a second channel that runs on top of HTTP iscalled Webhooks.

With Webhooks, you ask the API consumer to provide a way to receive a request from the server. When the server has information to be shared with the consumer, it uses the Webhook URL to push the available data. The consumer then receives the request and the information that they were waiting for. Another way of building an asynchronous API on top of the web is to use something called WebSockets. In this case, the API will be used by web browsers to communicate with the server. The goal is to open a direct communication channel that can be used by both the browser and the server to send information in both directions. This will allow the server to send information to the browser asynchronously. As an example, this is how some solutions, such as browser notifications, are implemented.

If we now move to other asynchronous protocols, there are different products that you can use. RabbitMQ is a product that provides an asynchronous communication broker that runs on the AMQP protocol. Mosquitto is another broker that in turn runs on MQTT. Another product that provides asynchronous messaging solutions is Kafka. Even though Kafka uses its own communication protocol and message format, it’s worth mentioning because it’s one of the most used asynchronous solutions.

Tools

By now, you know about the protocols, formats, and technologies that are available to help you build an API product. Continue reading to learn about the tools that you can use to get your API up and running. You have tools that let you design and define how your API will behave. Other tools help you validate your API and offer a mockup. There are also tools that convert your API definition into running code that can be deployed to a running server. Whichever tool you use, remember that it doesn’t replace your knowledge. You should be able to do things on your own by understanding the principles and the theory behind your actions. Let’s start by looking at API design tools. These are usually web applications that help you create an API definition document. These are important to you because they help you build your API product during the first stages of the process.

One of the most popular API design tools is Postman. This is a web and desktop application that can be used individually or by different members of a team to collaborate. Postman offers an interface that lets you create API definitions and automatically runs validation checks to make sure that your API follows industry best practices. With Postman, you can create an API mock from your API definition. You can share a link to the mock with your potential customers and use it for validating your API design. Your API clients can use the mock to try making requests to the API and seeing what it looks like before you actually release any product.

Another tool in the area of API design is Stoplight. The company offers a web and desktop application that lets you design and document your API. Among its features is the possibility of generating and customizing API portals to offer onboarding and documentation to developers. API design is the strong offering of Stoplight. It offers a unique visual approach to designing an API. Instead of typing your OpenAPI definition into a text editor, you can visually configure how your API will behave.

Swagger is probably the oldest API design tool still available. It offers a web application that lets you design your API by writing its OpenAPI definition. Even though it uses a text editor, it automatically renders the result of what you’re typing. You can interactively write your API definition and see the results immediately as API documentation. That’s an interesting approach because it gives you the perspective of what your API consumers will view.

Another area that is interesting to you is API documentation. API documentation gives you the ability to explain to your users what your API does and how they can use it. There are several tools in this area, ranging from simple documentation generators to more sophisticated API portals. In fact, all the aforementioned API design tools also allow you to publish API documentation. However, there are other tools that are more focused on API documentation.

One such tool is ReadMe. This is a web application that lets you build interactive API documentation. Anyone that accesses your API documentation will be able to interact with the API and engage with you, or your support team, if needed. It also lets you, the API owner, interact with your users by sharing updates whenever something changes. With ReadMe, you don’t have to install anything as the tool hosts the whole documentation.

Another tool that lets you build your API portal is Apiable. This lets you create your API documentation and goes further by letting you manage the signup and onboarding process. Apiable also lets you create what it calls “API products.” This feature lets you define signup processes and subscription plans for your API. Apiable manages the process so that you can focus on building and maintaining your API.

If you prefer to use an open source tool that you can run on your machine, you can look at Redoc. This is a tool that generates documentation from an OpenAPI definition document. Redoc follows a popular three-panel page layout with features such as a search bar and API request and response examples. You can install it on your machine and run it every time you update your OpenAPI definition.

The tools that I presented here are meant to be examples of what is available. Keep in mind that tools change—what’s important is that you stay updated with what exists. No tool can replace your own knowledge and how you’re able to build and maintain your API. However, having the right tools at hand makes your job much easier and more pleasant.

Summary

By now, you have a fair understanding of what APIs are, how they evolved, and how their history is connected to the history of computing and the internet. You also know which technologies and tools you can use to build your API product. Let’s look back at all the things you learned in this chapter.

You started by understanding the concept of API as a way to connect different pieces of software together, independently of their location or the communication protocol that they’re using. Then, you dived into local APIs, which run locally on the device and help the operating system and the applications that run on top of it communicate with each other. After that, you learned the difference between these local APIs and the remote ones that run on networks. Then, you walked through the history of APIs, seeing that, since the beginning, emphasis has been placed on the reusability of software. You saw how different people, such as von Neumann, Fielding, and Berners-Lee, influenced how APIs work and what they do. From there, you went through the existing technologies, protocols, and tools that are available for you to build an API product.

These are some of the concepts that you’ve learned in this chapter:

An API is a programmable way of interacting with an applicationAPIs offer reusable functionality that reduces the time it takes to build new applicationsSoftware modules and programming language libraries can be considered APIsAPIs exist on different types of networks, not just the webDifferent communication protocols provide different features to the APIs that run on them

The following are things to take into account when choosing between a synchronous and an asynchronous API approach:

The features that different API standards such as RPC, REST, gRPC, and GraphQL offerDifferent tools can help get your job done and help you with API design, documentation, validation, testing, and deployment

Thank you for reading this chapter. In the next chapter, you’ll focus on API user experience or API UX. You’ll be able to understand how to identify the users of your API and how to make sure they have the best possible experience. Keep reading to learn more about developer experience, API friction, and other topics related to API UX.