Mastering Prometheus E-Book

William Hegedus

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 430

Veröffentlichungsjahr: 2024

Beschreibung

With an increased focus on observability and reliability, establishing a scalable and reliable monitoring environment is more important than ever. Over the last decade, Prometheus has emerged as the leading open-source, time-series based monitoring software catering to this demand. This book is your guide to scaling, operating, and extending Prometheus from small on-premises workloads to multi-cloud globally distributed workloads and everything in between.
Starting with an introduction to Prometheus and its role in observability, the book provides a walkthrough of its deployment. You’ll explore Prometheus’s query language and TSDB data model, followed by dynamic service discovery for monitoring targets and refining alerting through custom templates and formatting. The book then demonstrates horizontal scaling of Prometheus via sharding and federation, while equipping you with debugging techniques and strategies to fine-tune data ingestion. Advancing through the chapters, you’ll manage Prometheus at scale through CI validations and templating with Jsonnet, and integrate Prometheus with other projects such as OpenTelemetry, Thanos, VictoriaMetrics, and Mimir.
By the end of this book, you’ll have practical knowledge of Prometheus and its ecosystem, which will help you discern when, why, and how to scale it to meet your ever-growing needs.

Leseprobe

Mastering Prometheus

Gain expert tips to monitoring your infrastructure, applications, and services

William Hegedus

Mastering Prometheus

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Preet Ahuja

Publishing Product Manager: Suwarna Rajput

Book Project Manager: Ashwin Kharwa

Senior Editor: Sayali Pingale

Technical Editor: Arjun Varma

Copy Editor: Safis Editing

Proofreader: Safis Editing

Indexer: Tejal Daruwale Soni

Production Designer: Nilesh Mohite

DevRel Marketing Coordinator: Rohan Dobhal

First published: May 2024

Production reference: 1220324

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK

ISBN 978-1-80512-566-2

www.packtpub.com

To Curtis John, for getting me started in my career, supporting me professionally and personally, and being willing to let me boss him around at multiple companies. To my wife, Kaylah, for her sacrifices, encouragement, love, and grace throughout the process of writing this book and all my life. Finally, to my sons, Luke, Joel, Zeppelin, and Chara, for always keeping my priorities straight.

– William Hegedus

Contributors

About the author

William Hegedus has worked in tech for over a decade in a variety of roles, culminating in site reliability engineering. He developed a keen interest in Prometheus and observability technologies during his time managing a 24/7 NOC environment and eventually became the first SRE at Linode, one of the foremost independent cloud providers.

Linode was acquired by Akamai Technologies in 2022, and now Will manages a team of SREs focused on building the internal observability platform for Akamai’s Connected Cloud. His team is responsible for a global fleet of Prometheus servers spanning over two dozen data centers and ingesting millions of data points every second, in addition to operating a suite of other observability tools.

Will is an open source advocate and contributor who has contributed code to Prometheus, Thanos, and many other CNCF projects related to Kubernetes and observability. He lives in central Virginia with his wonderful wife, four kids, three cats, two dogs, and a bearded dragon.

I want to thank my wife and kids for their support and understanding through the late nights and busy weekends of writing this book. Additionally, I want to thank the Prometheus community of maintainers and contributors who are so welcoming and helpful – with special thanks to Ben Kochie and Bartek Plotka, who were so influential in my early learning of Prometheus and Thanos. Finally, I want to thank my team at Akamai for being so supportive and encouraging and being my inspiration for writing this book.

About the reviewer

TJ Hoplock is an experienced SRE who specializes in observability. He was an early adopter of Prometheus while working for cloud hosting company Linode (now Akamai Connected Cloud) and a founding member of Linode’s Observability team. He is an active contributor to Prometheus (contributing to Linode service discovery, among other improvements), Alertmanager, and other open source tools. He was previously a technical reviewer for Prometheus: Up and Running, 2nd Edition by Julien Pivotto. He is currently working as a senior SRE at NS1 (an IBM company), playing a leading role working with tools such as OpenTelemetry, Prometheus, Thanos, Loki, Honeycomb, and Orb to improve NS1’s observability posture and improve services for users.

Table of Contents

Preface

Part 1: Fundamentals of Prometheus

1

Observability, Monitoring, and Prometheus

A brief history of monitoring

Nagios

A word on SNMP

Enter the cloud

Introduction to observability concepts

Metrics

Logs

Traces

Other signals

Tying signals together

Getting data out of systems

Prometheus’s role in observability

Alerting

Dashboarding

What Prometheus is not

Summary

Further reading

2

Deploying Prometheus

Technical requirements

Components of a Prometheus stack

Prometheus

Alertmanager

Node Exporter

Grafana

Provisioning Kubernetes

Configuring the linode-cli

Creating a Kubernetes cluster

Deploying Prometheus

Prometheus Operator overview

Deploying kube-prometheus

Summary

Further reading

3

The Prometheus Data Model and PromQL

Technical requirements

Prometheus’s data model

Prometheus’ TSDB

Head block

WAL

Blocks and chunks

Index

Compaction

PromQL basics

Syntax overview

Query operators

Query functions

Summary

Further reading

4

Using Service Discovery

Technical requirements

Service discovery overview

Using service discovery

Relabeling

Using service discovery in a cloud provider

Linode service discovery

Custom service discovery endpoints with HTTP SD

Summary

Further reading

5

Effective Alerting with Prometheus

Technical requirements

Alertmanager configuration and routing

Routing

Receivers

Inhibitions

Validating

Alertmanager templating

Configuring templates

Defining your own templates

Highly available (HA) alerting

Cluster sizing

Making robust alerts

Use logical/set binary operators

Use appropriate “for” durations

Use _over_time functions

Anomaly detection

Unit-testing alerting rules

Summary

Further reading

Part 2: Scaling Prometheus

6

Advancing Prometheus: Sharding, Federation, and High Availability

Technical requirements

Prometheus’ limitations

Cardinality

Long-term storage

Sharding Prometheus

Sharding by service

Sharding with relabeling

Federating Prometheus

Achieving high availability (HA) in Prometheus

HA via the Prometheus Operator

Cleanup

Summary

Further reading

7

Optimizing and Debugging Prometheus

Technical requirements

Controlling cardinality

Identifying cardinality issues

Remediating cardinality issues

Using limits

Recording rules

Recording rule conventions

Scrape jitter

Using pprof

Using promtool for pprof data

Query logging and limits

Query logging

Query limits

Tuning garbage collection

Using GOMEMLIMIT

Summary

Further reading

8

Enabling Systems Monitoring with the Node Exporter

Technical requirements

Node Exporter overview

What is in an exporter?

Default collectors

conntrack

cpu

diskstats

filesystem

loadavg

meminfo

netdev

pressure

Others

The textfile collector

Troubleshooting the Node Exporter

Summary

Further reading

Part 3: Extending Prometheus

9

Utilizing Remote Storage Systems with Prometheus

Technical requirements

Understanding remote write and remote read

Remote read

Remote write

Using VictoriaMetrics

Deployment methods

Deploying to Kubernetes

Using Grafana Mimir

Comparing to VictoriaMetrics

Deploying to Kubernetes

Summary

Further reading

10

Extending Prometheus Globally with Thanos

Technical requirements

Overview of Thanos

Why use Thanos?

Thanos Sidecar

Deploying Thanos Sidecar

Thanos Compactor

Vertical compaction

Downsampling

Deploying Thanos Compactor

Thanos Query

Deploying Thanos Query

Scaling Thanos Query

Thanos Query Frontend

Query sharding and splitting

Caching

Deploying Thanos Query Frontend

Thanos Store

Deploying Thanos Store

Scaling Thanos Store

Thanos Ruler

Stateless mode

Deploying Thanos Ruler

Thanos Receiver

Deploying Thanos Receiver

Thanos tools

Cleanup

Summary

Further reading

11

Jsonnet and Monitoring Mixins

Technical requirements

Overview of Jsonnet

Syntax

Using Jsonnet

Generating files

Formatting and linting

Monitoring Mixins

Mixin structure

Using and extending mixins

Summary

Further reading

12

Utilizing Continuous Integration (CI) Pipelines with Prometheus

Technical requirements

GitHub Actions

Validation in CI

Using promtool

Using amtool

Linting Prometheus rules with Pint

Configuring Pint

Integrating Pint with CI

Summary

Further reading

13

Defining and Alerting on SLOs

Technical requirements

Understanding SLIs, SLOs, and SLAs

Why SLOs matter

Types of SLOs

Defining SLOs with Prometheus data

Window-based SLOs

Alerting on SLOs

Using Sloth and Pyrra for SLOs

Sloth

Pyrra

Summary

Further reading

14

Integrating Prometheus with OpenTelemetry

Technical requirements

Introducing OpenTelemetry

OTel specification

OpenTelemetry line protocol

OpenTelemetry collector

Collecting Prometheus metrics with the OpenTelemetry collector

Sending metrics to Prometheus with the OpenTelemetry collector

Configuring Prometheus

Configuring OpenTelemetry collector

Summary

Further reading

15

Beyond Prometheus

Technical requirements

Extending observability past Prometheus

Logs

Traces

Connecting the dots across observability systems

Logging with Loki

Tracing with Tempo

Summary

Further reading

Index

Other Books You May Enjoy

Part 1: Fundamentals of Prometheus

In this part, we’ll get acquainted with observability concepts and understand the role that Prometheus serves in adding observability to your systems and services. Then, we’ll dive into Prometheus itself by deploying it and going through a detailed examination of its data model and query language. Finally, we’ll see how to leverage built-in Prometheus service discovery features to make the configuration of scrape targets dynamic, and how to make the most of its alerting features.

This part has the following chapters:

Chapter 1, Observability, Monitoring, and PrometheusChapter 2, Deploying PrometheusChapter 3, The Prometheus Data Model and PromQLChapter 4, Using Service DiscoveryChapter 5, Effective Alerting with Prometheus

1

Observability, Monitoring, and Prometheus

Observability and monitoring are two words that are often used synonymously but carry important distinctions. While this book is not focused on academic definitions and theories surrounding observability, it’s still useful to distinguish between observability and monitoring because it will provide you with a framework to get in the right mindset when thinking about how Prometheus works and what problems it solves. A screw and a nail can both hang a picture, and you can bang a screw into a wall with a hammer, but that doesn’t make it the best tool for the job. Likewise, with Prometheus, I’ve seen many people fall into the trap of trying to use Prometheus to cover all of their observability and monitoring needs – when you have a hammer, everything looks like a nail. Instead, let’s identify where Prometheus shines so that we can use it to its full effect throughout the rest of this book.

In this chapter, we’re going to cover the following main topics:

A brief history of monitoringIntroduction to observability conceptsPrometheus’s role in observability

Let’s get started!

A brief history of monitoring

In the beginning, there was Nagios… or, at least, so the story goes. Monitoring as we know it took off in the late 1990s and early 2000s with the introduction of tools such as Nagios, Cacti, and Zabbix. Sure, some things existed before that that focused on network monitoring such as Multi Router Traffic Grapher (MRTG) and its offshoot, rrdtool, but system monitoring – including servers – found its stride with Nagios. And it was good… for a time.

Nagios (and its ilk) served its purpose and – if your experience is anything like mine – it just won’t seem to go away. That’s because it does a simple job, and it does it fairly well. Let’s look a little closer at it, the philosophy it embodies, and where it differs from Prometheus.

Nagios

Early monitoring tools such as Nagios were check-based. You give it a script to run with some basic logic and it tells you whether things are good, bad, or really bad based on its exit code. This philosophy defined early monitoring – there wasn’t much nuance to it. Even more so, there wasn’t much detailto it.

In the early days of monitoring as a discipline, systems administrators were primarily focused on known unknowns. In other words, you’d set up checks for how you’d expect your system to break. High memory utilization? Got a check for that. CPU usage high? Got a check for that. Process stopped running? You know I’ve got a check for that.

This has obvious limitations: namely, you need to know in advance what ways your systems can and will break. It was primarily focused on cause-based alerting, in which you set up an alert for something that you know will cause a problem (get it?). However, Nagios and monitoring tools like it also differed in another key way: they assumed a fairly static set of monitoring targets.

Before the cloud, before virtualization took off, there was cold, hard bare-metal. Well, probably not cold if it’s doing any work, but you get the picture. Buying and provisioning servers was a far cry from the modern cloud-based experience we enjoy now, in which you can have a server running the latest release of Ubuntu halfway across the world created and booted in less than a minute. Consequently, the monitoring tools of the time didn’t have to worry about constantly changing, dynamic environments. Nor did they have to account for servers running across the globe. Adding or removing monitoring targets and checks was pretty infrequent, and you didn’t need a system that could do fancy stuff such as continuous, dynamic service discovery. You could tolerate a blip in monitoring while Nagios fully restarted to pick up a new configuration (something it still must do to this day).

Nevertheless, Nagios and other tools didn’t need that functionality. They weren’t pain points the way they are today. Nagios didn’t make a system “observable” – it just monitored if it was healthy or broken, and that was fine. For a time.

A word on SNMP

While Nagios was the first big tool to focus on systems monitoring, network monitoring tools existed for at least a decade before it. The first RFCs for defining Simple Network Management Protocol (SNMP) were submitted in 1988, and it is still in use widely today.

Originally, SNMP was designed primarily as a protocol to manage network devices rather than to monitor them. These days, though, SNMP is so synonymous with network monitoring that many people assume that the M in SNMP stands for monitoring. It can be used for things such as monitoring how many packets a particular network interface has or the chassis temperature of a switch. It’s the default way to monitor most networking equipment on the market and Prometheus is not a replacement for it. One of the most popular Prometheus exporters that is maintained by the core Prometheus team is the SNMP exporter.

Enter the cloud

While Nagios and similar tools were in their heyday in the 2000s, a disruptive new landscape was forming that we’d come to call cloud computing. Multiple advances in virtualization technologies such as Xen and KVM helped to enable the cloud as we know it today. At this point, it was possible and accessible to run virtual servers across the globe, which meant no more dealing with lead times to order and receive physical hardware and no more being physically restricted to nearby co-located data centers/the office basement/a spare closet. This marked a major challenge to monitoring as it had thus far been done.

As we previously covered, Nagios – and the tools developed around the same time – was (and is) well suited to static environments. However, with the cloud coming into its own, engineers started to realize that it took as long (or longer) to restart Nagios to pick up a new config as it did to create a new virtual machine (VM) from scratch. This wasn’t scalable.

Additionally, as systems became more complex – monoliths were broken into microservices, servers were created and destroyed as load ebbed and flowed – the need to have better observability into how these distributed systems were performing increased. No longer could a simple HTTP check suffice to know if the user experience of your site was good – the site was no longer just a single, carefully managed server. We needed to know things such as what our error rate looked like across all of our backends. Once you have that, you want to know if it’s gone up or down since the latest release was deployed. You know how it goes: if you give a mouse a cookie, they’re going to ask for a glass of milk. Simple monitoring couldn’t cut it anymore in these changing conditions – we needed observability.

Introduction to observability concepts

Observability both as a word and as a discipline is not unique to technology. The term is derived from control theory, which is traditionally more rooted in physical engineering disciplines such as robotics and nuclear engineering. It is, in essence, the ability to surmise the health of a system by observing its inputs and outputs. In nuclear engineering, you put in uranium and water, and you receive heat and steam. In software engineering, you put in an end user and an API call, and you receive a Jira ticket about how your API isn’t working. Err… well, hopefully not if your observability is doing its job.

Observability in systems engineering and software is primarily informed by and achieved with a handful of important telemetry signal types. You may have heard them referred to as “the three pillars of observability,” but that terminology has since fallen out of fashion as it elevates the act of gathering telemetry itself above the result. Make no mistake, you can have all three pillars without having an observable system. So, what are these pillars anyway? Well, they’re metrics, logs, and traces.

Other telemetry types

This does not preclude other forms of telemetry, though! For example, since 2020, there has been an explosion of exciting new technologies around continuous profiling through companies such as Polar Signals and Pyroscope (since acquired by Grafana Labs). Additionally, there is a line of thinking championed by observability sages such as Charity Majors, who argue that events are the foundational building blocks of observability.

Metrics

Metrics are perhaps the most fundamental type of telemetry. They are what most organizations start with in their observability journey. The reason is two-fold: they are cheap (in Prometheus, each data point for a metric takes up less than 2 bytes on average) and they are simple. Additionally, metrics are core to the time series-based alerting we mentioned earlier. So, what is a metric?

A metric is a numeric, time-based representation of one or more scoped data points. For example, you may have a metric on memory usage and watch how that changes over time, or a metric on error rates returned by an application. Metrics don’t have to be time series (though they often are), but they are always related to time since they have a relationship with the time in which they were observed/recorded.

They can also be viewed as statistics. In the business world, key performance indicators (KPIs) are also measured via metrics. A lot of advancements in the SRE space at the time of writing have to do with treating time series metrics as statistics and applying concepts from the mathematical branch of statistics to them to analyze distributions and compute z-scores (a novel way to detect anomalies by scoring how much of an outlier a data point is).

Concerning Prometheus, metrics are the primary observability signal that it is concerned with. Prometheus is purpose-built as a time series database for storing metric data efficiently.

Time series

If a time series is different from a metric, then how can you tell the difference between the two? The simplest explanation involves an example.

Recall when I mentioned tracking memory usage as a metric. Well, this is how that looks in Prometheus with a metric coming from the node_exporter process:

node_memory_MemFree_bytes

Each metric can have one or more time series associated with it. Each time series is an instance of a metric. If you’re familiar with object-oriented programming, you can think of a metric as a base class that multiple time series inherit from. Individual time series distinguish themselves through different metadata (that is, “labels”) attached to them. Consequently, individual time series in Prometheus look like this:

node_memory_MemFree_bytes{instance="server1.example.com"} node_memory_MemFree_bytes{instance="server2.example.com"}

Notice how the metric name is still the same, but the labels (in the curly braces) differ. For every metric, you can have tens, hundreds, or even thousands of individual time series for it (although this relates to cardinality, which we will discuss later).

Metrics, however, are limited in their amount of detail. By nature, many are aggregations. For example, counting the number of requests your API receives in total – not what each request looked like in terms of user-agent header, IP address, and so on. Consequently, you trade some level of detail in exchange for greater efficiency in querying and storage. However, the next two telemetry signals we’ll discuss regain some of that lost detail and are also critical to an observable system.

Logs

Logs are the OG telemetry signal. Before terminal screens existed, computers were printing log lines. They are the easiest to implement in a vacuum and are often quick-and-dirty (anyone else fall back on using printf to debug an issue instead of an actual debugger?). However, they are also inherently unstructured. Logs are typically meant to be read by humans rather than machines. Humans don’t want to read flattened JSON; we’re much more likely to add a long line that looks like, HEY!!! LOOK AT ME!! ERROR: <stacktrace>. Consequently, logs can be the most difficult signal to extract meaningful value from.

Logs are simply any bit of text emitted by a system designed to produce a record of something occurring. There are some common types of logs, such as access logs from Apache or Nginx, that have a consistent format. However, most logging is arbitrarily formatted. To maximize the usefulness of logs for observability, logs ought to be structured, which is a fancy way of saying that they should conform to a predictable format. To many people, that means writing out logs in JSON. To folks in the Go community, it’s often logfmt (what Prometheus uses). Regardless, the important piece is that structured formats provide a way to predictably extract key/value (K/V) pairs from log lines. These K/V pairs can then be used as a dimension upon which aggregations can be performed to identify trends and anomalies.

For comparison’s sake, here’s an example of a structured log from Prometheus using logfmt:

ts=2023-06-14T03:00:06.215Z caller=head.go:1191 level=info component=tsdb msg="WAL checkpoint complete" first=1059 last=1060 duration=1.949831517s

In a sense, logs are the closest thing that most companies have to what is referred to by some in the observability space as “events.” Events should be “arbitrarily wide,” as the aforementioned Charity Majors likes to say, which is to say that they should be able to have as many or as few K/V pairs as necessary without it mattering much. Structured logs fit that bill.

Traces

Traces are often the last observability signal to be adopted by organizations. It’s the least distorted signal, but also the most cumbersome to implement. An application trace can be thought of as a sort of custom stack trace that only tracks functions or steps in your code path that you care about. Each trace is made up of one or more spans, which represent individual units of work (often functions):

Figure 1.1 – An example of a trace with multiple spans being visualized in the open source Jaeger project (https://github.com/jaegertracing/jaeger)

A special but common type of trace is a distributed trace, which is a trace that encompasses multiple systems as a request travels to those distinct systems all while maintaining the same trace ID. It is an SRE’s dream – to be able to follow a request from the moment it hits your public load balancer to the time it hits your database from the 12th microservice it called.

However, all this wonderful detail is not without its drawbacks. Tracing is expensive. There is overhead from generating the trace, the cost to store and retrieve this detailed data, and the cost of running the tracing infrastructure itself (which is often far more involved than running something such as Prometheus). Consequently, tracing data is often sampled to reduce those costs. Depending on the workload and its expected requests per second, I’ve generally seen 1/100 as the sample rate for many production applications. This means that for every 100 requests, only 1 trace is stored. This can be problematic in theory since you may be missing 99 failed requests and only seeing the 1 successful request, or vice versa. Nevertheless, it’s typically sufficient to provide insight.

Prometheus and tracing

A neat, relatively new feature of Prometheus called exemplars allows Prometheus to integrate with tracing tools by attaching a trace ID to time series data points. This enables users to quickly jump to a trace from around the same time as a spike in a graph of your Prometheus data, which can be time-saving and is another way to illustrate the symbiotic connection between the various observability signals.

Other signals

Metrics, logs, and traces are not the be-all-and-end-all of observability – their mere presence does not make a system observable. Additionally, even more observability signals are emerging to continue to build a more complete picture of systems and thereby make them more observable. The two with the most traction behind them are events and profiles.

Events

We’ve touched briefly on events already, but to recap, events are point-in-time records of an action occurring. These could be logs (see the article on canonical log lines in the Further reading section), but they could also be purpose-built systems or even a database table or two. An event, however, can also be viewed as an all-encompassing signal that includes metrics, logs, and traces. An event may include a server’s memory and CPU utilization at the time of its generation (metrics), a tack on a field containing logs generated through the duration of the request (logs), and even information about a unique request ID/trace ID.

At the time of writing, there are no great open source observability tools for events (that I’m aware of), but plenty of SaaS vendors such as Honeycomb and New Relic are leading the charge in both defining and supporting this data type.

Profiling

Profiling has been around for a while, but only recently has it become a more prevalent observability signal in the form of continuous profiling. Profiling has long been a developer tool for analyzing application and system performance (for example, kprobes have been present in the mainline Linux kernel since as far back as 2005). However, it was such an intrusive, interruptive process that it was only used as needed. Now, almost two decades later, advances in computing have made it possible to profile a system continuously instead of just as needed.

This profiling data often looks like a trace, which makes sense – profiling involves running stack traces, and application traces are similar to stack traces. They are visualized in the same way (flame graphs) and stored similarly.

Unlike events, several open source continuous profiling tools exist, including Parca, from Polar Signals, a company started by a long-time core contributor to Prometheus, Thanos, and many other Prometheus-related projects.

Tying signals together

An observable system cannot be observable from only one point of view. A computer monitor that is only viewable from directly in front of it at a 90-degree angle to the screen would not sell well, and similarly, your systems should be observable from multiple angles. Going all in on logs without metrics or traces leaves gaps and inefficiencies in your observability. The same logic goes for any one signal. Instead, they all build off each other.

I like to think of the relationship between observability signals as different levels of drilling down into a problem. I start with metrics – they show trends and alert me when something is out of the ordinary. Then, I’ll look at the logs for a system. Ideally, the logs have a unique request ID attached to them, which I can then use to jump to look at the trace for an individual request and see where the issue occurred. From there, you can use the more detailed trace data to try to ascertain trends, such as “all failed requests went through this load balancer.” Each signal enables you to dig deeper into the next and in a more targeted manner.

Getting data out of systems

One of the core features that distinguishes Prometheus from Graphite, Nagios (when using passive checks), and most other monitoring systems is that it uses a pull-based model for extracting data from the systems it monitors. This contrasts with other systems that use a push-based model, in which you push metrics to a destination.

Prometheus constantly pulls metrics from targets, which offers several advantages. A major advantage is that Prometheus knows what it is supposed to be monitoring and can easily alert you to any issues it encounters with monitoring those things. This makes silent failures far less likely to occur compared to a push-based monitoring system where you not only need to track what the values of metrics are but also how recently they were been updated.

We’ve mentioned how Prometheus focuses on metrics, but where does Prometheus itself fit into observability as a whole? Let’s take a look.

Prometheus’s role in observability

Prometheus is objectively pretty great at what it does, but can we make a system fully observable with just Prometheus? Unfortunately, the answer to that question is no. Prometheus’s strength is also its weakness – it’s singularly focused on one thing: metrics.

Prometheus is only focused on the metrics aspect of an observable system. It is purpose-built to efficiently store numeric time series of varying types in a simple format. To the extent to which it interoperates with other observability signals, it is only to provide a link or bridge to some other purpose-built system.

However, Prometheus also provides some of the highest-value data that you can collect. It’s likely your go-to data source in tools such as Grafana to visualize how your systems are performing. Logging and tracing systems can certainly provide more detailed data, but to get them to provide that same level of value in analyzing trends, you need to strip away detail to aggregate data. Consequently, Prometheus is often going to be your best bet for alerting and dashboarding.

Alerting

Prometheus shines in the area of alerting. Alerting has to do more specifically with monitoring versus observability, but monitoring is inextricably tied to observability – the more observable your system is, the better the monitoring can be.

Monitoring can be thought of as the discipline of proactively identifying and alerting on the breaching of predetermined thresholds. Prometheus, by nature of being a time series database, excels in being able to alert on trends over time. When comparing Prometheus to Nagios, often, the first contrast I point to is just how much better Prometheus is at alerting.

It’s not some complex feature that makes Prometheus better than check-based monitoring and alerting software such as Nagios. It’s fairly simple. To me, Prometheus’s killer feature in alerting is just the ability to specify a for duration in alerts. For an alert to go from a pending to a firing state, it needs to meet the alerting criteria consistently across the length of a for duration.

While Nagios and other monitoring tools like it have the idea of hard and soft states and check retries, you can still get unlucky with check timing. Maybe Nagios just happens to check CPU usage every 5 minutes when a bursty cron job runs. With Prometheus, you’re almost certainly getting those metrics way more frequently than every 5 minutes, which means that you’re getting more comprehensive data more frequently, which leads to a more accurate picture of how a system is performing. All this leads to more reliable alerting from Prometheus with less “flappy” and noisy alerts, leading to less alert fatigue and better reliability.

Dashboarding

Dashboarding is arguably what people think of first when they think of observability. “An observable system is one where I can see a bunch of lines that go up and down.” There’s plenty of discourse on the value of dashboards overall, and certainly jumping from dashboard to dashboard to try to find the source of a problem is not ideal observability. Nevertheless, dashboards are the darling child of executives, SREs, and developers alike. They’re simple, intuitive, and easy to set up once you have the data. And where does that data come from? Why, Prometheus of course!

Prometheus is the most used data source plugin for the most popular open source dashboarding tool out there – Grafana. Because Prometheus is generally more high-level than logs or traces – since it deals with data in aggregate rather than per request – it can be the most useful data source to produce graphs and use in dashboards.

What Prometheus is not

We’ve covered how Prometheus directly relates to the signal type of metrics, but it’s worth mentioning a few specific things that Prometheus is explicitly not:

Prometheus is not an all-in-one monitoring solutionPrometheus is not something you configure once and forget aboutPrometheus is not going to automatically make your systems more observable

In fact, in terms of what telemetry signal provides the most detailed view of your system, metrics are the least specific. So, is it even worth running Prometheus? (Yes.) Should I just go all in tracing and forget the rest? (No.) Metrics – and, by extension, Prometheus – are pivotal to monitoring and can still contribute to making a system more observable (especially with custom instrumentation)! Throughout the rest of this book, we’ll look at how to use Prometheus to its maximum effect to make all your systems more observable.

Summary

In this chapter, we learned the abridged history of monitoring and observability, what observability is, and how Prometheus contributes to observability through metrics. With this new (or refreshed) frame of mind, we can approach our utilization of Prometheus in a way that maximizes its usefulness without trying to use it as a silver bullet to solve all our problems.

In the next chapter, we’ll cover deploying a Prometheus environment that we’ll use as the foundation that we build upon throughout the remainder of this book.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

Fast and flexible observability with canonical log lines: https://stripe.com/blog/canonical-log-lines

2

Deploying Prometheus

Now that we understand a little more about observability and monitoring, as well as their distinctions and how Prometheus fits into it all, we can deploy Prometheus. In this book, we’ll be deploying Prometheus to a managed Kubernetes environment using Linode Kubernetes Engine (LKE). However, you can substitute AWS’s EKS, Azure’s AKS, or any other engine as you see fit.

You may already have experience with deploying Prometheus, but we’re going to use the Prometheus environment that we’ll deploy in this chapter as the basis for future chapters in which we’ll be deploying additional applications that extend Prometheus and make configuration changes to Prometheus. Consequently, it is recommended that you follow along with the deployment, even if you have prior experience.

In this chapter, we will cover the following topics:

Components of a Prometheus stackProvisioning KubernetesDeploying the Prometheus Operator

Let’s get started!

Technical requirements

You’ll need to install the following programs and make them available in your PATH:

linode-cli: https://github.com/linode/linode-clikubectl: https://kubernetes.io/docs/tasks/tools/#kubectlhelm: https://helm.sh/docs/intro/install/

This chapter’s code is available at https://github.com/PacktPublishing/Mastering-Prometheus.

Components of a Prometheus stack

When discussing Prometheus, people rarely mean just the Prometheus project (https://github.com/prometheus/prometheus) itself. Instead, they refer to the following:

Prometheus as a full solution in which you deploy Prometheus for metrics collection, storage, and alerting rule evaluationPrometheus’s Alertmanager (https://github.com/prometheus/alertmanager) for routing alerts coming from PrometheusOne or more exporters, such as Node Exporter (https://github.com/prometheus/node_exporter), to expose metricsGrafana (https://github.com/grafana/grafana) to visualize metrics from Prometheus via dashboards

Each of these components fits together to form a complete metrics solution for observability.

Prometheus

Understandably, Prometheus is the core technology in a Prometheus stack. Prometheus is comprised of four main parts: the time series database (TSDB), the scrape manager, the rule manager, and the web UI (along with its REST API). There are – of course – additional components, as shown in the following figure, but these are the main ones you need to be concerned with:

Figure 2.1 – Prometheus’s internal architecture (source: https://github.com/prometheus/prometheus/blob/86a7064/documentation/internal_architecture.md)

Let’s quickly go through the main components.

TSDB

The defining feature of Prometheus is its TSDB. A TSDB is a special kind of database that is optimized for storing data points chronologically. The rest of Prometheus revolves around this TSDB – the scrape manager puts things in it, the rule manager evaluates its contents, and the web UI allows for it to be queried ad hoc style. We’ll dive more into the data model of the TSDB in the next chapter, so for now, all you need to know is that it’s where Prometheus stores all of its data.

Scrape manager

The scrape manager is part of Prometheus that handles pulling metrics from applications and exporters both by actually performing the scraping and maintaining an internal list of what things should be scraped by Prometheus. A “scrape” is just a Prometheus term for collecting metrics. It is akin to the concept of web scraping, similar to what is done by search engines, in which data is extracted from a website in an automated way. Since Prometheus pulls metrics over HTTP (as opposed to some special protocol format or gRPC), the core concept is the same.

Rule manager

The rule manager is the piece of Prometheus that handles evaluating alerting and recording rules in Prometheus. We’ll dig into recording and alerting rules later in this book, but you can think of recording rules as queries that produce new metrics (generally via aggregation) on a defined interval. On the other hand, alerting rules are queries that have a threshold attached to them to ensure that they only return data if something is wrong and an alert needs to be sent.

The rule manager handles evaluating rule groups comprised of alerting and/or recording rules on regular intervals (specified per rule group and/or via the global default). Additionally, it is responsible for passing along alerts that need to be sent to Alertmanager.

Web UI/API

The web UI is the portion of Prometheus that you can access via your browser. It provides helpful information for running Prometheus, such as its running configuration (including command-line flag values), TSDB information, and a simple query frontend that allows you to run ad hoc Prometheus Query Language (PromQL) queries to investigate your available metrics.

The Prometheus web UI is powered by the Prometheus REST API, which provides a standardized way for other programs – such as Grafana – to interact with Prometheus by querying metrics, displaying monitoring targets, and more.

Alertmanager

Alertmanager is an often overlooked piece of a Prometheus stack since those unfamiliar with Prometheus presume that Prometheus itself is responsible for sending alerts to Slack, PagerDuty, and other alerting destinations. In reality, Alertmanager handles all of that on its own through using a routing tree-based workflow. Prometheus itself is only responsible for evaluating alerting rules and sending alert payloads to Alertmanager, which then sends the alerts to their actual destinations. Consequently, Alertmanager is required if you want to enable alerting functionality in Prometheus.

Node Exporter

Node Exporter is the most prevalent Prometheus exporter. It covers (almost) anything you could need in terms of system metrics. For example, it covers metrics on CPU, RAM, disk, and network I/O in addition to a multitude of other metrics. Effectively, almost everything in the /proc pseudo-filesystem on Linux is exposed via Node Exporter (or can be via a PR to the project).

Node Exporter on Windows?

Node Exporter is only designed to be deployed on Linux, BSD, and Mac-based operating systems. For a Windows equivalent, please use Windows Exporter (https://github.com/prometheus-community/windows_exporter).

Naturally, this can be a significant amount of data, so many of Node Exporter’s collectors are disabled by default and require a command-line flag to be enabled. The most common collectors you will care about (CPU, disk, and so on) are enabled by default. In my experience, the only additional flags I need to set for Node Exporter are to enable the systemd collector and specify a directory for the textfile collector. For the most up-to-date list of default enabled/disabled collectors, reference the README file in this project’s GitHub repository or – if you have Node Exporter installed – the output of node_exporter --help.

Textfile collector

The textfile collector in Node Exporter is a unique collector that is worth drawing special attention to. A common question surrounding Prometheus is how to monitor short-lived processes and batch jobs. For this, two options exist: the textfile collector and Prometheus Pushgateway (https://github.com/prometheus/pushgateway). Of these, I generally recommend the textfile collector.

The textfile collector is enabled by default in Node Exporter but requires specifying a directory via the --collector.textfile.directory flag. Once a directory has been specified, Node Exporter will automatically read any files with a .prom file extension in that directory and include the metrics within them, alongside the typical metrics it exposes from /proc (presuming they are properly formatted).

Formatting text files for Prometheus

For the latest details on how to properly format text files for Prometheus’ consumption, reference the Prometheus documentation located at https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format.

Using this, you can enable short-lived processes such as cron jobs to write out Prometheus metrics and monitor them. For some examples of writing out metrics for the textfile collector, reference the Prometheus Community project’s repository, which includes helpful scripts: https://github.com/prometheus-community/node-exporter-textfile-collector-scripts.

Grafana

While the Prometheus web UI is great for running ad hoc queries, it has no way of storing commonly used queries, substituting variables to reuse queries, or presenting data without requiring prior knowledge of PromQL. Luckily for you, the Grafana project exists and does all of these things!