Building IoT Visualizations using Grafana E-Book

Rodrigo Juan Hernández

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 287

Veröffentlichungsjahr: 2022

Beschreibung

Grafana is a powerful open source software that helps you to visualize and analyze data gathered from various sources. It allows you to share valuable information through unclouded dashboards, run analytics, and send notifications.
Building IoT Visualizations Using Grafana offers how-to procedures, useful resources, and advice that will help you to implement IoT solutions with confidence. You’ll begin by installing and configuring Grafana according to your needs. Next, you’ll acquire the skills needed to implement your own IoT system using communication brokers, databases, and metric management systems, as well as integrate everything with Grafana. You’ll learn to collect data from IoT devices and store it in databases, as well as discover how to connect databases to Grafana, make queries, and build insightful dashboards. Finally, the book will help you implement analytics for visualizing data, performing automation, and delivering notifications.
By the end of this Grafana book, you’ll be able to build insightful dashboards, perform analytics, and deliver notifications that apply to IoT and IT systems.

Leseprobe

Building IoT Visualizations using Grafana

Power up your IoT projects and monitor with Prometheus, LibreNMS, and Elasticsearch

Rodrigo Juan Hernández

BIRMINGHAM—MUMBAI

Building IoT Visualizations Using Grafana

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Rahul Nair

Publishing Product Manager: Meeta Rajani

Senior Editor: Runcil Rebello

Content Development Editor: Yasir Ali Khan

Technical Editor: Shruthi Shetty

Copy Editor: Safis Editing

Project Coordinator: Ajesh Devavaram

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Shankar Kalbhor

Senior Marketing Coordinator: Sanjana Gupta

Marketing Coordinator: Nimisha Dua

First published: July 2022

Production reference: 1110722

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80323-612-4

www.packt.com

To my loved daughter, Ana Paula.

~ Rodrigo Juan Hernández

Contributors

About the author

Rodrigo Juan Hernández is an electronic engineer who was passionate about IoT even before it existed. He has been working on tech for more than 18 years. He has worked for local government, and has been involved in a wide variety of projects and technologies, including IP networks, IP video surveillance, IT services, energy systems, and many others.

For several years, he has been focusing on the IoT ecosystem. He is currently working as an IoT system consultant to clients around the world. He also produces content online about IoT and related subjects. This content is available on his blog, YouTube channel, and social networks – mainly LinkedIn.

He also writes for companies that need good-quality content about their products and services. His main objective nowadays is to help others to understand and implement IoT solutions.

I want to thank Vivi for her support in this project.

About the reviewer

Atif Ali is a mechatronics engineer specializing in automation, robotics, and IIoT, with a focus on electro-mechanical design and software integration for complex physical systems and processes. He is a passionate open-source advocate, focused on the intersection of hardware and software for perceptive physical world observability.

Atif has been heavily involved in both the industry and academia for the past several years, in sectors ranging from clean energy and web3, to cybersecurity and mining, holding positions at ETH-Zürich, UBC-Vancouver, and Grafana, among others. He also serves as a technical advisor for start-ups in the emerging market, and is on the faculty board for the Global Master’s in IoT at the Zigurat Global Institute of Technology.

Table of Contents

Preface

Part 1: Meeting Grafana

Chapter 1: Getting Started with Grafana

Technical requirements

Managing data

Performing visualizations

Transformations, alerts, and annotations

Installing Grafana on a Raspberry Pi

Installation instructions

Accessing Grafana for the first time

Installing Grafana on Ubuntu Server

Installation instructions

Running Grafana on Ubuntu Server

Installing Grafana with Docker

Deploying Grafana with Docker CLI

Initial configuration of Grafana

Configuration files

Environment variables

Maybe you prefer the cloud option

Summary

Invitation to join us on Discord

Chapter 2: Exploring Grafana

Technical requirements

Exploring the Grafana interface

The main menu

Organizations, dashboards, and panels

Organizations

Dashboards

Panels

How to get data into Grafana

Data sources

Making integrations easy with plugins

Data source plugins

Panel plugins

Users and permissions

Server level

Organization level

Teams

Users

Authentication methods

Grafana authentication

Short-lived tokens

Anonymous authentication

Authentication options

Proxy authentication

JWT authentication

LDAP authentication

GitHub OAuth2 authentication

Summary

Part 2: Collecting Data from IoT Devices

Chapter 3: Connecting IoT Devices

Technical requirements

Sending data from IoT devices

What is MQTT and what is it used for?

Topics in MQTT

QoS in MQTT

MQTT 3.1.1

MQTT 5

MQTT clients

Using an online MQTT broker

Mosquitto

HiveMQ

Running your MQTT broker

Sending data through HTTP

HTTP methods

HTTPS

HTTP use case

What is CoAP?

What about WebSocket?

How to select the right IoT protocol

Summary

Chapter 4: Data Sources for Grafana

What is a data source?

SQL versus NoSQL databases

SQL databases

What are NoSQL databases

What are TSDBs?

Google Sheets and CSV files

Google Sheets data source

Using files as data sources

How to use live data

Features

Live channels

Origin check

Use of resources

Maximum number of connections

Summary

Chapter 5: Using Time Series Databases

Installing and configuring InfluxDB

Installing InfluxDB

Configuring the InfluxDB service

Configuring the InfluxDB database

Creating and managing an InfluxDB database

Main concepts of InfluxDB databases

Creating an InfluxDB bucket

Storing values in the bucket

Reading data from InfluxDB

Installing and configuring OpenTSDB

Installing OpenTSDB

Configuring OpenTSDB

Writing data in an OpenTSDB database

Data schema

Writing data

Adding the data source to Grafana

Summary

Invitation to join us on Discord

Part 3: Connecting Data Sources and Building Dashboards

Chapter 6: Getting Data and Building Dashboards

Getting data from time-series databases

Showing time-series data in graph panels

Panel options

Time-series panel options

Building table data panels

Table panel options

Building and customizing a table with multiple queries

How to show categorical data

Using stat panels

Using bar charts

Building a georeferenced map

The Geomap panel

Histograms and heatmaps

The Histogram panel

The Heatmap panel

Summary

Chapter 7: Managing Plugins

Technical requirements

How to install and manage plugins

Installing a Grafana plugin on your server

Installing plugins in a Grafana Cloud instance

Plugins administration permissions

All the flavors of plugins

Signed versus unsigned plugins

Types of plugins

Summary

Chapter 8: Organizing and Managing Dashboards

Organizing panels and dashboards

Controlling the visualization

Building and showing playlists

Properties of dashboards

General dashboard properties

Annotations and variables

Annotations

Variables

Linking and sharing dashboards

Linking dashboards and panels

Sharing dashboards and panels

Exporting and importing dashboards

Exporting a dashboard

Importing a dashboard

Summary

Part 4: Performing Analytics and Notifications

Chapter 9: Performing Analytics in Grafana

Technical requirements

InfluxDB calculations

InfluxQL functions

Transformations in Grafana

Add field from calculation

Concatenate fields

Convert field type

Filter data by name

Filter data by query

Filter data by value

Group by

Labels to fields

Merge

Organize fields

Join by field (outer transformation)

Reduce

Building advanced plots with Plotly

Summary

Invitation to join us on Discord

Chapter 10: Alerting and Notifications in Grafana

Technical requirements

How Grafana manages alerts

Using Alertmanager in Grafana

Alert rule types

No data and error handling

Building numerical-based alerts

Time-series and tabular data

Notifications in Grafana

Adding a contact point

Connecting alerts and notifications

Grouping notification policies

Adding notification policies

Editing the root policy

Mute timings

Silences

Differences between mute timings and silences

Summary

Part 5: Integrating Grafana with Other Platforms

Chapter 11: Using Grafana with Prometheus

Technical requirements

What is Prometheus?

Architecture

Data model

Metric types

Installing Prometheus

Installing from binaries

Installing from source

Installing Prometheus using Docker

Feeding Prometheus with data

Prometheus clients

Exporters

Pushgateway

Scraping configuration

Integrating Prometheus and Grafana

Editing Prometheus queries in Grafana

Summary

Chapter 12: Using Grafana with OpenSearch

Technical requirements

What OpenSearch is and what it is used for

OpenSearch components

Installing OpenSearch

Disabling security for testing

Ingesting data in OpenSearch

Fluent Bit

Configuring Fluent Bit for receiving MQTT messages

Sending the data to OpenSearch

Integrating OpenSearch and Grafana

Adding the data source

Building dashboards with the OpenSearch data source

Interacting with OpenSearch using OpenSearch Dashboards

Summary

Invitation to join us on Discord

Chapter 13: Showing Data from LibreNMS in Grafana

Technical requirements

What LibreNMS is and how you can use it

SNMP

Configuring LibreNMS to use InfluxDB

Showing LibreNMS data on Grafana dashboards

Showing network topology diagrams with quasi-live data

Building diagrams with draw.io

Using the FlowCharting panel

Summary

Chapter 14: Integrations for Grafana Cloud

Technical requirements

What is the Grafana Agent?

Grafana Agent configuration

Grafana Cloud service integrations

Using the Linux Node integration

Using the Home Assistant integration

Using the RabbitMQ integration

Summary

Other Books You May Enjoy

Preface

In this book, you will learn how to implement an IoT system using open source tools.

The main focus of this book is the use of Grafana as an IoT platform. However, we will also cover many technologies involved in the transmission, storage, management, and visualization of data.

These technologies include IoT communication protocols, time-series databases, and Docker containers.

Also, you will learn how to integrate other systems with Grafana, including Prometheus, OpenSearch, LibreNMS, and Fluent Bit.

After finishing this book, you will have a clear idea about IoT systems, communications, and platforms, and you will be able to test and implement your IoT systems using Grafana according to specific requirements.

Who this book is for

People with a background in IT, technology, and systems who want to learn how to get data from sensors and devices and show it using Grafana, IoT developers who want to build visualizations and analytics for their projects and products, technicians from the embedded world who need to learn how to build systems and platforms using open source software, and technology enthusiasts who want to learn about IoT systems will benefit from this book.

What this book covers

Chapter 1, Getting Started with Grafana, explains what Grafana is and what it is useful for. You will also learn to install it and make basic configurations.

Chapter 2, Exploring Grafana, covers all the sections and capabilities of Grafana. Also, you will learn about user management, including groups, authentication, and permissions.

Chapter 3, Connecting IoT Devices, explains how to connect IoT devices and send data to IoT platforms using different protocols and systems.

Chapter 4, Data Sources for Grafana, explores different types of data sources. These data sources will be used in later chapters to feed data to Grafana.

Chapter 5, Using Time Series Databases, explains how to store data in time-series databases.

Chapter 6, Getting Data and Building Dashboards, discusses how to connect data sources and build visualization dashboards.

Chapter 7, Managing Plugins, covers the installation and management of Grafana plugins.

Chapter 8, Organizing and Managing Dashboards, explains how to manage dashboards, variables, annotations, links, and permissions. You will also learn how to export and share dashboards.

Chapter 9, Performing Analytics in Grafana, covers analytics using data source capabilities and plugins.

Chapter 10, Alerting and Notifications in Grafana, explains how to configure and use alerts and notifications in Grafana.

Chapter 11, Using Grafana with Prometheus, discusses getting data from Prometheus and displaying it in Grafana dashboards.

Chapter 12, Using Grafana with OpenSearch, covers integrating OpenSearch and Grafana.

Chapter 13, Showing Data from LibreNMS in Grafana, explains how to connect LibreNMS and Grafana, use data from LibreNMS, and display it with Grafana.

Chapter 14, Integrations for Grafana Cloud, discusses monitoring Home Assistant, RabbitMQ, and Linux nodes using the Grafana Cloud integrations.

To get the most out of this book

You will need at least basic knowledge of Linux use and administration. You will need access to a Linux server, local or in the cloud. You will get more from this book if you have a Raspberry Pi computer. You will need to have a Docker instance. You will need accounts in cloud services, such as Grafana, InfluxDB, and MQTT brokers.

All the software used in this book is free and open source.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

After reading this book, you will be able to implement your own IoT system using Grafana and many of the tools used in this book.

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/r9a1L.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "The app_mode variable accepts two options: development and production."

A block of code is set as follows:

{ "sensorType": "Thermometer",

"sensorModel": "AM2302",

"temp": 25,

"hum": 40}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

[default]exten => s,1,Dial(Zap/1|30)exten => s,2,Voicemail(u100)exten => s,102,Voicemail(b100)exten => i,1,Voicemail(s0)

Any command-line input or output is written as follows:

$ mkdir css

$ cd css

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "Select System info from the Administration panel."

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Building IoT Visualizations using Grafana, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Part 1: Meeting Grafana

In this part, you will learn what Grafana is and what it is used for. You will learn how to install it and make basic configurations.

This part contains the following chapters:

Chapter 1, Getting Started with GrafanaChapter 2, Exploring Grafana