DevOps for Databases E-Book

David Jambor

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Veröffentlichungsjahr: 2023

Beschreibung

In today's rapidly evolving world of DevOps, traditional silos are a thing of the past. Database administrators are no longer the only experts; site reliability engineers (SREs) and DevOps engineers are database experts as well. This blurring of the lines has led to increased responsibilities, making members of high-performing DevOps teams responsible for end-to-end ownership. This book helps you master DevOps for databases, making it a must-have resource for achieving success in the ever-changing world of DevOps.
You’ll begin by exploring real-world examples of DevOps implementation and its significance in modern data-persistent technologies, before progressing into the various types of database technologies and recognizing their strengths, weaknesses, and commonalities. As you advance, the chapters will teach you about design, implementation, testing, and operations using practical examples, as well as common design patterns, combining them with tooling, technology, and strategies for different types of data-persistent technologies. You’ll also learn how to create complex end-to-end implementation, deployment, and cloud infrastructure strategies defined as code.
By the end of this book, you’ll be equipped with the knowledge and tools to design, build, and operate complex systems efficiently.

Leseprobe

DevOps for Databases

A practical guide to applying DevOps best practices to data-persistent technologies

David Jambor

BIRMINGHAM—MUMBAI

DevOps for Databases

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Preet Ahuja

Publishing Product Manager: Arindam Majumder

Book Project Manager: Ashwin Kharwa

Senior Editor: Athikho Sapuni Rishana

Technical Editor: Rajat Sharma

Copy Editor: Safis Editing

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Prashant Ghare

DevRel Marketing Coordinator: Rohan Dobhal

First published: November 2023

Production reference: 1301123

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83763-730-0

www.packtpub.com

This book is lovingly dedicated to my family, friends, and colleagues, whose unwavering support and wisdom have been the bedrock of this journey. You have transformed my aspirations into reality. To my family, for your endless encouragement; to my friends, for your invaluable companionship; and to my colleagues, for your inspiring camaraderie – this accomplishment is as much yours as it is mine. Thank you for shaping my life and inspiring every page of this book. Without you, this would not have been possible.

Contributors

About the author

David Jambor is a seasoned technology expert with a 16-year career in building, designing, and managing large-scale, mission-critical systems. He has spent a decade honing his expertise in DevOps and data-persisting technologies, and he is widely regarded as an authority in the field. Currently serving as the head of DevOps, data, and analytics at Amazon Web Services in the UK, David brings with him a wealth of experience from previous roles at top-tier companies, such as Vodafone, Sky, Oracle, Symantec, Lufthansa, and IBM.

In addition to his professional achievements, David is a prominent figure in the DevOps community, frequently presenting technical and strategy-focused talks at various international events. He is also a respected judge and advisor for multiple DevOps awards, and he provides valuable support to technology vendors.

About the reviewer

Sandor Szabo has worked for over 16 years in different fields of IT, mainly focusing on supporting and maintaining large-scale systems worldwide. Several high-profile companies such as Lufthansa Systems, AT&T, Sky UK, and Huawei broadened his view on technologies and principles relating to highly available applications and databases, cloud technologies, DevOps, CI/CD, automation, business continuity, and project, team, and product ownership.

He received a BSc in engineering information technology from John Von Neumann Faculty of Informatics at the University of Obuda in Hungary. He is currently employed by Nokia full-time, where he is one of the product owners and security leads of Nokia Continuous Delivery software.

I’d like to thank my family, who understood that the effort and time invested in this project were important to me. I also would like to thank the author of this book, who shared his knowledge and valuable insights, and Packt Publishing, who made it possible for me to participate in such an interesting project! I also learned a lot from this book, and I hope you enjoy the content, just like I did while reviewing it!

Table of Contents

Preface

Part 1: Database DevOps

1

Data at Scale with DevOps

The modern data landscape

How do we generate data today?

Why speed matters

Data management strategies

The early days of DevOps

SRE versus DevOps

Engineering principles

Objectives – SLOs/SLIs

Summary

2

Large-Scale Data-Persistent Systems

A brief history of data

The early days of computing

The rise of relational databases

Conclusion

Database evolution

Hierarchical database models

Network database model

Relational databases

OO databases

NoSQL database paradigms

Data warehouses

Architecture

Data modeling

Integration

Data lakes

Architecture

Data ingestion and processing

Storage and retrieval

Security considerations

Conclusion

Summary

3

DBAs in the World of DevOps

The continuously evolving role of the DBA

The rise of data architecture and integration

The emergence of cloud computing and big data

The shift to DevOps

Conclusion

DevOps and DBAs

The role of the database expert in a DevOps-conscious team

Designing and implementing databases

Ensuring high availability and disaster recovery

Performance tuning

Security and compliance

Automation

A proven methodology with quantifiable benefits

Faster time to market

Improved quality

Reduced downtime

Increased scalability

Improved security

Summary

Part 2: Persisting Data in the Cloud

4

Cloud Migration and Modern Data(base) Evolution

What is cloud migration (and why are companies doing it)?

The importance of cloud migration

Steps to keep in mind before cloud migration

Key milestones through cloud migration

Types of cloud migrations

Lift-and-shift migration

Lift-and-reshape migration

Refactoring migration

Hybrid cloud migration

Multi-cloud migration

The process of cloud migration

Monolithic or distributed database systems?

What can a database expert help with during cloud migration?

Data migration strategies and their types

The Big Bang migration strategy

The phased migration strategy

The parallel migration strategy

The hybrid migration strategy

The reverse migration strategy

The ETL strategy

The replication strategy

Why are data migration strategies important during a database cloud migration project?

Taking your migration journey one step further

Summary

5

RDBMS with DevOps

Embracing DevOps

Provisioning and configuration management

Monitoring and alerting

Backup and disaster recovery

Performance optimization

DevSecOps

Summary

6

Non-Relational DMSs with DevOps

Activities and challenges

Data modeling

Denormalization

Nested and dynamic data

Data denormalization

Schema management

Schemaless data modeling

Dynamic schema evolution

Consistency and concurrency control

Deployment automation

Deployment of multiple database engines

Backup and disaster recovery

Capacity planning and scaling

Performance tuning

Data modeling for performance

Distributed query optimization

Network latency and data transfer

Data consistency

Lack of transactions

Eventual consistency

Data sharding

Security

Limited access control

Distributed denial of service attacks

Lack of encryption

Anti-patterns (what not to do…)

Overusing or misusing denormalization

Ignoring or underestimating data consistency

Failing to secure a database

Overlooking performance tuning

Neglecting to plan for growth

Summary

7

AI, ML, and Big Data

Definitions and applications of AI, ML, and big data

The relationship between AI, ML, and big data

The role of DevOps and engineering in AI, ML, and big data

Challenges of AI, ML, and big data

A deep dive into big data as a DevOps data expert

Big data infrastructure

Challenges with big data

A deep dive into ML as a DevOps data expert

How ML works

How ML differs from traditional software applications

Challenges with ML for a DevOps data expert

A deep dive into AI as a DevOps data- expert

Amazon SageMaker

Google Cloud AI platform

Microsoft Azure Machine Learning

Challenges with AI for a DevOps data expert

Summary

Part 3: The Right Tool for the Job

8

Zero-Touch Operations

Traditional versus zero-touch approaches

Automated configuration management

Automated release management

Automated monitoring and alerting

Increased operational efficiency

Automated database provisioning

Automated backup and recovery

Improved reliability and consistency

Automated configuration management

Automated release management

Accelerated deployment and time-to-market

CI/CD pipelines

IaC and orchestration

Enhanced scalability and elasticity

Automated resource provisioning

Container orchestration

Reduced downtime and faster recovery

Automated monitoring and alerting

Streamlined recovery processes

Improved compliance and security

Automated security configurations

Automated compliance checks

Sanity-checking our approach

Conclusion on ROI

Summary

9

Design and Implementation

Designing data-persistence technologies

Database design principles

RDBMS versus NoSQL versus NewSQL

Implementing data-persistence technologies

Installation, configuration, and management of database systems

Practical example – PostgreSQL database server installation, configuration, and management

Disaster recovery planning

Practical example – MongoDB replication and automatic failover

Database provisioning and Infrastructure as Code

Practical example – using Terraform to script the setup of a SQL Server database

Database version control and CI/CD

Importance of database version control

Practical example – using Liquibase to manage database schema changes

Role of the DevOps DBA in CI/CD pipelines

Practical example – Jenkins pipeline with database migrations using Flyway

Database performance tuning

Importance of performance tuning and common strategies

Practical example – optimizing a poorly performing query in Oracle

Security and compliance

Importance of security in database management

Common security measures

Practical example – best practices for securing a MySQL database and ensuring GDPR compliance

Collaboration and communication

Summary

10

Database Automation

Autonomous database management

Self-driving databases – a new horizon in DBMs

Automation of database administration tasks

The implications of self-driving databases

Challenges and future directions

Conclusion

The revolution of performance tuning – from manual to autonomous

Understanding performance tuning

The need for automated performance tuning

Technological foundations of automated performance tuning

The mechanics of automated performance tuning

The implications of automated performance tuning

Challenges and future directions

Conclusion

Automated data lineage tracking – a new era of transparency in data management

Understanding data lineage

The evolution from manual to automated data lineage tracking

The technological foundations of automated data lineage tracking

The process of automated data lineage tracking

The implications of automated data lineage tracking

Challenges and future directions

Conclusion

Data privacy automation – advancing the frontier of privacy compliance in the digital age

Understanding data privacy

The challenges of data privacy

Data masking and anonymization

The advent of data privacy automation

Technological underpinnings of data privacy automation

The process of data privacy automation

The benefits and implications of data privacy automation

Conclusion

Automated data discovery and cataloging – unveiling the hidden treasures in today’s data landscape

Understanding data discovery and cataloging

The growing need for automation in data discovery and cataloging

What is automated data discovery and cataloging?

The key features of automated data discovery and cataloging tools

The process of automated data discovery and cataloging

The benefits and implications of automated data discovery and cataloging

Conclusion

The ascendancy of DBaaS – transforming business efficiency and data utilization in the digital age

Understanding DBaaS

The mechanics of DBaaS

The impact of DBaaS on business operations

DBaaS – the future of database management

Conclusion

The emergence of serverless databases – revolutionizing DBaaS through on-demand scalability and cost efficiency

Understanding serverless databases

Why serverless databases? The driving forces

How serverless databases work

The impact of serverless databases on business operations

The future of serverless databases

Conclusion

Summary

Part 4: Build and Operate

11

End-to-End Ownership Model – a Theoretical Case Study

End-to-end ownership – a case study

Adoption of the end-to-end ownership model

Setting the stage

Project initiation

Formation of cross-functional teams

Defining end-to-end ownership

Design and development phase

Collaborative design and planning

Agile development practices

CI and continuous testing

Deployment and release

IaC

CD pipelines

Canary releases and feature flags

Monitoring and IM

Proactive monitoring and alerting

IR and post-mortems

Continuous improvement

Feedback and iteration

Gathering user feedback

Prioritizing and implementing changes

A/B testing and experiments

Scaling and challenges

Scaling the end-to-end ownership model

Managing dependencies

Balancing autonomy and alignment

Summary

12

Immutable and Idempotent Logic – A Theoretical Case Study

Introduction to immutable and idempotent logic

Immutable logic in data-persisting technologies

Understanding immutability in the context of data storage

Benefits and use cases of immutable data storage

Examples of immutable data storage approaches

Implementing immutable logic with data-persisting technologies

Idempotent logic in data-persisting technologies

Introduction to idempotent operations and their significance

Examples of idempotent operations in data persistence

Ensuring idempotency in data processing and transformations

Practical examples and use cases

Immutable and idempotent logic in relational databases

Immutable and idempotent approaches in NoSQL databases

Immutable and idempotent patterns in distributed storage systems

Considerations and best practices

Performance and scalability implications of immutable and idempotent approaches

Data consistency and integrity considerations

Handling failures and retries with idempotent logic

Managing data evolution and schema changes with immutability

Future trends and challenges

Emerging technologies and advancements in data persistence

Integrating immutable and idempotent logic with cloud-native architectures

Addressing complexities and trade-offs in large-scale data persistence systems

Summary

13

Operators and Self-Healing Data Persistent Systems

Self-healing systems

Components of a self-healing system

Importance of self-healing systems

Risks and limitations

Technical example of each core principle of a self-healing system

Operators in Kubernetes

Overview of Kubernetes and containerization

Understanding operators

Operator frameworks and ecosystem

Benefits of operators in Kubernetes

Use cases of operators in Kubernetes

Self-healing databases

Traditional database challenges

Self-healing mechanisms in databases

Benefits of self-healing databases

Risks and limitations

Factors influencing self-healing in different databases

Relational databases

NoSQL databases

NewSQL databases

Time-series databases

Self-healing in Kubernetes – implementation and best practices

Key components for self-healing in Kubernetes

Implementing self-healing in Kubernetes – best practices

Challenges and considerations

Conclusion

Case studies – self-healing databases in Kubernetes

Case study 1 – MySQL Operator

Case study 2 – MongoDB Operator

Case study 3 – Cassandra Operator

Benefits of self-healing databases in Kubernetes

Challenges and future directions

Challenges in self-healing systems

Future directions

Summary

14

Bringing Them Together

Alex’s AI journey

Introduction and project assignment

Software and infrastructure architecture decisions

Relational versus non-relational databases

Implementing caching, data lakes, and data warehouses

Security concerns and solutions

First status update

Implementation

Utilizing DevOps and SRE methodologies

The power of immutable and idempotent logic

Embracing zero-touch automation

Update 2

Implementing self-healing systems

Implementing load balancers and scaling

Update 3

Observability and operations

The art of canary deployments

Database scaling

Security and compliance in operations

Update 4

Version-controlled environment variables

Update 5

Lessons learned and future directions

Summary

Part 5: The Future of Data

15

Specializing in Data

Mastering data – bridging the gap between IT and business

Data-driven decision-making for business executives

Building a data-driven culture – an enterprise perspective

My first experience, Unix – 2009

The first signs of DevOps – 2010s

Support and software engineering in 2012

My first SRE team – 2015

Steep learning curves – 2017

Putting it all into practice – 2019

The landscape of 2023 – data and DevOps united

Integration of DevOps with data engineering

DataOps – revolutionizing data analytics pipelines

MLOps – bridging the gap between ML development and operations

AI-powered DevOps/SRE

Application of SRE principles to data systems

DevSecOps – security in the age of data

Summary

16

The Exciting New World of Data

Part 1 – the future of data-persisting technologies

The evolution of current data-persisting technologies

Emerging data-persisting technologies

Future use cases and challenges

Part 2 – anticipated changes in AI/ML DevOps

Advancements in MLOps

Future use cases and challenges

Career impacts and future skill requirements

Part 3 – evolving trends in SRE

Changes in the SRE landscape

The role of SRE in future IT operations

Part 4 – career outlook and emerging skill sets in SRE

SRE careers – opportunities, skills, and preparation

Career opportunities in SRE

Skill set requirements

Preparing for a career in SRE

Innovations in data persistence technologies

The current state of data persistence technologies

Future outlook – next-generation data persistence technologies

Preparing for the future – skills and strategies in data persistence technologies

Part 5 – the future of designing, building, and operating cutting-edge systems

Emerging technologies in system design and development

Potential implications and challenges in designing, building, and operating cutting-edge systems

Strategies for success – preparing for the future of designing, building, and operating cutting-edge systems

Summary

Index

Other Books You May Enjoy

Part 1: Database DevOps

Welcome to our insightful exploration of DevOps in the realm of data management and database technologies. This book is a journey through various facets of DevOps, blending theory with practical applications, and covering topics from the fundamentals of DevOps to advanced discussions on databases, big data, AI, and ML. Each chapter builds upon the last, offering a cohesive narrative for both new entrants and seasoned professionals in the field.

If you are new to the DevOps world, this part of the book will introduce you to the exciting world of DevOps. If you are an experienced database professional who has not yet had a chance to experience DevOps practices at scale, you are in for a great and surprising journey with this part. DevOps plays a core role in modern software development and operational best practices, unifying the building and running of systems with an emphasis on automation and monitoring at all stages. DevOps plays a key role in modern systems design and operation with a heavy focus on end-to-end ownership and autonomous operation. You will be introduced to DevOps fundamentals and best practices.

This part comprises the following chapters:

Chapter 1, Implementing DevOps in Real LifeChapter 2, Large-Scale Data Persistent SystemsChapter 3, DBAs in the World of DevOps