34,79 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Fachliteratur
- Sprache: Englisch
Docker Swarm serves as one of the crucial components of the Docker ecosystem and offers a native solution for you to orchestrate containers. It’s turning out to be one of the preferred choices for Docker clustering thanks to its recent improvements.
This book covers Swarm, Swarm Mode, and SwarmKit. It gives you a guided tour on how Swarm works and how to work with Swarm. It describes how to set up local test installations and then moves to huge distributed infrastructures. You will be shown how Swarm works internally, what’s new in Swarmkit, how to automate big Swarm deployments, and how to configure and operate a Swarm cluster on the public and private cloud.
This book will teach you how to meet the challenge of deploying massive production-ready applications and a huge number of containers on Swarm. You'll also cover advanced topics that include volumes, scheduling, a Libnetwork deep dive, security, and platform scalability.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 240
Veröffentlichungsjahr: 2016
Ähnliche
Table of Contents
Native Docker Clustering with Swarm
Native Docker Clustering with Swarm
Copyright © 2016 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: December 2016
Production reference: 1151216
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78646-975-5
www.packtpub.com
Credits
Authors
Fabrizio Soppelsa
Chanwit Kaewkasi
Copy Editors
Sneha Singh
Laxmi Subramanian
Reviewer
Baohua Yang
Project Coordinator
Kinjal Bari
Commissioning Editor
Wilson D'suoza
Proofreader
Safis Editing
Acquisition Editor
Prachi Bisht
Indexer
Pratik Shirodkar
Content Development Editor
Trusha Shriyan
Graphics
Kirk D'Penha
Technical Editor
Nirant Carvalho
Production Coordinator
Shantanu N. Zagade
About the Authors
Fabrizio Soppelsa works as an Escalations Engineer for Mirantis Inc., the OpenStack company. Docker activist and advocate since Docker 0.3 and author of several articles on the Docker tools in three languages, he is also a concrete contributor to the projects, especially Machine. He currently lives in Moscow, Russia, where he is the Docker Meetup organizer with his spider Mosha.
I would like to thank ClusterHQ folks for their help on Flocker, especially, Ryan Wallner. the Yandex team and Denis Kutin for having given me a free OpenStack lab on which I could easily access. Mirantis for creating the (I think) best OpenStack distribution. The Docker team and the Docker community, for all the fun.
Chanwit Kaewkasi is an Assistant Professor at the School of Computer Engineering, Suranaree University of Technology, Thailand. Chanwit started contributing to the Docker Swarm project since 0.1 where he co-designed and implemented the strategy filters, the ZooKeeper discovery, and other features. He currently serves as Docker Swarm Maintainer and Docker Captain.
I'd like to thank my wife, Pitchaya, for her encouragement and best support to everything I'm working on, including this book.
Special thanks to the Docker Engineering team for their great software, Suranaree University of Technlogy for being my great workplace, and my parents for their support and lastly, Fabrizio for inviting me to co-author this great book.
About the Reviewer
Baohua Yang is a senior researcher at IBM. His interests include key areas in Cloud Computing, Fintech, distributed system and analytics, especially on emerging technologies, e.g., SDN/NFV, Container, BigData, Blockchain and cognitive computing.
As the chief architect, he has lead the architecture design and system implementation of enterprise productions, and helps solve key technical challenges for industrial solutions.
As an Open-Source contributor, he has submitted code, proposals, and presentations to numbers of projects, e.g., OpenStack, Hyperledger, OpenvSwitch, Docker, OpenDaylight, Kubernetes, and has lead several projects including easyOVS, Hyperledger Fabric-SDK-py and Cello. He is now the chairperson of Hyperledger Technical Working Group in China.
He has published 10s of papers in top-quality international conference and journals (e.g., IEEE INFOCOM, IEEE Trans on Computers), with several technical books and patents. He is now TPC member of numbers of academic conferences and journals.
His homepage is at https://yeasy.github.com.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Dedication
I want to dedicate this book to the memory of my father, Donato.
Preface
Welcome to Native Docker clustering with Swarm! This is a book about containers and distributed systems. We’ll show how to use the native Docker tools to model micro services, spawn tasks, scale up the size of your application, and push your containers to the Docker clustering limit! In a word, we’ll discuss Docker orchestration.
With the recent rise of Swarm Mode and the enablement of Swarm inside the Docker Engine itself, it turned out that the best way to orchestrate Docker is… Docker!
Good, but what do we mean by “orchestrate Docker”? What is orchestration? Better, what is an orchestra?
An orchestra is an ensemble of musicians led by a conductor, who dictates tempo, rhythm and shapes the sound. Strings, woodwinds, percussions, keyboards and other instruments follow the conductor’s direction to perform an astonishing symphony, for example Beethoven’s The Ninth.
Similarly, in a containers orchestration system the musicians are tasks, and the conductor is a leader service (Swarm primitives). Tasks don’t play a symphony, or at least not only: More abstractly they execute some computational job, for example they run a webserver. The conductor, Swarm, is responsible for their provisioning, their availability, their linkage, their scaling. This (and more) is what we mean by “Docker orchestration”.
This book shows how to provision such Docker “orchestras”, how to guarantee the availability of the service, how to connect tasks and how to scale the platform, to play the symphony of your application.
What this book covers
Chapter 1, Welcome to Docker Swarm, introduces Swarm, and explains why you need a clustering solution for your containers. It illustrates the Swarm features, giving a high-level description of its architecture. We define some use cases and describe how Swarm is different from Fleet, Kubernetes and Mesos. The chapter proceeds with the Docker tools installation and finally with two Swarms provisionings: A local Swarm Standalone and a remote Swarm Mode cluster on DigitalOcean.
Chapter 2, Discover the Discovery Services, is a descriptive and mostly abstract chapter. We’ll learn what discovery mechanisms and consensus algorithms are, and why they are crucial for distributed systems. We’ll describe in detail Raft and its implementation Etcd, the consensus mechanism included in Swarm Mode. We will also show the limitations of the discovery mechanism used in Chapter 1, Welcome to Docker Swarm, by extending the local tiny example with Consul, re deploying it.
Chapter 3, Meeting Docker Swarm Mode, is about the new Docker kit that allows to create task clusters of any size. We will introduce Swarmit, the foundation of Docker Swarm Mode, showing how it works in Docker 1.12+, discuss its architecture, its concepts, how it’s different from the “old” Swarm, and how it organizes workloads by abstracting services and tasks.
Chapter 4, Creating a Production-Grade Swarm, shows and discusses the community-driven projects Swarm2k and Swarm3k, our 2,300 and 4,800 nodes Swarm clusters experiments, which ran hundreds of thousands of containers. We demonstrate how such huge clusters were planned, provisioned, and summarize the lessons we learned.
Chapter 5, Administer a Swarm Cluster, is a chapter about infrastructure. We will show how to increase or decrease Swarms sizes, how to promote and demote nodes, and how to update clusters and nodes properties. We’ll introduce Shipyard and Portainer.io as graphical UIs for Swarm.
Chapter 6, Deploy Real Applications on Swarm, is where we will put real applications in motion on Swarm and where we add to the discussion some notes about Compose, Docker Stacks and Docker Application Bundles. We will show the typical deployment workflow, how to filter and schedule containers over the cluster, launch them as services, handle containers as tasks. We’ll start defining a web service with Nginx, then we’ll deploy a mandatory Wordpress with MySQL example. We’ll finally move on with a more realistic app: Apache Spark.
Chapter 7, Scale Up Your Platform, will develop new topics from the previous chapter. Here we’ll introduce Flocker to add storage capacity to Spark on Swarm, and we’ll show how to install and use it automatically at a scale in conjunction with Swarm. We’ll refine our Spark example by running some real big data jobs and setting up a basic monitoring system for this infrastructure.
Chapter 8, Exploring Additional Features to Swarm, discusses some advanced topics important to Swarm, such as Libnetwork and Libkv.
Chapter 9, Securing a Swarm Cluster and Docker Software Supply Chain, will focus on security considerations for Swarm clusters. Among the arguments, certificates, firewalling concepts for platform, and a mention to Notary.
Chapter 10, Swarm and the Cloud, is a chapter illustrating the most popular options for running Swarm on cloud providers. We’ll install Swarm on AWS and Azure before introducing the Docker Datacenter, and we’ll finally move to OpenStack showing how to install and manage Swarms on the top of Magnum, the Container as a Service solution for OpenStack.
Chapter 11, What is Next?, concludes the discussion with an overview on the next Docker orchestration trends, such as software defined infrastructures, Infrakit, unikernels, Containers as a Service. The adventure continues!
What you need for this book
We assume the reader to have some experience using Docker from the command line: throughout the book we’ll continuously pull images, run containers, define services, expose ports and create networks.
Also, the ideal reader possesses a basic understanding of networking protocols and is familiar with public and private cloud concepts like virtual machines and tenant networks.
To follow the examples in the text, you will need Docker and its tools. Chapter 1, Welcome to Docker Swarm, covers their installation.
Also, to get the most from the examples, you will need access to a a public (for example AWS, Azure or DigitalOcean) or private (for example OpenStack) cloud to instantiate Virtual Machines.
Who this book is for
This book is for Docker users - developers and system administrators - who want to exploit the current Swarm and Swarmkit features for scaling massive applications with containers.
Conventions
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "When executing docker swarm init, just copy-paste the lines printed as output"
A block of code is set as follows:
digitalocean: image: “docker-1.12-rc4” region: nyc3 ssh_key_fingerprint: “your SSH ID” ssh_user: rootAny command-line input or output is written as follows:
# Set $GOPATH here go get https://github.com/chanwit/beltNew terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "The UI has the expected options, included a list of templates for launching containers, such as MySQL or a Private Registry, but at the moment of writing it doesn’t support Swarm services yet"
Note
Warnings or important notes appear in a box like this.
Tip
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at [email protected] with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
"Illustrations of Swarm Architecture, Minimal Swarm on Production, and Docker Eco System in Chapter 1, Welcome to Docker Swarm © 2015-2016 Docker, Inc. Used by permission."
Questions
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.
Chapter 1. Welcome to Docker Swarm
It's no mystery that Docker is one of the open-source technologies that have the most traction nowadays. The reasons are easy to understand, Docker makes the container technology available for all, and it comes with an included battery that is removable and is blessed by a vibrant community.
In the early days, users started working with Docker after being fascinated with this easy-to-use tool, which allowed them to sort out many challenges: pulling, packing, isolating, and making applications portable across systems with almost no burden.
A simplified Docker Ecosystem
You may notice a swarm of whales here plays nice with others. However, since the advent of containers, people have been looking for tools to efficiently orchestrate a huge number of them. The Docker team addressed this necessity with the release of Docker Swarm, hereinafter Swarm, one of the pieces of the Docker ecosystem, in 2015, alongside with Docker Machine and Docker Compose. The preceding image shows the simplified Docker Ecosystem, which consists of Docker Machine provisioning a new Docker-ready machine, then a set of machines will be formed into a Docker Swarm cluster. Later we will be able to use Docker Compose to deploy containers to the cluster, as if it were a normal Docker Engine.
The plan to make a cluster management system, natively, for Docker started in early 2014, as a communication protocol project called Beam. Later, it was implemented as a daemon to control heterogeneous distributed systems with the Docker API. The project had been renamed to libswarm and Swarmd is its daemon. Keeping the same concept of allowing any Docker client to connect to a pool of Docker Engines, the third generation of the project had been re-designed to use the same set of Docker Remote APIs and renamed to "Swarm" in November 2014. Basically, the most important part of Swarm are its remote APIs; the maintainers work hard to keep them 100% compatible with every version of Docker Engine. We'll call the first generation of Swarm as "Swarm v1".
In February 2016, after the core team found the scaling limitation of the centralized service, Swarm has been internally redesigned again as swarm.v2. This time, a decentralized cluster design has been taken into account. In June 2016, SwarmKit had been released as the orchestration toolkit for distributed service at any scale. Docker announced that SwarmKit was merged into Docker Engine at DockerCon 2016. We'll refer to this version of Swarm as "Swarm v2" or "Swarm mode".
As we'll see later, these three musketeers (Docker Swarm, Docker Machine, and Docker Compose) operate best when together and they are so seamlessly intertwined with each other that it is almost impossible to think of them as single pieces.
However, even despite this Machine and Compose are really direct in their goals and easy to use and understand, Swarm is a tool that indeed deserves a book for itself.
With Docker Machine, you can provision machines, both virtual and physical, on a number of cloud platforms as well as bare metal machines to run Docker containers. With Docker Compose, you can define Dockerfiles on steroids, by describing behaviors with the easy yet powerful syntax of YAML and launch applications by just "composing up" these files. Swarm is a powerful clustering tool that requires to be studied more in depth.
In this chapter, we will be taking a look at the following topics:
Clustering tools and container managers
A clustering tool is software that allows an operator to talk to a single end point and to command and orchestrate a set of resources, in our case containers. Instead of manually distributing workloads (containers) on a cluster, a clustering tool is used to automate this and many other tasks. It's the clustering tool that will decide where to start jobs (containers), how to store them, when to eventually restart them, and so on. The operator needs to only configure some behaviors, decide the cluster topology and size, tune settings, and enable or disable advanced features. Docker Swarm is an example of clustering tool for containers.
Beyond clustering tools, there is also a choice of container manager platforms. They do not provide container hosting, but interact with one or more existing systems; this kind of software usually offer good web interfaces, monitoring tools, and other visual or higher-level functionalities. Examples of container manager platforms are Rancher or Tutum (acquired in 2015 by Docker Inc.).
Swarm goals
Swarm is described by Docker itself as:
Docker Swarm is native clustering for Docker. It turns a pool of Docker hosts into a single, virtual Docker host.
Swarm is a tool that gives you the illusion to manage a single-huge Docker host made of many Docker hosts, as they were one and had one command-entry point. It allows you to orchestrate and operate a certain number of containers on these hosts, using the routine Docker tools, either using the Docker native or the python-docker client, even curl with the Docker Remote APIs.
This is what a minimal Swarm cluster in production looks similar to:
Why use Swarm
There are many reasons to use a clustering solution for your containers. As you will see your applications growing, you will face new mandatory requirements, such as scalability, manageability, and high availability.
There are many tools available out there; picking up Docker Swarm gives us some immediate advantages:
Real world use case examples
Docker Swarm is the choice of several projects, for example:
Pet versus cattle models
There are two opposite approaches when creating and utilizing infrastructures: pet versus cattle.
In the pet model, the administrator deploys servers or virtual machines or, in our case, containers and takes care of them. She or he logs in, installs software, configures it, and ensures that everything is working fine. As a result, this is her or his pet.
By contrast, the administrator doesn't really care about the destiny of his infrastructural components, when thinking of them as cattles. She or he doesn't log in to every single unit or handle it manually, rather, uses a bulk approach, deployment, configuration, and management are done with automation tools. If a server or container dies, it's automatically resurrected, or another is generated to substitute for the defunct. As a result, the operator is handling cattle.
In this book, we'll use the pet model in the very first chapter to introduce some basic concepts to the reader. But we'll follow the cattle pattern later, when it will be the time to do serious things.
Swarm features
The main purpose of Swarm was already defined, but how does it accomplish its goals? Here are its key features:
