35,99 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
This book is aimed at architects, solution providers, and those of the DevOps community who are looking to implement repeatable patterns for deploying and maintaining services in the Amazon cloud infrastructure. Prior experience using AWS is required as the book focuses more on the patterns and not on the basics of using AWS.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 237
Veröffentlichungsjahr: 2015
Ähnliche
Table of Contents
Implementing Cloud Design Patterns for AWS
Implementing Cloud Design Patterns for AWS
Copyright © 2015 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: April 2015
Production reference: 1240415
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78217-734-0
www.packtpub.com
Credits
Author
Marcus Young
Reviewers
João Ferreira Loff
Robert M. Marks
Somanath Nanda
Philip O'Toole
Fred Stluka
Commissioning Editor
Dipika Gaonkar
Acquisition Editor
Nikhil Karkal
Content Development Editor
Merwyn D'souza
Technical Editor
Mitali Somaiya
Copy Editors
Trishya Hajare
Sonia Mathur
Alpha Singh
Project Coordinator
Neha Bhatnagar
Proofreaders
Joanna McMahon
Paul Hindle
Indexer
Mariammal Chettiyar
Graphics
Abhinash Sahu
Production Coordinator
Aparna Bhagat
Cover Work
Aparna Bhagat
About the Author
Marcus Young recently graduated with a degree in computer science and mathematics before getting involved in system administration and DevOps. He currently works in software automation using open source tools and technologies. His hobbies include playing ice hockey and brewing homebrew beer. He also enjoys hardware projects based on microcontrollers and single board computers.
I'd like to thank my beautiful wife for putting up with the many projects and work items that make their way into my free time. Also to my son who continues to inspire me to keep pushing myself.
About the Reviewers
João Ferreira Loff has an MSc in Computer Science and Engineering with a major in software engineering from Instituto Superior Técnico (www.tecnico.ulisboa.pt), University of Lisboa, Portugal. His interest in Cloud computing emerged from his master's thesis, where he researched predictive elasticity for Cloud applications.
He currently collaborates with the Distributed Systems Group at INESC-ID Lisboa (www.inesc-id.pt), a nonprofit computer science and electronics research institute, where he researches the latest developments in Cloud computing provisioning, elasticity, and scalability.
As a part of his research he developed Vadara, a generic Cloud computing elasticity framework that allows for the development of elasticity strategies that are decoupled from Cloud providers (https://github.com/jfloff/vadara). The foundation of this framework has been the subject of a published work at a top tier Cloud computing conference.
You can read more about him at https://jfloff.github.io.
Robert M. Marks is an experienced software developer and has spent over 12 years of his career working for a variety of software companies, ranging from large companies, such as IBM, to small start-ups. He is passionate about crafting well-tested software using best practices such as TDD, layered design, dependency injection, and so on. He has contributed to various open source projects and was the creator of JOGRE (Java Online Gaming Real-time Engine).
He is currently the head of engineering at Adoreboard, a unique platform that measures how the world feels about your brand so that marketers can make better business decisions. In his work at Adoreboard, he is a key pioneer for the development of real-time scalable architectures using a combination of technologies, including Enterprise Java, Spring Framework, Cloud computing, and NoSQL databases such as MongoDB, Elasticsearch, Solr, and Redis.
Somanath Nanda has spent the past 3 and a half years in the IT industry developing innovative methods to build new products which can fill the gap between human requirements and technology. He is interested in learning new data usage techniques, high-performance computing, and storage-related technologies. He has worked in various Cloud and big data technologies and data analysis mechanisms. His areas of interest include storage mechanisms of data and new algorithms and computational strategies, followed by high-performance, various machine learning, and data science techniques. Previously, he was involved in reviewing AWS Development Essentials, 1st Ed, 2014.
I would like to thank my parents and friends for their support in making this review successful.
Philip O'Toole has developed software and led software development teams for more than 15 years for a variety of applications including embedded software, networking appliances, web services, and SaaS infrastructure. His most recent work with AWS includes having led the infrastructure design and development of Loggly's log analytics SaaS platform, which is entirely hosted in AWS. He is based in the San Francisco Bay Area and can be found online at http://www.philipotoole.com.
Fred Stluka is an avid computer programmer and has been a mentor to hundreds of people over his 30 plus years of professional experience. He is proud to be a "Fred" in the very best sense of the word. For more information, see http://bristle.com/~fred/MaximizingTheFredFactor.htm.
He wrote his first book in 1991, http://archive.adaic.com/docs/style-guide/83style/style-t.txt.
www.PacktPub.com
Support files, eBooks, discount offers, and more
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at <[email protected]> for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
https://www2.packtpub.com/books/subscription/packtlib
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.
Why subscribe?
Free access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.
Preface
Amazon Web Services (AWS) is arguably the most cutting-edge Cloud provider currently available. In the past, data centers were massive entities that often required days to provide resources for applications. With AWS, this barrier is nonexistent. Applications can be scaled almost instantly. Metrics can be gathered with little or no configuration. Moving into the Cloud, however, might not be easy.
This book will act as a small reference guide, with detailed implementation examples, to show how (and how not) to design your applications in a way that makes them tolerant of underlying hardware failures, resilient against an unexpected influx of data, and easy to manage and replicate. You will be able to see both the benefits and limitations of the current services available to you from the AWS infrastructure.
What this book covers
Chapter 1, Introduction, introduces you to AWS and the problems encountered when deploying and maintaining applications in the Cloud. Problems include upgrading databases, data replication, cache issues, and zero downtime SLAs.
Chapter 2, Basic Patterns, demonstrates some examples of basic patterns such as scaling instances, dynamic disk allocation, and more.
Chapter 3, Patterns for High Availability, demonstrates some examples of patterns for highly available services such as data center replication, floating IP address allocation, health checking, and more.
Chapter 4, Patterns for Processing Static Data, demonstrates some examples of patterns for static data such as cache distribution, direct hosting, web storage hosting, and more.
Chapter 5, Patterns for Processing Dynamic Data, demonstrates some examples of patterns for dynamic data such as state sharing, URL rewriting, rewrite/cache proxying, and more.
Chapter 6, Patterns for Uploading Data, provides some examples of patterns and solutions for object uploading, storage indexing, and write proxying.
Chapter 7, Patterns for Databases, provides some examples of patterns for data replication, in-memory caching, and sharding.
Chapter 8, Patterns for Data Processing, provides some examples of patterns for batch processing issues such as queuing chains, priority queues, and job observers.
Chapter 9, Patterns for Operation and Maintenance, provides some examples of patterns for server swapping, startup settings, backup patterns, and others.
Chapter 10, Patterns for Networking, provides some examples of patterns for multiload balancers, operational and functional firewalls, and on-demand NAT networking.
Chapter 11, Throw-away Environments, is the closing chapter and provides some examples of third-party tools such as CloudFormation, Terraform, and so on, which aid in infrastructure design.
What you need for this book
Who this book is for
This book is aimed at architects, solution providers, and those members of the DevOps community who are looking to implement repeatable patterns for deploying and maintaining services in the Amazon Cloud infrastructure. This book could be used by those new to the DevOps movement, as well as those who have embraced the movement and are looking to create reusable patterns. However, prior experience using AWS is required as the book focuses more on the patterns and not on the basics of using AWS.
Conventions
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows:"Once this volume is available, attach it as /dev/sdb to the instance."
A block of code is set as follows:
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
Any command-line input or output is written as follows:
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "Clicking the Next button moves you to the next screen".
Note
Warnings or important notes appear in a box like this.
Tip
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the erratasubmissionform link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
Questions
You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.
Chapter 1. Introduction
The paradigm for development of applications has shifted in many ways over the years. Instead of just developing pure applications, aimed at specific system configurations, the trend has moved towards web applications. These applications present a very different set of challenges not just for the developers, but also for the people who manage the systems that host them. The reaction to this need to build, test, and manage such web applications has been to develop an abstraction on top of the hardware that allows for the ability to bring up entire virtualized environments quickly and consistently.
Throughout these chapters, you will learn basic design principles for applications and known issues. These may not be completely compatible with all application types but should serve as a basic toolkit for bigger design patterns. It is also very important to note that AWS adds new services all the time, some of which by default solve these design patterns at the time of writing. If your software or services handle sensitive data and have in-flight or at-rest requirements, be very careful to read how each individual AWS-provided service handles data.
The topics that are covered in this chapter are:
Introduction to AWS
Amazon Web Services (AWS) is a very large suite of Cloud services provided by Amazon. AWS provides, at a base level, virtual machines and the services surrounding them. Many Cloud-based virtual machine services such as Google Compute Engine, DigitalOcean, Rackspace, Windows Azure, and so on provide the ability to bring up a machine from a supported base operating system image or snapshot, and it's up to the user to customize it further.
Amazon has made itself one of the leaders for Cloud-hosting by providing not just virtual machines, but configurable services and software implementations of hardware found in data centers. For most large-scale systems, the move to Cloud infrastructure brings to the table a huge set of questions on how to handle issues such as load balancing, content delivery networks, failover, and replication. The AWS suite can handle the same issues that a physical data center can, usually for a fraction of the cost. They can get rid of some of the red tape that comes with a data center such as requesting provisioning, repairs, and scheduling downtime.
Amazon is constantly offering new services to tackle new and unique problems encountered with Cloud infrastructure. However, this book may not cover every service offered by Amazon. The services that this book will cover include:
Tip
For information on other services provided by AWS that are not relevant to the information in this book visit http://aws.amazon.com/products/.
Cloud computing service models
AWS falls under the category of Cloud computing called Infrastructure as a Service. In Cloud computing there are three service models:
Infrastructure as a Service
IaaS can be described as a service that provides virtual abstractions for hardware, servers, and networking components. The service provider owns all the equipment and is responsible for its housing, running, and maintenance. In this case, AWS provides APIs, SDKs, and a UI for creating and modifying virtual machines, their network components, routers, gateways, subnets, load balancers, and much more. Where a user with a physical data center would incur charges for the hardware, shelving, and access, this is removed by IaaS with a payment model that is per-hour (or per-use) type.
Platform as a Service
While AWS itself is an IaaS provider, it contains a product named ElasticBeanstalk, which falls under the PaaS category for Cloud models. PaaS is described as the delivery of a computing platform, typically an operating system, programming language execution environment, database, or web server. With ElasticBeanStalk, a user can easily turn a code into a running environment without having to worry about any of the pieces underneath such as setting up and maintaining the database, web server, or code runtime versions. It also allows it to be scaled without having to do anything other than define scale policies through the configuration.
Software as a Service
AWS also provides a marketplace where a user can purchase official and third-party operating system images that provide configurable services such as databases, web applications, and more. This type of service falls under the SaaS model. The best interpretation for the SaaS model is on-demand software, meaning that the user need only configure the software to use and interact with it. The draw to SaaS is that there is no need to learn how to configure and deploy the software to get it working in a larger stack and generally the charges are per usage-hour.
The AWS suite is both impressive and unique in that it doesn't fall under any one of the Cloud service models as described previously. Until AWS made its name, the need to virtualize an entire environment or stack was usually not an easy task and consisted of a collection of different providers, each solving a specific part of the deployment puzzle. The cost of using many different providers to create a virtual stack might not be cheaper than the initial hardware cost for moving equipment into a data center. Besides the cost of the providers themselves, having multiple providers also created the problem of scaling in one area and notifying another of the changes. While making applications more resilient and scalable, this Frankenstein method usually did not simplify the problem as a whole.
Benefits of moving to the Cloud
There are many different answers to why moving to a Cloud-hosted environment might be beneficial but it depends on the end user. The shift may suit small teams but for mid-sized teams the effort saved begins to outweigh the cost. I start at mid-sized because this is the size that usually includes the two teams that benefit the most:
For a developer, the biggest benefit of Cloud providers is the ability to throw away entire environments. In a traditional developer setting, the developers usually develop their code locally, have access to a shared physical server, or have access to a virtual server of some type. Issues that usually arise out of these setups are that it's hard to enforce consistency and the servers can become stale over time. If each developer works locally, inconsistency can arise very quickly. Different versions of the core language or software could be used and might behave differently from machine to machine. One developer might use Windows and prefer registry look-ups while another developer may use Mac and prefer environment variables.
If the developers share a core server for development, other issues may arise such as permissions or possibly trying to modify services independent of each other causing race conditions. No matter what problems exist, known or unknown, they could be solved by always starting from the same base operating system state. The leading software for solving this issue is Vagrant. Vagrant provides the ability to spin up and destroy a virtual machine from a configuration file along with a configuration management suite such as Puppet, Chef, Docker, or Ansible. Vagrant itself is agnostic to the Cloud hosting tool in the sense that it does not require AWS. It can spin up instances at AWS given the credentials, but it can also spin up virtual machines locally from VirtualBox and VMWare.
Vagrant gives back consistency to the developers in the sense that it takes a base box (in AWS terms this is an Amazon Machine Image or AMI) and configures it via one of the configuration suites or shell to create a running virtual machine every time it is needed. If all the developers share the same configuration file then all of them are mostly guaranteed to work against the same environment. That environment can be destroyed just as easily as it was created, giving the resources back and incurring no charges until needed again.
The bringing up and destroying of the instances becomes a small invisible piece of their workflow. By virtue of enforcing a strategy like this on a team, a lot of issues can be found and resolved before they make their way up the chain to the testing or production environments.
Note
More information on Vagrant can be found at http://www.vagrantup.com.
The other team I mentioned that benefits from moving to the Cloud is the operations team. This team differs greatly in responsibility from company to company but it is safe to assume that the team is heavily involved with monitoring the applications and systems for issues and possible optimizations. AWS provides enough infrastructure for monitoring and acting on metrics and an entire book could be dedicated to the topic. However, I'll focus only on auto scaling groups and CloudWatch.
For AWS, an auto scaling group defines scaling policies for instances based on schedules, custom metrics, or base metrics such as disk usage, CPU utilization, memory usage, and so on. An auto scaling group can act on these thresholds and scale up or down depending on how they are configured. In a non-Cloud environment this same setup takes quite a bit of effort and depends on the software whereas, it's a core concept to AWS.
Auto scaling groups also automatically register instances with a load balancer and shift them into a round robin distribution. For an operations team, the benefit of moving to Amazon might justify itself only to alleviate all the work involved in duplicating this functionality elsewhere, allowing the team to focus on creating deeper and more meaningful system health checks.
Throw-away environments can also be beneficial to the operations teams. A sibling product to Vagrant, very recently released, is Terraform. Terraform, like Vagrant, is agnostic to the hosting environment but does not solely spin up virtual instances. Terraform is similar to CloudFormation in the sense that its goal is to take a central configuration file, which describes all the resources it needs. It then maps them into a dependency graph, optimizes, and creates an entire stack. A common example for Terraform would be to create a production environment from a few virtual machines, load balancers, Route53 DNS entries, and set auto scaling policies. This flexibility would be nearly impossible in traditional hardware settings and provides an on-demand mentality not just for the base application, but also for the entire infrastructure, leading to a more agile core.
Note
More information on Terraform can be found at http://www.terraform.io.
Common problems encountered at AWS
The previous sections have tried to make light of issues found in traditional settings, which might make moving to a Cloud infrastructure seem like a logical choice with no ramifications. But this is not true. While Cloud infrastructure aims to resolve many problems, it does bring up new issues to the user.
Underlying hardware failures
Some issues can be avoided while others may not. Some examples of issues that may not be avoided, other than user error, are underlying hardware issues that propagate themselves to the user. Hardware has a fail rate and can be guaranteed to fail at some point while the benefit of IaaS is that, even though the hardware is abstracted away, it is still a relevant topic to anyone using it.
AWS has a Service Level Agreement (SLA
