47,99 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
We often deploy our web applications into the cloud, and our code needs to interact with many third-party services. An efficient way to build applications to do this is through microservices architecture. But, in practice, it's hard to get this right due to the complexity of all the pieces interacting with each other.
This book will teach you how to overcome these issues and craft applications that are built as small standard units, using all the proven best practices and avoiding the usual traps. It's a practical book: you’ll build everything using Python 3 and its amazing tooling ecosystem. You will understand the principles of TDD and apply them.
You will use Flask, Tox, and other tools to build your services using best practices. You will learn how to secure connections between services, and how to script Nginx using Lua to build web application firewall features such as rate limiting. You will also familiarize yourself with Docker’s role in microservices, and use Docker containers, CoreOS, and Amazon Web Services to deploy your services.
This book will take you on a journey, ending with the creation of a complete Python application based on microservices. By the end of the book, you will be well versed with the fundamentals of building, designing, testing, and deploying your Python microservices.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 398
Veröffentlichungsjahr: 2017
Ähnliche
BIRMINGHAM - MUMBAI
< html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
Python Microservices Development
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: July 2017
Production reference: 1210717
ISBN 978-1-78588-111-4
www.packtpub.com
Credits
Author
Tarek Ziade
Copy Editor
Sonia Mathur
Reviewer
William Kahn-Greene
Project Coordinator
Vaidehi Sawant
Commissioning Editor
Aaron Lazar
Proofreader
Safis Editing
Acquisition Editor
Chaitanya Nair
Indexer
Mariammal Chettiyar
Content Development Editor
Rohit Kumar Singh
Graphics
Jason Monteiro
Technical Editor
Pavan Ramchandani
Production Coordinator
Nilesh Mohite
About the Author
Tarek Ziade is a Python developer, located in the countryside near Dijon, France. He works at Mozilla in the services team. He founded a French Python user group called Afpy, and he has written several books about Python in French and English. When he is not hacking on his computer or hanging out with his family, he's spending time between his two passions, running and playing the trumpet.You can visit his personal blog (Fetchez le Python) and follow him on Twitter (@tarek_ziade). You can also take a look at one of his books on Amazon, Expert Python Programming, published by Packt.
About the Reviewer
William Kahn-Greene has been writing Python and building applications on the web since the late 90s. He works in the crash-stats group on the crash ingestion pipeline at Mozilla and also maintains a variety of Python libraries, including bleach. When he is waiting for CI to test his code changes, he is building things with wood, tending to his tomato plant, and cooking for four.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBook.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at ;https://www.amazon.com/dp/1785881116.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
Understanding Microservices
Origins of Service-Oriented Architecture
The monolithic approach
The microservice approach
Microservice benefits
Separation of concerns
Smaller projects
Scaling and deployment
Microservices pitfalls
Illogical splitting
More network interactions
Data storing and sharing
Compatibility issues
Testing
Implementing microservices with Python
The WSGI standard
Greenlet and Gevent
Twisted and Tornado
asyncio
Language performances
Summary
Discovering Flask
Which Python?
How Flask handles requests
Routing
Variables and converters
The url_for function
Request
Response
Flask built-in features
The session object
Globals
Signals
Extensions and middlewares
Templates
Configuration
Blueprints
Error handling and debugging
Custom error handler
The debug mode
A microservice skeleton
Summary
Coding, Testing, and Documenting - the Virtuous Cycle
Different kinds of tests
Unit tests
Functional tests
Integration tests
Load tests
End-to-end tests
Using WebTest
Using pytest and Tox
Developer documentation
Continuous Integration
Travis-CI
ReadTheDocs
Coveralls
Summary
Designing Runnerly
The Runnerly application
User stories
Monolithic design
Model
View and Template
Background tasks
Strava token
Authentication and authorization
Putting together the monolithic design
Splitting the monolith
Data Service
Using Open API 2.0
More splitting
Summary
Interacting with Other Services
Synchronous calls
Using Session in a Flask app
Connection pooling
HTTP cache headers
Improving data transfer
GZIP compression
Binary payloads
Putting it together
Asynchronous calls
Task queues
Topic queues
Publish/subscribe
RPC over AMQP
Putting it together
Testing
Mocking synchronous calls
Mocking asynchronous calls
Mocking Celery
Mocking other asynchronous calls
Summary
Monitoring Your Services
Centralizing logs
Setting up Graylog
Sending logs to Graylog
Adding extra fields
Performance metrics
System metrics
Code metrics
Web server metrics
Summary
Securing Your Services
The OAuth2 protocol
Token-based authentication
The JWT standard
PyJWT
X.509 certificate-based authentication
The TokenDealer microservice
The POST/oauth/token implementation
Using TokenDealer
Web application firewall
OpenResty - Lua and nginx
Rate and concurrency limiting
Other OpenResty features
Securing your code
Asserting incoming data
Limiting your application scope
Using Bandit linter
Bringing It All Together
Building a ReactJS dashboard
The JSX syntax
React components
ReactJS and Flask
Using Bower, npm, and Babel
Cross-origin resource sharing
Authentication and authorization
Interacting with Data Service
Getting the Strava token
JavaScript authentication
Packaging and Running Runnerly
The packaging toolchain
A few definitions
Packaging
The setup.py file
The requirements.txt file
The MANIFEST.in file
Versioning
Releasing
Distributing
Running all microservices
Process management
Summary
Containerized Services
What is Docker?
Docker 101
Running Flask in Docker
The full stack - OpenResty, Circus and Flask
OpenResty
Circus
Docker-based deployments
Docker Compose
Introduction to Clustering and Provisioning
Summary
Deploying on AWS
AWS overview
Routing - Route53, ELB, and AutoScaling
Execution - EC2 and Lambda
Storage - EBS, S3, RDS, ElasticCache, and CloudFront
Messaging - SES, SQS, and SNS
Simple Email Service (SES)
Simple Queue Service (SQS)
Simple Notification Service (SNS)
Provisioning and deployment - CloudFormation and ECS
Deploying on AWS - the basics
Setting up your AWS account
Deploying on EC2 with CoreOS
Deploying with ECS
Route53
Summary
What Next?
Iterators and generators
Coroutines
The asyncio library
The aiohttp framework
Sanic
Asynchronous versus synchronous
Preface
If we try to deploy our web applications into the cloud, it requires our code to interact with many third-party services. Using microservice architectures, you can build applications that will allow you to manage these interactions. However, this comes with its own set of challenges, since each set has its own complexity, and getting their interaction right isn't easy. This easy-to-follow guide covers techniques to help you overcome these challenges. You will learn how to best design, write, test, and deploy your microservices. The real-world examples will help Python developers create their own Python microservices using the most efficient methods. By the end of this book, you will have acquired the skills to craft applications that are built as small standard units, using all the proven best practices and avoiding the usual traps. Also, this is a useful guide for the vast community of Python developers who are shifting from monolithic design to the new microservice-based development paradigm.
What this book covers
Chapter 1, Understanding Microservices, defines what microservices are, and their roles in modern web applications. It also introduces Python and explains why it's great for building microservices.
Chapter 2, Discovering Flask, introduces Flask and goes through its main features. It showcases the framework with a sample web application that will be the basis for building microservices.
Chapter 3, Coding, Testing, and Documenting - the Virtuous Cycle, describes the Test-Driven Development and Continuous Integration approach, and how to use it in practice to build and package Flask applications.
Chapter 4, Designing Runnerly, takes you through the app features and user stories, explains how it could be built as a monolithic app, then decomposes it into microservices and explains how they interact with the data. It will also introduce the Open API 2.0 specification (ex-Swagger), which can be used to describe HTTP APIs.
Chapter 5, Interacting with Other Services, explains how a service interacts with backend services, how to deal with network splits and other interaction problems, and how to test the service in isolation.
Chapter 6, Securing Your Services, explains how to secure your microservices and how to deal with user authentication, service-to-service authentication, as well as user management. It will also introduce the reader to fraud and abuse, and how to mitigate it.
Chapter 7, Monitoring Your Services, explains how to add logging and metrics in your code, and how to make sure you have a clear global understanding of what's going on in your application to track down issues and understand your services usage.
Chapter 8, Bringing It All Together, describes how to design and build a JavaScript application that leverages and uses the microservices in an end-user interface.
Chapter 9, Packaging and Running Runnerly, describes how to package, build, and run the whole Forrest application. As a developer, it's vital to be able to run all the parts that compose your application into a single dev box.
Chapter 10, Containerized Services, explains what is virtualization, how to use Docker, and also how to Dockerize your services.
Chapter 11, Deploying on AWS, introduces you to existing cloud service providers and then to the AWS world, and shows how to instantiate servers and use the major AWS services that are useful to run a microservices-based application. It also introduces CoreOS, a Linux distribution specifically created to deploy Docker containers in the cloud.
Chapter 12, What Next?, concludes the book by giving some hints on how your microservices can be built independently from specific cloud providers and virtualization technologies, to avoid the trap of putting all your eggs in the same basket. It emphasizes what you learned in Chapter 9, Packaging and Running Runnerly.
What you need for this book
To execute the commands and applications in this book, you will need Python 3.x, Virtualenv 1.x, and Docker CE installed on your system. Detailed instructions are given in the chapters where needed.
Who this book is for
If you are a developer who has basic knowledge of Python, the command line, and HTTP-based application principles, and who wants to learn how to build, test, scale, and manage Python 3 microservices, then this book is for you. No prior experience of writing microservices in Python is assumed.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the
SUPPORT
tab at the top.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on
Code Download
.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Python-Microservices-Development. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at [email protected] with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any aspect of this book, you can contact us at [email protected], and we will do our best to address the problem.
Understanding Microservices
We're always trying to improve how we build software, and since the punched-card era, we have improved a lot, to say the least.
The microservices trend is one improvement that has emerged in the last few years, partially based on companies' willingness to speed up their release cycles. They want to ship new products and new features to their customers as fast as possible. They want to be agile by iterating often, and they want to ship, ship, and ship again.
If thousands, or even millions, of customers use your service, pushing in production an experimental feature, and removing it if it does not work, is considered good practice rather than baking it for months before you publish it.
Companies such as Netflix are promoting their continuous delivery techniques where small changes are made very often into production, and tested on a subset of the user base. They've developed tools such as Spinnaker (http://www.spinnaker.io/) to automate as many steps as possible to update production, and ship their features in the cloud as independent microservices.
But if you read Hacker News or Reddit, it can be quite hard to detangle what's useful for you and what's just buzzwords-compliant journalistic-style info.
This chapter is going to help you understand what are microservices, and will then focus on the various ways in which you can implement them using Python. It's composed of the following few sections:
A word on Service-Oriented Architecture
Monolithic approach of building an application
Microservices approach of building applications
Benefits of microservices
Pitfalls in microservices
Implementing microservices with Python
Hopefully, once you've reached the end of the chapter, you will be able to dive into building microservices with a good understanding of what they are and what they aren't--and how you can use Python.
Origins of Service-Oriented Architecture
There are many definitions out there, since there is no official standard for microservices. People often mention Service-Oriented Architecture (SOA) when they are trying to explain what microservices are.
Each unit in this preceding definition is a self-contained service, which implements one facet of a business, and provides its feature through some interface.
While SOA clearly states that services should be standalone processes, it does not enforce what protocols should be used for those processes to interact with each other, and stays quite vague about how you deploy and organize your application.
If you read the SOA Manifesto (http://www.soa-manifesto.org) that a handful of experts published on the web circa 2009, they don't even mention if the services interact via the network.
SOA services could communicate via Inter-Process Communication (IPC) using sockets on the same machine, through shared memory, through indirect message queues, or even with Remote Procedure Calls (RPC). The options are extensive, and at the end of the day, SOA can be everything and anything as long as you are not running all your application code into a single process.
However, it is common to say that microservices are one specialization of SOA, which have started to emerge over the last few years, because they fulfill some of the SOA goals which are to build apps with standalone components that interact with each other.
Now if we want to give a complete definition of what are microservices, the best way to do it is to first look at how most software are architectured.
The monolithic approach
Let's take a very simple example of a traditional monolithic application: a hotel booking website.
Besides the static HTML content, the website has a booking feature that will let its users book hotels in any city in the world. Users can search for hotels, then book them with their credit cards.
When a user performs a search on the hotel website, the application goes through the following steps:
It runs a couple of SQL queries against its hotels' database.
An HTTP request to a partner's service is made to add more hotels to the list.
An HTML results page is generated using an HTML template engine.
From there, once the user has found the perfect hotel and clicked on it to book it, the application performs these steps:
The customer gets created in the database if needed, and has to authenticate.
Payment is carried out by interacting with the bank web service.
The app saves the payment details in the database for legal reasons.
A receipt is generated using a PDF generator.
A recap email is sent to the user using the email service.
A reservation email is forwarded to the third-party hotel using the email service.
A database entry is added to keep track of the reservation.
This process is a simplified model of course, but quite realistic.
The application interacts with a database that contains the hotel's information, the reservation details, the billing, the user information, and so on. It also interacts with external services for sending emails, making payments, and getting more hotels from partners.
In the good old LAMP (Linux-Apache-MySQL-Perl/PHP/Python) architecture, every incoming request generates a cascade of SQL queries on the database, and a few network calls to external services, and then the server generates the HTML response using a template engine.
The following diagram illustrates this centralized architecture:
This application is a typical monolith, and it has a lot of obvious benefits.
The biggest one is that the whole application is in a single code base, and when the project coding starts, it makes everything simpler. Building a good test coverage is easy, and you can organize your code in a clean and structured way inside the code base. Storing all the data into a single database also simplifies the development of the application. You can tweak the data model, and how the code will query it.
The deployment is also a no brainer: we can tag the code base, build a package, and run it somewhere. To scale it, we can run several instances of the booking app, and run several databases with some replication mechanism in place.
If your application stays small, this model works well and is easy to maintain for a single team.
But projects are usually growing, and they get bigger than what was first intended. And having the whole application in a single code base brings some nasty issues along the way. For instance, if you need to make a sweeping change that is large in scope such as changing your banking service or your database layer, the whole application gets into a very unstable state. These changes are a big deal in the project's life, and they necessitate a lot of extra testing to deploy a new version. And changes like this will happen in a project life.
Small changes can also generate collateral damage because different parts of the system have different uptime and stability requirements. Putting the billing and reservation processes at risk because the function that creates the PDF crashes the server is a bit of a problem.
Uncontrolled growth is another issue. The application is bound to get new features, and with developers leaving and joining the project, the code organization might start to get messy, the tests a bit slower. This growth usually ends up with a spaghetti code base that's hard to maintain, with a hairy database that needs complicated migration plans every time some developer refactors the data model.
Big software projects usually take a couple of years to mature, and then they slowly start to turn into an incomprehensible mess that's hard to maintain. And it does not happen because developers are bad. It happens because as the complexity grows, fewer people fully understand the implications of every small change they make. So they try to work in isolation in one corner of the code base, and when you take the 10,000-foot view of the project, you can see the mess.
We've all been there.
It's not fun, and developers who work on such a project dream of building the application from scratch with the newest framework. And by doing so, they usually fall into the same issues again--the same story is repeated.
The following points summarize the pros and cons of the monolithic approach:
Starting a project as a monolith is easy, and probably the best approach.
A centralized database simplifies the design and organization of the data.
Deploying one application is simple.
Any change in the code can impact unrelated features. When something breaks, the whole application may break.
Solutions to scale your application are limited: you can deploy several instances, but if one particular feature inside the app takes all the resources, it impacts everything.
As the code base grows, it's hard to keep it clean and under control.
There are, of course, some ways to avoid some of the issues described here.
The obvious solution is to split the application into separate pieces, even if the resulting code is still going to run in a single process. Developers do this by building their apps with external libraries and frameworks. Those tools can be in-house or from the Open Source Software (OSS) community.
Building a web app in Python if you use a framework like Flask, lets you focus on the business logic, and makes it very appealing to externalize some of your code into Flask extensions and small Python packages. And splitting your code into small packages is often a good idea to control your application growth.
For instance, the PDF generator described in the hotel booking app could be a separate Python package that uses Reportlab and some templates to do the work.
Chances are this package can be reused in some other apps, and maybe, even published to the Python Package Index (PyPI) for the community.
But you're still building a single application and some problems remain, like the inability to scale parts differently, or any indirect issue introduced by a buggy dependency.
You'll even get new challenges, because you're now using dependencies. One problem you can get is dependency hell. If one part of your application uses a library, but the PDF generator can only use a specific version of that library, there are good chances you will eventually have to deal with it with some ugly workaround, or even fork the dependency to have a custom fix there.
Of course, all the problems described in this section do not appear on day 1 when the project starts, but rather pile up over time.
Let's now look at how the same application would look like if we were to use microservices to build it.
The microservice approach
If we were to build the same application using microservices, we would organize the code into several separate components that run in separate processes. Instead of having a single application in charge of everything, we would split it into many different microservices, as shown in the following diagram:
Don't be afraid of the number of components displayed in this diagram. The internal interactions of the monolithic application are just being made visible by separate pieces. We've shifted some of the complexity and ended up with these seven standalone components:
Booking UI
: A frontend service, which generates the web user interface, and interacts with all the other microservices.
PDF reporting service
: A very simple service that would create PDFs for the receipts or any other document given a template and some data.
Search
: A service that can be queried to get a list of hotels given a city name. This service has its own database.
Payments
: A service that interacts with the third-party bank service, and manages a billing database. It also sends e-mails on successful payments.
Reservations:
Stores reservations, and generates PDFs.
Users
: Stores the user information, and interacts with users via emails.
Authentication
: An OAuth 2-based service that returns authentication tokens, which each microservice can use to authenticate when calling others.
Those microservices, along with the few external services like the email service, would provide a feature set similar to the monolithic application. In this design, each component communicates using the HTTP protocol, and features are made available through RESTful web services.
There's no centralized database, as each microservice deals internally with its own data structures, and the data that gets in and out uses a language-agnostic format like JSON. It could use XML or YAML as long as it can be produced and consumed by any language, and travel through HTTP requests and responses.
The Booking UI service is a bit particular in that regard, since it generates the User Interface (UI). Depending on the frontend framework used to build the UI, the Booking UI output could be a mix of HTML and JSON, or even plain JSON if the interface uses a static JavaScript-based client-side tool to generate the interface directly in the browser.
But besides this particular UI case, a web application designed with microservices is a composition of several microservices, which may interact with each other through HTTP to provide the whole system.
In that context, microservices are logical units that focus on a very particular task. Here's a full definition attempt:
This definition does not mention HTTP or JSON, because you could consider a small UDP-based service that exchanges binary data as a microservice for example.
But in our case, and throughout the book, all our microservices are just simple web applications that use the HTTP protocol, and consume and produce JSON when it's not a UI.
Microservice benefits
While the microservices architecture looks more complicated than its monolithic counterpart, its advantages are multiple. It offers the following:
Separation of concerns
Smaller projects to deal with
More scaling and deployment options
We will discuss them in more detail in the following sections.
Separation of concerns
First of all, each microservice can be developed independently by a separate team. For instance, building a reservation service can be a full project on its own. The team in charge can make it in whatever programming language and database, as long as it has a well-documented HTTP API.
That also means the evolution of the app is more under control than with monoliths. For example, if the payment system changes its underlying interactions with the bank, the impact is localized inside that service, and the rest of the application stays stable and is probably unaffected.
This loose coupling improves the overall project velocity a lot, as we apply, at the service level, a philosophy similar to the single responsibility principle.
The single responsibility principle was defined by Robert Martin to explain that a class should have only one reason to change; in other words, each class should provide a single, well-defined feature. Applied to microservices, it means that we want to make sure that each microservice focuses on a single role.
Smaller projects
The second benefit is breaking the complexity of the project. When you add a feature to an application such as PDF reporting, even if you do it cleanly, you make the base code bigger, more complicated, and sometimes, slower. Building that feature in a separate application avoids this problem, and makes it easier to write it with whatever tools you want. You can refactor it often, shorten your release cycles, and stay on top of things. The growth of the application remains under your control.
Dealing with a smaller project also reduces risks when improving the application: if a team wants to try out the latest programming language or framework, they can iterate quickly on a prototype that implements the same microservice API, try it out, and decide whether or not to stick with it.
One real-life example in mind is the Firefox Sync storage microservice. There are currently some experiments to switch from the current Python + MySQL implementation to a Go-based one, which stores users' data in standalone SQLite databases. That prototype is highly experimental, but since we have isolated the storage feature in a microservice with a well-defined HTTP API, it's easy enough to give it a try with a small subset of the user base.
Scaling and deployment
Finally, having your application split into components makes it easier to scale depending on your constraints. Let's say you start getting a lot of customers who book hotels daily, and the PDF generation starts to heat up the CPUs. You can deploy that specific microservice in some servers that have bigger CPUs.
Another typical example are RAM-consuming microservices like the ones that interact with memory databases like Redis or Memcache. You could tweak your deployments, consequently, by deploying them on servers with less CPU and a lot more RAM.
We can, thus, summarize the benefits of microservices as follows:
A team can develop each microservice independently, and use whatever technological stack makes sense. They can define a custom release cycle. All they need to define is a language-agnostic HTTP API.
Developers break the application complexity into logical components. Each microservice focuses on doing one thing well.
Since microservices are standalone applications, there's a finer control on deployments, which makes scaling easier.
The microservices architecture is good at solving a lot of the problems that may arise once your application starts to grow. However, we need to be aware of some of the new issues they also bring in practice.
Microservices pitfalls
As said earlier, building an application with microservices has a lot of benefits, but it's not a silver bullet by all means.
You need to be aware of these main problems you might have to deal with when coding microservices:
Illogical splitting
More network interactions
Data storing and sharing
Compatibility issues
Testing
These issues will be covered in detail in the following sections.
Illogical splitting
The first issue of a microservice architecture is how it gets designed. There's no way a team can come up with the perfect microservice architecture in the first shot. Some microservices like the PDF generator are an obvious use case. But as soon as you deal with the business logic, there are good chances that your code will move around before you get a good grasp of how to split things into the right set of microservices.
The design needs to mature with some try-and-fail cycles. And adding and removing microservices can be more painful than refactoring a monolithic application.
You can mitigate this problem by avoiding splitting your app in microservices if the split is not evident.
If there's any doubt that the split makes sense, keeping the code in the same app is the safe bet. It's always easier to split apart some of the code into a new microservice later than to merge back to two microservices in the same code base because the decision turned out to be wrong.
For instance, if you always have to deploy two microservices together, or if one change in a microservice impacts the data model of another one, the odds are that you did not split the application correctly, and that those two services should be reunited.
More network interactions
The second problem is the amount of network interactions added to build the same application. In the monolithic version, even if the code gets messy, everything happens in the same process, and you can send back the result without having to call too many backend services to build the actual response.
That requires extra attention on how each backend service is called, and raises a lot of questions like the following:
What happens when the Booking UI cannot reach the PDF reporting service because of a network split or a laggy service?
Does the Booking UI call the other services synchronously or asynchronously?
How will that impact the response time?
We will need to have a solid strategy to be able to answer all those questions, and we will address those in Chapter 5, Interacting with Other Services.
Data storing and sharing
Another problem is data storing and sharing. An effective microservice needs to be independent of other microservices, and ideally, should not share a database. What does this mean for our hotel booking app?
Again, that raises a lot of questions such as the following:
Do we use the same users' IDs across all databases, or do we have independent IDs in each service and keep it as a hidden implementation detail?
Once a user is added to the system, do we replicate some of her information in other services databases via strategies like data pumping, or is that overkill?
How do we deal with data removal?
These are hard questions to answer, and there are many different ways to solve those problems, as we'll learn throughout the book.
Compatibility issues
Another problem happens when a feature change impacts several microservices. If a change affects in a backward incompatible way the data that travels between services, you're in for some trouble.
Can you deploy your new service, and will it work with older versions of other services? Or do you need to change and deploy several services at once? Does it mean you've just stumbled on some services that should probably be merged back together?
A good versioning and API design hygiene help to mitigate those issues, as we will discover in the second part of the book when we'll build our application.
Testing
Last, when you want to do some end-to-end tests and deploy your whole app, you now have to deal with many bricks. You need to have a robust and agile deployment process to be efficient. You need to be able to play with your whole application when you develop it. You can't fully test things out with just one piece of the puzzle.
Hopefully, there are now many tools to facilitate deployments of applications that are built with several components, as we will learn about throughout this book. And all those tools probably helped in the success and adoption of microservices and vice versa.
The pitfalls of using microservices can be summarized as follows:
Premature splitting of an application into microservices can lead to architectural problems
Network interactions between microservices add weaknesses spots and additional overhead
Testing and deploying microservices can be complex
And the biggest challenge--data sharing between microservices is hard
You should not worry too much about all the pitfalls described in this section for now.
They may seem overwhelming, and the traditional monolithic application may look like a safer bet, but in the long term, splitting your project into microservices will make many of your tasks, as a developer or as an Operation person (Ops), easier.
Implementing microservices with Python
Python is an amazingly versatile language.
As you probably already know, it's used to build many different kinds of applications--from simple system scripts that perform tasks on a server to large object-oriented applications that run services for millions of users.
According to a study conducted by Philip Guo in 2014, published on the Association for Computing Machinery (ACM) website, Python has surpassed Java in top U.S. universities, and is the most popular language to learn computer science.
This trend is also true in the software industry. Python sits now in the top five languages in the TIOBE index (http://www.tiobe.com/tiobe-index/), and it's probably even bigger in the web development land, since languages like C are rarely used as main languages to build web applications.
However, some developers criticize Python for being slow and unfit for building efficient web services. Python is slow, and this is undeniable. But it still is a language of choice for building microservices, and many major companies are happily using it.
This section will give you some background on the different ways you can write microservices using Python, some insights on asynchronous versus synchronous programming, and conclude with some details on Python performances.
This section is composed of five parts:
The WSGI standard
Greenlet and Gevent
Twisted and Tornado
asyncio
Language performances
Greenlet and Gevent
The general principle of asynchronous programming is that the process deals with several concurrent execution contexts to simulate parallelism.
Asynchronous applications use an event loop that pauses and resumes execution contexts when an event is triggered--only one context is active, and they take turns. Explicit instruction in the code will tell the event loop that this is where it can pause the execution.
When that occurs, the process will look for some other pending work to resume. Eventually, the process will come back to your function and continue it where it stopped. Moving from an execution context to another is called switching.
The Greenlet project (https://github.com/python-greenlet/greenlet) is a package based on the Stackless
