39,59 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Fachliteratur
- Sprache: Englisch
- Veröffentlichungsjahr: 2018
Networks in your infrastructure set the foundation for how your application can be deployed, maintained, and serviced. Python is the ideal language for network engineers to explore tools that were previously available to systems engineers and application developers. In this second edition of Mastering Python Networking, you’ll embark on a Python-based journey to transition from traditional network engineers to network developers ready for the next-generation of networks.
This book begins by reviewing the basics of Python and teaches you how Python can interact with both legacy and API-enabled network devices. As you make your way through the chapters, you will then learn to leverage high-level Python packages and frameworks to perform network engineering tasks for automation, monitoring, management, and enhanced security. In the concluding chapters, you will use Jenkins for continuous network integration as well as testing tools to verify your network.
By the end of this book, you will be able to perform all networking tasks with ease using Python.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 469
Ähnliche
Mastering Python Networking Second Edition
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Vijin BorichaAcquisition Editor: Prachi BishtContent Development Editor: Deepti ThoreTechnical Editor:Varsha ShivhareCopy Editor: Safis EditingProject Coordinator: Kinjal BariProofreader: Safis EditingIndexer: Mariammal ChettiyarGraphics: Jisha ChirayilProduction Coordinator: Aparna Bhagat
First published: June 2017 Second edition: August 2018
Production reference: 1280818
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78913-599-2
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the author
Eric Chou is a seasoned technologist with over 18 years of industry experience. He has worked on and helped managed some of the largest networks in the industry while working at Amazon AWS, Microsoft Azure, and other companies. Eric is passionate about network automation, Python, and helping companies build better security postures. Eric is the author of several books and online classes on networking with Python and network security. He is the proud inventor of two patents in IP telephony. Eric shares his deep interest in technology through his books, classes, and his blog, and contributes to some of the popular Python open source projects.
About the reviewer
Rickard Körkkö, CCNP (Routing and Switching) and Cisco Network Programmability Design and Implementation Specialist, is a NetOps consultant at SDNit, where he's part of a group of experienced technical specialists with a great interest in and focus on emerging network technologies. His daily work includes working with orchestration tools such as Ansible to manage network devices. He's a self-taught programmer with a primary focus on Python. He has also served as a technical reviewer for the book A Practical Guide to Linux Commands, Editors, and Shell Programming, Third Edition by Mark G. Sobell.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Mastering Python Networking Second Edition
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
About the author
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Review of TCP/IP Protocol Suite and Python
An overview of the internet
Servers, hosts, and network components
The rise of data centers
Enterprise data centers
Cloud data centers
Edge data centers
The OSI model
Client-server model
Network protocol suites
The transmission control protocol
Functions and characteristics of TCP
TCP messages and data transfer
User datagram protocol
The internet protocol
The IP NAT and security
IP routing concepts
Python language overview
Python versions
Operating system
Running a Python program
Python built-in types
The None type
Numerics
Sequences
Mapping
Sets
Python operators
Python control flow tools
Python functions
Python classes
Python modules and packages
Summary
Low-Level Network Device Interactions
The challenges of the CLI
Constructing a virtual lab
Cisco VIRL
VIRL tips
Cisco DevNet and dCloud
GNS3
Python Pexpect library
Pexpect installation
Pexpect overview
Our first Pexpect program
More Pexpect features
Pexpect and SSH
Putting things together for Pexpect
The Python Paramiko library
Installation of Paramiko
Paramiko overview
Our first Paramiko program
More Paramiko features
Paramiko for servers
Putting things together for Paramiko
Looking ahead
Downsides of Pexpect and Paramiko compared to other tools
Idempotent network device interaction
Bad automation speeds bad things up
Summary
APIs and Intent-Driven Networking
Infrastructure as code
Intent-Driven Networking
Screen scraping versus API structured output
Data modeling for infrastructure as code
The Cisco API and ACI
Cisco NX-API
Lab software installation and device preparation
NX-API examples
The Cisco and YANG models
The Cisco ACI
The Python API for Juniper networks
Juniper and NETCONF
Device preparation
Juniper NETCONF examples
Juniper PyEZ for developers
Installation and preparation
PyEZ examples
The Arista Python API
Arista eAPI management
The eAPI preparation
eAPI examples
The Arista Pyeapi library
Pyeapi installation
Pyeapi examples
Vendor-neutral libraries
Summary
The Python Automation Framework – Ansible Basics
A more declarative framework
A quick Ansible example
The control node installation
Running different versions of Ansible from source
Lab setup
Your first Ansible playbook
The public key authorization
The inventory file
Our first playbook
The advantages of Ansible
Agentless
Idempotent
Simple and extensible
Network vendor support
The Ansible architecture
YAML
Inventories
Variables
Templates with Jinja2
Ansible networking modules
Local connections and facts
Provider arguments
The Ansible Cisco example
Ansible 2.5 connection example
The Ansible Juniper example
The Ansible Arista example
Summary
The Python Automation Framework – Beyond Basics
Ansible conditionals
The when clause
Ansible network facts
Network module conditional
Ansible loops
Standard loops
Looping over dictionaries
Templates
The Jinja2 template
Jinja2 loops
The Jinja2 conditional
Group and host variables
Group variables
Host variables
The Ansible Vault
The Ansible include and roles
The Ansible include statement
Ansible roles
Writing your own custom module
The first custom module
The second custom module
Summary
Network Security with Python
The lab setup
Python Scapy
Installing Scapy
Interactive examples
Sniffing
The TCP port scan
The ping collection
Common attacks
Scapy resources
Access lists
Implementing access lists with Ansible
MAC access lists
The Syslog search
Searching with the RE module
Other tools
Private VLANs
UFW with Python
Further reading
Summary
Network Monitoring with Python – Part 1
Lab setup
SNMP
Setup
PySNMP
Python for data visualization
Matplotlib
Installation
Matplotlib – the first example
Matplotlib for SNMP results
Additional Matplotlib resources
Pygal
Installation
Pygal – the first example
Pygal for SNMP results
Additional Pygal resources
Python for Cacti
Installation
Python script as an input source
Summary
Network Monitoring with Python – Part 2
Graphviz
Lab setup
Installation
Graphviz examples
Python with Graphviz examples
LLDP neighbor graphing
Information retrieval
Python parser script
Final playbook
Flow-based monitoring
NetFlow parsing with Python
Python socket and struct
ntop traffic monitoring
Python extension for ntop
sFlow
SFlowtool and sFlow-RT with Python
Elasticsearch (ELK stack)
Setting up a hosted ELK service
The Logstash format
Python helper script for Logstash formatting
Summary
Building Network Web Services with Python
Comparing Python web frameworks
Flask and lab setup
Introduction to Flask
The HTTPie client
URL routing
URL variables
URL generation
The jsonify return
Network resource API
Flask-SQLAlchemy
Network content API
Devices API
The device ID API
Network dynamic operations
Asynchronous operations
Security
Additional resources
Summary
AWS Cloud Networking
AWS setup
AWS CLI and Python SDK
AWS network overview
Virtual private cloud
Route tables and route targets
Automation with CloudFormation
Security groups and the network ACL
Elastic IP
NAT Gateway
Direct Connect and VPN
VPN Gateway
Direct Connect
Network scaling services
Elastic Load Balancing
Route53 DNS service
CloudFront CDN services
Other AWS network services
Summary
Working with Git
Introduction to Git
Benefits of Git
Git terminology
Git and GitHub
Setting up Git
Gitignore
Git usage examples
GitHub example
Collaborating with pull requests
Git with Python
GitPython
PyGitHub
Automating configuration backup
Collaborating with Git
Summary
Continuous Integration with Jenkins
Traditional change-management process
Introduction to continuous integration
Installing Jenkins
Jenkins example
First job for the Python script
Jenkins plugins
Network continuous integration example
Jenkins with Python
Continuous integration for Networking
Summary
Test-Driven Development for Networks
Test-driven development overview
Test definitions
Topology as code
Python's unittest module
More on Python testing
pytest examples
Writing tests for networking
Testing for reachability
Testing for network latency
Testing for security
Testing for transactions
Testing for network configuration
Testing for Ansible
Pytest in Jenkins
Jenkins integration
Summary
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
As Charles Dickens wrote in A Tale of Two Cities, "It was the best of times, it was the worse of times, it was the age of wisdom, it was the age of foolishness." His seemingly contradictory words perfectly describe the chaos and mood felt during a time of change and transition. We are no doubt experiencing a similar time with the rapid changes in the fields of network engineering. As software development becomes more integrated into all aspects of networking, the traditional command-line interface and vertically integrated network stack methods are no longer the best ways to manage today's networks. For network engineers, the changes we are seeing are full of excitement and opportunities and yet challenging, particularly for those who need to quickly adapt and keep up. This book has been written to help ease the transition for networking professionals by providing a practical guide that addresses how to evolve from a traditional platform to one built on software-driven practices.
In this book, we use Python as the programming language of choice to master network engineering tasks. Python is an easy-to-learn, high-level programming language that can effectively complement network engineers' creativity and problem-solving skills to streamline daily operations. Python is becoming an integral part of many large-scale networks, and through this book, I hope to share with you the lessons I've learned.
Since the publication of the first edition, I have been able to have interesting and meaningful conversations with many of the readers of the book. I am humbled by the success of the first edition of the book and took to the heart of the feedback I was given. In the second edition, I have tried to make the examples and technologies more relevant. In particular, the traditional OpenFlow SDN chapters were replaced with some of the Network DevOps tools. I sincerely hope the new addition is useful to you.
A time of change presents great opportunities for technological advancement. The concepts and tools in this book have helped me tremendously in my career, and I hope they can do the same for you.
Who this book is for
This book is ideal for IT professionals and operations engineers who already manage groups of network devices and would like to expand their knowledge on using Python and other tools to overcome network challenges. Basic knowledge of networking and Python is recommended.
What this book covers
Chapter 1, Review of TCP/IP Protocol Suite and Python, reviews the fundamental technologies that make up internet communication today, from the OSI and client-server model to TCP, UDP, and IP protocol suites. The chapter will review the basics of Python languages such as types, operators, loops, functions, and packages.
Chapter 2, Low-Level Network Device Interactions, uses practical examples to illustrate how to use Python to execute commands on a network device. It will also discuss the challenges of having a CLI-only interface in automation. The chapter will use the Pexpect and Paramiko libraries for the examples.
Chapter 3, APIs and Intent-Driven Networking, discusses the newer network devices that support Application Programming Interfaces (APIs) and other high-level interaction methods. It also illustrates tools that allow abstraction of low-level tasks while focusing on the intent of the network engineers. A discussion about and examples of Cisco NX-API, Juniper PyEZ, and Arista Pyeapi will be used in the chapter.
Chapter 4, The Python Automation Framework – Ansible Basics, discusses the basics of Ansible, an open source, Python-based automation framework. Ansible moves one step further from APIs and focuses on declarative task intent. In this chapter, we will cover the advantages of using Ansible, its high-level architecture, and see some practical examples of Ansible with Cisco, Juniper, and Arista devices.
Chapter 5, The Python Automation Framework – Beyond Basics, builds on the knowledge in the previous chapter and covers the more advanced Ansible topics. We will cover conditionals, loops, templates, variables, Ansible Vault, and roles. It will also cover the basics of writing custom modules.
Chapter 6, Network Security with Python, introduces several Python tools to help you secure your network. It will discuss using Scapy for security testing, using Ansible to quickly implement access lists, and using Python for network forensic analysis.
Chapter 7,Network Monitoring with Python – Part 1, covers monitoring the network using various tools. The chapter contains some examples using SNMP and PySNMP for queries to obtain device information. Matplotlib and Pygal examples will be shown for graphing the results. The chapter will end with a Cacti example using a Python script as an input source.
Chapter 8, Network Monitoring with Python – Part 2, covers more network monitoring tools. The chapter will start with using Graphviz to graph the network from LLDP information. We will move to use examples with push-based network monitoring using Netflow and other technologies. We will use Python to decode flow packets and ntop to visualize the results. An overview of Elasticsearch and how it can be used for network monitoring will also be covered.
Chapter 9, Building Network Web Services with Python, shows you how to use the Python Flask web framework to create our own API for network automation. The network API offers benefits such as abstracting the requester from network details, consolidating and customizing operations, and providing better security by limiting the exposure of available operations.
Chapter 10, AWS Cloud Networking, shows how we can use AWS to build a virtual network that is functional and resilient. We will cover virtual private cloud technologies such as CloudFormation, VPC routing table, access-list, Elastic IP, NAT Gateway, Direct Connect, and other related topics.
Chapter 11, Working with Git, we will illustrate how we can leverage Git for collaboration and code version control. Practical examples of using Git for network operations will be used in this chapter.
Chapter 12, Continuous Integration with Jenkins, uses Jenkins to automatically create operations pipelines that can save us time and increase reliability.
Chapter 13, Test-Driven Development for Networks, explains how to use Python's unittest and PyTest to create simple tests to verify our code. We will also see examples of writing tests for our network to verify reachability, network latency, security, and network transactions. We will also see how we can integrate the tests into continuous integration tools, such as Jenkins.
To get the most out of this book
To get the most out of this book, some basic hands-on network operation knowledge and Python is recommended. Most of the chapters can be read in any order, with the exceptions of chapters 4 and 5, which should be read in sequence. Besides the basic software and hardware tools introduced at the beginning of the book, new tools relevant to each of the chapters will be introduced.
It is highly recommended to follow and practice the examples shown in your own network lab.
Download the example code files
You can download the example code files for this book from your account atwww.packtpub.com. If you purchased this book elsewhere, you can visitwww.packtpub.com/supportand register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packtpub.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub athttps://github.com/PacktPublishing/Mastering-Python-Networking-Second-Edition.In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available athttps://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here:https://www.packtpub.com/sites/default/files/downloads/MasteringPythonNetworkingSecondEdition_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText:Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles.Here is an example:"The auto-config also generated vty access for both telnet and SSH."
A block of code is set as follows:
# This is a commentprint("hello world")
Any command-line input or output is written as follows:
$ python
Python 2.7.12 (default, Dec 4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "In the Topology Design option, I set the Management Network option to Shared Flat Network in order to use VMnet2 as the management network on the virtual routers."
Get in touch
Feedback from our readers is always welcome.
General feedback: [email protected] mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visitwww.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us [email protected] a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visitauthors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.
Review of TCP/IP Protocol Suite and Python
Welcome to the new age of network engineering. When I first started working as a network engineer 18 years ago, at the turn of the millennium, the role was distinctly different than other technical roles. Network engineers mainly possessed domain-specific knowledge to manage and operate local and wide area networks, while occasionally crossing over to systems administration, but there was no expectation to write code or understand programming concepts. This is no longer the case. Over the years, the DevOps and Software-Defined Networking (SDN) movement, among other factors, have significantly blurred the lines between network engineers, systems engineers, and developers.
The fact that you have picked up this book suggests that you might already be an adopter of network DevOps, or maybe you are considering going down that path. Maybe you have been working as a network engineer for years, just as I was, and want to know what the buzz around the Python programming language is about. Or you might already be fluent in Python but wonder what its applications are in network engineering. If you fall into any of these camps, or are simply just curious about Python in the network engineering field, I believe this book is for you:
Many books that dive into the topics of network engineering and Python have already been written. I do not intend to repeat their efforts with this book. Instead, this book assumes that you have some hands-on experience of managing networks, as well as a basic understanding of network protocols and the Python language. You do not need to be an expert in Python or network engineering, but should find that the concepts in this chapter form a general review. The rest of the chapter should set the level of expectation of the prior knowledge required to get the most out of this book. If you want to brush up on the contents of this chapter, there are lots of free or low-cost resources to bring you up to speed. I would recommend the free Khan Academy (https://www.khanacademy.org/) and the Python tutorials at: https://www.python.org/.
This chapter will pay a very quick visit to the relevant networking topics. From my experience working in the field, a typical network engineer or developer might not remember the exact TCP state machine to accomplish their daily tasks (I know I don't), but they would be familiar with the basics of the OSI model, the TCP and UDP operations, different IP headers fields, and other fundamental concepts.
We will also look at a high-level review of the Python language; just enough for those readers who do not code in Python on a daily basis to have ground to walk on for the rest of the book.
Specifically, we will cover the following topics:
An overview of the internet
The OSI and client-server model
TCP, UDP, and IP protocol suites
Python syntax, types, operators, and loops
Extending Python with functions, classes, and packages
Of course, the information presented in this chapter is not exhaustive; please do check out the references for further information.
An overview of the internet
What is the internet? This seemingly easy question might receive different answers depending on your background. The internet means different things to different people; the young, the old, students, teachers, business people, poets, could all give different answers to the question.
To a network engineer, the internet is a global computer network consisting of a web of inter-networks connecting large and small networks together. In other words, it is a network of networks without a centralized owner. Take your home network as an example. It might consist of a home Ethernet switch and a wireless access point connecting your smartphone, tablet, computers, and TV together for the devices to communicate with each other. This is your Local Area Network (LAN). When your home network needs to communicate with the outside world, it passes information from your LAN to a larger network, often appropriately named the Internet Service Provider (ISP). Your ISP often consists of edge nodes that aggregate the traffic to their core network. The core network's function is to interconnect these edge networks via a higher speed network. At special edge nodes, your ISP is connected to other ISPs to pass your traffic appropriately to your destination. The return path from your destination to your home computer, tablet, or smartphone may or may not follow the same path through all of these networks back to your device, while the source and destination remain the same.
Let's take a look at the components making up this web of networks.
Servers, hosts, and network components
Hosts are end nodes on the network that communicate to other nodes. In today's world, a host can be a traditional computer, or can be your smartphone, tablet, or TV. With the rise of the Internet of Things (IoT), the broad definition of a host can be expanded to include an IP camera, TV set-top boxes, and the ever-increasing type of sensors that we use in agriculture, farming, automobiles, and more. With the explosion of the number of hosts connected to the internet, all of them need to be addressed, routed, and managed. The demand for proper networking has never been greater.
Most of the time when we are on the internet, we make requests for services. This could be viewing a web page, sending or receiving emails, transferring files, and so on. These services are provided by servers. As the name implies, servers provide services to multiple nodes and generally have higher levels of hardware specification because of this. In a way, servers are special super-nodes on the network that provide additional capabilities to its peers. We will look at servers later on in the client-server model section.
If you think of servers and hosts as cities and towns, the network components are the roads and highways that connect them together. In fact, the term information superhighway comes to mind when describing the network components that transmit the ever increasing bits and bytes across the globe. In the OSI model that we will look at in a bit, these network components are layer one to three devices. They are layer two and three routers and switches that direct traffic, as well as layer one transports such as fiber optic cables, coaxial cables, twisted copper pairs, and some DWDM equipment, to name a few.
Collectively, hosts, servers, and network components make up the internet as we know it today.
The rise of data centers
In the last section, we looked at the different roles that servers, hosts, and network components play in the inter-network. Because of the higher hardware capacity that servers demand, they are often put together in a central location, so they can be managed more efficiently. We often refer to these locations as data centers.
Enterprise data centers
In a typical enterprise, the company generally has the need for internal tools such as emailing, document storage, sales tracking, ordering, HR tools, and a knowledge sharing intranet. These services translate into file and mail servers, database servers, and web servers. Unlike user computers, these are generally high-end computers that require a lot of power, cooling, and network connections. A byproduct of the hardware is also the amount of noise they make. They are generally placed in a central location, called the Main Distribution Frame (MDF), in the enterprise to provide the necessary power feed, power redundancy, cooling, and network connectivity.
To connect to the MDF, the user's traffic is generally aggregated at a location closer to the user, which is sometimes called the Intermediate Distribution Frame (IDF), before they are bundled up and connected to the MDF. It is not unusual for the IDF-MDF spread to follow the physical layout of the enterprise building or campus. For example, each building floor can consist of an IDF that aggregates to the MDF on another floor. If the enterprise consists of several buildings, further aggregation can be done by combining the buildings' traffic before connecting them to the enterprise data center.
Enterprise data centers generally follow the network design of three layers. These layers are access, distribution, and a core. The access layer is analogous to the ports each user connects to, the IDF can be thought of as the distribution layer, while the core layer consists of the connection to the MDF and the enterprise data centers. This is, of course, a generalization of enterprise networks, as some of them will not follow the same model.
Cloud data centers
With the rise of cloud computing and software, or infrastructure as a service, the data centers cloud providers build are at a hyper-scale. Because of the number of servers they house, they generally demand a much, much higher capacity for power, cooling, network speed, and feed than any enterprise data center. Even after working on cloud provider data centers for many years, every time I visit a cloud provider data center, I am still amazed at the scale of them. In fact, cloud data centers are so big and power-hungry that they are typically built close to power plants where they can get the cheapest power rate, without losing too much efficiency during the transportation of the power. Their cooling needs are so great, some are forced to be creative about where the data center is built, building in a generally cold climate just so they can just open the doors and windows to keep the server running at a safe temperature when needed. Any search engine can give you some of the astounding numbers when it comes to the science of building and managing cloud data centers for the likes of Amazon, Microsoft, Google, and Facebook:
At the cloud provider scale, the services that they need to provide are generally not cost efficient or able to feasibly be housed in a single server. They are spread between a fleet of servers, sometimes across many different racks, to provide redundancy and flexibility for service owners. The latency and redundancy requirements put a tremendous amount of pressure on the network. The number of interconnections equates to an explosive growth of network equipment; this translates into the number of times this network equipment needs to be racked, provisioned, and managed. A typical network design would be a multi-staged, CLOS network:
In a way, cloud data centers are where network automation becomes a necessity for speed and reliability. If we follow the traditional way of managing network devices via a Terminal and command-line interface, the number of engineering hours required would not allow the service to be available in a reasonable amount of time. This is not to mention that human repetition is error-prone, inefficient, and a terrible waste of engineering talent.
Cloud data centers are where I started my path of network automation with Python a number of years ago, and I've never looked back since.
Edge data centers
If we have sufficient computing power at the data center level, why keep anything anywhere else but at these data centers? All the connections from clients around the world can be routed back to the data center servers providing the service, and we can call it a day, right? The answer, of course, depends on the use case. The biggest limitation in routing the request and session all the way back from the client to a large data center is the latency introduced in the transport. In other words, large latency is where the network becomes a bottleneck. The latency number would never be zero: even as fast as light can travel in a vacuum, it still takes time for physical transportation. In the real world, latency would be much higher than light in a vacuum when the packet is traversing through multiple networks, and sometimes through an undersea cable, slow satellite links, 3G or 4G cellular links, or Wi-Fi connections.
The solution? Reduce the number of networks the end user traverses through. Be as closely connected to the user as possible at the edge where the user enters your network and place enough resources at the edge location to serve the request. Let's take a minute and imagine that you are building the next generation of video streaming service. In order to increase customer satisfaction with smooth streaming, you would want to place the video server as close to the customer as possible, either inside or very near to the customer's ISP. Also, the upstream of the video server farm would not just be connected to one or two ISPs, but all the ISPs that I can connect to to reduce the hop count. All the connections would be with as much bandwidth as needed to decrease latency during peak hours. This need gave rise to the peering exchange's edge data centers of big ISP and content providers. Even when the number of network devices is not as high as cloud data centers, they too can benefit from network automation in terms of the increased reliability, security, and visibility network automation brings.
We will cover security and visibility in later chapters of this book.
The OSI model
No network book is complete without first going over the Open System Interconnection (OSI) model. The model is a conceptional model that componentizes the telecommunication functions into different layers. The model defines seven layers, and each layer sits independently on top of another one, as long as they follow defined structures and characteristics. For example, in the network layer, IP, can sit on top of the different types of data link layers, such as Ethernet or frame relay. The OSI reference model is a good way to normalize different and diverse technologies into a set of common language that people can agree on. This greatly reduces the scope for parties working on individual layers and allows them to look at specific tasks in depth without worrying too much about compatibility:
The OSI model was initially worked on in the late 1970s and was later published jointly by the International Organization for Standardization (ISO) and what's now known as the Telecommunication StandardizationSector of the International Telecommunication Union (ITU-T). It is widely accepted and commonly referred to when introducing a new topic in telecommunication.
Around the same time period of the OSI model development, the internet was taking shape. The reference model the original designer used is often referred to as the TCP/IP model. The Transmission Control Protocol (TCP) and the Internet Protocol (IP) were the original protocol suites contained in the design. This is somewhat similar to the OSI model in the sense that they divide end-to-end data communication into abstraction layers. What is different is the model combines layers 5 to 7 in the OSI model in the Application layer, while the Physical and Data link layers are combined in the Link layer:
Both the OSI and TCP/IP models are useful for providing standards for end-to-end data communication. However, for the most part, we will refer to the TCP/IP model more, since that is what the internet was built on. We will specify the OSI model when needed, such as when we are discussing the web framework in upcoming chapters.
Client-server model
The reference models demonstrated a standard way for data to communicate between two nodes. Of course, by now, we all know that not all nodes are created equal. Even in its DARPA-net days, there were workstation nodes, and there were nodes with the purpose of providing content to other nodes. These server nodes typically have higher hardware specifications and are managed more closely by engineers. Since these nodes provide resources and services to others, they are typically referred to as servers. Servers typically sit idle, waiting for clients to initiate requests for their resources. This model of distributed resources that are asked for by the client is referred to as the client-server model.
Why is this important? If you think about it for a minute, the importance of networking is highlighted by this client-server model. Without it, there is really not a lot of need for network interconnections. It is the need to transfer bits and bytes from client to server that shines a light on the importance of network engineering. Of course, we are all aware of how the biggest network of them all, the internet, has been transforming the lives of all of us and continuing to do so.
How, you asked, can each node determine the time, speed, source, and destination every time they need to talk to each other? This brings us to network protocols.
Network protocol suites
In the early days of computer networking, protocols were proprietary and closely controlled by the company who designed the connection method. If you were using Novell's IPX/SPX protocol in your hosts, you would not able to communicate with Apple's AppleTalk hosts and vice versa. These proprietary protocol suites generally have analogous layers to the OSI reference model and follow the client-server communication method. They generally work great in Local Area Networks (LAN) that are closed, without the need to communicate with the outside world. When traffic does need to move beyond the local LAN, typically, an internet working device, such as a router, is used to translate from one protocol to another. An example would be a router connecting an AppleTalk network to an IP-based network. The translation is usually not perfect, but since most of the communication happens within the LAN in the early days, it is okay.
However, as the need for inter-network communication rises beyond the LAN, the need for standardizing the network protocol suites becomes greater. The proprietary protocols eventually gave way to the standardized protocol suites of TCP, UDP, and IP, which greatly enhanced the ability of one network to talk to another. The internet, the greatest network of them all, relies on these protocols to function properly. In the next few sections, we will take a look at each of the protocol suites.
The transmission control protocol
The Transmission Control Protocol (TCP) is one of the main protocols used on the internet today. If you have opened a web page or have sent an email, you have come across the TCP protocol. The protocol sits at layer 4 of the OSI model, and it is responsible for delivering the data segment between two nodes in a reliable and error-checked manner. The TCP consists of a 160-bit header consisting of, among others, source and destination ports, a sequence number, an acknowledgment number, control flags, and a checksum:
Functions and characteristics of TCP
TCP uses datagram sockets or ports to establish a host-to-host communication. The standard body, called Internet Assigned Numbers Authority (IANA) designates well-known ports to indicate certain services, such as port 80 for HTTP (web) and port 25 for SMTP (mail). The server in the client-server model typically listens on one of these well-known ports in order to receive communication requests from the client. The TCP connection is managed by the operating system by the socket that represents the local endpoint for connection.
The protocol operation consists of a state machine, where the machine needs to keep track of when it is listening for an incoming connection, during the communication session, as well as releasing resources once the connection is closed. Each TCP connection goes through a series of states such as Listen, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT, and CLOSED.
TCP messages and data transfer
The biggest difference between TCP and User Datagram Protocol (UDP), which is its close cousin on the same layer, is that it transmits data in an ordered and reliable fashion. The fact that the operation guarantees delivery is often referred to TCP as a connection-oriented protocol. It does this by first establishing a three-way handshake to synchronize the sequence number between the transmitter and the receiver, SYN, SYN-ACK, and ACK.
The acknowledgment is used to keep track of subsequent segments in the conversation. Finally, at the end of the conversation, one side will send a FIN message, and the other side will ACK the FIN message as well as sending a FIN message of its own. The FIN initiator will then ACK the FIN message that it received.
As many of us who have troubleshot a TCP connection can tell you, the operation can get quite complex. One can certainly appreciate that, most of the time, the operation just happens silently in the background.
A whole book could be written about the TCP protocol; in fact, many excellent books have been written on the protocol.
User datagram protocol
The User Datagram Protocol (UDP) is also a core member of the internet protocol suite. Like TCP, it operates on layer 4 of the OSI model that is responsible for delivering data segments between the application and the IP layer. Unlike TCP, the header is only 64-bit, which only consists of a source and destination port, length, and checksum. The lightweight header makes it ideal for applications that prefer faster data delivery without setting up the session between two hosts or needing reliable data delivery. Perhaps it is hard to imagine with today's fast internet connections, but the extra header made a big difference to the speed of transmission in the early days of X.21 and frame relay links. Although, as important as the speed difference is, not having to maintain various states, such as TCP, also saves computer resources on the two endpoints:
You might now wonder why UDP was ever used at all in the modern age; given the lack of reliable transmission, wouldn't we want all the connections to be reliable and error-free? If you think about multimedia video streaming or Skype calling, those applications benefit from a lighter header when the application just wants to deliver the datagram as quickly as possible. You can also consider the fast DNS lookup process based on the UDP protocol. When the address you type in on the browser is translated into a computer understandable address, the user will benefit from a lightweight process, since this has to happen before even the first bit of information is delivered to you from your favorite website.
Again, this section does not do justice to the topic of UDP, and the reader is encouraged to explore the topic through various resources if you are is interested in learning more about UDP.
The internet protocol
As network engineers will tell you, they live at the Internet Protocol (IP) layer, which is layer 3 on the OSI model. IP has the job of addressing and routing between end nodes, among others. The addressing of an IP is probably its most important job. The address space is divided into two parts: the network and the host portion. The subnet mask is used to indicate which portion in the network address consists of the network and which portion is the host by matching the network portion with a 1 and the host portion with a 0. Both IPv4 and, later, IPv6 expresses the address in the dotted notation, for example, 192.168.0.1. The subnet mask can either be in a dotted notation (255.255.255.0) or use a forward slash to express the number of bits that should be considered in the network bit (/24):
The IPv6 header, the next generation of the IP header of IPv4, has a fixed portion and various extension headers:
The Next Header field in the fixed header section can indicate an extension header to be followed that carries additional information. The extension headers can include routing and fragment information. As much as the protocol designer would like to move from IPv4 to IPv6, the internet today is still pretty much addressed with IPv4, with some of the service provider networks addressed with IPv6 internally.
The IP NAT and security
Network Address Translation (NAT) is typically used for translating a range of private IPv4 addresses into publicly routable IPv4 addresses. But it can also mean a translation between IPv4 to IPv6, such as at a carrier edge when they use IPv6 inside of the network that needs to be translated to IPv4 when the packet leaves the network. Sometimes, NAT6 to 6 is used as well for security reasons.
Security is a continuous process that integrates all the aspects of networking, including automation and Python. This book aims to use Python to help you manage the network; security will be addressed as part of the following chapters in the book, such as using SSHv2 over telnet. We will also look at how we can use Python and other tools to gain visibility in the network.
IP routing concepts
In my opinion, IP routing is about having the intermediate devices between the two endpoints transmit the packets between them based on the IP header. For all communication via the internet, the packet will traverse through various intermediate devices. As mentioned, the intermediate devices consist of routers, switches, optical gears, and various other gears that do not examine beyond the network and transport layer. In a road trip analogy, you might travel in the United States from the city of San Diego in California to the city of Seattle in Washington. The IP source address is analogous to San Diego and the destination IP address can be thought of as Seattle. On your road trip, you will stop by many different intermediate spots, such as Los Angeles, San Francisco, and Portland; these can be thought of as the routers and switches between the source and destination.
Why was this important? In a way, this book is about managing and optimizing these intermediate devices. In the age of mega data centers that span the size of multiple American football fields, the need for efficient, agile, reliable, and cost-effective ways to manage the network becomes a major point of competitive advantage for companies. In future chapters, we will dive into how we can use Python programming to effectively manage a network.
Python language overview
In a nutshell, this book is about making our network engineering lives easier with Python. But what is Python and why is it the language of choice of many DevOps engineers? In the words of the Python Foundation Executive Summary (https://www.python.org/doc/essays/blurb/):
If you are somewhat new to programming, the object-oriented, dynamic semantics mentioned previously probably do not mean much to you. But I think we can all agree that for rapid application development, simple, and easy-to-learn syntax sounds like a good thing. Python, as an interpreted language, means there is no compilation process required, so the time to write, test, and edit Python programs is greatly reduced. For simple scripts, if your script fails, a print statement is usually all you need to debug what was going on. Using the interpreter also means that Python is easily ported to different types of operating systems, such as Windows and Linux, and a Python program written on one operating system can be used on another.
The object-oriented nature encourages code reuse by breaking a large program into simple reusable objects, as well as other reusable formats with functions, modules, and packages. In fact, all Python files are modules that can be reused or imported into another Python program. This makes it easy to share programs between engineers and encourages code reuse. Python also has a batteries included mantra, which means that for common tasks, you need not download any additional packages. In order to achieve this without the code being too bloated, a set of standard libraries is installed when you install the Python interpreter. For common tasks such as regular expression, mathematics functions, and JSON decoding, all you need is the import statement, and the interpreter will move those functions into your program. This is what I would consider one of the killer features of the Python language.
Lastly, the fact that Python code can start in a relatively small-sized script with a few lines of code and grow into a full production system is very handy for network engineers. As many of us know, the network typically grows organically without a master plan. A language that can grow with your network in size is invaluable. You might be surprised to see a language that was deemed as a scripting language by many is being used for full production systems by many cutting-edge companies (organizations using Python, https://wiki.python.org/moin/OrganizationsUsingPython).
If you have ever worked in an environment where you have to switch between working on different vendor platforms, such as Cisco IOS and Juniper Junos, you know how painful it is to switch between syntaxes and usage when trying to achieve the same task. With Python being flexible enough for large and small programs, there is no such context switching, because it is just Python.
For the rest of the chapter, we will take a high-level tour of the Python language for a bit of a refresher. If you are already familiar with the basics, feel free to quickly scan through it or skip the rest of the chapter.
Python versions
As many readers are already aware, Python has been going through a transition from Python 2 to Python 3 for the last few years. Python 3 was released back in 2008, over 10 years ago, with active development with the most recent release of 3.7. Unfortunately, Python 3 is not backward compatible with Python 2. At the time of writing the second edition of this book, in the middle of 2018, the Python community has largely moved over to Python 3. The latest Python 2.x release, 2.7, was released over six years ago in mid-2010. Fortunately, both versions can coexist on the same machine. Personally, I use Python 2 as my default interpreter when I type in Python at the Command Prompt, and I use Python 3 when I need to use Python 3. More information is given in the next section about invoking Python interpreter, but here is an example of invoking Python 2 and Python 3 on an Ubuntu Linux machine:
$ pythonPython 2.7.12 (default, Dec 4 2017, 14:50:18)[GCC 5.4.0 20160609] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> exit()
$ python3Python 3.5.2 (default, Nov 23 2017, 16:37:01)[GCC 5.4.0 20160609] on linuxType "help", "copyright", "credits" or "license" for more information.>>> exit()
With the 2.7 release being end-of-life, most Python frameworks now support Python 3. Python 3 also has lots of good features, such as asynchronous I/O that can be taken advantage of when we need to optimize our code. This book will use Python 3 for its code examples unless otherwise stated. We will also try to point out the Python 2 and Python 3 differences when applicable.
If a particular library or framework is better suited for Python 2, such as Ansible (see the following information), it will be pointed out and we will use Python 2 instead.
Operating system
As mentioned, Python is cross-platform. Python programs can be run on Windows, Mac, and Linux. In reality, certain care needs to be taken when you need to ensure cross-platform compatibility, such as taking care of the subtle difference between backslashes in Windows filenames. Since this book is for DevOps, systems, and network engineers, Linux is the preferred platform for the intended audience, especially in production. The code in this book will be tested on the Linux Ubuntu 16.06 LTS machine. I will also try my best to make sure the code runs the same on the Windows and the MacOS platform.
If you are interested in the OS details, they are as follows:
$ uname -aLinux packt-network-python 4.13.0-45-generic #50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
