31,19 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Invent your own Python scripts to automate your infrastructure
Key FeaturesMake the most of Python libraries and modules to automate your infrastructureLeverage Python programming to automate server configurations and administration tasksEfficiently develop your Python skill setBook Description
Hands-On Enterprise Automation with Python starts by covering the set up of a Python environment to perform automation tasks, as well as the modules, libraries, and tools you will be using.
We’ll explore examples of network automation tasks using simple Python programs and Ansible. Next, we will walk you through automating administration tasks with Python Fabric, where you will learn to perform server configuration and administration, along with system administration tasks such as user management, database management, and process management. As you progress through this book, you’ll automate several testing services with Python scripts and perform automation tasks on virtual machines and cloud infrastructure with Python. In the concluding chapters, you will cover Python-based offensive security tools and learn how to automate your security tasks.
By the end of this book, you will have mastered the skills of automating several system administration tasks with Python.
What you will learnUnderstand common automation modules used in PythonDevelop Python scripts to manage network devicesAutomate common Linux administration tasks with Ansible and FabricManaging Linux processesAdministrate VMware, OpenStack, and AWS instances with PythonSecurity automation and sharing code on GitHubWho this book is for
Hands-On Enterprise Automation with Python is for system administrators and DevOps engineers who are looking for an alternative to major automation frameworks such as Puppet and Chef. Basic programming knowledge with Python and Linux shell scripting is necessary.
Bassem Aly is an experienced SDN/NFV solution consultant at Juniper Networks and has been working in the Telco industry for last 9 years. He focused on designing and implementing next generation by leveraging different automation and devops frameworks.Also he has extensive experience in architecting and deploying telco applications over the openstack. Bassem also conducts corporate training on network automation & network programmability using python and ansible.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 339
Veröffentlichungsjahr: 2018
Ähnliche
Hands-On Enterprise Automation with Python
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Vijin BorichaAcquisition Editor:Rohit RajkumarContent Development Editor:Ron KurienTechnical Editor:Manish D ShanbhagCopy Editor:Safis EditingProject Coordinator: Judie JoseProofreader: Safis EditingIndexer: Pratik ShirodkarGraphics:Tom ScariaProduction Coordinator: Aparna Bhagat
First published: June 2018
Production reference: 1270618
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78899-851-2
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the author
Bassem Aly is an experienced SDN/NFV solution consultant at Juniper Networks and has been working in the telco industry for the last 9 years. He has focused on designing and implementing next-generation solutions by leveraging different automation and DevOps frameworks. Also, he has extensive experience of architecting and deploying telco applications over OpenStack. He also conducts corporate training on network automation and network programmability using Python and Ansible.
About the reviewer
Jere Julian is a senior network automation engineer with nearly two decades of automation experience currently focused on workflow simplification through automation. The past few years have found him on the speaker circuit at DevOps Days and Interop ITX, as well as regularly contributing to network computing. He lives in NC with his wife and two boys and fights fire as a community volunteer as opposed to the data center. He can be contacted on Twitter at @julianje.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Hands-On Enterprise Automation with Python
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
About the author
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Setting Up Our Python Environment
An introduction to Python
Python versions
Why are there two active versions?
Should you only learn Python 3?
Does this mean I can't write code that runs on both Python 2 and Python 3?
Python installation
Installing the PyCharm IDE
Setting up a Python project inside PyCharm
Exploring some nifty PyCharm features
Code debugging
Code refactoring
Installing packages from the GUI
Summary
Common Libraries Used in Automation
Understanding Python packages
Package search paths
Common Python libraries
Network Python Libraries
System and cloud Python libraries
Accessing module source code
Visualizing Python code
Summary
Setting Up the Network Lab Environment
Technical requirements
When and why to automate the network
Why do we need automation?
Screen scraping versus API automation
Why use Python for network automation?
The future of network automation
Network lab setup
Getting ready – installing EVE-NG
Installation on VMware Workstation
Installation over VMware ESXi
Installation over Red Hat KVM
Accessing EVE-NG
Installing EVE-NG client pack
Loading network images into EVE-NG
Building an enterprise network topology
Adding new nodes
Connecting nodes together
Summary
Using Python to Manage Network Devices
Technical requirements
Python and SSH
Paramiko module
Module installation
SSH to the network device
Netmiko module
Vendor support
Installation and verification
Using netmiko for SSH
Configuring devices using netmiko
Exception handling in netmiko
Device auto detect
Using the telnet protocol in Python
Push configuration using telnetlib
Handling IP addresses and networks with netaddr
Netaddr installation
Exploring netaddr methods
Sample use cases
Backup device configuration
Building the python script
Creating your own access terminal
Reading data from an Excel sheet
More use cases
Summary
Extracting Useful Data from Network Devices
Technical requirements
Understanding parsers
Introduction to regular expressions
Creating a regular expression in Python
Configuration auditing using CiscoConfParse
CiscoConfParse library
Supported vendors
CiscoConfParse installation
Working with CiscoConfParse
Visualizing returned data with matplotLib
Matplotlib installation
Hands-on with matplotlib
Visualizing SNMP using matplotlib
Summary
Configuration Generator with Python and Jinja2
What is YAML?
YAML file formatting
Text editor tips
Building a golden configuration with Jinja2
Reading templates from the filesystem
Using Jinja2 loops and conditions
Summary
Parallel Execution of Python Script
How a computer executes your Python script
Python multiprocessing library
Getting started with multiprocessing
Intercommunication between processes
Summary
Preparing a Lab Environment
Getting the Linux operating system
Downloading CentOS
Downloading Ubuntu
Creating an automation machine on a hypervisor
Creating a Linux machine over VMware ESXi
Creating a Linux machine over KVM
Getting started with Cobbler
Understanding how Cobbler works
Installing Cobbler on an automation server
Provisioning servers through Cobbler
Summary
Using the Subprocess Module
The popen() subprocess
Reading stdin, stdout, and stderr
The subprocess call suite
Summary
Running System Administration Tasks with Fabric
Technical requirements
What is Fabric?
Installation
Fabric operations
Using run operation
Using get operation
Using put operation
Using sudo operation
Using prompt operation
Using reboot operation
Executing your first Fabric file
More about the fab tool
Discover system health using Fabric
Other useful features in Fabric
Fabric roles
Fabric context managers
Summary
Generating System Reports and System Monitoring
Collecting data from Linux
Sending generated data through email
Using the time and date modules
Running the script on a regular basis
Managing users in Ansible
Linux systems
Microsoft Windows
Summary
Interacting with the Database
Installing MySQL on an automation server
Securing the installation
Verifying the database installation
Accessing the MySQL database from Python
Querying the database
Inserting records into the database
Summary
Ansible for System Administration
Ansible terminology
Installing Ansible on Linux
On RHEL and CentOS
Ubuntu
Using Ansible in ad hoc mode
How Ansible actually works
Creating your first playbook
Understanding Ansible conditions, handlers, and loops
Designing conditions
Creating loops in ansible
Trigger tasks with handlers
Working with Ansible facts
Working with the Ansible template
Summary
Creating and Managing VMware Virtual Machines
Setting up the environment
Generating a VMX file using Jinja2
Building the VMX template
Handling Microsoft Excel data
Generating VMX files
VMware Python clients
Installing PyVmomi
First steps with pyvmomi
Changing the virtual machine state
There's more
Using Ansible playbook to manage instances
Summary
Interacting with the OpenStack API
Understanding RESTful web services
Setting up the environment
Installing rdo-OpenStack package
On RHEL 7.4
On CentOS 7.4
Generating answer file
Editing answer file
Run the packstack
Access the OpenStack GUI
Sending requests to the OpenStack keystone
Creating instances from Python
Creating the image
Assigning a flavor
Creating the network and subnet
Launching the instance
Managing OpenStack instances from Ansible
Shade and Ansible installation
Building the Ansible playbook
Running the playbook
Summary
Automating AWS with Boto3
AWS Python modules
Boto3 installation
Managing AWS instances
Instance termination
Automating AWS S3 services
Creating buckets
Uploading a file to a bucket
Deleting a bucket
Summary
Using the Scapy Framework
Understanding Scapy
Installing Scapy
Unix-based systems
Installing in Debian and Ubuntu
Installing in Red Hat/CentOS
Windows and macOS X Support
Generating packets and network streams using Scapy
Capturing and replaying packets
Injecting data inside packets
Packet sniffing
Writing the packets to pcap
Summary
Building a Network Scanner Using Python
Understanding the network scanner
Building a network scanner with Python
Enhancing the code
Scanning the services
Sharing your code on GitHub
Creating an account on GitHub
Creating and pushing your code
Summary
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
The book starts by covering the set up of a Python environment to perform automation tasks, as well as the modules, libraries, and tools you will be using. We'll explore examples of network automation tasks using simple Python programs and Ansible. Next, we will walk you through automating administration tasks with Python Fabric, where you will learn to perform server configuration and administration along with system administration tasks such as user management, database management, and process management. As you progress through this book, you'll automate several testing services with Python scripts and perform automation tasks on virtual machines and the cloud infrastructure with Python. In the concluding chapters, you will cover Python-based offensive security tools and learn to automate your security tasks. By the end of this book, you will have mastered the skills of automating several system administration tasks with Python.
Who this book is for
Hands-On Enterprise Automationwith Python is for system administrators and DevOps engineers who are looking for an alternative to major automation frameworks such as Puppet and Chef. Basic programming knowledge with Python and Linux shell scripting is necessary.
What this book covers
Chapter 1, Setting Up Python Environment, explores how to download and install the Python interpreter along with the Python Integrated Development Environment, called JetBrains PyCharm. The IDE provides you with smart autocompletion, intelligent code analysis, powerful refactoring and integrates with Git, virtualenv, Vagrant, and Docker. This will help you to write professional and robust Python code.
Chapter 2, Common Libraries Used in Automation, covers the Python libraries that are available today and that are used for automation. We will categorize them based on their usage (system, network, and cloud) and provide a simple introduction. As you progress through the book, you will find yourself deep diving into each of them and understanding their usage.
Chapter 3, Setting up Your Network Lab Environment, discusses the merits of network automation and how network operators use it today to automate the current devices. We will explore popular libraries that are used today to automate network nodes from Cisco, Juniper, and Arista. This chapter covers how to build a networking lab to apply the Python script on. We will use an open source network emulation tool called EVE-NG.
Chapter 4, Using Python to Manage Network Devices, dives into managing networking devices through telnet and SSH connections using netmiko, paramiko, and telnetlib. We will learn how to write the Python code to access switches and routers and execute commands on the terminal and then return the output. We will also learn how to utilize different Python techniques to back up and push configuration. The chapter ends with some use cases used today in modern network environment.
Chapter 5, Extracting Useful Data from Network Devices, explains how to use different tools and techniques inside Python to extract useful data from returned output and act upon it. Also, we will use a special library called CiscoConfParse to audit the configuration. Then we will learn how to visualize data to generate appealing graphs and reports with matplotlib.
Chapter 6, Configuration Generator with Python and Jinja2, explains how to generate a common configuration for a site with hundreds of network nodes. We will learn how to write a template and use it to generate a golden configuration with a templating language called Jinja2.
Chapter 7, Parallel Execution of the Python Script, covers how to instantiate and execute your Python code in parallel. This will allow us to finish the automation workflow faster as long as it is not interdependent.
Chapter 8, Preparing a Lab Environment, covers the installation process and preparation for our lab environment. We will install our automation server either in CentOS or Ubuntu over different hypervisors. Then we will learn how to automate the operating system installation with Cobbler.
Chapter 9, Using the Subprocess Module, explains how to send a command from a Python script directly to the operating system shell and investigate the returning output.
Chapter 10, Running System Administration Tasks with Fabric, introduces Fabric, which is a Python library for executing system administration tasks through SSH. Also, it's used in large deployment of applications. We will learn how to utilize and leverage this library to execute tasks on remote servers.
Chapter 11, Generating System Reports, Managing Users, and System Monitoring, explains that collecting data and generating recurring reports from the system is an essential task for any system administrator, and automating this task will help you to discover issues early and provide a solutions for them. In this chapter, we will see some proven ways to automate data collection from servers and generate formal reports. We will learn how to manage new and existing users using Python and Ansible. Also, we will dive into monitoring the system KPI and logs analysis. You can also schedule the monitoring scripts to run on a regular basis and send the result to your mail inbox.
Chapter 12, Interacting with the Database, states that if you're a database administrator or database developer, then Python provides a wide range of libraries and modules that cover managing and working on popular DBMSes such as MySQL, Postgress, and Oracle. In this chapter, we will learn how to interact with DBMSes using Python connectors.
Chapter 13, Ansible for System Administration, explores one of the most powerful tools in configuration management software. Ansible is very powerful when it comes to system administration and can be used to make sure the configuration is replicated exactly across hundreds or even thousands of servers at the same time.
Chapter 14, Creating and Managing VMWare Virtual Machines, explains how to automate VM creation on a VMWare hypervisor. We will discover different ways to create and manage virtual machines over ESXi using VMWare's official binding library.
Chapter 15, Interacting with Openstack API, explains that OpenStack was very popular in creating private IaaS when it came to private cloud. We will use Python modules such as requests to create REST calls and interact with OpenStack services such as nova, cinder, and neutron, and create the required resources over OpenStack. Later in the chapter, we will use Ansible playbooks for the same workflow.
Chapter 16, Automating AWS with Python and Boto3, covers how to automate common AWS services such as EC2 and S3 using official Amazon binindgs (BOTO3), which provides an easy-to-use API for services access.
Chapter 17, Using the SCAPY Framework, introduces SCAPY, which is a powerful Python tool used to build and craft packets and then send them on the wire. You can build any type of network stream and send it on the wire. It can also help you to capture network packets and replay them to the wire.
Chapter 18, Building Network Scanner Using Python, provides a complete example of building a network scanner using Python. You can scan a complete subnet for different protocols and ports and generate a report for each scanned host. Then, we will learn how to share the code with the open source community (GitHub) by leveraging Git.
To get the most out of this book
The reader should be acquainted with the basic programming paradigm of Python programming language and should have basic knowledge of Linux and Linux shell scripting.
Download the example code files
You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packtpub.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Hands-On-Enterprise-Automation-with-Python. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/HandsOnEnterpriseAutomationwithPython_ColorImages.pdf.
Get in touch
Feedback from our readers is always welcome.
General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.
Setting Up Our Python Environment
In this chapter, we will provide a brief introduction to the Python programming language and the differences between the current versions. Python ships in two active versions, and making a decision on which one to use during development is important. In this chapter, we will download and install Python binaries into the operating system.
At the end of the chapter, we will install one of the most advanced Integrated Development Editors (IDEs) used by professional developers around the world: PyCharm. PyCharm provides smart code completion, code inspections, on-the-fly error highlighting and quick fixes, automated code refactoring, and rich navigation capabilities, which we will go over throughout this book, as we write and develop Python code.
The following topics will be covered in this chapter:
An introduction to Python
Installing the PyCharm IDE
Exploring some nifty PyCharm features
An introduction to Python
Python is a high-level programming language that provides a friendly syntax; it is easy to learn and use, for both beginner and expert programmers.
Python was originally developed by Guido van Rossum in 1991; it depends on a mix of C, C++, and other Unix shell tools. Python is known as a language for general purpose programming, and today it's used in many fields, such as software development, web development, network automation, system administration, and scientific fields. Thanks to its large number of modules available for download, covering many fields, Python can cut development time down to a minimum.
The Python syntax was designed to be readable; it has some similarities to the English language, while the code construction itself is beautiful. Python core developers provide 20 informational rules, called the Zen of Python, that influenced the design of the Python language; most of them involve building clean, organized, and readable code. The following are some of the rules:
You can read more about the Zen of Python at https://www.python.org/dev/peps/pep-0020/.
Python versions
Python comes with two major versions: Python 2.x and Python 3.x. There are subtle differences between the two versions; the most obvious is the way their print functions treat multiple strings. Also, all new features will only be added to 3.x, while 2.x will receive security updates before full retirement. This won't be an easy migration, as many applications are built on Python 2.x.
Why are there two active versions?
I will quote the reason from the official Python website:
You can read more about this topic at https://wiki.python.org/moin/Python2orPython3.
Should you only learn Python 3?
It depends. Learning Python 3 will future-proof your code, and you will use up-to-date features from the developers. However, note that some third-party modules and frameworks lack support for Python 3 and will continue to do so for the near future, until they completely port their libraries to Python 3.
Also, note that some network vendors, such as Cisco, provide limited support for Python 3.x, as most of the required features are already covered in Python 2.x releases. For example, the following are the supported Python versions for Cisco devices; you will see that all devices support 2.x, not 3.x:
Does this mean I can't write code that runs on both Python 2 and Python 3?
No, you can, of course, write your code in Python 2.x and make it compatible with both versions, but you will need to import a few libraries first, such as the __future__ module, to make it backward compatible. This module contains some functions that tweak the Python 2.x behavior and make it exactly like Python 3.x. Take a look at the following examples to understand the differences between the two versions:
#python 2 only
"Welcome to Enterprise Automation"
The following code is for Python 2 and 3:
# python 2 and 3
(
"Welcome to Enterprise Automation"
)
Now, if you need to print multiple strings, the Python 2 syntax will be as follows:
# python 2, multiple strings
"welcome"
,
"to"
,
"Enterprise"
,
"Automation"
# python 3, multiple strings
(
"welcome"
,
"to"
,
"Enterprise"
,
"Automation"
)
If you try to use parentheses to print multiple strings in Python 2, it will interpret it as a tuple, which is wrong. For that reason, we will import the __future__ module at the beginning of our code, to prevent that behavior and instruct Python to print multiple strings.
The output will be as follows:
Python installation
Whether you choose to go with a popular Python version (2.x) or build future-proof code with Python 3.x, you will need to download the Python binaries from the official website and install them in your operating system. Python provides support for different platforms (Windows, Mac, Linux, Raspberry PI, and so on):
Go to
https://www.python.org/downloads/
and choose the latest version of either 2.x or 3.x:
Choose your platform from the
Download
page, and either the x86 or x64 version:
Install the package as usual. It's important to select the
Add python to the path
option during the installation, in order to access Python from the command line (in the case of Windows). Otherwise, Windows won't recognize the Python commands and will throw an error:
Verify that the installation is complete by opening the command line or terminal in your operating system and typing
python
. This should access the Python console and provide a verification that Python has successfully installed on your system:
Installing the PyCharm IDE
PyCharm is a fully fledged IDE, used by many developers around the world to write and develop Python code. The IDE is developed by the Jetbrains company and provides rich code analysis and completion, syntax highlighting, unit testing, code coverage, error discovery, and other Python linting operations.
Also, PyCharm Professional Edition supports Python web frameworks, such as Django, web2py, and Flask, beside integrations with Docker and vagrant for running a code over them. It provides amazing integration with multiple version control systems, such as Git (and GitHub), CVS, and subversion.
In the next few steps, we will install PyCharm Community Edition:
Go to the PyCharm download page (
https://www.jetbrains.com/pycharm/download/
) and choose your platform. Also, choose to download either the Community Edition (free forever) or the Professional Edition (the Community version is completely fine for running the codes in this book):
Install the software as usual, but make sure that you select the following options:
32-
or
64-bit
launcher (depending on your operating system).
Create Associations
(this will make PyCharm the default application for Python files).
Download and install JRE x86 by JetBrains
:
Wait until PyCharm downloads the additional packages from the internet, and installs it, then choose
Run PyCharm Community Edition
:
Since this is a new and fresh installation, we won't import any settings from
Select the desired UI theme (either the
default
or
darcula
,
for dark mode). You can install some additional plugins, such as
Markdown
and
BashSupport,
which will make PyCharm recognize and support those languages. When you finish, click on
Start Using PyCharm
:
Setting up a Python project inside PyCharm
Inside PyCharm, a Python project is a collection of Python files that you have developed and Python modules that are either built in or were installed from a third party. You will need to create a new project and save it to a specific location inside your machine before starting to develop your code. Also, you will need to choose the default interpreter for this project. By default, PyCharm will scan the default location on the system and search for the Python interpreter. The other option is to create a completely isolated environment, using Python virtualenv. The basic problem with the virtualenv address is its package dependencies. Let's assume that you're working on multiple different Python projects, and one of them needs a specific version of x package. On the other hand, one of the other projects needs a completely different version from the same package. Notice that all installed Python packages go to /usr/lib/python2.7/site-packages, and you can't store different versions of the same package. The virtualenv will solve this problem by creating an environment that has its own installation directories and its own package; each time you work on either of the two projects, PyCharm (with the help of virtualenv) will activate the corresponding environment to avoid any conflict between packages.
Follow these steps to set up the project:
Choose
Create New Project
:
Choose the project settings:
Select the type of project; in our case, it will be
Pure Python
.
Choose the project's location on the local hard drive.
Choose the Project Interpreter. Either use the existing Python installation in the default directory, or create a new virtual environment tied specifically to that project.
Click on
Create
.
Create a new
Python File
inside the project:
Right-click on the project name and select
New
.
Choose
Python File
from the menu, then choose a filename.
A new, blank file is opened, and you can write a Python code directly into it. Try to import the __future__ module, for example, and PyCharm will automatically open a pop-up window with all possible completions available as shown in the following screenshot:
Run your code:
Enter the code that you wish to run.
Choose
Edit Configuration
to configure the runtime settings for the Python file.
Configure new Python settings for running your file:
Click on the + sign to add a new configuration, and choose
Python
.
Choose the configuration name.
Choose the script path inside your project.
Click on
OK
.
Run the code:
Click on the
play
button beside the configuration name.
PyCharm will execute the code inside the file specified in the configuration, and will return the output to the terminal.
Exploring some nifty PyCharm features
In this section, we will explore some of PyCharm's features. PyCharm has a huge collection of tools out of the box, including an integrated debugger and test runner, Python profiler, a built-in Terminal, integration with major VCS and built-in database tools, remote development capabilities with remote interpreters, an integrated SSH Terminal, and integration with Docker and Vagrant. For a list of other features, please check the official site (https://www.jetbrains.com/pycharm/features/).
Code debugging
Code debugging is a process that can help you to understand the cause of an error, by providing an input to the code and walking through each line of the code and seeing how it evaluates at the end. The Python language contains some debugging tools to get insights from the code, starting with a simple print function, assert command till a complete unit testing for the code. PyCharm provides an easy way to debug the code and see the evaluated values.
To debug code in PyCharm (say, a nested for loop with if clauses), you need to set a breakpoint on the line at which you want PyCharm to stop the program execution. When PyCharm hits this line, it will pause the program and dump the memory to see the contents of each variable:
Notice that the value of each variable is printed besides it, on the first iteration:
Also, you can right-click on the breakpoint and add a specific condition for any variable. If the variable is evaluated to a specific value, then a log message will be printed:
Code refactoring
Refactoring the code is the process of changing the structure of a specific variable name inside your code. For example, you may choose a name for your variable and use it for a project that consists of multiple source files, then later decide to rename the variable to something more descriptive. PyCharm provides many refactoring techniques, to make sure that the code can be updated without breaking the operation.
PyCharm does the following:
The refactoring itself
Scans every file inside the project and makes sure that the references to the variables are updated
If something can't be updated automatically, it will give you a warning and open a menu, so you can decide what to do
Saves the code before refactoring it, so you can revert it later
Let's look at an example. Assume that we have three Python files in our project, called refactor_1.py, refactor_2.py, and refactor_3.py. The first file contains important_funtion(x), which is also used in both refactor_2.py and refactor_3.py.
Copy the following code in a refactor_1.py file:
def
important_function(
x
)
:
(
x
)
Copy the following code in a refactor_2.py file:
from
refactor_1
import
important_function
important_function
(
2
)
Copy the following code in a refactor_3.py file:
from
refactor_1
import
important_function
important_function
(
10
)
To perform the refactoring, you need to right-click on the method itself, select Refactor | Rename, and enter the new name for the method:
Notice that a window opens at the bottom of the IDE, listing all references of this function, the current value for each one, and which file will be affected after the refactoring:
If you choose Do Refactor, all of the references will be updated with the new name, and your code will not be broken.
Installing packages from the GUI
PyCharm can be used to install packages for existing interpreters (or the virtualenv) using the GUI. Also, you can see a list of all installed packages, and whether upgrades are available for them.
First, you need to go to File | Settings | Project | Project Interpreter:
As shown in the preceding screenshot, PyCharm provides a list of installed packages and their current versions. You can click on the + sign to add a new package to the project interpreter, then enter the package initials into the search box:
You should see a list of available packages, containing a name and description for each one. Also, you can specify a specific version to be installed on your interpreter. Once you have clicked on Install Package, PyCharm will execute a pip command on your system (and may ask you for a permission); then, it will download the package onto the installation directory and execute the setup.py file.
Summary
In this chapter, you learned the differences between Python 2 and Python 3, and how to decide which one to use, based on your needs. Also, you learned how to install a Python interpreter and how to use PyCharm as an advanced editor to write and manage your code's life cycle.
In the next chapter, we will discuss the Python package structure and the common Python packages used in automation.
Common Libraries Used in Automation
This chapter will walk you through how Python packages are structured and the common libraries used today to automate the system and network infrastructure. There's a long growing list of Python packages that cover network automation, system administration, and managing public and private clouds.
Also, it's important to understand how to access the module source code and how the small pieces inside the Python package are related to each other so we can modify the code, add or remove features, and share the code again with the community.
The following topics will be covered in this chapter:
Understanding Python packages
Common Python libraries
Accessing module source code
Understanding Python packages
Python core code is actually small by design to maintain simplicity. Most functionalities will be through adding third-party packages and modules.
Module is a Python file that contains functions, statements, and classes that will be used inside your code. The first thing to do is import the module then start to use its functions.
On other hand, a package collects related modules connected to each other and puts them in a single hierarchy. Some large packages such as matplotlib or django have hundreds of modules inside them, and developers usually categorize the related modules into a sub-directories. For example, the netmiko package contains multiple sub-directories and each one contains modules to connect to network devices from different vendors:
Doing that gives the package maintainer the flexibility to add or remove features from each module without breaking the global package operation.
Package search paths
Typically, Python searches for modules in some specific system paths. You can print these paths by importing the sys module and printing the sys.path
