36,59 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Refine your Python programming skills and build professional grade applications with this comprehensive guide
Key Features
- Create manageable code that can run in various environments with different sets of dependencies
- Implement effective Python data structures and algorithms to write optimized code
- Discover the exciting new features of Python 3.7
Book Description
Python is a dynamic programming language that's used in a wide range of domains thanks to its simple yet powerful nature. Although writing Python code is easy, making it readable, reusable, and easy to maintain is challenging. Complete with best practices, useful tools, and standards implemented by professional Python developers, the third edition of Expert Python Programming will help you overcome this challenge.
The book will start by taking you through the new features in Python 3.7. You'll then learn the advanced components of Python syntax, in addition to understanding how to apply concepts of various programming paradigms, including object-oriented programming, functional programming, and event-driven programming. This book will also guide you through learning the best naming practices, writing your own distributable Python packages, and getting up to speed with automated ways of deploying your software on remote servers. You'll discover how to create useful Python extensions with C, C++, Cython, and CFFI. Furthermore, studying about code management tools, writing clear documentation, and exploring test-driven development will help you write clean code.
By the end of the book, you will have become an expert in writing efficient and maintainable Python code.
What you will learn
- Explore modern ways of setting up repeatable and consistent development environments
- Package Python code effectively for community and production use
- Learn modern syntax elements of Python programming such as f-strings, enums, and lambda functions
- Demystify metaprogramming in Python with metaclasses
- Write concurrent code in Python
- Extend Python with code written in different languages
- Integrate Python with code written in different languages
Who this book is for
This book will appeal to you if you're a programmer looking to take your Python knowledge to the next level by writing efficient code and learning the latest features of version 3.7 and above.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 859
Veröffentlichungsjahr: 2019
Ähnliche
Expert Python Programming Third Edition
Copyright © 2019 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Kunal ChaudhariAcquisition Editor: Chaitanya NairContent Development Editor: Zeeyan PinheiroTechnical Editor: Ketan KambleCopy Editor: Safis EditingProject Coordinator: Vaidehi SawantProofreader: Safis EditingIndexer: Priyanka DhadkeGraphics: Alishon MendonsaProduction Coordinator: Shraddha Falebhai
First published: September 2008 Second edition: May 2016 Third edition: April 2019
Production reference: 1270419
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78980-889-6
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
Packt.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the authors
Michał Jaworski has 10 years' of professional experience in Python. He has been in various roles at different companies, from an ordinary full-stack developer, through software architect, to VP of engineering in a fast-paced start-up company. He is currently a senior backend engineer at Showpad. He is highly experienced in designing high-performance distributed services. He is also an active contributor to many open source Python projects.
Tarek Ziadé is a Python developer located in the countryside near Dijon, France. He works at Mozilla in the services team. He founded a French Python user group called Afpy, and has written several books about Python in French and English. When he is not hacking on his computer or hanging out with his family, he's spending time between his two passions, running and playing the trumpet.
You can visit his personal blog (Fetchez le Python) and follow him on Twitter (tarek_ziade).
About the reviewer
Cody Jackson is a disabled military veteran, the founder of Socius Consulting, an IT and business management consulting company in San Antonio, and a co-founder of Top Men Technologies. He is currently employed at CACI International as the lead ICS/SCADA modeling and simulations engineer. He has been involved in the tech industry since 1994, when he joined the Navy as a nuclear chemist and radcon technician. Prior to CACI, he worked at ECPI University as a computer information systems adjunct professor. A self-taught Python programmer, he is the author of Learning to Program Using Python and Secret Recipes of the Python Ninja. He holds an Associate in Science degree, a Bachelor of Science degree, and a Master of Science degree.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Expert Python Programming Third Edition
Dedication
About Packt
Why subscribe?
Packt.com
Contributors
About the authors
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Section 1: Before You Start
Current Status of Python
Technical requirements
Where are we now and where we are going to?
Why and how Python changes
Being up-to-date with changes ­by following PEP documents
Python 3 adoption at the time of writing this book
The main differences between Python 3 and Python 2
Why should I care?
The main syntax differences and common pitfalls
Syntax changes
Changes in the standard library
Changes in data types and collections and string literals
The popular tools and techniques used for maintaining cross-version compatibility
Not only CPython
Why should I care?
Stackless Python
Jython
IronPython
PyPy
MicroPython
Useful resources
Summary
Modern Python Development Environments
Technical requirements
Installing additional Python packages using pip
Isolating the runtime environment
Application-level isolation versus system-level isolation
Python's venv
venv versus virtualenv
System-level environment isolation
Virtual development environments using Vagrant
Virtual environments using Docker
Containerization versus virtualization
Writing your first Dockerfile
Running containers
Setting up complex environments
Useful Docker recipes for Python
Reducing the size of containers
Addressing services inside of a Compose environment
Communicating between multiple Compose environments
Popular productivity tools
Custom Python shells – ipython, bpython, ptpython, and so on
Setting up the PYTHONSTARTUP environment variable
IPython
bpython
ptpython
Incorporating shells in your own scripts and programs
Interactive debuggers
Summary
Section 2: Python Craftsmanship
Modern Syntax Elements - Below the Class Level
Technical requirements
Python's built-in types
Strings and bytes
Implementation details
String concatenation
Constant folding, the peephole optimizer, and the AST optimizer
String formatting with f-strings
Containers
Lists and tuples
Implementation details
List comprehensions
Other idioms
Dictionaries
Implementation details
Weaknesses and alternatives
Sets
Implementation details
Supplemental data types and containers
Specialized data containers from the collections module
Symbolic enumeration with the enum module
Advanced syntax
Iterators
Generators and yield statements
Decorators
General syntax and possible implementations
As a function
As a class
Parametrizing decorators
Introspection preserving decorators
Usage and useful examples
Argument checking
Caching
Proxy
Context provider
Context managers – the with statement
The general syntax and possible implementations
As a class
As a function – the contextlib module
Functional-style features of Python
What is functional programming?
Lambda functions
map(), filter(), and reduce()
Partial objects and partial() functions
Generator expressions
Function and variable annotations
The general syntax
The possible uses
Static type checking with mypy
Other syntax elements you may not know of yet
The for ... else ... statement
Keyword-only arguments
Summary
Modern Syntax Elements - Above the Class Level
Technical requirements
The protocols of the Python language – dunder methods and attributes
Reducing boilerplate with data classes
Subclassing built-in types
MRO and accessing methods from superclasses
Old-style classes and super in Python 2
Understanding Python's Method Resolution Order
Super pitfalls
Mixing super and explicit class calls
Heterogeneous arguments
Best practices
Advanced attribute access patterns
Descriptors
Real-life example – lazily evaluated attributes
Properties
Slots
Summary
Elements of Metaprogramming
Technical requirements
What is metaprogramming?
Decorators – a method of metaprogramming
Class decorators
Using __new__() for overriding the instance creation process
Metaclasses
The general syntax
New Python 3 syntax for metaclasses
Metaclass usage
Metaclass pitfalls
Code generation
exec, eval, and compile
Abstract syntax tree (AST)
Import hooks
Projects that use code generation patterns
Falcon's compiled router
Hy
Summary
Choosing Good Names
Technical requirements
PEP 8 and naming best practices
Why and when to follow PEP 8?
Beyond PEP 8 – Team-specific style guidelines
Naming styles
Variables
Constants
Naming and usage
Public and private variables
Functions and methods
The private controversy
Special methods
Arguments
Properties
Classes
Modules and packages
The naming guide
Using the has/is prefixes for Boolean elements
Using plurals for variables that are collections
Using explicit names for dictionaries
Avoid generic names and redundancy
Avoiding existing names
Best practices for arguments
Building arguments by iterative design
Trusting the arguments and your tests
Using *args and **kwargs magic arguments carefully
Class names
Module and package names
Useful tools
Pylint
pycodestyle and flake8
Summary
Writing a Package
Technical requirements
Creating a package
The confusing state of Python packaging tools
The current landscape of Python packaging thanks to PyPA
Tool recommendations
Project configuration
setup.py
setup.cfg
MANIFEST.in
Most important metadata
Trove classifiers
Common patterns
Automated inclusion of version string from package
README file
Managing dependencies
The custom setup command
Working with packages during development
setup.py install
Uninstalling packages
setup.py develop or pip -e
Namespace packages
Why is it useful?
PEP 420 - implicit namespace packages
Namespace packages in previous Python versions
Uploading a package
PyPI - Python Package Index
Uploading to PyPI - or other package index
.pypirc
Source packages versus built packages
sdist
bdist and wheels
Standalone executables
When standalone executables useful?
Popular tools
PyInstaller
cx_Freeze
py2exe and py2app
Security of Python code in executable packages
Making decompilation harder
Summary
Deploying the Code
Technical requirements
The Twelve-Factor App
Various approaches to deployment automation
Using Fabric for deployment automation 
Your own package index or index mirror
PyPI mirroring
Bundling additional resources with your Python package
Common conventions and practices
The filesystem hierarchy
Isolation
Using process supervision tools
Application code running in user space
Using reverse HTTP proxies
Reloading processes gracefully
Code instrumentation and monitoring
Logging errors – Sentry/Raven
Monitoring system and application metrics
Dealing with application logs
Basic low-level log practices
Tools for log processing
Summary
Python Extensions in Other Languages
Technical requirements
Differentiating between the C and C++ languages
Loading extensions in C or C++
The need to use extensions
Improving the performance in critical code sections
Integrating existing code written in different languages
Integrating third-party dynamic libraries
Creating custom datatypes
Writing extensions
Pure C extensions
A closer look at Python/C API
Calling and binding conventions
Exception handling
Releasing GIL
Reference counting
Writing extensions with Cython
Cython as a source-to-source compiler
Cython as a language
Challenges with using extensions
Additional complexity
Debugging
Interfacing with dynamic libraries without extensions
The ctypes module
Loading libraries
Calling C functions using ctypes
Passing Python functions as C callbacks
CFFI
Summary
Section 3: Quality over Quantity
Managing Code
Technical requirements
Working with a version control system
Centralized systems
Distributed systems
Distributed strategies
Centralized or distributed?
Use Git if you can
GitFlow and GitHub Flow
Setting up continuous development processes
Continuous integration
Testing every commit
Merge testing through CI
Matrix testing
Continuous delivery
Continuous deployment
Popular tools for continuous integration
Jenkins
Buildbot
Travis CI
GitLab CI
Choosing the right tool and common pitfalls
Problem 1 – Complex build strategies
Problem 2 – Long building time
Problem 3 – External job definitions
Problem 4 – Lack of isolation
Summary
Documenting Your Project
Technical requirements
The seven rules of technical writing
Write in two steps
Target the readership
Use a simple style
Limit the scope of information
Use realistic code examples
Use a light but sufficient approach
Use templates
Documentation as code
Using Python docstrings
Popular markup languages and styles for documentation
Popular documentation generators for Python libraries
Sphinx
Working with the index pages
Registering module helpers
Adding index markers
Cross-references
MkDocs
Documentation building and continuous integration
Documenting web APIs
Documentation as API prototype with API Blueprint
Self-documenting APIs with Swagger/OpenAPI
Building a well-organized documentation system
Building documentation portfolio
Design
Usage
Recipe
Tutorial
Module helper
Operations
Your very own documentation portfolio
Building a documentation landscape
Producer's layout
Consumer's layout
Summary
Test-Driven Development
Technical requirements
I don't test
Three simple steps of test-driven development
Preventing software regression
Improving code quality
Providing the best developer documentation
Producing robust code faster
What kind of tests?
Unit tests
Acceptance tests
Functional tests
Integration tests
Load and performance testing
Code quality testing
Python standard test tools
unittest
doctest
I do test
unittest pitfalls
unittest alternatives
nose
Test runner
Writing tests
Writing test fixtures
Integration with setuptools and plugin system
Wrap-up
py.test
Writing test fixtures
Disabling test functions and classes
Automated distributed tests
Wrap-up
Testing coverage
Fakes and mocks
Building a fake
Using mocks
Testing environment and dependency compatibility
Dependency matrix testing
Document-driven development
Writing a story
Summary
Section 4: Need for Speed
Optimization - Principles and Profiling Techniques
Technical requirements
The three rules of optimization
Making it work first
Working from the user's point of view
Keeping the code readable and maintainable
Optimization strategy
Looking for another culprit
Scaling the hardware
Writing a speed test
Finding bottlenecks
Profiling CPU usage
Macro-profiling
Micro-profiling
Profiling memory usage
How Python deals with memory
Profiling memory
objgraph
C code memory leaks
Profiling network usage
Tracing network transactions
Summary
Optimization - Some Powerful Techniques
Technical requirements
Defining complexity
Cyclomatic complexity
The big O notation
Reducing complexity by choosing proper data structures
Searching in a list
Using sets
Using collections
deque
defaultdict
namedtuple
Using architectural trade-offs
Using heuristics and approximation algorithms
Using task queues and delayed processing
Using probabilistic data structures
Caching
Deterministic caching
Non-deterministic caching
Cache services
Memcached
Summary
Concurrency
Technical requirements
Why concurrency?
Multithreading
What is multithreading?
How Python deals with threads
When should we use threading?
Building responsive interfaces
Delegating work
Multiuser applications
An example of a threaded application
Using one thread per item
Using a thread pool
Using two-way queues
Dealing with errors and rate limiting
Multiprocessing
The built-in multiprocessing module
Using process pools
Using multiprocessing.dummy as the multithreading interface
Asynchronous programming
Cooperative multitasking and asynchronous I/O
Python async and await keywords
asyncio in older versions of Python
A practical example of asynchronous programming
Integrating non-asynchronous code with async using futures
Executors and futures
Using executors in an event loop
Summary
Section 5: Technical Architecture
Event-Driven and Signal Programming
Technical requirements
What exactly is event-driven programming?
Event-driven != asynchronous
Event-driven programming in GUIs
Event-driven communication
Various styles of event-driven programming
Callback-based style
Subject-based style
Topic-based style
Event-driven architectures
Event and message queues
Summary
Useful Design Patterns
Technical requirements
Creational patterns
Singleton
Structural patterns
Adapter
Interfaces
Using zope.interface
Using function annotations and abstract base classes
Using collections.abc
Proxy
Facade
Behavioral patterns
Observer
Visitor
Template
Summary
reStructuredText Primer
reStructuredText
Section structure
Lists
Inline markup
Literal block
Links
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
Python is a dynamic programming language, used in a wide range of domains thanks to its simple yet powerful nature. Although writing Python code is easy, making it readable, reusable, and easy to maintain is challenging. Complete with best practices, useful tools, and standards implemented by professional Python developers, the third version of Expert Python Programming will help you overcome this challenge.
The book will start by taking you through the new features in Python 3.7. You'll learn the Python syntax and understand how to apply advanced object-oriented concepts and mechanisms. You'll also explore different approaches to implement metaprogramming. This book will guide you in following best naming practices when writing packages, and creating standalone executables easily, alongside using powerful tools such as buildout and virtualenv to deploy code on remote servers. You'll discover how to create useful Python extensions with C, C++, Cython, and Pyrex. Furthermore, learning about code management tools, writing clear documentation, and test-driven development will help you write clean code.
By the end of the book, you will have become an expert in writing efficient and maintainable Python code.
Who this book is for
This book is written for Python developers who wish to go further in mastering Python. And by developers, I mean mostly professionals, so programmers who write Python software for their living. This is because it focuses mostly on tools and practices that are crucial for creating performant, reliable, and maintainable software in Python.
It does not mean that hobbyists won't find anything interesting. This book should be great for anyone who is interested in learning advanced-level concepts with Python. Anyone who has basic Python skills should be able to follow the content of the book, although it might require some additional effort from less experienced programmers. It should also be a good introduction to Python 3.7 for those who are still a bit behind and continue to use Python version 2.7 or older.
Finally, the groups that should benefit most from reading this book are web developers and backend engineers. This is because of two topics featured in here that are especially important in their areas of work: reliable code deployments and concurrency.
What this book covers
Chapter 1, Current Status of Python, showcases the current state of the Python language and its community. We will see how Python is constantly changing, why it is changing, and also why these facts are important for anyone who wants to call themselves a Python professional. We will also take a look at the most popular and canonical ways for working on written in Python—popular productivity tools and conventions that are de facto standards now.
Chapter 2, Modern Python Development Environments, describes modern ways of setting up repeatable and consistent development environments for Python programmers. We will concentrate on two popular tools for environment isolation: virtualenv-type environments and Docker containers.
Chapter 3, Modern Syntax Elements – Below the Class Level, focuses on best practices for writing code in Python (language idioms) and also provides a summary of selected elements of Python syntax that may be new for intermediate Python users or those experienced with older versions of Python. We will also take a look at useful notes about internal CPython-type implementations and their computational complexities as a rationale for provided idioms.
Chapter 4, Modern Syntax Elements – Above the Class Level, covers more advanced object-oriented concepts and mechanisms available in Python.
Chapter 5, Elements of Metaprogramming, presents an overview of common approaches to metaprogramming available to Python programmers.
Chapter 6, Choosing Good Names, explains what is the most widely-adopted style guide for Python code (PEP-8) and when and why developers should follow it. We will also take a look at some of the author's general advice for naming things.
Chapter 7, Writing a Package, describes the current state of Python packaging and best practices for creating packages that are to be distributed as open source code in the Python Package Index (PyPI). We will also cover an often overlooked topic of Python – standalone executables.
Chapter 8, Deploying Code, presents some common lightweight tools for deploying Python code on remote servers. Deployment is one of the fields where Python shines are backends for web-based services and applications.
Chapter 9, Python Extensions in Other Languages, explains why writing extensions in C and C++ for Python can sometimes be a good solution and shows that it is not as hard as it seems, as long as the proper tools are used.
Chapter 10, Managing Code, describes how to properly manage a code base and why version control systems should be used. We will also leverage the power of version control systems (especially Git) in implementing continuous processes, such as continuous integration and continuous delivery.
Chapter 11, Documenting Your Project, describes the general rules for writing technical documentation that may be applied to software written in any language, and various tools that are especially useful for creating documentation of your Python code.
Chapter 12, Test-Driven Development, advocates the usage of test-driven development and provides more information on how to use popular Python tools designed for testing.
Chapter 13, Optimization – Principles and Profiling Techniques, discusses the most basic rules of optimization that every developer should be aware of. We will also learn how to identify application performance bottlenecks and use common profiling tools.
Chapter 14, Optimization – Some Powerful Techniques, shows how to use that knowledge to actually make your application run faster or be more efficient in terms of used resources.
Chapter 15, Concurrency, explains how to implement concurrency in Python using different approaches and libraries.
Chapter 16, Event-Driven and Signal Programming, describes what event-driven/signal programming is and how it relates to asynchronous programming and different concurrency models. We will present the various approaches to event-driven programming available to Python programmers, along with useful libraries that enable these patterns.
Chapter 17, Useful Design Patterns, implements a set of useful design patterns and example implementations in Python.
Appendix A, reStructuredText Primer, provides a brief tutorial on how to use reStructuredText markup language.
To get the most out of this book
This book is written for developers who work under any operating system for which Python 3 is available.
This is not a book for beginners, so I assume you have Python installed in your environment or know how to install it. Anyway, this book takes into account the fact that not everyone needs to be fully aware of the latest Python features or officially recommended tools. This is why the first chapter provides a recap on common utilities (such as virtual environments and pip) that are now considered standard tools of professional Python developers.
Download the example code files
You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packt.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packt.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Expert-Python-Programming-Third-Edition. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789808896_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Any attempt to run the code that has such issues will immediately cause the interpreter to fail, raising a SyntaxError exception."
A block of code is set as follows:
print("hello world") print "goodbye python2"
Any command-line input or output is written as follows:
$ python3 script.py
Get in touch
Feedback from our readers is always welcome.
General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packt.com.
Section 1: Before You Start
This part prepares the user for the modern Python development routine. It explains how Python has changed over the last few years and what the common development tools used by modern Python programmers are.
The following chapters are included in this section:
Chapter 1
,
Current Status of Python
Chapter 2
,
Modern Python Development Environments
Current Status of Python
Python is amazing.
For a very long time, one of the most important virtues of Python was interoperability. No matter what operating system you or your customers were using, if a Python interpreter was available for that system, your software that was written in Python would work there. And, most importantly, your software would work the same way. However, that's not uncommon anymore. Modern languages such as Ruby and Java provide similar interoperability capabilities. But, interoperability isn't the most important quality of programming language nowadays. With the advent of cloud computing, web-based applications, and reliable virtualization software, it isn't that important to have a programming language that works the same no matter the operating system. What is still important is the tools that allow programmers to efficiently write reliable and maintainable software. Fortunately, Python is still one of the languages that allows programmers the most efficiency, and is definitely a smart choice for a company's primary development language.
Python stays relevant for so long because it is constantly evolving. This book is focused on the latest Python 3.7 version, and all code examples are written in this version of the language unless another version is explicitly mentioned. Because Python has a very long history, and there are still programmers using Python 2 on a daily basis, this book starts with a chapter that describes the current status quo of Python 3. In this chapter, you'll find how and why Python changes, and will learn how to write software that is compatible with both the historic and latest versions of Python.
In this chapter, we will cover the following topics:
Where are we now and where we are going to?
Why and how Python changes
Being up-to-date with changes to PEP documentation
Python 3 adoption at the time of writing this book
The main difference between Python 3 and Python 2
Not only CPython
Useful resources
Technical requirements
You can download the latest version of Python from https://www.python.org/downloads/ for this chapter.
Alternative Python interpreter implementations can be found at the following sites:
Stackless Python:
https://github.com/stackless-dev/stackless
PyPy:
https://pypy.org
Jython:
https://www.jython.org
IronPython:
https://ironpython.net
MicroPython:
https://micropython.org
The code files for this chapter can be found at https://github.com/PacktPublishing/Expert-Python-Programming-Third-Edition/tree/master/chapter1.
Where are we now and where we are going to?
Python history starts somewhere in the late 1980s, but its 1.0 release date was in the year 1994. So, it isn't a young language. There could be a whole timeline of major Python releases mentioned here, but what really matters is a single date: Python 3.0—December 3, 2008.
At the time of writing, almost ten years have passed since the first Python 3 release. It is also seven years since the creation of PEP 404—the official document that un-released Python 2.8 and officially closed the 2.x branch. Although a lot of time has passed, there is a specific dichotomy in the Python community—while the language is developing very fast, there is a large group of its users that do not want to move forward with it.
Why and how Python changes
The answer is simple—Python changes because there is such a need. The competition does not sleep. Every few months, a new language pops out, out of nowhere, claiming to solve every problem of all its predecessors. Most projects like these lose the developers' attention very shortly, and their popularity is often driven by sudden hype.
This is a sign of some bigger problem. People design new languages because they find that existing ones do not solve their problems in the best way possible. It would be silly to not recognize such a need. Also, more and more widespread usage of Python shows that it could, and should, be improved on in many places.
Many improvements in Python are driven by the needs of particular fields where it is being used. The most significant one is web development. Thanks to the ever-increasing demand for speed and performance in this area, we've seen that ways to deal with concurrency in Python have been drastically improved over the time.
Other changes are simply caused by the age and maturity of the Python project. Throughout the years, it collected some of the clutter in the form of disorganized and redundant standard library modules, or some bad design decisions. First, the Python 3 release aimed to bring with it a major cleanup and refreshment to the language. Unfortunately, time showed that this plan backfired a bit. For a long time, Python 3 was treated by many developers only like a curiosity. Hopefully, this is changing.
Being up-to-date with changes by following PEP documents
The Python community has a well-established way of dealing with changes. While speculative Python language ideas are mostly discussed on specific mailing lists ([email protected]), nothing major ever gets changed without the existence of a new document, called a Python Enhancement Proposal (PEP).
It is a formalized document that describes, in detail, the proposal of change to be made in Python. It is also the starting point for the community discussion. The whole purpose, format, and workflow around these documents is also standardized in the form of a PEP—precisely the PEP 1 document (http://www.python.org/dev/peps/pep-0001).
PEP documentation is very important for Python, and, depending on the topic, they serve different purposes:
Informing
: They summarize the information needed by core Python developers, and notify about Python release schedules
Standardizing
: They provide code style, documentation, or other guidelines
Designing
: They describe the proposed features
A list of all proposed PEPs are available in a living PEP 0 document (https://www.python.org/dev/peps/). Since they are easily accessible in one place, and the actual URL is also very easy to guess, they are usually referred to by the number in the book.
The PEP 0 document is a great source of information for those who are wondering what direction Python language is heading in, but do not have time to track every discussion on Python mailing lists. It shows which documents were already accepted but not yet implemented, and also which are still under consideration.
PEPs also serve additional purposes. Very often, people ask questions like the following:
Why does feature
A
work that way?
Why does Python not have feature
B
?
In most such cases, the extensive answer is already available in specific PEP documents where such a feature was already mentioned. There is a lot of PEP documentation describing Python language features that were proposed but not accepted. This documentation is left as a historical reference.
Python 3 adoption at the time of writing this book
So, thanks to new, exciting features, is Python 3 well adopted among its community? It's hard to say. The once-popular page, Python 3 Wall of Superpowers (https://python3wos.appspot.com), that tracked the compatibility of the most popular packages with the Python 3 branch was, at the beginning, named Python 3 Wall of Shame.
The site is no longer maintained, but in the list from the last time it was updated, on April 22, 2018, it shows that exactly 191 from 200 of the most popular Python packages at that time were compatible within Python 3. So, we can see that Python 3 seems to be finally well-adopted in the community of open source Python programmers. Still, this does not mean that all teams building their applications are finally using Python 3. At least, since most of the popular Python packages are available in Python 3, the popular excuse packages that we use have not been ported yet is no longer valid.
The main reason for such a situation is that porting the existing application from Python 2 to Python 3 is always a challenge. There are tools such as 2to3 that can perform automated code translation, but they do not assure that the result will be 100% correct. Also, such translated code may not perform as well as in its original form without manual adjustments. Moving existing complex code bases to Python 3 might involve tremendous effort, and a cost that some organizations may not be able to afford. Fortunately, such costs can be split over time. Some good software architecture design methodologies, such as service-oriented architecture or microservices, can help to achieve this goal gradually. New project components (services or microservices) can be written using the new technology, and existing ones can be ported one at a time.
In the long run, moving to Python 3 can have only beneficial effects on a project. According to PEP 404, there won't be another 2.8 release in the 2.x branch of Python, and the official end-of-life for Python 2 is scheduled for 2020. Until that time, we can expect only patch version updates for major security issues, but nothing more. Also, there may be a time in the future when all major projects, such as Django, Flask, and NumPy will drop any 2.x compatibility and will be available only in Python 3. Django has already made that step, and since version 2.0.0 was released, it no longer supports Python 2.7.
My personal opinion on this topic can be considered controversial. I think that the best incentive for the community would be to completely drop Python 2 support when creating new packages. This, of course, limits a range of such software, but may be the only right way to change the way of thinking in those who insist on sticking to Python 2.x.
We'll take a look at the main differences between Python 3 and Python 2 in the next section.
The main differences between Python 3 and Python 2
It has already been stated that Python 3 breaks backward compatibility with Python 2 on a syntax level. Still, it is not a complete redesign. Also, it does not cause every Python module written for some 2.x release to stop working under Python 3. It is possible to write completely cross-compatible code that will run on both major releases without additional tools or techniques, but usually it is possible only for simple applications.
Why should I care?
Despite my personal opinion on Python 2 compatibility that I exposed earlier in this chapter, it is impossible to simply forget about it at this time. There are still some useful packages that are really worth using, but are not likely to be ported in the very near future.
Also, sometimes, we may be constrained by the organization we work in. The existing legacy code may be so complex that porting it is not economically feasible. So, even if we decide to move on and live only in the Python 3 world from now on, it will be impossible to live completely without Python 2 for some time.
Nowadays, it is very hard to call yourself a professional developer without giving something back to the community. So, helping the open source developers add Python 3 compatibility to the existing packages is a good way to pay off the moral debt incurred by using them. This, of course, cannot be done without knowing the differences between Python 2 and Python 3. By the way, this is also a great exercise for those new to Python 3.
The main syntax differences and common pitfalls
The Python documentation is the best reference for differences between every Python release. However, for your convenience, this section summarizes the most important ones. This does not change the fact that the documentation is mandatory reading for those not familiar with Python 3 yet (see https://docs.python.org/3.0/whatsnew/3.0.html).
The breaking changes that were introduced by Python 3 can be generally divided into three groups:
Syntax changes, where some syntax elements were removed/changed and other elements were added
Changes in the standard library
Changes in datatypes and collections
Changes in the standard library
Breaking changes in the standard library are the second easiest to catch after syntax changes. Each subsequent version of Python adds, deprecates, improves, or completely removes standard library modules. Such a process was also common in the older branches of Python (1.x and 2.x), so it does not come as a shock in Python 3. In most cases, depending on the module that was removed or reorganized (such as urlparse being moved to urllib.parse), it will raise exceptions on the import time just after it is interpreted. This makes such issues so easy to catch. In order to be sure that all such issues are covered, full test code coverage is essential. In some cases (for example, when using lazily loaded modules), the issues that are usually noticed at import time will not appear before some modules are used in code as function calls. This is why it is so important to make sure that every line of code is actually executed during tests suite.
Changes in data types and collections and string literals
Changes in how Python represents datatypes and collections require the most effort when the developer tries to maintain compatibility or simply ports existing code to Python 3. While incompatible syntax or standard library changes are easily noticeable and often easy to fix, changes in collections and types are either non-obvious or require a lot of repetitive work. The list of such changes is long and the official documentation is the best reference.
Still, this section must cover the change in how string literals are treated in Python 3, because it seems to be the most controversial and discussed change in Python 3, despite being a very good move that makes things more explicit.
All string literals are now Unicode, and bytestring literals require b or B prefix. For Python 3.0 and 3.1, the old Unicode u prefix (like u"foo") is illegal and will raise a syntax error. Dropping off that prefix was the main reason for most of the controversies. It made it really hard to create code compatible with different branches of Python—Python in version 2.x relied on these prefixes in order to create Unicode literals. This prefix was brought back in Python 3.3 to ease the integration process, although it now lacks any syntactic meaning.
The popular tools and techniques used for maintaining cross-version compatibility
Maintaining compatibility between versions of Python is a challenge. It may add a lot of additional work depending on the size of the project, but is definitely doable and worth doing. For packages that are meant to be reused in many environments it is absolutely a must-have. Open source packages without well-defined and tested compatibility bounds are very unlikely to become popular, but closed third-party code that never leaves the company network can also greatly benefit from being tested in different environments.
It should be noted here that, while this part focuses mainly on compatibility between various versions of Python, these approaches apply for maintaining compatibility with external dependencies such as different package versions, binary libraries, systems, or external services.
The whole process can be divided into three main areas, ordered by their importance:
Defining and documenting target compatibility bounds and how they will be managed
Testing in every environment and with every dependency version declared as compatible
Implementing actual compatibility code
Declaration of what is considered compatible is the most important part of the whole process because it gives your code users (developers) the ability to have expectations and make assumptions on how it works and how it can change in the future. Our code can be used as a dependency in different projects that may also strive to manage compatibility, so the ability to reason how it behaves is crucial.
While this book tries to always give a few choices and not to give absolute recommendations on specific options, here is one of the few exceptions. The best way to define how compatibility may change in the future is by using proper approach to versioning numbers using Semantic Versioning (semver) (http://semver.org/). It describes a broadly accepted standard for marking scope of changes in code by the version specifier, consisting only of three numbers. It also gives some advice on how to handle deprecation policies. Here is an excerpt from its summary (licensed under Creative Commons - CC BY 3.0):
When it comes to testing the sad truth, that is, to be sure that code is compatible with every declared dependency version and in every environment (here Python version), it must be tested in every combination of these. This, of course, may not be possible when the project has a lot of dependencies, because the number of combinations grows rapidly with every new dependency version. So, typically some trade-off needs to be made so that running full compatibility tests does not need to take ages. The selection of tools that help testing in so-called matrixes is presented in Chapter 12, Test-Driven Development, which discusses testing in general.
The implementation of the compatibility layer is the last, and also the least important, step of the process if the bounds of that compatibility are well-defined and rigorously tested. Still, there are some tools and techniques that every programmer interested in such a topic should know.
The most basic is Python's __future__ module. It backports some features from newer Python releases back into the older ones and takes the form of an import statement:
from __future__ import <feature>
Features provided by the future statements are syntax-related elements that cannot be easily handled by different means. This statement affects only the module where it was used. Here is an example of a Python 2.7 interactive session that brings Unicode literals from Python 3.0:
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit
(Intel)] on win32 Type "help", "copyright", "credits" or "license" for more
information. >>> type("foo")
# old literals <type 'str'>
>>> from __future__ import unicode_literals >>> type("foo")
# now is Unicode <type 'unicode'>
Here is a list of all the available __future__ statement options that developers concerned with two-thirds compatibility should know:
division
: This adds a Python 3 division operator (PEP 238)
absolute_import
: This makes every form of an
import
statement not starting from dot character be interpreted as absolute imports (PEP 328)
print_function
: This changes a
statement into a function call so that parentheses around
become mandatory (PEP 3112)
unicode_literals
: This makes every string literal be interpreted as Unicode literals (PEP 3112)
A list of the __future__ statement options is very short, and it covers only a few syntax features. The other things that have changed, such as the metaclass syntax (which is an advanced feature that's covered in Chapter 5, Elements of Metaprogramming), are a lot harder to maintain. Reliable handling of multiple standard library reorganizations also cannot be solved by the future statements. Fortunately, there are some tools that aim to provide a consistent layer of ready-to-use compatibility code. The most well-known of these is Six (https://pypi.python.org/pypi/six/), which provides a whole common two-thirds compatibility boilerplate as a single module. The other promising, but slightly less popular, tool is the future module (http://python-future.org/).
In some situations, developers may not want to include additional dependencies in some small packages. A common practice is the additional module that gathers all the compatibility code, usually named compat.py. Here is an example of such compat modules taken from the python-gmaps project (https://github.com/swistakm/python-gmaps):
# -*- coding: utf-8 -*-"""This module provides compatibility layer forselected things that have changed across Python versions.""" import sys if sys.version_info < (3, 0, 0): import urlparse # noqa def is_string(s): """Return True if given value is considered string""" return isinstance(s, basestring) else: # note: urlparse was moved to urllib.parse in Python 3 from urllib import parse as urlparse # noqa def is_string(s): """Return True if given value is considered string""" return isinstance(s, str)
Such compat.py modules are popular, even in projects that depend on Six for two-thirds compatibility, because it is a very convenient way to store code that handles compatibility with different versions of packages being used as dependencies.
In the next section, we'll take a look at what CPython is.
Not only CPython
The reference Python interpreter implementation is called CPython and, as its name suggests, it is written entirely in the C language. It was always C and probably will be still for a very long time. That's the implementation that most Python programmers choose because it is always up to date with the language specification and is the interpreter that most libraries are tested on. But, besides C, Python interpreter was written in a few other languages. Also, there are some modified versions of CPython interpreter available under different names and tailored exactly for some niche applications. Most of them are a few milestones behind CPython, but provide a great opportunity to use and promote the language in a specific environment.
In this section, we will discuss some of the most prominent and interesting alternative Python implementations.
Why should I care?
There are plenty of alternative Python implementations available. The Python wiki page on that topic (https://wiki.python.org/moin/PythonImplementations) features dozens of different language variants, dialects, or implementations of Python interpreter built with something other than C. Some of them implement only a subset of the core language syntax, features, and built-in extensions, but there are at least a few that are almost fully compatible with CPython. The most important thing to know is that, while some of them are just toy projects or experiments, most of them were created to solve some real problems – problems that were either impossible to solve with CPython or required too much of the developer's effort.
Examples of such problems are as follows:
Running Python code on embedded systems
Integration with code written for runtime frameworks, such as Java or .NET, or in different languages
Running Python code in web browsers
The following sections provide a short description of, subjectively, the most popular and up-to-date choices that are currently available for Python programmers.
Stackless Python
Stackless Python advertises itself as an enhanced version of Python. Stackless is named so because it avoids depending on the C call stack for its own stack. It is, in fact, a modified CPython code that also adds some new features that were missing from the core Python implementation at the time Stackless was created. The most important of these are microthreads, which are managed by the interpreter as cheap and lightweight alternatives to ordinary threads, that must depend on system kernel context switching and task scheduling.
The latest available versions are 2.7.15 and 3.6.6 and implement 2.7 and 3.6 versions of Python, respectively. All the additional features provided by Stackless are exposed as a framework within this distribution through the built-in stackless module.
Stackless isn't the most popular alternative implementation of Python, but it is worth knowing, because some of the ideas that were introduced in it had a strong impact on the language community. The core switching functionality was extracted from Stackless and published as an independent package named greenlet, which is now the basis for many useful libraries and frameworks. Also, most of its features were re-implemented in PyPy—another Python implementation that will be featured later. The official online documentation of Stackless Python can be found at https://stackless.readthedocs.io and the project wiki can be found at https://github.com/stackless-dev/stackless.
Jython
Jython is a Java implementation of the language. It compiles the code into Java byte code, and allows the developers to seamlessly use Java classes within their Python modules. Jython allows people to use Python as the top-level scripting language on complex application systems, for example, J2EE. It also brings Java applications into the Python world. Making Apache Jackrabbit (which is a document repository API based on JCR; see http://jackrabbit.apache.org) available in a Python program is a good example of what Jython allows.
The main differences of Jython compared to the CPython implementation are as follows:
True Java's garbage collection instead of reference counting
Lack of
global interpreter lock
(
GIL
) allows better utilization of multiple cores in multi-threaded applications
The main weakness of this implementation of the language is the lack of support for C Python Extension APIs, so no Python extensions written in C will work with Jython.
The latest available version of Jython is Jython 2.7, and this corresponds to the 2.7 version of the language. It is advertised as implementing nearly all of the core Python standard library and using the same regression test suite. Unfortunately, Jython 3.x was never released, and the project can be now safely considered dead. However, Jython is still worth mentioning, even if it is not developed anymore, because it was very unique implementation at the time and had meaningful impact on other alternative Python implementations.
The official project page can be found at http://www.jython.org.
IronPython
IronPython brings Python into the .NET Framework. The project is supported by Microsoft, where IronPython's lead developers work. It is quite an important implementation for the promotion of a language. Besides Java, the .NET community is one of the biggest developer communities out there. It is also worth noting that Microsoft provides a set of free development tools that turn Visual Studio into a full-fledged Python IDE. This is distributed as Visual Studio plugins named Python Tools for Visual Studio (PVTS), and is available as open source code on GitHub (http://microsoft.github.io/PTVS).
The latest stable release is 2.7.8, and it is compatible with Python 2.7. Unlike Jython, we can observe active development on both 2.x and 3.x branches of the interpreter, although Python 3 support still hasn't been officially released yet. Despite the fact that .NET runs primarily on Microsoft Windows, it is also possible to run IronPython on macOS and Linux. This is thanks to Mono, a cross platform, open source .NET implementation.
The main differences and advantages of IronPython compared to CPython are as follows:
Similar to Jython, the lack of
global interpreter lock
(
GIL
) allows for better utilization of multiple cores in multi-threaded applications
Code written in C# and other .NET languages can be easily integrated in IronPython and vice versa
It can be run in all major web browsers through Silverlight (although Microsoft will stop supporting Silverlight in 2021)
