Advanced Python Programming E-Book

Quan Nguyen

• Herausgeber: Packt Publishing
• Kategorie: Lebensstil
• Sprache: Englisch
• Veröffentlichungsjahr: 2022

Seitenzahl: 691

Beschreibung

Python's powerful capabilities for implementing robust and efficient programs make it one of the most sought-after programming languages.
In this book, you'll explore the tools that allow you to improve performance and take your Python programs to the next level.
This book starts by examining the built-in as well as external libraries that streamline tasks in the development cycle, such as benchmarking, profiling, and optimizing. You'll then get to grips with using specialized tools such as dedicated libraries and compilers to increase your performance at number-crunching tasks, including training machine learning models.
The book covers concurrency, a major solution to making programs more efficient and scalable, and various concurrent programming techniques such as multithreading, multiprocessing, and asynchronous programming.
You'll also understand the common problems that cause undesirable behavior in concurrent programs.
Finally, you'll work with a wide range of design patterns, including creational, structural, and behavioral patterns that enable you to tackle complex design and architecture challenges, making your programs more robust and maintainable.
By the end of the book, you'll be exposed to a wide range of advanced functionalities in Python and be equipped with the practical knowledge needed to apply them to your use cases.

Leseprobe

Advanced Python Programming

Second Edition

Accelerate your Python programs using proven techniques and design patterns

Quan Nguyen

BIRMINGHAM—MUMBAI

Advanced Python Programming

Second Edition

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Richa Tripathi

Publishing Product Manager: Gebin George

Senior Editor: Ruvika Rao

Content Development Editor: Nithya Sadanandan

Technical Editor: Maran Fernandes

Copy Editor: Safis Editing

Project Coordinator: Manisha Singh

Proofreader: Safis Editing

Indexer: Subalakshmi Govindhan

Production Designer: Aparna Bhagat

Marketing Coordinator: Sonakshi Bubbar

First published: February 2019

Second edition: March 2022

Production reference: 1180222

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80181-401-0

www.packt.com

To Julie. Please learn Python—it's better than R, I promise.

– Quan Nguyen

Contributors

About the author

Quan Nguyen is a Python programmer and machine learning enthusiast. He is interested in solving decision-making problems under uncertainty. Quan has authored several books on Python programming and scientific computing. He is currently pursuing a Ph.D. degree in computer science at Washington University in St. Louis, researching Bayesian methods in machine learning.

About the reviewers

Dhruv Thakkar is a full-stack developer with 7+ years of experience working in multiple Fortune 500 companies in industries such as telecoms, information technology, and finance. He holds a masters in computer science from the University of Bridgeport, Connecticut, and a Bachelor of Science (information technology) from the University of Mumbai.

Having started his career working as a frontend developer, he later transitioned to backend technologies such as Python, PHP, Perl, and Node.js to name a few. He also has experience working on DevOps tools such as Git, Jenkins, Ansible, and Docker. He is always open to learning new skills and is also passionate about open source and blockchain technologies.

Marius Iulian Mihailescu, Ph.D. is an associate professor at Spiru Haret University and a software project manager at the company Institute for Computers, both based in Bucharest, Romania. As an associate professor at Spiru Haret University, Marius is responsible for information security, functional programming, and developing IoT application courses. For more than 6 years, he has served as a lecturer at well-known national and international universities and different companies.

His main activity involves fields such as information security and cryptography related business/industry and research projects, ethical hacking, and developing software projects using the latest technologies, such as DevOps, IoT, cloud computing, big data, C#, F#, Java, Haskell, and Python.

Marius has authored and coauthored more than 30 articles in conference proceedings, 25 articles in journals, and 6 books.

Table of Contents

Preface

Section 1: Python-Native and Specialized Optimization

Chapter 1: Benchmarking and Profiling

Technical requirements

Designing your application

Building a particle simulator

Visualizing the simulation

Writing tests and benchmarks

Timing your benchmark

Writing better tests and benchmarks with pytest-benchmark

Finding bottlenecks with cProfile

Graphically analyzing profiling results

Profiling line by line with line_profiler

Optimizing our code

Using the dis module

Profiling memory usage with memory_profiler

Summary

Questions

Further reading

Chapter 2: Pure Python Optimizations

Technical requirements

Using the right algorithms and data structures

Lists and deques

Dictionaries

Sets

Heaps

Tries

Improved efficiency with caching and memoization

Joblib

Efficient iteration with comprehensions and generators

Summary

Questions

Further reading

Chapter 3: Fast Array Operations with NumPy, Pandas, and Xarray

Technical requirement

Getting started with NumPy

Creating arrays

Accessing arrays

Broadcasting

Mathematical operations

Calculating the norm

Rewriting the particle simulator in NumPy

Reaching optimal performance with numexpr

Working with database-style data with pandas

pandas fundamentals

Database-style operations with pandas

High-performance labeled data with xarray

Analyzing concentration

The xarray library

Improved performance

Plotting with xarray

Summary

Questions

Further reading

Chapter 4: C Performance with Cython

Technical requirements

Compiling Cython extensions

Adding static types

Declaring variables

Declaring functions

Declaring classes

Sharing declarations

Working with arrays

C arrays and pointers

Working with NumPy arrays

Working with typed memoryviews

Using a particle simulator in Cython

Profiling Cython

Using Cython with Jupyter

Summary

Questions

Chapter 5: Exploring Compilers

Technical requirements

Getting started with Numba

Using Numba decorators

Type specializations

Object mode versus native mode

Numba and NumPy

JIT classes

Limitations in Numba

The PyPy project

Setting up PyPy

Running a particle simulator in PyPy

Other interesting projects

Summary

Questions

Further reading

Chapter 6: Automatic Differentiation and Accelerated Linear Algebra for Machine Learning

A crash course in machine learning

Model parameters

Loss function

Loss minimization

Getting JAX up and running

Installing JAX

Using Google Colab

Automatic differentiation for loss minimization

Making the dataset

Building a linear model

Gradient descent with automatic differentiation

Just-In-Time compilation for improved efficiency

Automatic vectorization for efficient kernels

Data that is not linearly separable

The kernel method in machine learning

Automatic vectorization for kernelized models

Summary

Questions

Further reading

Section 2: Concurrency and Parallelism

Chapter 7: Implementing Concurrency

Technical requirements

Asynchronous programming

Waiting for input/output

Concurrency

Callbacks

Futures

Event loops

The asyncio framework

Coroutines

Converting blocking code into non-blocking code

Reactive programming

Observables

Useful operators

Hot and cold observables

Building a CPU monitor

Summary

Questions

Further reading

Chapter 8: Parallel Processing

Technical requirements

Introduction to parallel programming

GPUs

Using multiple processes

The Process and Pool classes

The Executor interface

Monte Carlo approximation of pi

Synchronization and locks

Parallel Cython with OpenMP

Automatic parallelism

Getting started with Theano

Profiling Theano

TensorFlow

Running code on a GPU

Summary

Questions

Chapter 9: Concurrent Web Requests

The basics of web requests

HTML

HTTP requests

HTTP status code

The requests module

Making a request in Python

Running a ping test

Concurrent web requests

Spawning multiple threads

Refactoring request logic

The problem with timeouts

Support from httpstat.us and simulation in Python

Timeout specifications

Good practices in making web requests

Consider the terms of service and data-collecting policies

Error handling

Update your program regularly

Avoid making a large number of requests

Summary

Questions

Further reading

Chapter 10: Concurrent Image Processing

Technical requirements

Image processing fundamentals

Python as an image processing tool

Computer image basics

OpenCV API

Image processing techniques

Applying concurrency to image processing

Good concurrent image processing practices

Choosing the correct way (out of many)

Spawning an appropriate number of processes

Processing input/output concurrently

Summary

Questions

Further reading

Chapter 11: Building Communication Channels with asyncio

Technical requirements

The ecosystem of communication channels

Communication protocol layers

Asynchronous programming for communication channels

Transports and protocols in asyncio

The big picture of asyncio's server client

Getting started with Python and Telnet

Starting a server

Installing Telnet

Simulating a connection channel

Sending messages back to clients

Closing transports

Client-side communication with aiohttp

Installing aiohttp and aiofiles

Fetching a website's HTML code

Writing files asynchronously

Summary

Questions

Further reading

Chapter 12: Deadlocks

Technical requirements

The concept of deadlocks

The dining philosophers problem

A deadlock in a concurrent system

Python simulation

Approaches to deadlock situations

Implementing ranking among resources

Ignoring locks and sharing resources

The concept of livelocks

Summary

Questions

Further reading

Chapter 13: Starvation

Technical requirements

Understanding starvation

What is starvation?

Scheduling

Causes of starvation

Starvation's relationship to deadlock

Approaching the readers-writers problem

Problem statement

The first readers-writers problem

The second readers-writers problem

The third readers-writers problem

Solutions to starvation

Summary

Questions

Further reading

Chapter 14: Race Conditions

Technical requirements

The concept of race conditions

Critical sections

How race conditions occur

Simulating race conditions in Python

Locks as a solution to race conditions

The effectiveness of locks

Implementation in Python

The downside of locks

Race conditions in real life

Security

Operating systems

Networking

Summary

Questions

Further reading

Chapter 15: The Global Interpreter Lock

Technical requirements

Introducing the GIL

Analyzing memory management in Python

The problem that the GIL addresses

Problems raised by the GIL

The potential removal of the GIL from Python

Working with the GIL

Implementing multiprocessing, rather than multithreading

Getting around the GIL with native extensions

Utilizing a different Python interpreter

Summary

Questions

Further reading

Section 3: Design Patterns in Python

Chapter 16: The Factory Pattern

Technical requirements

Understanding design patterns

Implementing the factory method

Real-world examples

Use cases

Implementing the factory method

Applying the abstract factory

Real-world examples

Use cases

Implementing the abstract factory pattern

Summary

Questions

Chapter 17: The Builder Pattern

Technical requirements

Understanding the builder pattern

Real-world examples

Use cases

Implementing an ordering application

Summary

Questions

Chapter 18: Other Creational Patterns

Technical requirements

Implementing the prototype pattern

Real-world examples

Use cases

Implementation

Implementing the singleton pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Further reading

Chapter 19: The Adapter Pattern

Technical requirements

Understanding the adapter pattern

Real-world examples

Use cases

Implementation

Summary

Chapter 20: The Decorator Pattern

Technical requirements

Introducing the decorator pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Chapter 21: The Bridge Pattern

Technical requirements

Real-world examples

Use cases

Implementation

Summary

Questions

Chapter 22: The Façade Pattern

Technical requirements

Understanding the façade pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Further reading

Chapter 23: Other Structural Patterns

Technical requirements

Implementing the flyweight pattern

Real-world examples

Use cases

Implementation

Implementing the model-view-controller pattern

Real-world examples

Use cases

Implementation

Applying the proxy pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Chapter 24: The Chain of Responsibility Pattern

Technical requirements

Understanding the Chain of Responsibility pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Chapter 25: The Command Pattern

Technical requirements

Understanding the command pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Chapter 26: The Observer Pattern

Technical requirements

Understanding the observer pattern

Real-world examples

Use cases

Implementation

Summary

Questions

Assessments

Other Books You May Enjoy

Section 1: Python-Native and Specialized Optimization

Python, along with its many libraries and packages, offers specialized data structures and classes that facilitate highly optimized operations. You will learn how to navigate through these different tools and use them appropriately to accelerate your Python program.

This section contains the following chapters:

Chapter 1, Benchmarking and ProfilingChapter 2, Pure Python OptimizationsChapter 3, Fast Array Operations with NumPy, Pandas, and XarrayChapter 4, C Performance with CythonChapter 5, Exploring CompilersChapter 6, Automatic Differentiation and Accelerated Linear Algebra for Machine Learning