Learn Python Programming E-Book

Fabrizio Romano

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 849

Veröffentlichungsjahr: 2024

Beschreibung

Learn Python Programming, Fourth Edition, provides a comprehensive, up-to-date introduction to Python programming, covering fundamental concepts and practical applications. This edition has been meticulously updated to include the latest features from Python versions 3.9 to 3.12, new chapters on type hinting and CLI applications, and updated examples reflecting modern Python web development practices. This Python book empowers you to take ownership of writing your software and become independent in fetching the resources you need. By the end of this book, you will have a clear idea of where to go and how to build on what you have learned from the book.

Through examples, the book explores a wide range of applications and concludes by building real-world Python projects based on the concepts you have learned. This Python book offers a clear and practical guide to mastering Python and applying it effectively in various domains, such as data science, web development, and automation.

Leseprobe

Learn Python Programming

Fourth Edition

A comprehensive, up-to-date, and definitive guide to learning Python

Fabrizio Romano

Heinrich Kruger

Packt and this book are not officially connected with Python. This book is an effort from the Python community of experts to help more developers.

Learn Python Programming

Fourth Edition

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Senior Publishing Product Manager: Denim Pinto

Acquisition Editor – Peer Reviews: Swaroop Singh

Project Editor: Amisha Vathare

Content Development Editor: Shazeen Iqbal

Copy Editor: Safis Editing

Technical Editor: Tejas Mhasvekar

Proofreader: Safis Editing

Indexer: Tejal Soni

Presentation Designer: Ganesh Bhadwalkar

Developer Relations Marketing Executive: Deepak Kumar

First published: December 2015

Second edition: June 2018

Third edition: October 2021

Fourth edition: November 2024

Production reference: 1261124

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83588-294-8

www.packt.com

To Margherita e Graziano, cuore del mio cuore. Thank you for all the love, help, and advice you’ve given me in the past twenty years

– Fabrizio Romano

To my wife, Debi, without whose love, support, and endless patience, I would not have been able to do this

– Heinrich Kruger

Contributors

About the authors

Fabrizio Romano was born in Italy in 1975. He holds a master’s degree in computer science engineering from the University of Padova. He’s been working as a professional software developer since 1999. Fabrizio has been working at Sohonet since 2016, where he currently serves as a development manager. In 2020, the Television Academy honored him and his team with an Emmy Award in Engineering Development for advancing remote collaboration.

I would like to thank all the people at Packt, in particular Amisha Vathare for her professionalism and kindness. Thanks to Stefano and Javier, the reviewers, for their insightful comments, and to Heinrich. To my wife Elisa goes my deepest gratitude: thank you for your love and support.

Heinrich Kruger was born in South Africa in 1981. He holds a master’s degree in computer science from Utrecht University in the Netherlands. He has been working as a professional software developer since 2014. Heinrich has been working alongside Fabrizio in the Product Team at Sohonet since 2017. In 2020, the Television Academy honored them with an Emmy Award in Engineering Development for advancing remote collaboration.

I want to thank Fabrizio for asking me to help him with this book. It’s been a great experience working with you, my friend. I would also like to thank Stefano Chinellato and Javier Navarro for their helpful comments, as well as everyone at Packt who helped us along the way. Most of all, I thank my wife Debi for all her love, encouragement, and support.

About the reviewer

Javier Muñoz was born in Valladolid, Spain, where he nurtured a deep interest in computers from an early age. He holds a degree in Telecommunications Engineering with a specialization in Telematics and a Master’s degree in Big Data Science from the Universidad de Valladolid, graduating with First Class Honours.

After his studies, Javier moved to Dublin, Ireland, where he worked as a software engineer and data analyst. His career progressed when he joined Optum, a Fortune 5 healthcare technology company, where he focused on AI. At Optum, he developed and deployed AI models to enhance medical diagnosis and treatment, collaborating with healthcare professionals to improve patient outcomes while ensuring compliance with regulatory standards like HIPAA and GDPR.

In parallel with his work at Optum, Javier founded Saoi Tech Solutions, a startup that delivered large-scale machine learning solutions for multinational corporations. His projects included pioneering research in facial recognition, and he demonstrated expertise in deploying AI models at scale using advanced cloud and containerization technologies.

Currently, Javier works in the United Kingdom as a software engineer at Sohonet, where he focuses on Python-based applications written in Django, primarily for the cinema and entertainment industry. His work at Sohonet involves developing tools that facilitate collaboration and enhance production workflows in film and television.

Javier’s technical expertise, particularly in Python and AI, has earned him a role as a technical reviewer for Learn Python Programming, Fourth Edition, where he ensures the content is practical, accurate, and relevant for today’s programmers.

I would like to sincerely thank my friends and colleagues Fabrizio and Heinrich, the authors of this book, for their incredible support, collaboration, and the opportunity to contribute as a technical reviewer. I am also grateful to the rest of the Sohonet team, as well as my past colleagues and mentors, whose guidance has been invaluable throughout my journey. This experience has been both rewarding and enriching, and I deeply appreciate everyone who has been part of it.

Stefano Chinellato is a software engineer residing in Italy. He holds a Master’s degree in Computer Science Engineering from the University of Padova. With over 15 years of experience in software development, he focuses on web applications and has a strong interest in algorithms and statistics.

Contents

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

A Gentle Introduction to Python

A brief introduction to programming

Enter the Python

About Python

Portability

Coherence

Developer productivity

An extensive library

Software quality

Software integration

Data science

Satisfaction and enjoyment

What are the drawbacks?

Who is using Python today?

Setting up the environment

Installing Python

Useful installation resources

Installing Python on Windows

Installing Python on macOS

Installing Python on Linux

The Python console

About virtual environments

Your first virtual environment

Installing third-party libraries

The console

How to run a Python program

Running Python scripts

Running the Python interactive shell

Running Python as a service

Running Python as a GUI application

How is Python code organized?

How do we use modules and packages?

Python’s execution model

Names and namespaces

Scopes

Guidelines for writing good code

Python culture

A note on IDEs

A word about AI

Summary

Built-In Data Types

Everything is an object

Mutability

Numbers

Integers

Booleans

Real numbers

Complex numbers

Fractions and decimals

Immutable sequences

Strings and bytes

Encoding and decoding strings

Indexing and slicing strings

String formatting

Tuples

Mutable sequences

Lists

Bytearrays

Set types

Mapping types: dictionaries

Data types

Dates and times

The standard library

Third-party libraries

The collections module

namedtuple

defaultdict

ChainMap

Enums

Final considerations

Small value caching

How to choose data structures

About indexing and slicing

About names

Summary

Conditionals and Iteration

Conditional programming

The if statement

A specialized else: elif

Nesting if statements

The ternary operator

Pattern matching

Looping

The for loop

Iterating over a range

Iterating over a sequence

Iterators and iterables

Iterating over multiple sequences

The while loop

The break and continue statements

A special else clause

Assignment expressions

Statements and expressions

Using the walrus operator

A word of warning

Putting all this together

A prime generator

Applying discounts

A quick peek at the itertools module

Infinite iterators

Iterators terminating on the shortest input sequence

Combinatoric generators

Summary

Functions, the Building Blocks of Code

Why use functions?

Reducing code duplication

Splitting a complex task

Hiding implementation details

Improving readability

Improving traceability

Scopes and name resolution

The global and nonlocal statements

Input parameters

Argument-passing

Assignment to parameter names

Changing a mutable object

Passing arguments

Positional arguments

Keyword arguments

Iterable unpacking

Dictionary unpacking

Combining argument types

Defining parameters

Optional parameters

Variable positional parameters

Variable keyword parameters

Positional-only parameters

Keyword-only parameters

Combining input parameters

More signature examples

Avoid the trap! Mutable defaults

Return values

Returning multiple values

A few useful tips

Recursive functions

Anonymous functions

Function attributes

Built-in functions

Documenting your code

Importing objects

Relative imports

One final example

Summary

Comprehensions and Generators

The map, zip, and filter functions

map

zip

filter

Comprehensions

Nested comprehensions

Filtering a comprehension

Dictionary comprehensions

Set comprehensions

Generators

Generator functions

Going beyond next

The yield from expression

Generator expressions

Some performance considerations

Do not overdo comprehensions and generators

Name localization

Generation behavior in built-ins

One last example

Summary

OOP, Decorators, and Iterators

Decorators

A decorator factory

OOP

The simplest Python class

Class and object namespaces

Attribute shadowing

The self argument

Initializing an instance

OOP is about code reuse

Inheritance and composition

Accessing a base class

Multiple inheritance

Method resolution order

Class and static methods

Static methods

Class methods

Private methods and name mangling

The property decorator

The cached_property decorator

Operator overloading

Polymorphism—a brief overview

Data classes

Writing a custom iterator

Summary

Exceptions and Context Managers

Exceptions

Raising exceptions

Defining your own exceptions

Tracebacks

Handling exceptions

Exception groups

Not only for errors

Context managers

Class-based context managers

Generator-based context managers

Summary

Files and Data Persistence

Working with files and directories

Opening files

Using a context manager to open a file

Reading from and writing to a file

Reading and writing in binary mode

Protecting against overwriting an existing file

Checking for file and directory existence

Manipulating files and directories

Manipulating pathnames

Temporary files and directories

Directory content

File and directory compression

Data interchange formats

Working with JSON

Custom encoding/decoding with JSON

I/O, streams, and requests

Using an in-memory stream

Making HTTP requests

Persisting data on disk

Serializing data with pickle

Saving data with shelve

Saving data to a database

Configuration files

Common formats

The INI configuration format

The TOML configuration format

Summary

Cryptography and Tokens

The need for cryptography

Useful guidelines

Hashlib

HMAC

Secrets

Random objects

Token generation

Digest comparison

JSON Web Tokens

Registered claims

Time-related claims

Authentication-related claims

Using asymmetric (public key) algorithms

Useful references

Summary

Testing

Testing your application

The anatomy of a test

Testing guidelines

Unit testing

Writing a unit test

Mock objects and patching

Assertions

Testing a CSV generator

Boundaries and granularity

Testing the export function

Final considerations

Test-driven development

Summary

Debugging and Profiling

Debugging techniques

Debugging with print

Debugging with a custom function

Using the Python debugger

Inspecting logs

Other techniques

Reading tracebacks

Assertions

Where to find information

Troubleshooting guidelines

Where to inspect

Using tests to debug

Monitoring

Profiling Python

When to profile

Measuring execution time

Summary

Introduction to Type Hinting

Python approach to types

Duck typing

History of type hinting

Benefits of type hinting

Type annotations

Annotating functions

The Any type

Type aliases

Special forms

Optional

Union

Generics

Annotating variables

Annotating containers

Annotating tuples

Fixed-length tuples

Tuples with named fields

Tuples of arbitrary length

Abstract base classes (ABCs)

Special typing primitives

The Self type

Annotating variable parameters

Protocols

The Mypy static type checker

Some useful resources

Summary

Data Science in Brief

IPython and Jupyter Notebook

Using Anaconda

Starting a Notebook

Dealing with data

Setting up the Notebook

Preparing the data

Cleaning the data

Creating the DataFrame

Unpacking the campaign name

Unpacking the user data

Renaming columns

Computing some metrics

Cleaning everything up

Saving the DataFrame to a file

Visualizing the results

Where do we go from here?

Summary

Introduction to API Development

The Hypertext Transfer Protocol

How does HTTP work?

Response status codes

APIs – An introduction

What is an API?

What is the purpose of an API?

API protocols

API data-exchange formats

The railway API

Modeling the database

Main setup and configuration

Application settings

Station endpoints

Reading data

Creating data

Updating data

Deleting data

User authentication

Documenting the API

Where do we go from here?

Summary

CLI Applications

Command-line arguments

Positional arguments

Options

Sub-commands

Argument parsing

Building a CLI client for the railway API

Interacting with the railway API

Creating the command-line interface

Configuration files and secrets

Creating sub-commands

Implementing sub-commands

Other resources and tools

Summary

Packaging Python Applications

The Python Package Index

Packaging with Setuptools

Project layout

Development installation

Changelog

License

README

pyproject.toml

Package metadata

Versioning and dynamic metadata

Specifying dependencies

Project URLs

Scripts and entry points

Defining the package contents

Accessing metadata in your code

Building and publishing packages

Building

Publishing

Advice for starting new projects

Other files

Alternative tools

Further reading

Summary

Programming Challenges

Advent of Code

Camel Cards

Part one – problem statement

Part one – solution

Part two – problem statement

Part two – solution

Cosmic Expansion

Part one – problem statement

Part one – solution

Part two – problem statement

Part two – solution

Final considerations

Other programming challenge websites

Summary

Other Books You May Enjoy

Index

Landmarks

Cover

Index

Preface

It is our great pleasure to introduce you to the fourth edition of our book. The first one came out in 2015, and since then the book has been a top seller all around the world.

Ten years ago, being a programmer meant working with certain tools, implementing certain paradigms. Today, the landscape is different. Developers tend to be even more specialized than before, and there is great focus on things like APIs, and distributed applications. We have tried to capture the current trends, and to offer you the best foundational layer we could think of, drawing from our experience as developers who work in a fast-paced industry.

Each new edition has brought about some kind of change. Obsolete chapters were removed, new ones were added, and others again have been amended to reflect the modern ways in which software is written.

This edition features three new chapters. The first one discusses the topic of type hinting. Type hinting is not new to Python, but now we feel it has become such an established practice that the book wouldn’t have been complete without it.

The second one, which discusses the topic of CLI applications, steps in by replacing an old chapter that was about GUIs. Most of what we do with a computer today happens in a browser, and many desktop applications have been built, or rewritten, by leveraging browser components, so we felt that a whole chapter dedicated to GUIs was perhaps a bit obsolete.

Finally, the third one explores the topic of competitive programming.

The remaining chapters have been updated to reflect the latest additions to the language, and improved to make the presentation even simpler and more fluid, while still aiming at offering interesting examples to the reader.

The soul of the book, its essence, is still intact. It shouldn’t feel like yet another Python book. It is, first and foremost, about programming. It tries to convey as much information as possible and, sometimes, when the page count couldn’t allow it, it points to the resources you need to further your knowledge.

It is designed to last. It explains concepts and information in a way that should stand the test of time, for as long as possible. There is great amount of work, thinking, and meetings, that goes into making sure of that.

It is also radically different than its first edition. It is much more mature, more professional, and focuses more on the language and slightly less on projects. We think the line that strikes the balance between these two parts has been drawn in the right place.

It will require you to focus and work hard. All the code is available for download. You can clone the repository from GitHub, if you like. Please check it out. It will help you cement what you will learn when reading these pages. Code is not a static thing. It is very much alive. It changes, it morphs. You will learn much more if you take the time to explore it, change it, and break it. We have left several guidelines in the book, to help you do that.

In closing, I want to express my gratitude to my co-author, Heinrich.

This book is now as much his as it is mine. Every chapter is infused with his talent, his creativity, and the depth of his knowledge. He is also gifted with tremendous memory: he can spot a redundant piece of information in chapters that are 200 pages apart. I can’t do that.

Like me, he has spent many long nights and weekends making sure everything was presented in the best possible way. It is because I have shared this journey with him, that I have so much confidence in the quality of this work.

Our advice for you is to study these pages well, and experiment with the source code. Once you are confident in your Python skills, please don’t stop learning. Try to go beyond the language, transcend it. A senior developer should know certain concepts and master certain skills that cannot be contained within one language, it’s just not possible. Studying other languages helps to learn how to discriminate between those features that pertain to a certain language, and others that are instead more generic, related to programming. Hopefully this book will help you get there.

Enjoy the journey and, whatever you learn, please share it with others.

Fabrizio

Who this book is for

This book is for people who have some programming experience, but not necessarily with Python. Some knowledge of basic programming concepts will be useful, although it is not a strict requirement.

Even if you already have some experience with Python, this book can still be useful to you, both as a reference to Python’s fundamentals, and for providing a wide range of considerations and suggestions collected over four combined decades of experience.

What this book covers

Chapter 1, A Gentle Introduction to Python, introduces you to fundamental programming concepts and constructs of the Python language. It also guides you through getting Python up and running on your computer.

Chapter 2, Built-In Data Types, introduces you to Python built-in data types. Python has a very rich set of native data types, and this chapter will give you a description and examples for each of them.

Chapter 3, Conditionals and Iteration, teaches you how to control the flow of the code by inspecting conditions, applying logic, and performing loops.

Chapter 4, Functions, the Building Blocks of Code, teaches you how to write functions. Functions are essential to code reuse, to reducing debugging time, and, in general, to writing higher quality code.

Chapter 5, Comprehensions and Generators, introduces you to the functional aspects of Python programming. This chapter teaches you how to write comprehensions and generators, which are powerful tools that you can use to write faster, more concise code, and save memory.

Chapter 6, OOP, Decorators, and Iterators, teaches you the basics of object-oriented programming with Python. It shows you the key concepts and all the potentials of this paradigm. It also shows you one of the most useful features of the language: decorators.

Chapter 7, Exceptions and Context Managers, introduces the concept of exceptions, which represent errors that occur in applications, and how to handle them. It also covers context managers, which are very useful when dealing with resources.

Chapter 8, Files and Data Persistence, teaches you how to deal with files, streams, data interchange formats, and databases.

Chapter 9, Cryptography and Tokens, touches upon the concepts of security, hashes, encryption, and tokens, which are essential for writing secure software.

Chapter 10, Testing, teaches you the fundamentals of testing, and guides you through a few examples on how to test your code, in order to make it more robust, fast and reliable.

Chapter 11, Debugging and Profiling, shows you the main methods for debugging and profiling code and some examples of how to apply them.

Chapter 12, Introduction to Type Hinting, guides you through the syntax and main concepts of type hinting. Type hinting has become more and more popular in recent years because it enriches both the language and the tools that are part of its ecosystem.

Chapter 13, Data Science in Brief, illustrates a few key concepts by means of a comprehensive example, using the powerful Jupyter Notebook.

Chapter 14, Introduction to API Development, introduces API development using the FastAPI framework.

Chapter 15, CLI Applications, introduces command-line interface applications. They are run in a console or terminal, and are a common and natural way in which developers write several of their own day-to-day tools.

Chapter 16, Packaging Python Applications, guides you through the process of preparing a project to be published, and shows you how to upload the result on the Python Package Index (PyPI).

Chapter 17, Programming Challenges, introduces the concept of competitive programming, by showing you how to solve two problems from the Advent of Code website.

To get the most out of this book

You are encouraged to follow the examples in this book. You will need a computer, an internet connection, and a browser. The book is written for Python 3.12, but it should also work, for the most part, with any recent version of Python 3. We have given guidelines on how to install Python on your operating system. The procedures to do that normally get out of date quickly, so we recommend you refer to the most up-to-date guide on the web to find precise setup instructions. We have also explained how to install all the extra libraries used in the various chapters. No particular editor is required to type the code; however, we suggest that those who are interested in following the examples should consider adopting a proper coding environment. We have offered suggestions on this matter in the first chapter.

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781835882948.

Download the example code files

The code bundle for the book is hosted on GitHub at https://github.com/PacktPublishing/Learn-Python-Programming-Fourth-Edition. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. For example: “The as_integer_ratio() method has also been added to integers and Booleans.”

Bold: Indicates a new term, an important word, or words that you see on the screen. For instance, words in menus or dialog boxes appear in the text like this. For example: “They are immutable sequences of Unicode code points.”

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email [email protected] and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you reported this to us. Please visit http://www.packtpub.com/submit-errata, click Submit Errata, and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.

Share your thoughts

Once you’ve read Learn Python Programming, Fourth Edition, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily.

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below:

https://packt.link/free-ebook/9781835882948

Submit your proof of purchase.That’s it! We’ll send your free PDF and other benefits to your email directly.

1

A Gentle Introduction to Python

”Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime.”

—Chinese proverb

Computer programming, or coding, as it is sometimes known, is telling a computer to do something using a language it understands.

Computers are very powerful tools, but unfortunately, they cannot think for themselves. They need to be told everything: how to perform a task; how to evaluate a condition to decide which path to follow; how to handle data that comes from a device, such as a network or a disk; and how to react when something unforeseen happens, in the case of, say, something being broken or missing.

You can code in many different styles and languages. Is it hard? We would say yes and no. It is a bit like writing—it is something that everybody can learn. But what if you want to become a poet? Writing alone is not enough. You have to acquire a whole other set of skills, and this will involve a longer and greater effort.

In the end, it all comes down to how far you want to go down the road. Coding is not just putting together some instructions that work. It is so much more!

Good code is short, fast, elegant, easy to read and understand, simple, easy to modify and extend, easy to scale and refactor, and easy to test. It takes time to be able to write code that has all these qualities at the same time, but the good news is that you are taking the first step toward it at this very moment by reading this book. And we have no doubt you can do it. Anyone can; in fact, we all program all the time, only we are not aware of it.

Let’s say, for example, that you want to make instant coffee. You have to get a mug, the instant coffee jar, a teaspoon, water, and a kettle. Even if you are not aware of it, you are evaluating a lot of data. You are making sure that there is water in the kettle and that the kettle is plugged in, that the mug is clean, and that there is enough coffee in the jar. Then you boil the water and, maybe in the meantime, you put some coffee in the mug. When the water is ready, you pour it into the mug, and stir.

So, how is this programming?

Well, we gathered resources (the kettle, coffee, water, teaspoon, and mug) and we verified some conditions concerning them (the kettle is plugged in, the mug is clean, and there is enough coffee). Then we started two actions (boiling the water and putting coffee in the mug), and when both of them were completed, we finally ended the procedure by pouring water into the mug and stirring.

Can you see the parallel? We have just described the high-level functionality of a coffee program. It was not that hard because this is what the brain does all day long: evaluate conditions, decide to take actions, carry out tasks, repeat some of them, and stop at some point.

All you need now is to learn how to deconstruct all those actions you do automatically in real life so that a computer can actually make some sense of them. You need to learn a language as well so that the computer can be instructed.

So, this is what this book is for. We will show you one way in which you can code successfully, and we will try to do that by means of many simple but focused examples (our favorite kind).

In this chapter, we are going to cover the following:

Python’s characteristics and ecosystemGuidelines on how to get up and running with Python and virtual environmentsHow to run Python programsHow to organize Python code and its execution model

A brief introduction to programming

We love to make references to the real world when we teach coding; we believe they help people to better retain the concepts they are learning. However, now is the time to be a bit more rigorous and see what coding is from a more technical perspective.

When we write code, we are instructing a computer about the things it has to do. Where does the action happen? In many places: the computer memory, hard drives, network cables, the CPU, and so on. It is a whole world, which most of the time is the representation of a subset of the real world.

If you write a piece of software that allows people to buy clothes online, you will have to represent real people, real clothes, real brands, sizes, and so on and so forth, within the boundaries of a program.

To do this, you will need to create and handle objects in your program. A person can be an object. A car is an object. A pair of trousers is an object. Luckily, Python understands objects very well.

The two key features any object has are properties and methods. Let us take the example of a person as an object. Typically, in a computer program, you will represent people as customers or employees. The properties that you store against them are things like a name, a social security number, an age, whether they have a driving license, an email, and so on. In a computer program, you store all the data needed in order to use an object for the purpose that needs to be served. If you are coding a website to sell clothes, you probably want to store the heights and weights as well as other measures of your customers so that the appropriate clothes can be suggested to them. So, properties are characteristics of an object. We use them all the time: Could you pass me that pen? —Which one? —The black one. Here, we used the color (black) property of a pen to identify it (most likely it was being kept alongside different colored pens for the distinction to be necessary).

Methods are actions that an object can perform. As a person, I have methods such as speak, walk, sleep, wake up, eat, dream, write, read, and so on. All the things that I can do could be seen as methods of the objects that represent me.

So, now that you know what objects are, that they provide methods that can be run and properties that you can inspect, you are ready to start coding. Coding, in fact, is simply about managing those objects that live in the subset of the world we’re reproducing in our software. You can create, use, reuse, and delete objects as you please.

According to the Data Model chapter on the official Python documentation (https://docs.python.org/3/reference/datamodel.html):

“Objects are Python’s abstraction for data. All data in a Python program is represented by objects or by relations between objects.”

We will take a closer look at Python objects in Chapter 6, OOP, Decorators, and Iterators. For now, all we need to know is that every object in Python has an ID (or identity), a type, and a value.

Object-Oriented Programming (OOP) is just one of many programming paradigms. In Python, we can write code using a functional or imperative style, as well as object-oriented. However, as we previously stated, everything in Python is an object, therefore we employ them all the time, regardless of the chosen coding style.

Once created, the ID of an object never changes. It is a unique identifier for it, and it is used behind the scenes by Python to retrieve the object when we want to use it. The type also never changes. The type states what operations are supported by the object and the possible values that can be assigned to it. We will see Python’s most important data types in Chapter 2, Built-In Data Types. The value of some objects can change. Such objects are said to be mutable. If the value cannot be changed, the object issaid to be immutable.

How, then, do we use an object? We give it a name, of course! When you give an object a name, then you can use the name to retrieve the object and use it. In a more generic sense, objects, such as numbers, strings (text), and collections, are associated with a name. In other languages, the name is normally called a variable. You can see the variable as being like a box, which you can use to hold data.

Objects represent data. It is stored in databases or sent over network connections. It is what you see when you open any web page, or work on a document. Computer programs manipulate that data to perform all sorts of actions. They regulate its flow, evaluate conditions, react to events, and much more.

To do all this, we need a language. That is what Python is for. Python is the language we will use together throughout this book to instruct the computer to do something for us.

Enter the Python

Python is the marvelous creation of Guido Van Rossum, a Dutch computer scientist and mathematician who decided to gift the world with a project he was playing around with over Christmas 1989. The language appeared to the public somewhere around 1991, and since then has evolved to be one of the leading programming languages used worldwide today.

We (the authors) started programming when we were both very young. Fabrizio started at the age of 7, on a Commodore VIC-20, which was later replaced by its bigger brother, the Commodore 64. The language it used was BASIC. Heinrich started when he learned Pascal in high school. Between us, we’ve programmed in Pascal, Assembly, C, C++, Java, JavaScript, Visual Basic, PHP, ASP, ASP .NET, C#, and plenty of others we can’t even remember; only when we landed on Python did we finally get the feeling that you go through when you find the right couch in the shop, when all of your body is yelling: Buy this one! This one is perfect!

It took us about a day to become accustomed to it. Its syntax is a bit different from what we were used to, but after getting past that initial feeling of discomfort (like having new shoes), we both just fell in love with it. Deeply. Let us see why.

About Python

Before we get into the gory details, let us get a sense of why someone would want to use Python. It embodies the following qualities.

Portability

Python runs everywhere, and porting a program from Linux to Windows or Mac is usually just a matter of fixing paths and settings. Python is designed for portability, and it takes care of specific operating system (OS) quirks behind interfaces that shield you from the pain of having to write code tailored to a specific platform.

Coherence

Python is extremely logical and coherent. You can see it was designed by a brilliant computer scientist. Most of the time, you can just guess what a method is called if you do not know it.

You may not realize how important this is right now, especially if you are not that experienced as a programmer, but this is a major feature. It means less clutter in your head, as well as less skimming through the documentation, and less need for mappings in your brain when you code.

Developer productivity

According to Mark Lutz (Learning Python, 5th Edition, O’Reilly Media), a Python program is typically one-fifth to one-third the size of equivalent Java or C++ code. This means the job gets done faster. And faster is good. Faster means being able to respond more quickly to the market. Less code not only means less code to write, but also less code to read (and professional coders read much more than they write), maintain, debug, and refactor.

Another important aspect is that Python runs without the need for lengthy and time-consuming compilation and linkage steps, so there is no need to wait to see the results of your work.

An extensive library

Python has an incredibly extensive standard library (it is said to come with batteries included). If that wasn’t enough, the Python international community maintains a body of third-party libraries, tailored to specific needs, which you can access freely at the Python Package Index (PyPI). When you code in Python and realize that a certain feature is required, in most cases, there is at least one library where that feature has already been implemented.

Software quality

Python is heavily focused on readability, coherence, and quality. The language’s uniformity allows high readability, and this is crucial nowadays, as coding is more of a collective effort than a solo endeavor. Another important aspect of Python is its intrinsic multiparadigm nature. You can use it as a scripting language, but you can also employ object-oriented, imperative, and functional programming styles—it is extremely versatile.

Software integration

Another important aspect is that Python can be extended and integrated with many other languages, which means that even when a company is using a different language as their mainstream tool, Python can come in and act as a gluing agent between complex applications that need to talk to each other in some way. This is more of an advanced topic, but in the real world, this feature is important.

Data science

Python is among the most popular (if not the most popular) languages used in the fields of data science, machine learning, and artificial intelligence today. Knowledge of Python is therefore almost essential for those who want to have a career in these fields.

Satisfaction and enjoyment

Last, but by no means least, there is the fun of it! Working with Python is fun; we can code for eight hours and leave the office happy and satisfied, unaffected by the struggle other coders have to endure because they use languages that do not provide them with the same amount of well-designed data structures and constructs. Python makes coding fun, no doubt about it, and fun promotes motivation and productivity.

These are the major reasons why we would recommend Python to everyone. Of course, there are many other technical and advanced features that we could have mentioned, but they do not really pertain to an introductory section like this one. They will come up naturally, chapter after chapter, as we learn about Python in greater detail.

Now, let’s look at what the potential limitations of Python are.

What are the drawbacks?

Aside from personal preferences, the primary drawback of Python lies in its execution speed. Typically, Python is slower than its compiled siblings. The standard implementation of Python produces, when you run an application, a compiled version of the source code called byte code (with the extension .pyc), which is then run by the Python interpreter. The advantage of this approach is portability, which we pay for with increased runtimes because Python is not compiled down to the machine level, as other languages are.

Despite this, Python speed is rarely a problem today, hence its wide use regardless of this downside. What happens is that, in real life, hardware cost is no longer a problem, and usually you can gain speed by parallelizing tasks. Moreover, many programs spend a great proportion of the time waiting for I/O operations to complete; therefore, the raw execution speed is often a secondary factor to the overall performance.

It is worth noting that Python’s core developers have put great effort into speeding up operations on the most common data structures in the last few years. This effort, in some cases very successful, has somewhat alleviated this issue.

In situations where speed really is crucial, one can switch to faster Python implementations, such as PyPy, which provides, on average, just over a four-fold speedup by implementing advanced compilation techniques (check https://pypy.org/ for reference). It is also possible to write performance-critical parts of your code in faster languages, such as C or C++, and integrate that with your Python code. Libraries such as pandas and NumPy (which are commonly used for doing data science in Python) use such techniques.

There are a few different implementations of the Python language. In this book, we will use the reference implementation, known as CPython. You can find a list of other implementations at https://www.python.org/download/alternatives/.

If that is not convincing enough, you can always consider that Python has been used to drive the backend of services such as Spotify and Instagram, where performance is a concern. From this, it can be seen that Python has done its job perfectly well.

Who is using Python today?

Python is used in many different contexts, such as system programming, web and API programming, GUI applications, gaming and robotics, rapid prototyping, system integration, data science, database applications, real-time communication, and much more. Several prestigious universities have also adopted Python as their main language in computer science courses.

Here is a list of major companies and organizations that are known to use Python in their technology stack, product development, data analysis, or automation processes:

Tech industryGoogle: Uses Python for many tasks including backend services, data analysis, and artificial intelligence (AI)Facebook: Utilizes Python for various purposes, including infrastructure management and operational automationInstagram: Relies heavily on Python for its backend, making it one of the largest Django (a Python web framework) usersSpotify: Employs Python mainly for data analysis and backend servicesNetflix: Uses Python for data analysis, operational automation, and securityFinancial sectorJP Morgan Chase: Uses Python for financial models, data analysis, and algorithmic tradingGoldman Sachs: Employs Python for various financial models and applicationsBloomberg: Uses Python for financial data analysis and its Bloomberg Terminal interfaceTechnology and softwareIBM: Utilizes Python for AI, machine learning, and cybersecurityIntel: Uses Python for hardware testing and development processesDropbox: The desktop client is largely written in PythonSpace and researchNASA: Uses Python for various purposes, including data analysis and system integrationCERN: Employs Python for data processing and analysis in physics experimentsRetail and e-CommerceAmazon: Uses Python for data analysis, product recommendations, and operational automationeBay: Utilizes Python for various backend services and data analysisEntertainment and mediaPixar: Uses Python for animation software and scripting in the animation processIndustrial Light & Magic (ILM): Employs Python for visual effects and image processingEducation and learning platformsCoursera: Utilizes Python for web development and backend servicesKhan Academy: Uses Python for educational content delivery and backend servicesGovernment and non-profitThe United States Federal Government: Has various departments and agencies using Python for data analysis, cybersecurity, and automationThe Raspberry Pi Foundation: Uses Python as a primary programming language for educational purposes and projects

Setting up the environment

On our machines (MacBook Pro), this is the latest Python version:

>>> import sys>>> print(sys.version) 3.12.2 (main, Feb 14 2024, 14:16:36) [Clang 15.0.0 (clang-1500.1.0.2.5)]

So, you can see that the version is 3.12.2, which was out on October 2, 2023. The preceding text is a little bit of Python code that was typed into a console. We will talk about this in a moment.

All the examples in this book will be run using Python 3.12. If you wish to follow the examples and download the source code for this book, please make sure you are using the same version.

Installing Python

The process of installing Python on your computer depends on the operating system you have. First of all, Python is fully integrated and, most likely, already installed in almost every Linux distribution. If you have a recent version of macOS, it is likely that Python 3 is already there as well, whereas if you are using Windows, you probably need to install it.

Regardless of Python being already installed in your system, you will need to make sure that you have version 3.12 installed.

To check if you have Python already installed on your system, try typing python --version or python3 --version in a command prompt (more on this later).

The place you want to start is the official Python website: https://www.python.org. This website hosts the official Python documentation and many other resources that you will find very useful.

Useful installation resources

The Python website hosts useful information regarding the installation of Python on various operating systems. Please refer to the relevant page for your operating system.

Windows and macOS:

https://docs.python.org/3/using/windows.htmlhttps://docs.python.org/3/using/mac.html

For Linux, please refer to the following links:

https://docs.python.org/3/using/unix.htmlhttps://ubuntuhandbook.org/index.php/2023/05/install-python-3-12-ubuntu/

Installing Python on Windows

As an example, this is the procedure to install Python on Windows. Head to https://www.python.org/downloads/ and download the appropriate installer according to the CPU of your computer.

Once you have it, you can double-click on it in order to start the installation.

Figure 1.1: Starting the installation process on Windows

We recommend choosing the default install, and NOT ticking the Add python.exe to PATH option to prevent clashes with other versions of Python that might be installed on your machine, potentially by other users.

For a more comprehensive set of guidelines, please refer to the link indicated in the previous paragraph.

Once you click on Install Now, the installation procedure will begin.

Figure 1.2: Installation in progress

Once the installation is complete, you will land on the final screen.

Figure 1.3: Installation complete

Click on Close to finish the installation.

Now that Python is installed on your system, open a command prompt and run the Python interactive shell by typing py. This command will select the latest version of Python installed on your machine. At the time of writing, 3.12 is the latest available version of Python. If you have a more recent version installed, you can specify the version with the command py -3.12.

To open the command prompt in Windows, go to the Start menu and type cmd in the search box to start your terminal up. Alternatively, you can also use Powershell.

Installing Python on macOS

On macOS, the installation procedure is similar to that of Windows. Once you have downloaded the appropriate installer for your machine, complete the installation steps, and then start a terminal by going to Applications > Utilities > Terminal. Alternatively, you can install it through Homebrew.

Once in the terminal window, you can type python. If that launches the wrong version, you can try and specify the version with either python3 or python3.12.

Installing Python on Linux

The process of installing Python on Linux is normally a bit more complex than that for Windows or macOS. The best course of action, if you are on a Linux machine, is to search for the most up-to-date set of steps for your distribution online. These will likely be quite different from one distribution to another, so it is difficult to give an example that would be relevant for everyone. Please refer to the link in the Useful installation resources section for guidance.

The Python console

We will use the term console interchangeably to indicate the Linux console, the Windows Command Prompt or Powershell, and the macOS Terminal. We will also indicate the command-line prompt with the default Linux format, like this:

$ sudo apt-get update

If you are not familiar with that, please take some time to learn the basics of how a console works. In a nutshell, after the $ sign, you will type your instructions. Pay attention to capitalization and spaces, as they are very important.

Whatever console you open, type python at the prompt (py on Windows) and make sure the Python interactive shell appears. Type exit() to quit. Keep in mind that you may have to specify python3 or python3.12 if your OS comes with other Python versions preinstalled.

We often refer to the Python interactive shell simply as the Python console.

This is roughly what you should see when you run Python (some details will change according to the version and OS):

$ python Python 3.12.2 (main, Feb 14 2024, 14:16:36) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

Now that Python is set up and you can run it, it is time to make sure you have the other tool that will be indispensable to follow the examples in the book: a virtual environment.

About virtual environments

When working with Python, it is very common to use virtual environments. Let us see what they are and why we need them by means of a simple example.

You install Python on your system, and you start working on a website for client X. You create a project folder and start coding. Along the way, you also install some libraries, for example, the Django framework. Let us say the Django version you installed for Project X is 4.2.

Now, your website is so good that you get another client, Y. She wants you to build another website, so you start Project Y and, along the way, you need to install Django again. The only issue is that now the Django version is 5.0 and you cannot install it on your system because this would replace the version you installed for Project X. You do not want to risk introducing incompatibility issues, so you have two choices: either you stick with the version you have currently on your machine, or you upgrade it and make sure the first project is still fully working correctly with the new version.

Let us be honest; neither of these options is very appealing, right? Definitely not. But there is a solution: virtual environments!

Virtual environments are isolated Python environments, each of which is a folder that contains all the necessary executables to use the packages that a Python project would need (think of packages as libraries for the time being).

So, you create a virtual environment for Project X, install all the dependencies, and then you create a virtual environment for Project Y, and install all its dependencies without the slightest worry because every library you install ends up within the boundaries of the appropriate virtual environment. In our example, Project X will hold Django 4.2, while Project Y will hold Django 5.0.

It is of great importance that you never install libraries directly at the system level. Linux, for example, relies on Python for many different tasks and operations, and if you fiddle with the system installation of Python, you risk compromising the integrity of the entire system. So, take this as a rule: always create a virtual environment when you start a new project.

When it comes to creating a virtual environment on your system, there are a few different methods to carry this out. Since Python 3.5, the suggested way to create a virtual environment is to use the venv module. You can look it up on the official documentation page (https://docs.python.org/3/library/venv.html) for further information.

If you are using a Debian-based distribution of Linux, for example, you will need to install the venv module before you can use it:

$ sudo apt-get install python3.12-venv

Another common way of creating virtual environments is to use the virtualenv third-party Python package. You can find it on its official website: https://virtualenv.pypa.io.

In this book, we will use the recommended technique, which leverages the venv module from the Python standard library.

Your first virtual environment

It is very easy to create a virtual environment, but depending on how your system is configured and which Python version you want the virtual environment to run on, you need to run the command properly. Another thing you will need to do when you want to work with it is to activate it. Activating virtual environments produces some path juggling behind the scenes so that when you call the Python interpreter from that shell, it will come from within the virtual environment, instead of the system. We will show you a full example on macOS and Windows (on Linux, it will be very similar to that of macOS). We will:

Open a terminal and change into the folder (directory) we use as root for our projects (our folder is code). We are going to create a new folder called my-project and change into it.Create a virtual environment called lpp4ed.After creating the virtual environment, we will activate it. The methods are slightly different between Linux, macOS, and Windows.Then, we will make sure that we are running the desired Python version (3.12.X) by running the Python interactive shell.Finally, we will deactivate the virtual environment.

Some developers prefer to call all virtual environments with the same name (for example, .venv). This way, they can configure tools and run scripts against any virtual environment by just knowing their location. The dot in .venv is there because in Linux/macOS, prepending a name with a dot makes that file or folder “invisible.”

These steps are all you need to start a project.

We are going to start with an example on macOS (note that you might get a slightly different result, according to your OS, Python version, and so on). In this listing, lines that start with a hash, #, are comments, spaces have been introduced for readability, and an arrow, →, indicates where the line has wrapped around due to lack of space:

fab@m1:~/code$ mkdir my-project # step 1 fab@m1:~/code$ cd my-project fab@m1:~/code/my-project$ which python3.12 # check system python /usr/bin/python3.12 # <-- system python3.12 fab@m1:~/code/my-project$ python3.12 -m venv lpp4ed # step 2 fab@m1:~/code/my-project$ source ./lpp4ed/bin/activate # step 3 # check python again: now using the virtual environment's one (lpp4ed) fab@m1:~/code/my-project$ which python /Users/fab/code/my-project/lpp4ed/bin/python (lpp4ed) fab@m1:~/code/my-project$ python # step 4 Python 3.12.2 (main, Feb 14 2024, 14:16:36) → [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> exit() (lpp4ed) fab@m1:~/code/my-project$ deactivate # step 5 fab@m1:~/code/my-project$

Each step has been marked with a comment, so you should be able to follow along quite easily.

Something to notice here is that to activate the virtual environment, we need to run the lpp4ed/bin/activate script, which needs to be sourced. When a script is sourced, it means that it is executed in the current shell, and its effects last after the execution. This is very important. Also notice how the prompt changes after we activate the virtual environment, showing its name on the left (and how it disappears when we deactivate it).

On a Windows 11 PowerShell, the steps are as follows:

PS C:\Users\H\Code> mkdir my-project # step 1 PS C:\Users\H\Code> cd .\my-project\ # check installed python versions PS C:\Users\H\Code\my-project> py --list-paths -V:3.12 * → C:\Users\H\AppData\Local\Programs\Python\Python312\python.exe PS C:\Users\H\Code\my-project> py -3.12 -m venv lpp4ed # step 2 PS C:\Users\H\Code\my-project> .\lpp4ed\Scripts\activate # step 3 # check python versions again: now using the virtual environment's (lpp4ed) PS C:\Users\H\Code\my-project> py --list-paths * → C:\Users\H\Code\my-project\lpp4ed\Scripts\python.exe -V:3.12 → C:\Users\H\AppData\Local\Programs\Python\Python312\python.exe (lpp4ed) PS C:\Users\H\Code\my-project> python # step 4 Python 3.12.2 (tags/v3.12.2:6abddd9, Feb 6 2024, 21:26:36) → [MSC v.1937 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more → information. >>> exit() (lpp4ed) PS C:\Users\H\Code\my-project> deactivate # step 5 PS C:\Users\H\Code\my-project>

Notice how, on Windows, after activating the virtual environment, you can either use the py command or, more directly, python.

At this point, you should be able to create and activate a virtual environment. Please try and create another one on your own. Get acquainted with this procedure—it is something that you will always be doing: we never work system-wide with Python, remember? Virtual environments are extremely important.

The source code for the book contains a dedicated folder for each chapter. When the code shown in the chapter requires third-party libraries to be installed, we will include a requirements.txt file (or an equivalent requirements folder with more than one text file inside) that you can use to install the libraries required to run that code. We suggest that when experimenting with the code for a chapter, you create a dedicated virtual environment for that chapter. This way, you will be able to get some practice in the creation of virtual environments, and the installation of third-party libraries.

Installing third-party libraries

In order to install third-party libraries, we need to use the Python Package Installer, known as pip. Chances are that it is already available to you within your virtual environment, but if not, you can learn all about it on its documentation page: https://pip.pypa.io.

The following example shows how to create a virtual environment and install a couple of third-party libraries taken from a requirements file:

fab@m1:~/code$ mkdir my-project fab@m1:~/code$ cd my-project fab@m1:~/code/my-project$ python3.12 -m venv lpp4ed fab@m1:~/code/my-project$ source ./lpp4ed/bin/activate (lpp4ed) fab@m1:~/code/my-project$ cat requirements.txt django==5.0.3 requests==2.31.0 # the following instruction shows how to use pip to install # requirements from a file (lpp4ed) fab@m1:~/code/my-project$ pip install -r requirements.txt Collecting django==5.0.3 (from -r requirements.txt (line 1)) Using cached Django-5.0.3-py3-none-any.whl.metadata (4.2 kB) Collecting requests==2.31.0 (from -r requirements.txt (line 2)) Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) ... more collecting omitted ... Installing collected packages: ..., requests, django Successfully installed ... django-5.0.3 requests-2.31.0 (lpp4ed) fab@m1:~/code/my-project$

As you can see at the bottom of the listing, pip has installed both libraries that are in the requirements file, plus a few more. This happened because both django and requests have their own list of third-party libraries that they depend upon, and therefore pip will install them automatically for us.

Using a requirements.txt file is just one of the many ways of installing third-party libraries in Python. You can also just specify the names of the packages you want to install directly, for example, pip install django.

You can find more information in the pip user guide: https://pip.pypa.io/en/stable/user_guide/.