Pandas Basics E-Book

Oswald Campesato

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 262

Veröffentlichungsjahr: 2024

Beschreibung

This book is essential for aspiring data scientists and anyone needing to perform data cleaning using Pandas and NumPy. It offers numerous code samples and comprehensive coverage of NumPy and Pandas features, including writing regular expressions. Chapter 3 introduces fundamental statistical concepts, while Chapter 7 delves into data visualization using Matplotlib and Seaborn. Companion files with code are available for download from the publisher.
Starting with an introduction to Python, the course progresses through working with data, and then moves into Pandas, covering its functionalities in three detailed chapters. The statistical concepts provided are crucial for analyzing data effectively, while the visualization techniques enhance the ability to present data insights clearly.
By the end of this course, users will have a solid foundation in data manipulation and cleaning, statistical analysis, and data visualization, enabling them to tackle real-world data science tasks confidently and efficiently.

Leseprobe

PANDAS BASICS

LICENSE, DISCLAIMER OF LIABILITY, AND LIMITED WARRANTY

By purchasing or using this book and companion files (the “Work”), you agree that this license grants permission to use the contents contained herein, including the disc, but does not give you the right of ownership to any of the textual content in the book / disc or ownership to any of the information or products contained in it. This license does not permit uploading of the Work onto the Internet or on a network (of any kind) without the written consent of the Publisher. Duplication or dissemination of any text, code, simulations, images, etc. contained herein is limited to and subject to licensing terms for the respective products, and permission must be obtained from the Publisher or the owner of the content, etc., in order to reproduce or network any portion of the textual material (in any media) that is contained in the Work.

MERCURY LEARNING AND INFORMATION (“MLI” or “the Publisher”) and anyone involved in the creation, writing, or production of the companion disc, accompanying algorithms, code, or computer programs (“the software”), and any accompanying Web site or software of the Work, cannot and do not warrant the performance or results that might be obtained by using the contents of the Work. The author, developers, and the Publisher have used their best efforts to ensure the accuracy and functionality of the textual material and/or programs contained in this package; we, however, make no warranty of any kind, express or implied, regarding the performance of these contents or programs. The Work is sold “as is” without warranty (except for defective materials used in manufacturing the book or due to faulty workmanship).

The author, developers, and the publisher of any accompanying content, and anyone involved in the composition, production, and manufacturing of this work will not be liable for damages of any kind arising out of the use of (or the inability to use) the algorithms, source code, computer programs, or textual material contained in this publication. This includes, but is not limited to, loss of revenue or profit, or other incidental, physical, or consequential damages arising out of the use of this Work.

The sole remedy in the event of a claim of any kind is expressly limited to replacement of the book and/or disc, and only at the discretion of the Publisher. The use of “implied warranty” and certain “exclusions” vary from state to state, and might not apply to the purchaser of this product.

Companion files for this title are available by writing to the publisher at [email protected].

PANDAS BASICS

Oswald Campesato

MERCURY LEARNING AND INFORMATION

Dulles, Virginia

Boston, Massachusetts

New Delhi

Copyright ©2023 by MERCURY LEARNING AND INFORMATION LLC. All rights reserved.

This publication, portions of it, or any accompanying software may not be reproduced in any way, stored in a retrieval system of any type, or transmitted by any means, media, electronic display or mechanical display, including, but not limited to, photocopy, recording, Internet postings, or scanning, without prior permission in writing from the publisher.

Publisher: David Pallai

MERCURY LEARNING AND INFORMATION

22841 Quicksilver Drive

Dulles, VA 20166

[email protected]

www.merclearning.com

800-232-0223

O. Campesato. Pandas Basics.

ISBN: 978-1-68392-826-3

The publisher recognizes and respects all marks used by companies, manufacturers, and developers as a means to distinguish their products. All brand names and product names mentioned in this book are trademarks or service marks of their respective companies. Any omission or misuse (of any kind) of service marks or trademarks, etc. is not an attempt to infringe on the property of others.

Library of Congress Control Number: 2022945212

222324321     This book is printed on acid-free paper in the United States of America.

Our titles are available for adoption, license, or bulk purchase by institutions, corporations, etc. For additional information, please contact the Customer Service Dept. at 800-232-0223(toll free).

All of our titles are available in digital format at academiccourseware.com and other digital vendors. Companion files (figures and code listings) for this title are available by [email protected]. The sole obligation of MERCURY LEARNING AND INFORMATION to the purchaser is to replace the disc, based on defective materials or faulty workmanship, but not based on the operation or functionality of the product.

I’d like to dedicate this book to my parents —may this bring joy and happiness into their lives.

CONTENTS

Preface

Chapter 1: Introduction to Python

Tools for Python

easy_install and pip

virtualenv

IPython

Python Installation

Setting the PATH Environment Variable (Windows Only)

Launching Python on Your Machine

The Python Interactive Interpreter

Python Identifiers

Lines, Indentation, and Multi-lines

Quotations and Comments

Saving Your Code in a Module

Some Standard Modules

The help() and dir() Functions

Compile Time and Runtime Code Checking

Simple Data Types

Working with Numbers

Working with Other Bases

The chr() Function

The round() Function

Formatting Numbers

Working with Fractions

Unicode and UTF-8

Working with Unicode

Working with Strings

Comparing Strings

Formatting Strings

Uninitialized Variables and the Value None

Slicing and Splicing Strings

Testing for Digits and Alphabetic Characters

Search and Replace a String in Other Strings

Remove Leading and Trailing Characters

Printing Text without NewLine Characters

Text Alignment

Working with Dates

Converting Strings to Dates

Exception Handling

Handling User Input

Command-line Arguments

Summary

Chapter 2: Working with Data

Dealing with Data: What Can Go Wrong?

What is Data Drift?

What are Datasets?

Data Preprocessing

Data Types

Preparing Datasets

Discrete Data Versus Continuous Data

Binning Continuous Data

Scaling Numeric Data via Normalization

Scaling Numeric Data via Standardization

Scaling Numeric Data via Robust Standardization

What to Look for in Categorical Data

Mapping Categorical Data to Numeric Values

Working with Dates

Working with Currency

Working with Outliers and Anomalies

Outlier Detection/Removal

Finding Outliers with NumPy

Finding Outliers with Pandas

Calculating Z-scores to Find Outliers

Finding Outliers with SkLearn (Optional)

Working with Missing Data

Imputing Values: When is Zero a Valid Value?

Dealing with Imbalanced Datasets

What is SMOTE?

SMOTE extensions

The Bias-Variance Tradeoff

Types of Bias in Data

Analyzing Classifiers (Optional)

What is LIME?

What is ANOVA?

Summary

Chapter 3: Introduction to Probability and Statistics

What is a Probability?

Calculating the Expected Value

Random Variables

Discrete versus Continuous Random Variables

Well-known Probability Distributions

Fundamental Concepts in Statistics

The Mean

The Median

The Mode

The Variance and Standard Deviation

Population, Sample, and Population Variance

Chebyshev’s Inequality

What is a p-value?

The Moments of a Function (Optional)

What is Skewness?

What is Kurtosis?

Data and Statistics

The Central Limit Theorem

Correlation versus Causation

Statistical Inferences

Statistical Terms: RSS, TSS, R^2, and F1 Score

What is an F1 score?

Gini Impurity, Entropy, and Perplexity

What is the Gini Impurity?

What is Entropy?

Calculating the Gini Impurity and Entropy Values

Multi-dimensional Gini Index

What is Perplexity?

Cross-Entropy and KL Divergence

What is Cross-Entropy?

What is KL Divergence?

What’s Their Purpose?

Covariance and Correlation Matrices

The Covariance Matrix

Covariance Matrix: An Example

The Correlation Matrix

Eigenvalues and Eigenvectors

Calculating Eigenvectors: A Simple Example

Gauss Jordan Elimination (Optional)

PCA (Principal Component Analysis)

The New Matrix of Eigenvectors

Well-known Distance Metrics

Pearson Correlation Coefficient

Jaccard Index (or Similarity)

Local Sensitivity Hashing (Optional)

Types of Distance Metrics

What is Bayesian Inference?

Bayes’ Theorem

Some Bayesian Terminology

What is MAP?

Why Use Bayes’ Theorem?

Summary

Chapter 4: Introduction to Pandas (1)

What is Pandas?

Pandas Options and Settings

Pandas Data Frames

Data Frames and Data Cleaning Tasks

Alternatives to Pandas

A Pandas Data Frame with a NumPy Example

Describing a Pandas Data Frame

Pandas Boolean Data Frames

Transposing a Pandas Data Frame

Pandas Data Frames and Random Numbers

Reading CSV Files in Pandas

Specifying a Separator and Column Sets in Text Files

Specifying an Index in Text Files

The loc() and iloc() Methods in Pandas

Converting Categorical Data to Numeric Data

Matching and Splitting Strings in Pandas

Converting Strings to Dates in Pandas

Working with Date Ranges in Pandas

Detecting Missing Dates in Pandas

Interpolating Missing Dates in Pandas

Other Operations with Dates in Pandas

Merging and Splitting Columns in Pandas

Reading HTML Web Pages in Pandas

Saving a Pandas Data Frame as an HTML Web Page

Summary

Chapter 5: Introduction to Pandas (2)

Combining Pandas Data Frames

Data Manipulation with Pandas Data Frames (1)

Data Manipulation with Pandas Data Frames (2)

Data Manipulation with Pandas Data Frames (3)

Pandas Data Frames and CSV Files

Managing Columns in Data Frames

Switching Columns

Appending Columns

Deleting Columns

Inserting Columns

Scaling Numeric Columns

Managing Rows in Pandas

Selecting a Range of Rows in Pandas

Finding Duplicate Rows in Pandas

Inserting New Rows in Pandas

Handling Missing Data in Pandas

Multiple Types of Missing Values

Test for Numeric Values in a Column

Replacing NaN Values in Pandas

Summary

Chapter 6: Introduction to Pandas (3)

Threshold Values and Outliers

The Pandas Pipe Method

Pandas query() Method for Filtering Data

Sorting Data Frames in Pandas

Working with groupby() in Pandas

Working with apply() and mapapply() in Pandas

Handling Outliers in Pandas

Pandas Data Frames and Scatterplots

Pandas Data Frames and Simple Statistics

Aggregate Operations in Pandas Data Frames

Aggregate Operations with the titanic.csv Dataset

Save Data Frames as CSV Files and Zip Files

Pandas Data Frames and Excel Spreadsheets

Working with JSON-based Data

Python Dictionary and JSON

Python, Pandas, and JSON

Window Functions in Pandas

Useful One-line Commands in Pandas

What is pandasql?

What is Method Chaining?

Pandas and Method Chaining

Pandas Profiling

Alternatives to Pandas

Summary

Chapter 7: Data Visualization

What is Data Visualization?

Types of Data Visualization

What is Matplotlib?

Lines in a Grid in Matplotlib

A Colored Grid in Matplotlib

Randomized Data Points in Matplotlib

A Histogram in Matplotlib

A Set of Line Segments in Matplotlib

Plotting Multiple Lines in Matplotlib

Trigonometric Functions in Matplotlib

Display IQ Scores in Matplotlib

Plot a Best-Fitting Line in Matplotlib

The Iris Dataset in Sklearn

Sklearn, Pandas, and the Iris Dataset

Working with Seaborn

Features of Seaborn

Seaborn Built-in Datasets

The Iris Dataset in Seaborn

The Titanic Dataset in Seaborn

Extracting Data from the Titanic Dataset in Seaborn (1)

Extracting Data from the Titanic Dataset in Seaborn (2)

Visualizing a Pandas Dataset in Seaborn

Data Visualization in Pandas

What is Bokeh?

Summary

Index

PREFACE

WHAT IS THE PRIMARY VALUE PROPOSITION FOR THIS BOOK?

This book contains a fast-paced introduction to as much relevant information about Pandas as possible that can be reasonably included in a book of this size. Moreover, you will learn about data types, data cleaning tasks, statistical concepts, imbalanced datasets, and data visualization.

However, you will be exposed to a variety of features of NumPy and Pandas, how to write regular expressions, and how to perform many data cleaning tasks. Keep in mind that some topics are presented in a cursory manner, which is for two main reasons. First, it’s important that you be exposed to these concepts. In some cases you will find topics that might pique your interest, and hence motivate you to learn more about them through self-study; in other cases you will probably be satisfied with a brief introduction. In other words, you will decide whether or not to delve into more detail regarding the topics in this book.

Second, a full treatment of all the topics that are covered in this book would significantly increase the size of this book, and few people are interested in reading technical tomes.

THE TARGET AUDIENCE

This book is intended primarily for people who have a solid background as software developers. Specifically, this book is for developers who are accustomed to searching online for more detailed information about technical topics. If you are a beginner, there are other books that are more suitable for you, and you can find them by performing an online search.

This book is also intended to reach an international audience of readers with highly diverse backgrounds in various age groups. This book uses standard English rather than colloquial expressions that might be confusing to those readers. As you know, many people learn by different types of imitation, which includes reading, writing, or hearing new material. This book takes these points into consideration in order to provide a comfortable and meaningful learning experience for the intended readers.

WHAT WILL I LEARN FROM THIS BOOK?

The first chapter contains a quick tour of basic Python3, followed by a chapter that introduces you to data types and data cleaning tasks, such as working with datasets that contain different types of data, and how to handle missing data.

The third chapter contains fundamental statistical concepts, how to handle imbalanced features (SMOTE), how to analyze classifiers, variance and correlation matrices, dimensionality reduction (including SVD and t-SNE), and a section that discusses Gini impurity, entropy, and KL-divergence.

The fourth, fifth, and sixth chapters concentrate on a multitude of features of Pandas (and many code samples). The final chapter of this book delves into data visualization with Matplotlib and Seaborn.

DO I NEED TO LEARN THE THEORY PORTIONS OF THIS BOOK?

Once again, the answer depends on the extent to which you plan to become involved in data analytics. For example, if you plan to study machine learning, then you will probably learn how to create and train a model, which is a task that is performed after data cleaning tasks. In general, you will probably need to learn everything that you encounter in this book if you are planning to become a machine learning engineer.

GETTING THE MOST FROM THIS BOOK

Some programmers learn well from prose, others learn well from sample code (and lots of it), which means that there’s no single style that can be used for everyone.

Moreover, some programmers want to run the code first, see what it does, and then return to the code to delve into the details (and others use the opposite approach).

Consequently, there are various types of code samples in this book: some are short, some are long, and other code samples “build” from earlier code samples.

WHAT DO I NEED TO KNOW FOR THIS BOOK?

Current knowledge of Python 3.x is the most helpful skill. Knowledge of other programming languages (such as Java) can also be helpful because of the exposure to programming concepts and constructs. The less technical knowledge that you have, the more diligence will be required in order to understand the various topics that are covered.

If you want to be sure that you can grasp the material in this book, glance through some of the code samples to get an idea of how much is familiar to you and how much is new for you.

DOES THIS BOOK CONTAIN PRODUCTION-LEVEL CODE SAMPLES?

The primary purpose of the code samples in this book is to show you Python-based libraries for solving a variety of data-related tasks in conjunction with acquiring a rudimentary understanding of statistical concepts. Clarity has higher priority than writing more compact code that is more difficult to understand (and possibly more prone to bugs). If you decide to use any of the code in this book in a production website, you ought to subject that code to the same rigorous analysis as the other parts of your code base.

WHAT ARE THE NON-TECHNICAL PREREQUISITES FOR THIS BOOK?

Although the answer to this question is more difficult to quantify, it’s important to have strong desire to learn about data analytics, along with the motivation and discipline to read and understand the code samples.

HOW DO I SET UP A COMMAND SHELL?

If you are a Mac user, there are three ways to do so. The first method is to use Finder to navigate to Applications > Utilities and then double click on the Utilities application. Next, if you already have a command shell available, you can launch a new command shell by typing the following command:

open /Applications/Utilities/Terminal.app

A second method for Mac users is to open a new command shell on a MacBook from a command shell that is already visible simply by clicking command+n in that command shell, and your Mac will launch another command shell.

If you are a PC user, you can install Cygwin (open source https://cygwin.com/) that simulates bash commands, or use another toolkit such as MKS (a commercial product). Please read the online documentation that describes the download and installation process. Note that custom aliases are not automatically set if they are defined in a file other than the main start-up file (such as .bash_login).

COMPANION FILES

All the code samples in this book may be obtained via downloading by writing to the publisher at [email protected].

WHAT ARE THE “NEXT STEPS” AFTER FINISHING THIS BOOK?

The answer to this question varies widely, mainly because the answer depends heavily on your objectives. If you are interested primarily in NLP, then you can learn more advanced concepts, such as attention, transformers, and the BERT-related models.

If you are primarily interested in machine learning, there are some subfields of machine learning, such as deep learning and reinforcement learning (and deep reinforcement learning) that might appeal to you. Fortunately, there are many resources available, and you can perform an Internet search for those resources. One other point: the aspects of machine learning for you to learn depend on who you are: the needs of a machine learning engineer, data scientist, manager, student, or software developer, are all different.