Artificial Intelligence with Power BI E-Book

Mary-Jo Diepeveen

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 396

Veröffentlichungsjahr: 2022

Beschreibung

The artificial intelligence (AI) capabilities in Power BI enable organizations to quickly and easily gain more intelligent insights from unstructured and structured data.
This book will teach you how to make use of the many AI features available today in Power BI to quickly and easily enrich your data and gain better insights into patterns that can be found in your data.
You’ll begin by understanding the benefits of AI and how it can be used in Power BI. Next, you’ll focus on exploring and preparing your data for building AI projects and then progress to using prominent AI features already available in Power BI, such as forecasting, anomaly detection, and Q&A. Later chapters will show you how to apply text analytics and computer vision within Power BI reports. This will help you create your own Q&A functionality in Power BI, which allows you to ask FAQs from another knowledge base and then integrate it with PowerApps. Toward the concluding chapters, you’ll be able to create and deploy AutoML models trained in Azure ML and consume them in Power Query Editor. After your models have been trained, you’ll work through principles such as privacy, fairness, and transparency to use AI responsibly.
By the end of this book, you’ll have learned when and how to enrich your data with AI using the out-of-the-box AI capabilities in Power BI.

Leseprobe

Artificial Intelligence with Power BI

Take your data analytics skills to the next level by leveraging the AI capabilities in Power BI

Mary-Jo Diepeveen

BIRMINGHAM—MUMBAI

Artificial Intelligence with Power BI

Copyright © 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Publishing Product Manager: Reshma Raman

Senior Editor: Tazeen Shaikh

Content Development Editor: Sean Lobo

Technical Editor: Devanshi Ayare

Copy Editor: Safis Editing

Project Coordinator: Aparna Ravikumar Nair

Proofreader: Safis Editing

Indexer: Tejal Daruwale Soni

Production Designer: Shankar Kalbhor

Marketing Coordinator: Priyanka Mhatre

First published: April 2022

Production reference: 1240322

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80181-463-8

www.packt.com

Contributors

About the author

Mary-Jo Diepeveen is an AI expert at Microsoft who focuses primarily on creating educational content around data and AI. Mary-Jo's passion for AI stems from her experience in neuroscience, where collecting and analyzing brain data to understand unconscious behavioral patterns led to an interest in machine learning and AI. This interest has continued to grow during her years at Microsoft, and she is often invited to talk and inspire others to get involved in tech.

About the reviewers

Divya Sardana serves as the lead AI/ML engineer at Nike. Previously, she was a senior data scientist at Teradata Corp. She holds a Ph.D. in computer science from the University of Cincinnati, OH. She has experience working on end-to-end ML and deep learning problems involving techniques such as regression and classification. She has further experience in moving the developed models to production and ensuring scalability. Her interests include solving complex big data and ML/deep learning problems in real-world domains. She is actively involved in the peer review of journals and books in the area of ML. She has also served as a session chair at ML conferences such as ICMLA 2021 and BDA 2021.

Art Tennick is the author of 20 computer books and over 1,000 magazine and LinkedIn articles, including one of the first books on DAX, written in 2009. He is an independent freelance consultant on Power BI (since 2009), Analysis Services (since 1998), and SQL Server (since 1993). His main interests are in Analysis Services (on-premises, Azure, and in the Power BI service), Power BI, Power BI Paginated, and the integration of Python and R data science with Power BI, Analysis Services, and SQL Server. You can find him on LinkedIn.

I would like to thank Rita Mendoza, Emma, and Lorna for their love and support over the years.

Ricardo Franco da Silva has worked at Microsoft as a consultant for the BI stack and Azure for almost 20 years. Ricardo has hasparticipated in projects for sectors including oil, banking, aviation, retail, and governments worldwide, managing big data and getting insights using many tools, especially SQL Server, Analysis Services, and Power BI, among many others. One of the most challenging projects was for a bank in Switzerland. Ricardo cooperated with Packt to review this book, and he also wrote a novella a few years ago. With over 30 years of experience, Ricardo is currently engaged in projects related to AI and recently concluded a course in fintech AI and open banking at the University of Oxford.

Table of Contents

Preface

Part 1: AI Fundamentals

Chapter 1: Introducing AI in Power BI

What do we expect from a data analyst?

What is a data analyst?

Connecting to data

Visualizing data

What is AI?

Understanding the definition of AI

Understanding machine learning

Understanding deep learning

Understanding supervised and unsupervised learning

Understanding algorithms

What is the data science process?

Why should we use AI in Power BI?

The problems with implementing AI

Why AI in Power BI is the solution

What are our options for AI in Power BI?

Out-of-the-box options

Creating your own models

Summary

Chapter 2: Exploring Data in Power BI

Technical requirements

Using the sample dataset on world happiness

How to interpret this dataset

Importing the world happiness dataset into Power BI

What to look for in your data

Understanding data quantity

Understanding data quality

Using data profiling tools

Column quality

Column distribution

Column profile

Using visuals to explore your data

Line charts

Bar charts

Histograms

Scatter plots

matplotlib

Summary

Chapter 3: Data Preparation

Fixing the structure of your data

Working with structured data

Fixing the structure of semi-structured data

Fixing the structure when working with images

Working with missing data

How do you find missing data?

What do you do with missing data?

Mitigating bias

How to find bias

How to mitigate bias in your dataset

Handling outliers

Summary

Part 2: Out-of-the-Box AI Features

Chapter 4: Forecasting Time-Series Data

Technical requirements

Data requirements for forecasting

Why use forecasting?

Time-series data

Using an example – tourism data

Algorithms used for forecasting

The benefit of using an out-of-the-box feature

Understanding how forecasting is calculated in Power BI

Optimizing forecasting accuracy in Power BI

Using forecasting in Power BI

Summary

Further reading

Chapter 5: Detecting Anomalies in Your Data Using Power BI

Technical requirements

Which data is suitable for anomaly detection?

Why use anomaly detection?

Data requirements for anomaly detection

Understanding the logic behind anomaly detection

The algorithms behind Microsoft's anomaly detection feature

No need to label your data

Fast and powerful analysis

Using anomaly detection in Power BI

Importing the sample dataset into Power BI

Enabling anomaly detection in Power BI

Summary

Further reading

Chapter 6: Using Natural Language to Explore Data with the Q&A Visual

Technical requirements

Understanding natural language processing

Using natural language in programs

Understanding natural language for data exploration

Preparing data for natural language models

Creating a Q&A visual in Power BI

Adding a Q&A visual

Using the Q&A visual

Optimizing your Q&A visual

Exploring the Q&A setup

Improving the Q&A experience

Using feedback to improve the model over time

Summary

Further reading

Chapter 7: Using Cognitive Services

Technical requirements

Understanding Azure's Cognitive Services

Creating a Cognitive Services resource

Using Cognitive Services for LU

Using Azure's Text Analytics

Creating question answering from a knowledge base

Using Cognitive Services for CV

Understanding Azure's Computer Vision

Using Azure's Custom Vision

Using the Face service

Summary

Chapter 8: Integrating Natural Language Understanding with Power BI

Technical requirements

Using Language APIs in Power BI Desktop

Using AI Insights

Using Power Query Editor

Visualizing insights from text in reports

Visualizing text with a Word Cloud

Summary

Chapter 9: Integrating an Interactive Question and Answering App into Power BI

Technical requirements

Creating a question answering service

Understanding the application of question answering

Configuring a question answering service

Creating an FAQ app with Power Apps

Creating a new app with Power Apps

Adding Power Automate to call the question answering service

Connecting Power Automate to Power Apps

Integrating the FAQ app with Power BI

Improving the question answering model

Summary

Chapter 10: Getting Insights from Images with Computer Vision

Technical requirements

Getting insights with Computer Vision using AI Insights

Using the Vision option of AI Insights

Configuring Custom Vision

Preparing the data for Custom Vision

Training the model in Custom Vision

Evaluating classification models

Publishing your Custom Vision model

Integrating Computer Vision or Custom Vision with Power BI

Using visuals to show a reel of images in a report

Storing data and ensuring it is anonymously accessible

Improving the Custom Vision model

Summary

Part 3: Create Your Own Models

Chapter 11: Using Automated Machine Learning with Azure and Power BI

Technical requirements

Understanding AutoML

Understanding the ML process

Improving the performance of an ML model

When to use AutoML

Creating an AutoML experiment in Azure ML

Creating an Azure ML workspace and resources

Configuring an AutoML run

Deploying a model to an endpoint

Integrating the model with Power BI

Summary

Chapter 12: Training a Model with Azure Machine Learning

Technical requirements

Understanding how to train a model

Understanding the machine learning process

Working with Azure ML

Creating Azure ML assets

Training a model with Azure ML Designer

Configuring an Azure ML Designer pipeline

Deploying a model for batch or real-time predictions

Generating batch predictions

Generating real-time predictions

Integrating an endpoint with Power BI to generate predictions

Summary

Chapter 13: Responsible AI

Understanding responsible AI

Protecting privacy when using personal data

Removing personally identifiable information

Using differential privacy on personal data

Creating transparent models

Using algorithms that are transparent by design

Explaining black-box models

Creating fair models

Identifying unfairness in models

Mitigating unfairness in models

Summary

Other Books You May Enjoy

Preface

Even though many organizations are embracing Artificial Intelligence (AI), the field remains mystical for many. AI is a large field with various applications, which makes it difficult to completely comprehend if you are new to the field.

The focus of AI is often to find the best-performing model to predict new data. However, equally, if not more, important is understanding how to apply those models to ensure the predictions are properly consumed by the targeted users.

Power BI is a great tool for extracting the value out of AI models. Data analysts are already familiar with working with data and are the ideal candidates to integrate a model's predictions into a story to tell decision-makers.

With this book, I hope to demystify AI and make it more approachable for data analysts working with Power BI. The aim of this book is not to become a full-fledged AI expert or data scientist, but to understand what a data scientist deals with to create models. Based on the fundamental understanding of AI, you should be able to consciously decide whether AI will enrich your reports in Power BI.

Since I wrote this book to show how AI can be applied to create value, there is a large focus on walkthroughs that you can replicate. If you prefer to learn by doing, and are keen to explore how to apply AI, this book is for you.

Who this book is for

The book focuses on applying AI in Power BI, which means some experience or familiarity with Power BI is preferred. The AI features will be introduced assuming you have no prior knowledge. Most of the features are accompanied by a walkthrough, to help you understand the concepts that are explained in each chapter.

Although this book will introduce you to many AI concepts, it is not meant to teach you how to do data science. Feel free to use this as a stepping board but know that there is much more to learn in order to become a data scientist.

What this book covers

Chapter 1, Introducing AI in Power BI, introduces basic concepts associated with AI and Power BI.

Chapter 2, Exploring Data in Power BI, introduces some Power BI features to explore data as is often done in data science.

Chapter 3, Data Preparation, introduces some data requirements when implementing AI and how those requirements can be met within Power BI.

Chapter 4, Forecasting Time-Series Data, introduces forecasting as a data science method and how to use the built-in forecasting feature in Power BI.

Chapter 5, Detecting Anomalies in Your Data Using Power BI, introduces anomaly detection as a data science method and how to use the built-in anomaly detection feature in Power BI.

Chapter 6, Using Natural Language to Explore Data with the Q&A Visual, introduces natural language querying and how to use the built-in Q&A visual in Power BI.

Chapter 7, Using Cognitive Services, goes through the various AI models offered through Azure Cognitive Services and how they can be of use.

Chapter 8, Integrating Natural Language Understanding with Power BI, further explores Cognitive Services for Language and how to apply its models in Power BI.

Chapter 9, Integrating an Interactive Question and Answering App into Power BI, focuses on one of the Cognitive Services for Language models and how you can create a power app to create an interactive Q&A visual in Power BI.

Chapter 10, Getting Insights from Images with Computer Vision, further explores Cognitive Services for Vision and how to apply its models in Power BI.

Chapter 11, Using Automated Machine Learning with Azure and Power BI, introduces Azure Machine Learning and how to use its automated machine learning feature to quickly train multiple models. The best-performing model will be integrated with Power BI.

Chapter 12, Training a Model with Azure Machine Learning, covers how to train your own model in Azure Machine Learning, and how to integrate a model with Power BI.

Chapter 13, Responsible AI, discusses important considerations when working with AI to ensure its fair and responsible use.

To get the most out of this book

All walkthroughs will require Power BI Desktop installed on a computer. Next to that, you'll need an Azure subscription to access the Azure portal through a browser.

Not all chapters' walkthroughs require an Azure subscription. You'll be informed in each chapter when you need access to Azure and instructions on how to use it are included.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book's GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Artificial-Intelligence-with-Power-BI. If there's an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781801814638_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "If we look at the Country name, we can see we have 149 distinct and 7 unique values."

A block of code is set as follows:

import matplotlib

dataset.boxplot("Life Ladder", showmeans=True, showfliers=False)

plt.show()

Any command-line input or output is written as follows:

pip install pandas

pip install matplotlib

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: "Select Transform Data, this will open the Power Query Editor."

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you've read Artificial Intelligence with Power BI, we'd love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we're delivering excellent quality content.

Part 1: AI Fundamentals

This part of the book will help you understand what AI can do and how it can be used in Power BI. It will show you how to make sure your data is prepared and that you understand which AI feature to use.

This section includes the following chapters:

Chapter 1, Introducing AI in Power BIChapter 2, Exploring Data in Power BIChapter 3, Data Preparation

Chapter 1: Introducing AI in Power BI

Everyone wants to be working with data. Organizations are keen on relying more on data-driven decisions instead of intuition-driven decisions. To be driven by data, we need to extract insights from data. Thankfully, Power BI is a great tool to visualize and share what the data tells us. To better understand what trends and learning we can derive from data, we can use techniques from the field of data science.

Data science and artificial intelligence (AI) are becoming increasingly popular approaches to extracting insights from data. Among other things, this is because these tools allow us to work with unstructured data, which we couldn't work with before. And this helps us to more quickly find complicated trends and patterns in the data.

In this book, we will focus on using Microsoft Power BI as a data exploration and visualization tool. And we will take some parts of the Azure cloud to give us the power to train models and integrate this with Power BI.

But first, let's start with some of the groundwork. We need to understand what AI is, to properly scope our projects and run them successfully. We need to know what is possible and how we go from a simple dataset to an AI model before we get into the details of each step of the process. That is why we will first dive into the following questions:

What do we expect from a data analyst?What is AI?Why should we use AI in Power BI?What are our options for AI in Power BI?

Let's start with covering the basics.

What do we expect from a data analyst?

Every company is looking for different insights and is working with different types and sets of data. Even though you will find data analysts across several organizations, their actual day-to-day work can greatly differ. When reading this book, you will pick up whatever is useful to you, and you will probably skip over irrelevant parts. Nevertheless, it is good to go over what we expect you to know and be familiar with.

First, we will discuss what it means to be a data analyst, the assumptions we make, and why we chose to use this job title. Next, we will go over what you should already know about Power BI and where you can find this information if you feel you need to brush up on that knowledge.

What is a data analyst?

You could call yourself a business intelligence engineer, a business intelligence specialist, a database administrator, or simply a data analyst. Whatever your job title is, you picked up this book because you work with Power BI and want to learn more about it. With all these different titles nowadays, it becomes challenging to understand what your base knowledge should be. For simplicity and consistency, we will refer to a person working with Power BI as a data analyst.

Why a data analyst? Because in this book, we will assume that you are familiar with working with data in Power BI and are able to do the following:

Prepare dataModel data (creating a data model in Power BI, not a machine learning model)Visualize dataAnalyze dataDeploy and maintain Power BI deliverables

On the other hand, we'll assume you're unfamiliar with the field of data science. We'll approach all AI features in Power BI from the point of view of this persona. In this book, we'll introduce the machine learning models behind the AI features, to make sure enough is understood to use the features correctly. However, we won't go into the full complexities of all models, as this is not a book targeted at the data scientist, who already has a lot of knowledge about mathematical and statistical methods used in AI.

There are two main skills important for when you are venturing into AI in Power BI: connecting to and visualizing data. Let's elaborate a little bit more on these two topics so that you know what is expected of you before you continue.

Connecting to data

So, assuming we are all data analysts, let's look at our core tasks. The very first thing we need to do to work with data, is get access to data. From a technical perspective, we can very easily connect Power BI to various data sources, whether we have data stored in cloud databases, Azure or otherwise, or whether we have local files we want to connect to. Power BI will allow us to do so and will even allow us to schedule an automatic refresh to visualize new data as long as we set up a gateway connection between the network within which the data resides and the Power BI service.

What kind of data can you work with? Any kind! You can connect to structured data, formatted nicely in tables, semi-structured data, often in the form of JSON, or even unstructured data to insert images into your Power BI reports. This also means data can come from a variety of sources. You can collect Twitter data (semi-structured) which contains tweet text, date of creation, number of retweets, likes, and hashtags used. You can collect sales and marketing data to understand which products you have sold, when you sold them, and which ad campaigns you were running that may have had an effect on your sales. Or maybe you are looking at the supply and demand for your warehouses and stores to make sure you plan the logistics of stocking your stores accordingly.

Since data can be generated by so many different sources, and can come in so many different formats, we also want to think about how we extract that data and get it ready to build reports on. Power BI has a lot of standard connectors to allow you to connect to data. A best practice here, however, is that you have a pipeline handling data orchestration before you even connect Power BI to it. Such a process is often called an ETL (Extract-Transform-Load) or ELT (Extract-Load-Transform) pipeline in which we connect to our sources generating data, extract the data, load it into a database, and transform it if necessary. Although similar tasks can be done by Power BI, we prefer working with ETL tools such as Azure Data Factory to handle these kind of orchestration pipelines when we are working with large amounts of data.

ETL or ELT?

ETL is an established and commonly used approach to extract data from sources. The purpose is often to transform it and load it into a structured database such as an Azure SQL Database and fit the data nicely into columns and rows. This is very well suited for transactional data, where we want to get quick results. However, with the emergence of cloud, the newer approach, ETL, is getting more traction. By extracting and loading the data into a data storage that can store unstructured data, such as an Azure Storage Account or Azure Data Lake, we can dump data in the cloud without having to worry about schemas. This also makes it easier to reuse the same data and transform it in different ways, depending on the insights you want to extract.

Long story short, a lot can and probably should already happen to your data before you even open Power BI. Be aware of the process that may have occurred before data enters Power BI. Whatever source we use for Power BI will influence the options we have within Power BI, as well as the performance of the reports we create. In this book, we will work mostly with preprocessed data, available through files stored on public websites or cloud databases. For some projects with AI, however, we will need to process data before bringing it into Power BI to make sure we can work with it.

Visualizing data

Lastly, the reason we are using Power BI is because we want to tell a story with our data. We want to convert data into valuable and intuitive insights that everyone within our organization can read for their own use. In the context of AI, this may be one of the most important skills of a data analyst, as will be explained in later sections.

So, what do we mean by telling a story? Humans are much more susceptible to stories than listening to data. If we say there is a 70% chance it is going to rain, do you bring an umbrella? It's hard to decide. If someone tells you to bring an umbrella, you will probably do it. Even though we want to make more data-driven decisions, humans are not naturally driven by data. We are driven by stories, which we find more intuitive. That also means that we cannot just give AI insights to humans. We need to translate the output of an AI model to make sure it is understandable for people. That means we need to use the data to tell stories.

The way we can do it within Power BI is by making use of the visuals it offers. We have the option to use standard visuals, import visuals from the marketplace, or create our own visuals using Python or R. Understanding when to use what and how to combine different visuals into a report is an important skill of a data analyst, and a skill we assume you have when reading this book. Throughout the book, remember that to gain people's trust, we need to speak their language and not just throw numbers at them to make them change their behavior.

In this book, we will focus on how we can use Power BI to perform AI. That means that you, as a data analyst, are already familiar with the different types of data you can get into Power BI from the different sources available to you. You should already be familiar with how to create visuals to form a report in Power BI. In later chapters, we will highlight features that are relevant when preparing your data for AI and to actually implement AI. But first, let's talk more about what AI is.

What is AI?

AI is a term often used to show that organizations use state-of-the-art technologies. Interestingly, this term has already existed for over 60 years. Back then, it was defined as the science and engineering of making intelligent machines (Professor John McCarthy, Stanford University, accessed June 2021, http://jmc.stanford.edu/artificial-intelligence/what-is-ai/). Unfortunately for us, that leaves a lot open for interpretation, which is also why the term AI has had so many different meanings over the years.

AI often goes hand in hand with data science, which is a field combining science, engineering, mathematics, and statistics. Its purpose is very much in the name: it's the science of extracting insights from data, to make sense out of the very raw data we might get from any applications or databases we have. Using this field, we can get data, clean it up, train a model based on that data, and integrate that model into our applications to generate predictions on new incoming data.

To fully grasp what AI can do, we need to understand a couple of different terms, often used together with AI: machine learning, deep learning, supervised learning, and unsupervised learning. Next to that, it helps to be familiar with a typical structure of the process it takes to create an AI model.

Understanding the definition of AI

We can find many different definitions for AI, which generally have three main aspects in common:

ComputersExecuting an intelligent taskLike a human would do it

Computers come in different forms and can mean software or hardware; it can be an application running locally on someone's laptop or an actual robot. The part of the definition of AI which is more open to interpretation is the intelligent task executed like a human would. What do we consider intelligent? If we think about an intelligent task performed by a human, we could also take the example of a calculator. The more expensive calculators are able to make complex calculations within seconds, which would take a mathematician some time to figure out. However, if you asked someone whether a calculator should be considered AI, the answer would most probably be no.

So, then the question arises: what is intelligence? Fortunately, there are many philosophers who are spending their academic life on answering this question, so let's assume that is outside the scope of this book. Instead, let's agree that the threshold of what is considered to be AI evolves over the years. With new developments come new expectations. Whereas we first considered beating the world champion in chess to be the ultimate level of AI, we now wonder whether we can create fully autonomous self-driving cars.

Some of these new developments have been new types of algorithms that allow us to train even more intelligent models. These algorithms are often categorized as machine learning and deep learning algorithms and are important to understand to know when to use what. We will explore both of these terms in more detail.

Understanding machine learning

If we think back to our simple yet high-performing calculator, you can imagine the intelligence of such a machine being created by a rule-based system. Adding one plus one is always two. This kind of mathematical rule and many others can be programmed into a calculator to empower it to count. This approach is also called using regular expressions and can still be very useful today. It is considered the most rudimentary approach to accomplishing AI but can still yield quick and clear results.

If you want smarter AI, however, you might want to work with techniques where a model is not fully programmed based on rules we humans decide on, but instead is self-learning. Because of this term, it is often thought AI is self-improving and will continuously improve over time until it reaches the singularity. What self-learning actually means is that we do not have to explicitly tell AI how to interpret data coming in. Instead, we show AI a lot of examples and, based on those examples, the model we train will decide how a pattern of variable values influence a specific prediction.

For example, what if you sell laptops and you want to advertise the right laptop to the right person? You could work with a rule-based system where you would create groups based on demographic data, such as women younger than 30, women who are 30 or older, and the same for men younger than 30 and men who are 30 or over. We would have four different groups we would use different marketing strategies on, assuming that every woman younger than 30 has the same requirements when buying a new laptop.

Instead of this, we of course want to pick up on patterns we may not have realized ourselves but can still be found in the data. That is when we would use machine learning to have a self-learning model that looks at the data and learns which variables or features make you interested in specific laptop requirements. It could very well be that, based on this self-learning, we find out that we have different groups we should use different marketing strategies on. For example, we may have a subgroup of both men and women under 30 who love to play online games and need different requirements than men and women under 30 who only use their laptop for work.

Compared to regular expressions, using machine learning to accomplish AI is considered a more sophisticated approach. To create a machine learning model, we take our data, and we choose an algorithm to train a model. These terms are visualized in Figure 1.1:

Figure 1.1 – Creating a machine learning model

As seen in Figure 1.1, the data you have and the algorithm you choose are the inputs, and the model you train is the output. The algorithm we select will decide how we want to look at our data. Do we want to classify our data? Do we want to create a regression model where we want to predict a numeric value? Or do we want to cluster our data into different groups? This information is encapsulated in the algorithm you choose when training a model.

Now that we understand what machine learning is, how does it then differ from deep learning?

Understanding deep learning

Machine learning has already opened up a world of possibilities for data scientists. Instead of spending hours and hours exploring data and calculating correlations, covariances, and other statistical metrics to find patterns in the data, we could just train a model to find that pattern for us.

Machine learning was initially done with structured data fitting nicely into columns and rows. Soon, people wanted more. Data scientists wanted to also be able to classify images or to understand patterns in large text documents. Unfortunately, the algorithms within machine learning could not handle these kinds of unstructured data very well, mostly because of the complexity of the data itself. It is said that an image says a thousand words and it is indeed true that even one single pixel of an image holds a lot of information that can be analyzed in many different ways.

With the emergence of cloud computing and the improvements in processing units, a new subfield within machine learning arrived. Instead of the simpler CPUs (Central Processing Units), we now have the more powerful GPUs (Graphical Processing Units) that can process complex data such as images at a much faster rate. With more power comes more cost, but thanks to the cloud, we have GPUs available on demand and only have to pay for when we use them.

Once we had the processing power, we still needed different algorithms to extract the patterns in these kinds of unstructured data. Since we wanted to perform these tasks just like humans would do it, researchers turned to the brain and looked at how the brain processes information. Our brain is made up of cells that we call neurons, which process information on many different layers. So, when looking at unstructured data, researchers tried to recreate a simplified artificial neural network in which these neurons and layers are simulated. This turned out to work very well and resulted in the subfield of deep learning. Now, we can take images and classify them or detect objects in them using the subfield we call Computer Vision. And we can use Natural Language Processing (NLP) to extract insights from text.

We have now talked about AI, machine learning, and deep learning. In Figure 1.2, a visual overview is shown of these three terms:

Figure 1.2 – Overview of AI, machine learning, and deep learning

As seen in Figure 1.2, the three terms we just discussed are related to each other. AI is often considered to be the umbrella term for anything we do to let a machine execute an intelligent task like a human would do it. One approach to this has been machine learning, in which we train models not by teaching them the rules, but by letting the models learn themselves. All we do is offer the data and algorithm (which defines the task) to train a model. And lastly, we have deep learning as a subfield of machine learning, in which special algorithms are used to better handle complex tasks such as understanding patterns in unstructured data.

Besides these three main terms that are necessary to understand before working with AI in Power BI, we also need to make a distinction between supervised and unsupervised learning. These two approaches divide the algorithms we will use into two categories. Understanding the difference between them helps us to know what we need to include in our datasets that we will use as input for a model we want to train.

Understanding supervised and unsupervised learning

The purpose of AI is that you want to predict something. You want to predict something such as whether someone is more likely to buy a washing machine or a new fridge. Or you want to predict how many apples you will sell on a given day so that you know how much to supply your store with. What we want to predict is often called a label or tag. Sometimes we have training data that includes that label, and sometimes we do not.

If we run a store that sells apples, we can take our historical data, which we can combine with extra data such as weather data and day of the week. On cold Mondays, no one may want to buy apples but on sunny Fridays, you may run out of apples before noon. Because we can see in our historical data how many apples we have sold in the past when specific conditions were met, we have training data that includes a label, namely number of apples sold. If we know the label, we call this supervised learning.

What about the laptops we were selling? Let's say we have customer data including demographic information such as age and gender. But we may also have data on what they use the laptops for: online games or work. In this case, we do not know how many different groups we should create different marketing strategies for. So, the groups we want to categorize them into do not exist in the training data. Therefore, we have no labels in our training data and thus are doing unsupervised learning.

It is good to make the distinction between these different terms because it will help you understand what is required from your data and what you can expect from the model. Finally, let's zoom in to the different types of algorithms that we can expect to use throughout this book.

Understanding algorithms

When you have decided you want to apply AI to your data, we now know that we need to have data and an algorithm to create a model. We will discuss the requirements of the data in later chapters in much more detail. That leaves us with understanding how to use algorithms. Understanding how to work with algorithms is considered to be the data scientist's expertise as it lies at the cross-section of mathematics and statistics. However, even if we do not build models ourselves, or aspire to becoming full-on data scientists, it is still beneficial to understand the main types of algorithms we can work with.

The most important thing we need to know is that by choosing the right algorithm, we dictate how we want to look at the data and what kind of pattern should be detected by the model. If we talk about supervised learning – where we know the label we want to predict – we often talk about regression or classification. With regression, we try to predict a numeric value, whereas with classification the label is categorical (two or more categories).

Understanding regression algorithms

Imagine you work for a company that supplies electricity to households all across the country. You will have collected a lot of data on your customers, such as where they live, what type of house they live in, the number of people who make up that household, and the size of the house. For existing customers, you know how much energy they have consumed in previous years. For new customers, you want to predict what their energy consumption will be so that you can make a good estimation on what the costs will be for them.

Our data may look something like the table shown in Figure 1.3, where we have historical data for two existing customers, and we want to predict the energy consumption in kWh for the third, new customer:

Figure 1.3 – Data for three customers on household characteristics as well as energy consumption

In this example, we know the label: energy consumption. So, we know that we are doing supervised learning. The variable we want to predict has a numerical value (for example 1990 kWh, 4320 kWh, or anything in between). Therefore, this is a simple example of a regression model. There are different algorithms we could choose within the subset of regression to train such a model. The choice may depend on things such as how complicated you want to allow your model to be, how explainable you want your model to be, and how much compute and time you want to spend on training your model. Some examples of such algorithms are linear regression, decision forest regression, and boosted decision tree regression.

Understanding classification algorithms

After we trained our model with one of these algorithms and our historical data, we were able to correctly predict for our new customer what the energy consumption will be and how much they will be spending per month. Our new customer has agreed to buy electricity from us but wants to know more about potential ways to save energy. This brings us to the idea of working more proactively with solar panels. If people have solar panels on their roofs, they can generate electricity themselves on sunny days and save money. We, of course, want to help them with this investment and the installation of solar panels.

Some customers may already have solar panels, some may approach us to talk about solar panels, and some may have never given it a thought. We want to reach out to customers and advertise the solar panels that we sell, but we do not want to annoy or spend marketing budget on customers who already have them.

So, what we want to do now is make an inventory of which of our customers already has solar panels. Sure, we could check with each and every household to see whether they have solar panels, but that seems like a big task that takes too much time and energy. And not every household may react to the survey we would send out to collect that data. We therefore decide we are going to see whether we can predict it. We can collect some sample data, train a model that can predict whether that customer has solar panels, and use those insights to target the right households.

This sample data may look like the data shown in Figure 1.4. We again have historical or known data, and we have a label. Since we know the label for our sample data, we are doing supervised learning. In this case, however, we are not trying to predict a numeric value. The label we try to predict is solar panels, and can either be yes or no. So, we have two categories, making it a classification problem; more specifically, a binary or two-class classification problem:

Figure 1.4 – Household characteristics and the solar panels label

Again, there are different algorithms we could choose from when we know we want to do classification. First of all, it matters whether we have two or three or more classes for our label. Next, we again can choose how complex or explainable we want our model to be with options such as a two-class logistic regression or decision forest algorithm to train our model with.

Finally, let's take look at one more simple example so that we have seen different types of algorithms we can work with. Imagine we are still supplying electricity to people all across the country, but now a new competitor has entered the market. We are afraid this new business will take away some of our customers, so we want to offer people nice benefits to convince them to stay with us. We do not want to offer this to all our customers, to save money, meaning we need to make a good assessment on who will leave us.

We have not had many customers leaving us yet, and we want to prevent this from happening. This does mean, however, that we do not have a label; we do not have enough data on the variable we want to predict. In this case, we might want to cluster our customers, and divide them into different groups: those who will leave us for the competitor, and those who will not. We may have some customers that have left us. Based on the little data we have, we can use an algorithm such as K-means clustering to find similar data points to the customers who left us to create these groups and target those that are on the verge of leaving us with attractive deals to make sure they stay with us.

Working with algorithms requires an understanding of the mathematics and statistics behind them. To use them to train models, we rely on data scientists bringing that knowledge to our team so that we can make more data-driven decisions. To work with the AI features in Power BI, we are not expected to become data science experts. It does, however, help to understand the choices being made to realize what the potential and restrictions are of using AI on our data.

Now that we know what AI is and how we can use different algorithms to train our model, let's take a step back and have a look at the complete data science process. How do we take data from beginning to end? And how we can we do so successfully?

What is the data science process?

Just like every project, training a model consists of multiple phases. And just like many projects, these phases are not necessarily linear. Instead, we want to take an iterative approach when developing an AI solution. First, let´s have a look at what the phases of the data science process look like.

The very first thing we need to think about is what we are doing this for. Why do we want to use AI? What is the model going to do? Even though it is good to drive innovation, we want to avoid using AI just because everyone else is doing it. Nevertheless, identifying an appropriate use case for AI can be challenging as many applications are relatively new and unknown. So, what is then a good use case? Of course, it depends on the area of business, but in each area, there is some low-hanging fruit that we can identify. Most commonly, we can think of using AI to predict how to invest marketing budgets, how to increase sales, how to monitor for predictive maintenance, or how to find outliers and anomalies in your data such as risk assessments.

After deciding on the use case, and essentially the scope, it becomes easier to think about what data to use and what metrics to evaluate the model on to know whether it will be successful or not. The next step is then to actually