Practical Guide to Azure Cognitive Services E-Book

Chris Seferlis

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 615

Veröffentlichungsjahr: 2023

Beschreibung

Streamline your complex processes and optimize your organization's operational efficiency, cost-effectiveness, and customer experience by unlocking the potential of Microsoft Azure Cognitive Services and OpenAI

Purchase of the print or Kindle book includes a free PDF eBook

Key Features

• Minimize costs and maximize operations by automating mundane activities using AI tools
• Ideate solutions using real-world examples for manufacturing process improvement with AI
• Master TCO and ROI analysis for implementing AI solutions, automating operations, and ideating innovative manufacturing solutions with real-world examples

Book Description

Azure Cognitive Services and OpenAI are a set of pre-built artificial intelligence (AI) solution APIs that can be leveraged from existing applications, allowing customers to take advantage of Microsoft’s award-winning Vision, Speech, Text, Decision, and GPT-4 AI capabilities.

With Practical Guide to Azure Cognitive Services, you’ll work through industry-specific examples of implementations to get a head-start in your production journey. You’ll begin with an overview of the categorization of Azure Cognitive Services and the benefits of embracing AI solutions for practical business applications. After that, you’ll explore the benefits of using Azure Cognitive Services to optimize efficiency and improve predictive capabilities. Then, you’ll learn how to leverage Vision capabilities for quality control, Form Recognizer to streamline supply chain nuances, language understanding to improve customer service, and Cognitive Search for next-generation knowledge-mining solutions.

By the end of this book, you’ll be able to implement various Cognitive Services solutions that will help you enhance efficiency, reduce costs, and improve the customer experience at your organization. You’ll also be well equipped to automate mundane tasks by reaping the full potential of OpenAI.

What you will learn

• Master cost-effective deployment of Azure Cognitive Services
• Develop proven solutions from an architecture and development standpoint
• Understand how Cognitive Services are deployed and customized
• Evaluate various uses of Cognitive Services with different mediums
• Disseminate Azure costs for Cognitive Services workloads smoothly
• Deploy next-generation Knowledge Mining solutions with Cognitive Search
• Explore the current and future journey of OpenAI
• Understand the value proposition of different AI projects

Who this book is for

This book is for data scientists, technology leaders, and software engineers looking to implement Azure Cognitive Services with the help of sample use cases derived from success stories. Experience with Python as well as an overall understanding of the Azure Portal with related services such as Azure Data Lake Storage and Azure Functions will help you make the most of this book.

Leseprobe

Practical Guide to Azure Cognitive Services

Leverage the power of Azure OpenAI to optimize operations, reduce costs, and deliver cutting-edge AI solutions

Chris Seferlis

Christopher Nellis

Andy Roberts

BIRMINGHAM—MUMBAI

Practical Guide to Azure Cognitive Services

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Ali Abidi

Publishing Product Manager: Ali Abidi

Senior Editor: David Sugarman

Content Development Editor: Priyanka Soam

Technical Editor: Rahul Limbachiya

Copy Editor: Safis Editing

Project Coordinator: Farheen Fathima

Proofreader: Safis Editing

Indexer: Hemangini Bari

Production Designer: Alishon Mendonca

Marketing Coordinator: Shifa Ansari and Vinishka Kalra

First published: April 2023

Production reference: 1280423

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80181-291-7

www.packtpub.com

Contributors

About the authors

Chris Seferlis is a technology strategist at Microsoft. He has over 20 years of experience working in IT and using technology to accomplish business objectives. He has an MBA from UMass, is a part-time professor at Boston University, and brings a mix of business acumen and practical technology solutions.

Christopher Nellis is a senior infrastructure engineer and is experienced in deploying large-scale infrastructure for organizations. He has a passion for automation and MLOps and enjoys working with people to solve problems and make things better.

Andy Roberts is a seasoned data and AI architect. His career has included roles such as developer, DBA, architect, project lead, and pre-sales, always revolving around data. He is experienced in acquiring, shaping, moving, protecting, and using data to predict future outcomes, and processing them efficiently.

About the reviewer

Sumit Garg is a product manager at Azure Cognitive Search, Microsoft. Sumit has over 17 years of experience in tech. He started as a developer and consultant and moved on to product management in the last decade. He has worked extensively with B2B PaaS and SaaS products. In his free time, Sumit likes to travel and is a big fan of print. He is an amateur runner and intends to run a marathon someday.

Table of Contents

Preface

Part 1: Ocean Smart – an AI Success Story

1

How Azure AI Changed Ocean Smart

Choosing Azure Cognitive Services

The Ocean Smart story

Dealing with paperwork using Knowledge Mining

Using Form Recognizer to process financial documents

Using Anomaly Detector for discovering abnormalities

Using Computer Vision to detect product quality issues early

Applying Content Moderator to avoid the posting of inappropriate material

Using the Personalizer service for product recommendations

Applying Speech services for call center improvements

Building your case: calculating ROI and TCO

Building your justification

Proving out your solution

Summary

2

Why Azure Cognitive Services?

Exploring the history of Azure Cognitive Services

Exploring Azure Cognitive Services

Decision

Speech

Language

Vision

Reviewing past and future investments

Accessibility

Sustainability

Operational efficiencies

Summary

3

Architectural and Cost Optimization Considerations

Exploring core Azure services and costs

Storage accounts

Azure Virtual Network (VNet)

Network Security Groups

Databases

Azure App Service

Azure Functions

Estimating the costs for a proof-of-concept solution, including all services

Understanding data orchestration for loading data into Azure

Summary

Part 2: Deploying Next-Generation Knowledge Mining Solutions with Azure Cognitive Search

4

Deriving Value from Knowledge Mining Solutions in Azure

Reviewing a brief history of document collection solutions

Understanding the drawbacks of traditional data collection systems

Exploring the purpose and benefits of knowledge mining solutions

Using cognitive services to develop knowledge mining solutions

Summary

Further reading

5

Azure Cognitive Search Overview and Implementation

Technical requirements

Understanding how Azure Cognitive Search is built

Search types

Underlying index and search activities

Exploring what services will be used as part of a Cognitive Search solution

Pairing common services with Cognitive Search for your KM solution

Summary

6

Exploring Further Azure Cognitive Services for Successful KM Solutions

Technical requirements

Exploring native AI enrichments in Azure Cognitive Search

Reviewing advancements in OCR and image recognition

Understanding OCR enhancements

Exploring image recognition enhancements

Considering other cognitive services commonly used in KM

Skills arrays

Custom skills

Adding additional skills to your Cognitive Search indexer

Summary

7

Pulling It All Together for a Complete KM Solution

Technical requirements

Getting your Azure environment set up to support your KM solution

Deploying the Azure Cognitive Search service

Azure storage account deployment

Deploying Cognitive Search for your KM solution

Ingesting your data for indexing and related activities

Azure Speech-to-Text for audio transcription

Loading files in Form Recognizer for processing

Bulk loading of everything else

Connecting to and deploying related services

Batch processing audio files with Cognitive Speech

Deploying the KM solution with monitoring and notification considerations

Showing the code

Summary

Part 3: Other Cognitive Services That Will Help Your Company Optimize Operations

8

Decluttering Paperwork with Form Recognizer

Technical requirements

Understanding Form Recognizer machine learning model options

Building custom models

Using Form Recognizer Development Studio

Exploring the AP process at Ocean Smart

Deploying Form Recognizer for mostly automating the AP process

Creating Azure resources

Deploying Azure App Service

Integrating the complete solution for production use

Building the demonstration

Example code

Summary

9

Identifying Problems with Anomaly Detector

Technical requirements

Overview of the Anomaly Detector service

Univariate Anomaly Detector

Multivariate Anomaly Detector

Using the Anomaly Detector algorithms with your data

Async API

Sync API

Configuring and refining monitoring of data in your environment

Defining and detecting anomalies in your organization

Building the complete solution in Azure

The SimulateChillerEvents function

The ProcessChillerEvents function

The Azure Stream Analytics job

Summary

10

Streamlining the Quality Control Process with Custom Vision

Technical requirements

Understanding the quality control process at Ocean Smart

Evolving the quality control process with Custom Vision

Training and improving the model

Creating your first project

Uploading and tagging images

Model training

Model performance

Using Smart Labeler for faster image labeling

Sending a notification of an irregular product

Building the complete Custom Vision solution

The processNewImage function

The TrainNewModel function

The PublishLatestIteration function

Deploying the functions

Using the Ocean Smart portal

Summary

11

Deploying a Content Moderator

Technical requirements

Overview of the Content Moderator service

Pricing considerations when deploying the service

Applying the service to text moderation

Using a custom text list

Applying the service to image moderation

Applying the service to video moderation

Flagging and reviewing inappropriate content

Building the complete solution

Summary

12

Using Personalizer to Cater to Your Audience

Technical requirements

The Personalizer service and its origins

Receiving and scoring consumer feedback

Using Reinforcement Learning for recommendations

Configuring Content, Actions, and Context to make recommendations

Using evaluations to increase the effectiveness of your Personalized Service

Building a completed solution

Data used for this example

RewardAction

Testing and deploying the functions

Using a test script

Summary

13

Improving Customer Experience with Speech to Text

Technical requirements

Overview of Azure Speech services

Working with real-time audio and batch speech-to-text data

Real-time speech-to-text

Batch speech-to-text

Improving speech-to-text accuracy with Custom Speech

Working with different languages

Using the translation services

Building a complete batch solution

TranscribeAudio function

Retrieve transcription results Logic Apps app

Failed transcription result

Successful transcription results

Capturing sentiment and reporting

Summary

14

Using Language Services in Chat Bots and Beyond

Technical requirements

Using a chat bot for enhanced customer service

Tying the Cognitive Service for Language to the chat bot for NLP

Using the translator services for worldwide solutions

Advanced capabilities of Language Understanding

Orchestration workflow

Question answering

None intents

Building a bot using language Understanding

Creating our bot project

A quick tour of Bot Framework Composer

Adding the check my order flow

Allowing a user to add shipping notes – language translation

Summary

15

Surveying Our Progress

Highlighting recent advancements in AI

The lessons learned and returns on investments

Industry ripples

Evaluating future opportunities for optimization

The ethics of AI

Summary

16

Appendix – Azure OpenAI Overview

Reviewing industry trends

Understanding what Azure OpenAI services are

Clarifying common misconceptions

Summary

Index

Other Books You May Enjoy

Preface

Azure Cognitive Services is a set of pre-built AI solution APIs that can be leveraged from existing applications, allowing customers to take advantage of Microsoft’s award-winning vision, speech, text, and decision AI capabilities. Developers working with Azure Cognitive Services will be able to get hands-on with this practical guide and deploy AI solutions. The book provides industry-specific examples of implementations to get you into production in no time. You’ll begin with an overview of how Azure Cognitive Services have been categorized and the benefits of embracing AI solutions for practical business applications. Next, we’ll dive into Ocean Smart, a technically advanced seafood distributor that has adopted all the benefits the Azure platform has to offer by using Cognitive Services to attain operational efficiencies and predictive capabilities. Later, you’ll learn how they implemented the Vision capabilities for quality control, Forms Recognizer to streamline supply chain nuances, Language Understanding to improve their customer service, Cognitive Search for a next-generation knowledge mining solution, and so on. By the end of this book, you will be able to implement various Cognitive Services solutions that will help you to drive efficiencies, reduce costs, and improve the customer experience at your organization.

Who is this book for?

The readership for this book includes technology leaders, data scientists, and software engineers looking to implement Azure Cognitive Services with sample use cases derived from success stories. Experience with Python will be required, as well as an overall understanding of the Azure portal, and related services such as Azure Data Lake Storage and Azure Functions are also needed to get the best from this book.

What this book covers

Chapter 1, How Azure AI Changed Ocean Smart, will give a brief overview of Ocean Smart, a fictitious company, with real-world examples of how Azure Cognitive Services digitally transformed the company to achieve operational excellence. From there, we’ll give an overview of Azure Cognitive Services, ways to consider costs in terms of ROI and TCO implications, and some architectural considerations to reference from the whole Azure deployment.

Chapter 2, Why Azure Cognitive Services?, will give a history of Azure Cognitive Services, with an overview of the intent in developing these services, the focus areas that Cognitive Services targets, and the investments Microsoft is making in the services. At the end of this chapter, you will understand what services have been developed and why, as well as gain an understanding of what to expect from future enhancements.

Chapter 3, Architectural and Cost Optimization Considerations, offers an overview of how Cognitive Services are costed, with examples of workloads that Ocean Smart developed for estimation purposes. We will also talk about ways Ocean Smart was able to optimize its Azure architecture and deployments to save costs and reduce complexity for the entire set of solutions it deployed. We will also describe aspects of data ingestion as part of the overall development process. At the end of this chapter, you will better understand what common architectures exist for deployments of cognitive services.

Chapter 4, Deriving Value from Knowledge Mining Solutions in Azure, describes briefly the history of knowledge mining solutions and their purpose. We will then compare the additional functions and features that can be added with Azure AI, using example solutions, and see what value they provide to enhance traditional deployments, allowing for enhanced searches and the analysis of stored documents, reducing significant time spent tagging and describing documents inputted by humans. By invoking Cognitive Search and other Cognitive Services, we’ll discuss how there tremendous amounts of information were uncovered across the whole document library at Ocean Smart.

Chapter 5, Azure Cognitive Search Overview and Implementation, offers an overview of Azure Cognitive Search, how it works with related Azure services, how the product is built architecturally, and how it was deployed for Ocean Smart. This chapter will introduce the search mechanism and how related services such as OCR, key phrase extraction, and named entity recognition help staff at Ocean Smart more quickly answer questions they have about their documents and data.

Chapter 6, Exploring Further Azure Cognitive Services for Successful KM Solutions, provides an overview of other Cognitive Services Ocean Smart used that are commonly combined with Azure Cognitive Search to build a fully comprehensive knowledge mining solution. We will look at how technologies such as OCR and image recognition have increased in effectiveness to add considerable value to traditional knowledge mining solutions, as well as the services that can be used for enhancement.

Chapter 7, Pulling It All Together for a Complete KM Solution, provides step-by-step instructions on how Ocean Smart deployed a complete knowledge mining solution in Azure, including related Cognitive Services and how they’re all integrated. The related web services that call the APIs for execution will also be covered, in order to apply additional context on how all of the components work together.

Chapter 8, Decluttering Paperwork with Form Recognizer, offers an overview of the Forms Recognizer service, deployment considerations, and tools to accompany the service, which Ocean Smart used to optimize processes. We will describe the differences between prebuilt and custom form types for the accurate extraction of data from forms within their organization. This will serve as a foundation to streamline operations that currently require human intervention but can be automated with a high level of confidence with Forms Recognizer.

Chapter 9, Identifying Problems with Anomaly Detector, discusses how Ocean Smart uses the univariate and multivariate Anomaly Detector Cognitive Service to identify areas of concern in the organization and processes by analyzing data for anomalous activity. These activities can be related to quality concerns, security concerns, or equipment failures within the organization, which can cost significant amounts of money.

Chapter 10, Streamlining the Quality Control Process with Custom Vision, explores how Ocean Smart puts many of its species of seafood products through a rigorous quality control process, identifying irregular products before they are packaged and shipped to their customers. This process uses the Custom Vision service to identify the color, shape, size, and more information about the products for comparison with expected information about the species, which is produced with notification capabilities for production supervisors to take action.

Chapter 11, Deploying a Content Moderator, explains how, by using the Content Moderator service, Ocean Smart can ensure that all website content is published without the potential to offend any consumers of the related material on blogs, social media, and the customer service portal. These postings can take the form of text, images, and video, and deployment saves human inspectors significant time and effort.

Chapter 12, Using Personalizer to Cater to Your Audience, explores how Ocean Smart was able to use the Personalizer service to build capabilities into its sales portal and make it a mainstay, including the ability to relate, couple, and recommend products and services to customers through e-commerce platforms. We will go through the development process, cover reinforcement learning, and configure the Personalizer loop and improvements to the deployed service.

Chapter 13, Improving Customer Experience with Speech to Text, delves into how a great customer experience is becoming more and more critical for successful businesses in this climate of on-demand everything. If a person has a poor experience with a company, they’re sure to let the world know as quickly as possible, using as many social media outlets as possible. Because of this, Ocean Smart wanted a better system to improve how customer calls were handled, setting a precedent for training customer service representatives. This chapter will describe how, by using Azure Speech services to capture customer call transcripts, Ocean Smart was able to dramatically improve the experience for its customers.

Chapter 14, Using Language Services in Chat Bots and Beyond, explains how to further improve the customer experience. Ocean Smart deployed a chatbot within their website to help customers get support or find what they need. To make the chat bot more effective, they deployed the Language Understanding and Translator Cognitive Services, which helps any customer enter a question in any of the supported languages and receive logical answers and receive support for their questions without any human intervention, only being directed to a human if necessary.

Chapter 15, Surveying Our Progress, gives an overview of how Ocean Smart was able to dramatically improve operational efficiencies by deploying Cognitive Services and the great accomplishments made. We will also explore future projects that can be undertaken to further take advantage of what this advanced technological suite offers.

Chapter 16, Appendix – Azure OpenAI Overview, provides a baseline understanding of the OpenAI API services in Azure. We will explore the various APIs and models, as well as their use cases within organizations. Finally, we will discuss where we feel AI is going and where enhancements can be made, and dispel some misconceptions about the services.

To get the most out of this book

When you get started with your own workloads, using the various chapters as a reference, there are several areas you will want to have experience with. First, for a user new to Azure, you will want to have a good understanding of navigating the portal, how subscriptions are organized, and ways to control costs. Although it is not required to pass the exam, using the Microsoft AZ-900 exam preparation material will give you a good foundation for best practices. You will also want to have a minimum of Contributor permissions to be able to freely deploy resources as required. The following is a list of other tools that will be used commonly to complete the work:

Base application

Operating system requirements

Visual Studio Code with Azure extensions

Windows, macOS, or Linux

Azure data tools with Azure extensions

Windows, macOS, or Linux

Python versions specific to each service

Windows, macOS, or Linux

Each of the preceding tools is free to download and easy to find with a simple search online. There are situations where having the full version of Visual Studio may offer more capabilities than the free tools listed here, so it may be advantageous to check with your organization whether you can be assigned a license.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Practical-Guide-to-Azure-Cognitive-Services. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “The demo app will also require a script to run when the container is executed. Create a file called docker-entrypoint.sh with the following contents."

A block of code is set as follows:

#!/bin/bash python manage.py collectstatic --noinput python manage.py migrate python manage.py createsuperuser --noinput

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

pipenv install azure-ai-formrecognizer

Any command-line input or output is written as follows:

az cognitiveservices account keys list \ --name OceanSmartCh14ContentModerator \ --resource-group Chapter14

Bold: Indicates a new term, an important word, or words that you see on screen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “After you remind the user not to send personal information to a bot, repeat the dialog by adding Dialog Management -> Repeat this dialog.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packtpub.com.

Share Your Thoughts

Once you’ve read Practical Guide to Azure Cognitive Services, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781801812917

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: Ocean Smart – an AI Success Story

In this part, you will understand how Ocean Smart successfully deployed Azure Cognitive Services with cost-effective implementations on the Azure AI platform and the impact it has had on operational efficiencies and the bottom line.

This section has the following chapters:

Chapter 1, How Azure AI Changed Ocean SmartChapter 2, Why Azure Cognitive Services?Chapter 3, Architectural and Cost Optimization Considerations

1

How Azure AI Changed Ocean Smart

This book is intended to teach you how to apply Azure Cognitive Services to everyday business opportunities where some efficiency can be gained, some cost can be saved, or some unknown or misunderstood insight can be unveiled. We will start with a brief introduction of why and how Azure Cognitive Services is helping with those processes.

Next, we will give an overview of our fictitious seafood company, Ocean Smart. The operational challenges that are outlined are based on the authors' experience working in the seafood industry for over 15 years in various capacities as consultants, technicians, developers, process improvers, and an executive. The examples provided, however, could be applied to many organizations, whether they are manufacturing companies or not. Of course, some of the examples are specific to manufacturing but can be applied to many different examples across various industries and verticals.

From there, we will take the examples provided and develop a plan for evaluating and building a case to present what the return on investment (ROI) and total cost of ownership (TCO) will be to implement these types of solutions. This will be done with guidance on ideating ways to create insight around the value of the implementations—in other words, what efficiency can be gained, how much cost can be saved, or how many insights can be brought forward in tangible examples. These examples can then be presented to the key stakeholders and project sponsors in your organization to get their approval to move ahead with the project.

These examples, of course, will be applied differently at every organization as there are variable values related to building each example that will be different depending on the industry, country, or even region of the country in which you reside.

Wrapping up the chapter, we will look at some architectural guidelines based on successfully implemented use cases that align with the examples we present later in the book to get a taste of which other services will be deployed as part of the overall solution and technologies that will accompany the Cognitive Services you will be configuring in each of the use cases.

We'll go through the following sections:

Choosing Azure Cognitive ServicesThe Ocean Smart storyBuilding your case: calculating ROI and TCO

Choosing Azure Cognitive Services

Artificial intelligence (AI) is not exactly a new topic of discussion: we have always desired computers that are able to do more for us. In the tools we use every day, AI is present. Whether you are using your smartphone's virtual assistant or recommendations it is making to you based on the patterns it has cataloged from your activities, you are using AI. For instance, I do my grocery shopping on Sunday mornings, and my phone knows where I am going and tells me how long it will take to get there. This is becoming more pervasive in our lives every day, whether using our phone, computer, thermostat, sprinkler system, or the far too many more applications to mention. Unless you are willing to up and move "off the grid", AI is here to stay, so you had better get used to it!

Many organizations are looking to employ AI to help streamline operations and help their businesses be even better at what they are already good at. With this in mind, Microsoft created Azure Cognitive Services as a quick and easy way to implement AI for deployment in a multitude of ways. By giving customers the ability to implement cognitive services and connect to them various applications the business is using, it helps streamline the process of taking advantage of what AI can do for an organization.

As we go through the book and provide samples and solutions, keep an eye out for areas where we see the most improvement. The reason behind this is likely to be a better design choice than building a solution from scratch. The development of these products took many years and cannot be replicated in a reasonably timely fashion. Of course, there will always be unique situations that cannot be satisfied with "canned" software products, but the heart of the conversation of this book is building value for your business in both a cost-effective and a timely manner.

Now, let's get started by looking at our fictional scenario: Ocean Smart.

The Ocean Smart story

Ocean Smart is a fictitious seafood procuring, processing, distribution, and retailing company with locations scattered throughout the United States (US) and Canada that specializes in quality lobster, scallops, clams, crab, and various species of fish. Depending on the type of product and the medium by which customers would procure the product, they may go to their local supermarket to purchase live lobster, clams or crabs, fresh scallops or fish, and even "value-added" products that use any combination of the raw materials that can be processed into some sort of seafood pie, frozen and breaded fillets of fish, or a seafood medley to be served over rice or pasta. This encompasses the retail outlets from which a standard purchaser could buy a product to make a meal at home; however, it is just the beginning of where Ocean Smart-produced products may go. As we go deeper into the book, we will discuss how Ocean Smart is able to take advantage of Azure Cognitive Services to streamline operations; improve quality control (QC), customer support, engagement, and services; and predict what intermediate customers (distributors and wholesalers) and end customers will purchase.

The company was founded in the early 1980s when there was not a whole lot of regulation in the industry, and standards were dramatically different from company to company. This meant that the quality of seafood provided ranged widely depending on the source and how the product was handled in transit. Ocean Smart is just as much a success story in the world of logistics as it is in the other areas of expertise and quality standards achieved over time. This section will serve as an overview of company operations for how products are procured right through to how they are sold to end consumers and includes how the company strives to keep customers happy. We will see an overview of each of the areas in which Azure Cognitive Services has been able to help streamline operations and the other results Ocean Smart has realized because of these implementations.

As Ocean Smart decides it wants to use AI to improve operations, it evaluates the ability to build its own solutions by creating machine learning (ML) models and developing code. The company is able to wrap around those ML models and create development-operations (DevOps) and ML-operations (MLOps) practices to ensure standardization. It even goes as far as to compare public cloud vendors in what their offerings are regarding data science and related tools. What the company finds is that there is significant overhead—personnel, training, and much more—required to build all these types of solutions from scratch. Plus, there is a need to have in-house experts on business operations as well as data scientists who could either work with those experts directly or have at least some specific domain knowledge to be able to work independently to build the solutions. As a result of this discovery, Ocean Smart decides that it will be dramatically easier to use Azure Cognitive Services because of the service's ability to quickly and easily plug into any software that is running within the organization via the application programming interfaces (APIs) that are provided. This also helps the company to take advantage of the award-winning ML models that are already developed and deployed by Microsoft researchers, many of which have or are remarkably close to human parity, leading in many of the categories that all tools available in the space are compared against. The company wants to be able to compare how close to a human a machine can get in recognizing the same types of patterns, whether they be in vision, speech, text, or other areas of interest.

Now, let's take a look at a quick overview of some of the operations of the company and areas where Azure Cognitive Services can help streamline operations.

Dealing with paperwork using Knowledge Mining

Today, the company consists of boats for fishing, wharves for receiving products from its own and other fishermen, trucks for transportation of products, live, fresh, or frozen at sea, production facilities for processing of seafood, short-term holding facilities for keeping the product alive, and cold-storage facilities for longer-term storage of the products after processing into a consumable form. Each step of this process requires paperwork, comprising the following:

Purchase receipt of the product off the boatTrucking paperwork, customs documents, and cost receipts for transportationStorage facility receiving and shipping documentsReceiving documents at production facilities containing critical date code and weight informationPaperwork required to capture information related to disassembly of the product and downstream productionShipping paperwork for transport to a storage facility or customerSales information about the product when soldQuality reports of samples, protein levels, bacterial concerns, and so on

And this is just an idea of the amount of paperwork related to each transaction to ensure the safety of the consumers who will ultimately end up eating the product. Of course, there are also the regulatory requirements that need to be satisfied for the US Food and Drug Administration (FDA) and other governing bodies, who will want all traceability information about the products from source to table with accompanying paperwork.

When we see a glut of documents of any kind, but especially ones from a variety of sources including both typed and handwritten text, as well as documents containing images and other areas of analysis, it immediately becomes a great use case for a knowledge mining solution. Traditional knowledge mining or document management solutions are great for collecting documents, capturing some detail from within them, and adding tags and references to later be able to pull some details from the documents. However, when using Azure Cognitive Search with related cognitive services, the information that can be unlocked brings the solution to a whole different category of enterprise search. It really is the next-generation engine for relating documents, finding tone in the documents, and gathering significant details of everything that is contained within.

A knowledge mining solution is a large undertaking but produces tremendous value for the companies who are willing to take the time and implement the solution so that it spans the entire organization and becomes a resource for all. The following diagram shows a sample architecture containing some of the services that will be used alongside the Cognitive Search functions, which we will elaborate further on in the second part of the book:

Figure 1.1 – Knowledge mining sample architecture

Next, we are going to use Form Recognizer to process financial documents.

Using Form Recognizer to process financial documents

Another area we will focus on is process- and expense-related documents for automatic processing using the Form Recognizer cognitive service. Form Recognizer allows us to capture relevant details in these documents that we can then store in a database or elsewhere for further use—for instance, processing an invoice using an accounts payable system and feeding the appropriate data we capture into the fields required, with appropriate validation steps along the way. This also helps develop an ML model in the background that can be trained to provide more accurate model training over time.

The Form Recognizer cognitive service offers the ability to streamline how documents, forms, receipts, and other assets are processed within an organization and is a key component in how robotic process automation (RPA) systems are implemented for optimized operations. Some common uses of these technologies are listed here:

Expense report processingAccounts payable/accounts receivable approvalsOther uses where data needs to be extracted from a form and feeds a system downstream

The following diagram shows a sample architecture using Form Recognizer, some related cognitive services, and other Azure services commonly deployed together to build a full solution:

Figure 1.2 – Form Recognizer sample architecture

The next service we will discuss will help you to understand anomalous behavior among processes in your network.

Using Anomaly Detector for discovering abnormalities

The operations at Ocean Smart are complex, and overhead costs are incredibly significant, especially when it comes to the product market being among the most volatile in the whole food industry. The supply is limited to what is caught, and much more challenging to produce than a traditional farm-raised product such as cattle, pork, chicken, and other land-based animals. Ironically, the industry does not follow the traditional economic models of supply and demand. The independent fishermen are resistant to any one organization, or a small group of organizations, having control of market prices, so buying in greater volume does not necessarily mean buying prices can be reduced as a result. In a very cutthroat industry, relationships matter, but do not necessarily guarantee preference when it comes to purchasing power.

Natural disasters can wreak havoc on the entire industry and can cause significant cost fluctuations for the end consumer. Even today, you can walk into many seafood restaurants throughout the world and find seafood prices marked as Market. This volatility also causes challenges in planning what will be produced and sold at the manufacturing stage of the overall process and what can be promised to distributors. When it comes to finished goods, in whatever form (fresh, frozen, or live), however, there is much opportunity to take advantage of pricing based on supply and demand. The process by which purchase prices are settled can depend on which distributor has which product. There is a weekly, and sometimes daily even, tug of war in the industry between buyers and sellers to determine product pricing that requires significant visibility into which product is moving where and how much of each type is available. As the details have been captured over time regarding the settled prices, fishers, product type, product quality, product region, time of year, and more details, we could use the Anomaly Detector service to test if one of our parameters is somehow out of a "normal" range.

The Anomaly Detector service monitors all the data being collected and calls out areas of concern to the procurement team, giving them the ability to accept the anomaly and provide feedback to the system. Again, this will improve an ML model that has been developed over time or allow the team to go in a different direction with the transaction.

The following diagram provides a sample architecture for Anomaly Detector and other common services in Azure used alongside it for building a solution.

Figure 1.3 – Anomaly Detector sample architecture

The process pictured in the preceding diagram in a simplified version is outlined here:

Data is ingested from some source in Azure.Some compute—in this case, Azure Databricks—is used to manipulate and prepare the data and serve it to the Anomaly Detector service for monitoring.When anomalous behavior is detected, an event is triggered and sent to Azure Service Bus for further action.An alert is sent to appropriate parties for follow-up action using the service of the developer's choosing.The output of the data is then logged in Azure Data Lake Storage (ADLS).Power BI is used to visualize the results of the logged data.

Next, we are going to discuss how human sight can be augmented using the Computer Vision service.

Using Computer Vision to detect product quality issues early

After procurement, the manufacturing (or production process, as it is commonly known in the industry) is when the raw materials are disassembled, filleted, prepared, or enhanced by adding seasonings or breading, then cooked, frozen, and packed for distribution. A key part of this process is determining the quality of the product while it is being produced. Simply put, we take the incoming weight of the product and the cost of procurement, including landing the product at a production facility, do whatever production is to be performed to add further value to the product, and understand the overall cost of goods sold as a basis for what the selling price will be. Knowing that the selling price will fluctuate depending on demand and seasonality means that we need to take considerable care when setting these prices. Of course, each of the parts has a different value, similarly to the beef industry where various parts of the animal have different values (Filet mignon is highly sought after compared to chuck steak) so, for instance, the tail of a lobster is valued significantly higher and generates more saleable product than the legs of that same lobster.

As an example, the cost of a pound of tail meat would be four to five times the cost of a pound of leg meat to a grocery store. Because of the value assigned to each of the parts, it is critical to know if the product is somehow damaged upon procurement or within the production process. A tail missing a flipper or having an obvious crack is not going to be sold at the full price but will not be thrown out either. It will either be put in a "B-grade" batch or pushed to a different location to be processed in a different way— perhaps cooking. On the other hand, if a leg were to be crushed, cracked, or similar, it would likely just be thrown away because it is not worth salvaging. This is true for many of the varieties of saleable products, and each will be handled in a way that responds to their market value.

A process for capturing quality issues at the production level will significantly help avoid downstream customer complaints and returns, meaning the product would need to be destroyed and Ocean Smart needs to provide a refund to the customer. As a result, we will employ the Custom Vision cognitive service to capture potential quality issues that get past initial QC testers. By capturing images of lobster tails or irregular scallops prior to packaging, and alerting key personnel to the concern, the downstream losses can be avoided in the future.

The Computer Vision API gives the ability to handle images in several ways. Here are just a few examples of how the service can be used:

Classifying images based on the content and aligning with similar imagesGrabbing data from screenshots or images captured for cataloging event-related informationComparing captured images from cameras and comparing against "known good" images for QC purposes

The Computer Vision service has ML models already developed for image classification and other uses but we can also use Custom Vision for training and deploying custom ML models to compare against.

The following diagram shows a sample architecture of the Custom Vision service and a simplified flow of how the service processes images it receives and compares or other activities downstream:

Figure 1.4 – Computer Vision sample architecture

Next, we are going to discuss how to protect corporate internet services from inappropriate content being displayed.

Applying Content Moderator to avoid the posting of inappropriate material

Once all goods are produced, they are ready for sale to a distributor, restaurant, grocery chain, fish market, via direct sales through a website, or another outlet. Ocean Smart is concerned about having a recognizable brand to consumers, and it typically runs various social campaigns and has a modern, interactive website, plus other marketing materials that will help make the brand stand out to consumers. But this comes with the caveat that the company maintains its reputation for quality. If a review of Ocean Smart's product is published in a forum somewhere or on a social networking website, positive or negative, it is important for the sales and marketing teams to be aware. The last thing the team wants to hear is that the public thinks poorly of their product or the Chief Executive Officer (CEO) comes to them with negative publicity. The company needs to be very aware of any situations that could enhance or serve as a detriment to the Ocean Smart family of brands in the market. All these channels need to be monitored for content as well as responded to in a timely manner to avoid any escalation of a negative situation. With data that is collected from social media outlets and review websites, we can use the Text Analytics cognitive service to gauge sentiment analysis in positive, neutral, or negative form, then build in automated responses as well as information being reported to customer services for appropriate action to be taken. We can also use the Content Moderator cognitive service to ensure no inappropriate content can be published to any of the Ocean Smart digital platforms or brand websites.

The Content Moderator cognitive service has three key areas that can help to moderate content: images, text, and video. The intent behind the service is to ensure that content does not get displayed on a website or other digital outlet where one of the preceding media types could be consumed by an inappropriate audience. When it is deployed, several other services can be expected to be used as part of the architecture, as follows:

Event Hubs will capture live data streams.Azure Functions will parse text and content.Azure web services will host the application.

The architecture shown in the following diagram gives a good overview of the flow of information when the Content Moderator service is deployed for the protection of a regional chat service online:

Figure 1.5 – Content Moderator sample architecture

Next, we're going to discuss a service that companies can deploy on their website for making personal recommendations to customers.

Using the Personalizer service for product recommendations

The end consumer of a product is not generally known by the team on the production floor except perhaps by the type of packaging that is being used. There are several ways a product can be packed, depending on if it is going to a distributor who will then sell the product to a restaurant, chain of restaurants, or another type of food producer. There are also products that are specifically packaged for optimal display in retail outlets such as grocery stores, fish markets, and specialty stores where the product is much more visible and presented in a way that will try to entice a consumer, as with most retail packaging.

Customers can order the product of their choice directly from Ocean Smart through its e-commerce website. As a customer makes purchases and patterns are determined, the website can either make recommendations for previously purchased products or suggestions for comparable items the customer may like based on their history. There is also a capability that allows for additional recommendations when the customer puts a product into their "cart" such as a marinade or additional product from the store that might pair well with the product they have already added to their cart.

The Azure Personalizer service is a useful tool for just this type of application. Whether looking to make recommendations for other projects or to show commonly paired products that are purchased together by customers, the Personalizer service helps to build those recommendations. The service will improve its accuracy over time, and more of a user profile is built about the consumer of the content. Also included with the service are tools for building a "mock" scenario that could be presented to a user of the site, leading to an improved recommendation.

The Personalizer cognitive service is a resource that helps organizations to align products or make recommendations for location placement on screen. These recommendations are based on optimal positioning backed by an ML model that is developed over time and offers the ability to improve the model with a feedback capability. Because the service is embedded within a web application, many of the services required will be a part of a traditional website deployment.

The architecture in the following diagram gives an overview of how the Personalizer service is used to add recommendations in real time based on what a customer has chosen to add to their cart or explored on the website:

Figure 1.6 – Sample Personalizer service architecture

By implementing the Personalizer service, there was a 15% increase in profit from website sales at Ocean Smart due to the paired items that were offered and new items the customer had not purchased in the past when these were suggested. This increase in profit also drove the direct sales teams to start making similar suggestions when making sales to their customers to also increase sales. We will discuss this in more detail in Chapter 12, Using Personalizer to Cater to Your Audience.

Next, we will explore some of the options and capabilities you have when applying Speech cognitive services to your organization.

Applying Speech services for call center improvements

With any quality product being sold, there needs to be a reputation for backing that product with quality customer service. In the case of Ocean Smart, if a customer has a poor experience with one of its products and they contact customer services for some sort of resolution, there are several ways they can approach this. The website has a "contact us" form where a user can leave feedback for the team. There is a need to route the feedback to the correct employee or team for handling the feedback in the appropriate manner, but most important is how communication is handled where a customer complains about the quality of the product directly. Another more likely way customers will complain is by calling customer services directly.

As has become standard for corporations, Ocean Smart has an automated attendant phone system that helps to triage and route the nature of calls coming in. An example of this could be the following:

"If you would like to let us know that your experience with our product is not quite what you expected, press 1, or say 'issue with product'.""If you would like to tell us how great our product is, press 2, or say 'this product is great'.""If you would like to make a suggestion, press 3, 'I'd like to make a suggestion'."

Each of these responses, whether verbally or through the keypad, will then route the call to a team/person or, depending on the time of day, for example, route the call to a voicemail box that is likely to be emailed to a team/person for handling during normal business hours. Using Speech cognitive services, the system also recognizes several different languages to help customers who do not speak English fluently, further enhancing the customer service experience. There has been considerable care in the way these types of concerns can be expressed so that customers feel as though they are being heard by the company and that the company will make it right. Ocean Smart is using the following services together to build a "call center" reference architecture:

Speech-to-textLanguage Understanding Intelligent Service (LUIS)Text-to-speechSentimentTranslator

By employing this group of services, Ocean Smart can greatly enhance the customer service experience and maintain brand loyalty.

The Speech services can be applied to a host of activities related to the translation of speech, speech-to-text, text-to-speech, and language recognition activities such as keyword, speaker, and intent from recorded or real-time audio. There are many existing use cases for how speech services can be used, so the example later in Part 3, The other Cognitive Services that will help your company optimize operations, of this book will cover a call center reference architecture for improving the customer service experience.

The following diagram shows a common architecture deployed when working with the Speech services for transcribing call center activities into a text-readable format. These files are stored in Blob storage for use with later activities such as sentiment analysis, translation, and other related services downstream:

Figure 1.7 – Speech services sample architecture

LUIS is used to take natural language and process it to literally put words into action. Rudimentary versions of this service involve calling an automated phone system and being prompted "Say yes or no", for example. Depending on the response, the caller is then prompted to respond to the next question as the flow of information continues within the automated system until the caller is routed to the correct person, department, answering service, and so on. The following diagram shows a reference architecture representative of a solution that can be used to deploy an intelligent chatbot with relatedLUIS services:

Figure 1.8 – LUIS sample architecture

Next, let's look at how we can calculate ROI

Building your case: calculating ROI and TCO

When you are considering an AI workload in your business, there are many factors you must consider as part of the process. This section of the book will give you some helpful hints as you strive to deploy solutions to do the following:

Recognize areas where you may be able to take advantage of AI easier than in other areas by understanding the complexity of deployment of each of the solutions.Calculate the ROI, because we all know that we have a boss who will want to know what it is going to take to recoup what is spent on this project.An overview of what you can expect your TCO to be when the solution is fully deployed, and calculating cost growth as your solution grows.How to pull all the necessary information together to ensure you have a complete solution that will bring value to your business and make you an AI hero!.

When we begin to discuss the areas where AI and cognitive services can be of benefit to an organization, we really need to put on our business analyst (BA) hat. Or, perhaps a BA has already started the process of identifying key areas where there are major inefficiencies in your organization. Each chapter of the book will include some helpful hints on how to be sure you are optimizing the services for cost and simplicity. That said, you will need to evaluate the value of implementing a cognitive service to help streamline a process, reduce human intervention, or be able to outline the benefit your organization will receive as a result.

Building your justification

There are costs associated with the implementation of these services, and without a full analysis of what the TCO and ROI are for the implementation, it may be very challenging to justify the out-of-pocket expenses required. Try to avoid deploying technology for the sake of technology. What I mean by this is that too many technologists really want to embrace new technology because it is "cool" or "exciting" to learn and build something new.

As an alternative scenario, you may have an executive or key stakeholder from your organization who really thinks that by implementing "x" technology, the business will be greatly enhanced. This is a dangerous scenario to have to deal with when it comes to your career at your organization, and you will have to be very careful with how you proceed. In reality, there may not be much value for the use case they are pursuing, or the data available for the use case isn't clean, accurate, or good enough. In situations such as these, you will have to decide whether it is worth your reputation to push back, or even go forward with, such an implementation.

To avoid the possible fallout from a failure, it is advisable that you build your TCO and ROI models with as little conjecture as possible. Break down the use case into simple math with parameters that can help you justify your position either way. Present your findings to the stakeholder and state your position. Let that person then make a decision on whether to proceed, hopefully reducing the pressure on you and the team you are working with.

In either case, you need to get a starting point for how to build these models. Here are some thoughts on ways to get started with the data you will need for doing so using a Form Recognizer scenario to automate the accounts payable process. Be sure to work with someone in finance or accounting to understand what the average overhead is of one of these individuals to build a baseline of your averages:

How much, on average, does an accounts payable clerk cost the company per hour?How much time, on average, gets spent per week on manually entering the details of that accounts payable transaction into whatever system is being used for paying vendors?Is there already a process for capturing forms in a digital format, such as scanning to a repository? This could add additional costs to your calculation if there is not, or if equipment is required for such a process to be implemented.What are the costs required to insert data into the system, and with how much confidence can it be inserted? Are you getting handwritten slips, as in the seafood industry, making it harder to be accurate, or is everything typed?Is there already an electronic data interchange (EDI) process in place, and the only human intervention required is validation? Find your alternate scenarios and boil costs down in the same way so that you are sure to capture the full picture.What is the risk or cost if payment of an incorrect amount is made or it is made to the wrong vendor? Many of these systems can be built with very high accuracy; however, they are not completely infallible, so knowing what the downstream effects are as a result may need to be factored in.What is the cost of the development time required to build the solution? Can you put together a sample project plan with approximate timelines to evaluate development expenses?What will the solution cost the organization after it is implemented? There will be recurring costs, which we will cover in each chapter, that will need to be factored in.Do you have clean enough data to build an ML model to run against when processing invoices for payment? Good, clean data is critical for any deployment of AI in any scenario.

This list should provide a good starting point for where you can begin to evaluate what will be required when building out your cost models. Work through the costs and the savings to build the complete model and understand how long it will take to recoup the costs that are required to the point where you start to save the company money. Is it 3 months or 3 years? That break-even point can be a critical point that helps your solution float or sink, so be sure to have a trusted advisor look at your completed analysis.

Proving out your solution

After a complete evaluation, if you still feel it makes sense to push forward with building the solution, you will also likely have to build a proof of concept (PoC) for demonstration. Microsoft provides many tutorials for getting started with the technology through their documentation and GitHub, or you can use the solution we provide as your baseline with relevant documents for demonstration.

You have built your models, checked them with another trustworthy source, built a PoC, and tested your demonstration, so now, it is showtime. Be mindful—there is certainly no exact science to building a perfect presentation for management, as each team will have its own priorities. However, the goal is to try to make sure that you are as prepared as possible to be able to start one of these projects and build in some alternatives or answer those curveball questions that come at us when we are being challenged. The steps in the preceding list should hopefully prepare you for these challenges and will hopefully help you avoid chasing after a solution that won't provide as much value to the organization as you or the stakeholders originally thought.

Of course, there is always the possibility that the budget may not be available for any investment. If we believe there is still a compelling case to be made and it is worth pursuing, this is when we should flip the conversation to "this is how much revenue we believe can be expected as a result of implementing these solutions", if this is feasible. There is certainly a softer side of this conversation, as sometimes we need to evaluate best-case scenarios where maximum value can be achieved, without having the hard data to back it up. If "selling" your solution isn't one of your core skills, enlist a teammate, friend, advisor, or sponsor that you trust and see if they are willing to join your crusade of helping your company be more efficient. With time, the technology will improve and the costs will decrease for implementation, so maybe it is best, instead, to wait for the next budget cycle, possibly? Use your best judgment on how to have the most impact on your organization.

Summary

In this chapter, we have seen an introduction to Ocean Smart