Advanced Natural Language Processing with TensorFlow 2 E-Book

Ashish Bansal

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 474

Veröffentlichungsjahr: 2021

Beschreibung

Recently, there have been tremendous advances in NLP, and we are now moving from research labs into practical applications. This book comes with a perfect blend of both the theoretical and practical aspects of trending and complex NLP techniques.
The book is focused on innovative applications in the field of NLP, language generation, and dialogue systems. It helps you apply the concepts of pre-processing text using techniques such as tokenization, parts of speech tagging, and lemmatization using popular libraries such as Stanford NLP and SpaCy. You will build Named Entity Recognition (NER) from scratch using Conditional Random Fields and Viterbi Decoding on top of RNNs.
The book covers key emerging areas such as generating text for use in sentence completion and text summarization, bridging images and text by generating captions for images, and managing dialogue aspects of chatbots. You will learn how to apply transfer learning and fine-tuning using TensorFlow 2.
Further, it covers practical techniques that can simplify the labelling of textual data. The book also has a working code that is adaptable to your use cases for each tech piece.
By the end of the book, you will have an advanced knowledge of the tools, techniques and deep learning architecture used to solve complex NLP problems.

Leseprobe

Advanced Natural Language Processing with TensorFlow 2

Build effective real-world NLP applications using NER, RNNs, seq2seq models, Transformers, and more

Ashish Bansal

BIRMINGHAM - MUMBAI

Advanced Natural Language Processing with TensorFlow 2

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Producer: Tushar Gupta

Acquisition Editor – Peer Reviews: Divya Mudaliar

Content Development Editor: Alex Patterson

Technical Editor: Gaurav Gavas

Project Editor: Mrunal Dave

Proofreader: Safis Editing

Indexer: Rekha Nair

Presentation Designer: Sandip Tadge

First published: February 2021

Production reference: 1290121

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-80020-093-7

www.packt.com

packt.com

Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionalsLearn better with Skill Plans built especially for youGet a free eBook or video every monthFully searchable for easy access to vital informationCopy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.Packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

At www.Packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Contributors

About the author

Ashish Bansal is the Director of Recommendations at Twitch, where he works on building scalable recommendation systems across a variety of product surfaces, connecting content to people. He has worked on recommendations systems at multiple organizations, most notably Twitter, where he led Trends and Events recommendations, and at Capital One, where he worked on B2B and B2C products. Ashish was also a co-founder of GALE Partners, a full-service digital agency in Toronto, and spent over 9 years at SapientNitro, a leading digital agency.

In many years of work building hybrid recommendation systems balancing collaborative filtering signals with content-based signals, he has spent a lot of time building NLP systems for extracting content signals. In digital marketing, he built systems to analyze coupons, offers, and subject lines. He has worked on messages, tweets, and news articles among other types of textual data and applying cutting edge NLP techniques.

He has over 20 years of experience, with over a decade building ML and Deep Learning systems. Ashish is a guest lecturer at IIT BHU teaching Applied Deep Learning. He has a bachelor's in technology from IIT BHU, and an MBA in marketing from Kellogg School of Management.

My father, Prof. B. B. Bansal, said that the best way to test understanding of a subject is to explain it to someone else. This book is dedicated to him, and my Gurus – my mother, my sister, who instilled the love of reading, and my wife, who taught me consider all perspectives. I would like to mention Aditya sir, who instilled the value of hard work, which was invaluable in writing this book while balancing a full-time job and family. I would like to mention Ajeet, my manager at Twitter, and Omar, my manager at Twitch, for their support during the writing of this book. Ashish Agrawal and Subroto Chakravorty helped me tide over issues in code.

I would like to thank the technical reviewers for ensuring the quality of the book and the editors for working tirelessly on the book. Tushar Gupta, my acquisitions editor, was instrumental in managing the various challenges along the way. Alex – your encouraging comments kept my morale high!

About the reviewers

Tony Mullen is an Associate Teaching Professor at The Khoury College of Computer Science at Northeastern University in Seattle. He has been involved in language technology for over 20 years and holds a master's degree in Linguistics from Trinity College, Dublin, and a PhD in natural language processing from the University of Groningen. He has published papers in the fields of sentiment analysis, named entity recognition, computer-assisted language learning, and ontology development, among others. Recently, in addition to teaching and supervising graduate computer science, he has been involved in NLP research in the medical domain and consulted for a startup in language technology.

Kumar Shridhar is an NLP researcher at ETH Zürich and founder of NeuralSpace. He believes that an NLP system should comprehend texts as humans do. He is working towards the design of flexible NLP systems making them more robust and interpretable. He also believes that NLP systems should not be restricted to few languages, and with NeuralSpace he is extending NLP capabilities to low-resource languages.

Contents

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Reviews

Essentials of NLP

A typical text processing workflow

Data collection and labeling

Collecting labeled data

Development environment setup

Enabling GPUs on Google Colab

Text normalization

Modeling normalized data

Tokenization

Segmentation in Japanese

Modeling tokenized data

Stop word removal

Modeling data with stop words removed

Part-of-speech tagging

Modeling data with POS tagging

Stemming and lemmatization

Vectorizing text

Count-based vectorization

Modeling after count-based vectorization

Term Frequency-Inverse Document Frequency (TF-IDF)

Modeling using TF-IDF features

Word vectors

Pretrained models using Word2Vec embeddings

Summary

Understanding Sentiment in Natural Language with BiLSTMs

Natural language understanding

Bi-directional LSTMs – BiLSTMs

RNN building blocks

Long short-term memory (LSTM) networks

Gated recurrent units (GRUs)

Sentiment classification with LSTMs

Loading the data

Normalization and vectorization

LSTM model with embeddings

BiLSTM model

Summary

Named Entity Recognition (NER) with BiLSTMs, CRFs, and Viterbi Decoding

Named Entity Recognition

The GMB data set

Loading the data

Normalizing and vectorizing data

A BiLSTM model

Conditional random fields (CRFs)

NER with BiLSTM and CRFs

Implementing the custom CRF layer, loss, and model

A custom CRF model

A custom loss function for NER using a CRF

Implementing custom training

Viterbi decoding

The probability of the first word label

Summary

Transfer Learning with BERT

Transfer learning overview

Types of transfer learning

Domain adaptation

Multi-task learning

Sequential learning

IMDb sentiment analysis with GloVe embeddings

GloVe embeddings

Loading IMDb training data

Loading pre-trained GloVe embeddings

Creating a pre-trained embedding matrix using GloVe

Feature extraction model

Fine-tuning model

BERT-based transfer learning

Encoder-decoder networks

Attention model

Transformer model

The bidirectional encoder representations from transformers (BERT) model

Tokenization and normalization with BERT

Pre-built BERT classification model

Custom model with BERT

Summary

Generating Text with RNNs and GPT-2

Generating text – one character at a time

Data loading and pre-processing

Data normalization and tokenization

Training the model

Implementing learning rate decay as custom callback

Generating text with greedy search

Generative Pre-Training (GPT-2) model

Generating text with GPT-2

Summary

Text Summarization with Seq2seq Attention and Transformer Networks

Overview of text summarization

Data loading and pre-processing

Data tokenization and vectorization

Seq2seq model with attention

Encoder model

Bahdanau attention layer

Decoder model

Training the model

Generating summaries

Greedy search

Beam search

Decoding penalties with beam search

Evaluating summaries

ROUGE metric evaluation

Summarization – state of the art

Summary

Multi-Modal Networks and Image Captioning with ResNets and Transformer Networks

Multi-modal deep learning

Vision and language tasks

Image captioning

MS-COCO dataset for image captioning

Image processing with CNNs and ResNet50

CNNs

Convolutions

Pooling

Regularization with dropout

Residual connections and ResNets

Image feature extraction with ResNet50

The Transformer model

Positional encoding and masks

Scaled dot-product and multi-head attention

VisualEncoder

Decoder

Transformer

Training the Transformer model with VisualEncoder

Loading training data

Instantiating the Transformer model

Custom learning rate schedule

Loss and metrics

Checkpoints and masks

Custom training

Generating captions

Improving performance and state-of-the-art models

Summary

Weakly Supervised Learning for Classification with Snorkel

Weak supervision

Inner workings of weak supervision with labeling functions

Using weakly supervised labels to improve IMDb sentiment analysis

Pre-processing the IMDb dataset

Learning a subword tokenizer

A BiLSTM baseline model

Tokenization and vectorizing data

Training using a BiLSTM model

Weakly supervised labeling with Snorkel

Iterating on labeling functions

Naïve-Bayes model for finding keywords

Evaluating weakly supervised labels on the training set

Generating unsupervised labels for unlabeled data

Training BiLSTM on weakly supervised data from Snorkel

Summary

Building Conversational AI Applications with Deep Learning

Overview of conversational agents

Task-oriented or slot-filling systems

Question-answering and MRC conversational agents

General conversational agents

Summary

Epilogue

Installation and Setup Instructions for Code

GitHub location

Chapter 1 installation instructions

Chapter 2 installation instructions

Chapter 3 installation instructions

Chapter 4 installation instructions

Chapter 5 installation instructions

Chapter 6 installation instructions

Chapter 7 installation instructions

Chapter 8 installation instructions

Chapter 9 installation instructions

Other Books You May Enjoy

Index

Landmarks

Cover

Index