60,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
A reliable, cost-effective approach to extracting priceless business information from all sources of text Excavating actionable business insights from data is a complex undertaking, and that complexity is magnified by an order of magnitude when the focus is on documents and other text information. This book takes a practical, hands-on approach to teaching you a reliable, cost-effective approach to mining the vast, untold riches buried within all forms of text using R. Author Ted Kwartler clearly describes all of the tools needed to perform text mining and shows you how to use them to identify practical business applications to get your creative text mining efforts started right away. With the help of numerous real-world examples and case studies from industries ranging from healthcare to entertainment to telecommunications, he demonstrates how to execute an array of text mining processes and functions, including sentiment scoring, topic modelling, predictive modelling, extracting clickbait from headlines, and more. You'll learn how to: * Identify actionable social media posts to improve customer service * Use text mining in HR to identify candidate perceptions of an organisation, match job descriptions with resumes, and more * Extract priceless information from virtually all digital and print sources, including the news media, social media sites, PDFs, and even JPEG and GIF image files * Make text mining an integral component of marketing in order to identify brand evangelists, impact customer propensity modelling, and much more Most companies' data mining efforts focus almost exclusively on numerical and categorical data, while text remains a largely untapped resource. Especially in a global marketplace where being first to identify and respond to customer needs and expectations imparts an unbeatable competitive advantage, text represents a source of immense potential value. Unfortunately, there is no reliable, cost-effective technology for extracting analytical insights from the huge and ever-growing volume of text available online and other digital sources, as well as from paper documents--until now.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 475
Veröffentlichungsjahr: 2017
Ähnliche
Table of Contents
Cover
Title Page
Copyright
Dedication
Foreword
Chapter 1: What is Text Mining?
1.1 What is it?
1.2 Why We Care About Text Mining
1.3 A Basic Workflow – How the Process Works
1.4 What Tools Do I Need to Get Started with This?
1.5 A Simple Example
1.6 A Real World Use Case
1.7 Summary
Chapter 2: Basics of Text Mining
2.1 What is Text Mining in a Practical Sense?
2.2 Types of Text Mining: Bag of Words
2.3 The Text Mining Process in Context
2.4 String Manipulation: Number of Characters and Substitutions
2.5 Keyword Scanning
2.6 String Packages stringr and stringi
2.7 Preprocessing Steps for Bag of Words Text Mining
2.10 DeltaAssist Wrap Up
2.11 Summary
Chapter 3: Common Text Mining Visualizations
3.1 A Tale of Two (or Three) Cultures
3.2 Simple Exploration: Term Frequency, Associations and Word Networks
3.3 Simple Word Clusters: Hierarchical Dendrograms
3.4 Word Clouds: Overused but Effective
3.5 Summary
Chapter 4: Sentiment Scoring
4.1 What is Sentiment Analysis?
4.2 Sentiment Scoring: Parlor Trick or Insightful?
4.3 Polarity: Simple Sentiment Scoring
4.4 Emoticons – Dealing with These Perplexing Clues
4.5 R's Archived Sentiment Scoring Library
4.6 Sentiment the Tidytext Way
4.7 Airbnb.com Boston Wrap Up
4.8 Summary
Chapter 5: Hidden Structures: Clustering, String Distance, Text Vectors and Topic Modeling
5.1 What is clustering?
5.2 Calculating and Exploring String Distance
5.3 LDA Topic Modeling Explained
5.4 Text to Vectors using text2vec
5.5 Summary
Chapter 6: Document Classification: Finding Clickbait from Headlines
6.1 What is Document Classification?
6.2 Clickbait Case Study
6.3 Summary
Chapter 7: Predictive Modeling: Using Text for Classifying and Predicting Outcomes
7.1 Classification vs Prediction
7.2 Case Study I: Will This Patient Come Back to the Hospital?
7.3 Case Study II: Predicting Box Office Success
7.4 Summary
Chapter 8: The OpenNLP Project
8.1 What is the OpenNLP project?
8.2 R's OpenNLP Package
8.3 Named Entities in Hillary Clinton's Email
8.4 Analyzing the Named Entities
8.6 Summary
Chapter 9: Text Sources
9.1 Sourcing Text
9.2 Web Sources
9.3 Getting Text from File Sources
9.4 Summary
Index
End User License Agreement
Pages
xi
xii
1
2
3
4
5
6
7
8
10
11
12
13
14
15
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
37
38
39
40
41
42
43
44
45
46
47
48
49
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
205
206
207
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
305
306
307
Guide
Cover
Table of Contents
Foreword
Begin Reading
List of Illustrations
Chapter 1: What is Text Mining?
Figure 1.1 Possible enterprise uses of text min.
Figure 1.2 A gratuitous word cloud for Chapter 1.
Figure 1.3 Text mining is the transition from an unstructured state to a structured understandable state.
Chapter 2: Basics of Text Mining
Figure 2.1 Recall, text mining is the process of taking unorganized sources of text, and applying standardized analytical steps, resulting in a concise insight or recommendation. Essentially, it means going from an unorganized state to a summarized and structured state.
Figure 2.2 The sentence is parsed using simple part of speech tagging. The collected contextual data has been captured as tags, resulting in more information than the bag of words methodology captured.
Figure 2.3 The section of the term document matrix from the code above.
Chapter 3: Common Text Mining Visualizations
Figure 3.1 The bar plot of individual words has expected words like please, sorry and flight confirmation.
Figure 3.2 Showing that the most associated word from DeltaAssist's use of apologies is “delay”.
Figure 3.3 A simple word network, illustrating the node and edge attributes.
Figure 3.4 The matrix result from an R console of the matrix multiplication operator.
Figure 3.5 A larger matrix is returned with the answers to each of the multiplication inputs.
Figure 3.7 Calling the single qdap function yields a similar result, saving coding time.
Figure 3.8 The word association function's network.
Figure 3.9 The city rainfall data expressed as a dendrogram.
Figure 3.10 A reduced term DTM, expressed as a dendrogram for the @DeltaAssist corpus.
Figure 3.11 A modified dendrogram using a custom visualization. The dendrogram confirms the agent behavior asking for customers to follow and dm (direct message) the team with confirmation numbers.
Figure 3.12 The circular dendrogram highlighting the agent behavioral insights.
Figure 3.13 A representation of the three word cloud functions from the wordcloud package.
Figure 3.14 A simple word cloud with 100 words and two colors based on Delta tweets.
Figure 3.15 The words in common between Amazon and Delta customer service tweets.
Figure 3.16 A comparison cloud showing the contrasting words between Delta and Amazon customer service tweets.
Figure 3.17 An example polarized tag plot showing words in common between corpora. R will plot a larger version for easier viewing.
Chapter 4: Sentiment Scoring
Figure 4.1 Plutchik's wheel of emotion with eight primary emotional states.
Figure 4.2 Top 50 unique terms from ~2.5million tweets follows Zipf's distribution.
Figure 4.3 Qdap's polarity function equals 0.68 on this single sentence.
Figure 4.4 The original word cloud functions applied to various corpora.
Figure 4.5 Polarity based subsections can be used to create different corpora for word clouds.
Figure 4.6 Histogram created by ggplot code – notice that the polarity distribution is not centered at zero.
Figure 4.7 The sentiment word cloud based on a scaled polarity score and a TFIDF weighted TDM.
Figure 4.8 The sentiment word cloud based on a polarity score without scaling and a TFIDF weighted TDM.
Figure 4.9 some common image based emoji used for smart phone messaging.
Figure 4.10 Smartphone sarcasm emote.
Figure 4.11 Twitch's kappa emoji used for sarcasm.
Figure 4.12 The 10k Boston Airbnb reviews skew highly to the emotion joy.
Figure 4.13 A sentiment-based word cloud based on the 10k Boston Airbnb reviews. Apparently staying in Malden or Somerville leaves people in a state of disgust.
Figure 4.14 Bar plot of polarity as the Wizard of Oz story unfolds.
Figure 4.15 Smoothed polarity for the Wizard of Oz.
Chapter 5: Hidden Structures: Clustering, String Distance, Text Vectors and Topic Modeling
Figure 5.1 The five example documents in two-dimensional space.
Figure 5.2 The added centroids and partitions grouping the documents.
Figure 5.3 Centroid movement to equalizing the “gravitational pull” from the documents, thereby minimizing distances.
Figure 5.4 The final k-means partition with the correct document clusters.
Figure 5.5 The k-means clustering with three partitions on work experiences.
Figure 5.6 The plotcluster visual is overwhelmed by the second cluster and shows that partitioning was not effective.
Figure 5.7 The k-means clustering silhouette plot dominated by a single cluster.
Figure 5.8 The comparison clouds based on prototype scores.
Figure 5.9 A comparison of the k-means and spherical k-means distance measures.
Figure 5.10 The cluster assignments for the 50 work experiences using spherical k-means.
Figure 5.11 The spherical k-means cluster plot with some improved document separation.
Figure 5.12 The spherical k-means silhouette plot with three distinct clusters.
Figure 5.13 The spherical k-means comparison cloud improves on the original in the previous section.
Figure 5.14 K-mediod cluster silhouette showing a single cluster with 49 of the documents.
Figure 5.15 Mediod prototype work experiences 15 & 40 as a comparison cloud.
Figure 5.16 The six OSA operators needed for the distance measure between “raspberry” and “pear”.
Figure 5.17 The example fruit dendrogram.
Figure 5.18 “Highlighters” used to capture three topics in a passage.
Figure 5.19 The log likelihoods from the 25 sample iterations, showing it improving and then leveling off.
Figure 5.20 A screenshot portion for the resulting topic model visual.
Figure 5.21 Illustrating the articles' size, polarity and topic grouping.
Figure 5.22 The vector space of single word documents and a third document sharing both terms.
Chapter 6: Document Classification: Finding Clickbait from Headlines
Figure 6.1 The training step where labeled data is fed into the algorithm.
Figure 6.2 The algorithm is applied to new data and the output is a predicted value or class.
Figure 6.3 Line 42 of the trace window where “Acronym” needs to be changed to “acronym.”
Figure 6.4 Training data is split into nine training sections, with one holdout portion used to evaluate the classifier's performance. The process is repeated while shuffling the holdout partitions. The resulting performance measures from each of the models are averaged so the classifier is more reliable.
Figure 6.5 The interaction between lambda values and misclassification rates for the lasso regression model.
Figure 6.6 The ROC using the lasso regression predictions applied to the training headlines.
Figure 6.7 A comparison between training and test sets.
Figure 6.8 The kernel density plot for word coefficients.
Figure 6.9 Top and bottom terms impacting clickbait classifications.
Figure 6.10 The probability scatter plot for the lasso regression.
Chapter 7: Predictive Modeling: Using Text for Classifying and Predicting Outcomes
Figure 7.1 The text-only GLMNet model accuracy results.
Figure 7.2 The numeric and dummy patient information has an improved AUC compared to the text only model.
Figure 7.3 The cross validated model AUC improves close to 0.8 with all inputs.
Figure 7.4 The additional lift provided using all available data instead of throwing out the text inputs.
Figure 7.5 Wikipedia's intuitive explanation for precision and recall.
Figure 7.6 Showing the minimum and “1 se” lambda values that minimize the model's mse.
Figure 7.7 The box plot comparing distributions between actual and predicted values.
Figure 7.8 The scatter plot between actual and predicted values shows a reasonable relationship.
Figure 7.9 The test set actual and predicted values show a wider dispersion than with the training set.
Chapter 8: The OpenNLP Project
Figure 8.1 A sentence parsed syntactically with various annotations.
Figure 8.2 The most frequent organizations identified by the named entity model.
Figure 8.3 The base map with ggplot's basic color and axes.
Figure 8.4 The worldwide map with locations from 551 Hillary Clinton emails.
Figure 8.5 The Google map of email locations.
Figure 8.6 Black and white map of email locations.
Figure 8.7 A normal distribution alongside a box and whisker plot with three outlier values.
Figure 8.8 A box and whisker plot of Hillary Clinton emails containing “Russia,” “Senate” and “White House.”
Figure 8.9 The Quantmod's simple line chart for Microsoft's stock price.
Figure 8.10 Entity polarity over time.
Chapter 9: Text Sources
Figure 9.1 The methodology breakdown for obtaining text exemplified in this chapter.
Figure 9.2 An Amazon help forum thread mentioning Prime movies.
Figure 9.3 A portion of the Amazon forum page with SelectorGadget turned on and the thread text highlighted.
Figure 9.4 A portion of Amazon's general help forum.
Figure 9.5 The forum scraping workflow showing two steps requiring scraping information.
Figure 9.6 A visual representation of the amzn.forum list.
Figure 9.7 The row bound list resulting in a data table.
Figure 9.8 A typical Google News feed for Amazon Echo.
Figure 9.9 A portion of a newspaper image to be sent to the OCR service.
Figure 9.10 The final OCR text containing the image text.
List of Tables
Chapter 1: What is Text Mining?
Table 1.1 Example use cases and recommendations to use or not use text mining.
Chapter 2: Basics of Text Mining
Table 2.1 An abbreviated document term matrix, showing simple word counts contained in the three-tweet corpus.
Table 2.2 The term document matrix contains the same information as the document term matrix but is the transposition. The rows and columns have been switched.
Table 2.3 @DeltaAssist agent workload –
The abbreviated Table demonstrates a simple text mining analysis that can help with competitive intelligence and benchmarking for customer service workloads.
Table 2.4 Common text-preprocessing functions from R's
tm
package with an example of the transformation's impact.
Table 2.5 In common English writing, these words appear frequently but offer little insight. As a result, they are often removed to prepare a document for text mining.
Chapter 3: Common Text Mining Visualizations
Table 3.1 A small term document matrix, called all to build an example word network.
Table 3.2 The adjacency matrix based on the small TDM in Table 3.1.
Table 3.3 A small data set of annual city rainfall that will be used to create a dendrogram.
Table 3.4 The ten terms of the Amazon and Delta TDM.
Table 3.5 The tail of the common.words matrix.
Chapter 4: Sentiment Scoring
Table 4.1 An example subjectivity lexicon from University of Pittsburgh's MQPA Subjectivity Lexicon.
Table 4.2 The top terms in a word frequency matrix show an expected distribution.
Table 4.3 The polarity function output
Table 4.4 Polarity output with the custom and non-custom subjectivity lexicon.
Table 4.5 A portion of the TDM using frequency count
Table 4.6 A portion of the Term Document Matrix using TFIDF
Table 4.7 A small sample of the hundreds of punctuation and symbol based emoticons.
Table 4.8 Example native emoticons expressed as Unicode and byte strings in R.
Table 4.9 Common punctuation based emoticons
Table 4.10 Pre-constructed punctuation-based emoticon dictionary from qdap.
Table 4.11 Common emoji with Unicode and R byte representations.
Table 4.12 The last six emotional words in the sentiment lexicon.
Table 4.13 The first six Airbnb reviews and associated sentiments data frame.
Table 4.14 Excerpt from the sentiments data frame.
Table 4.15 A portion of the tidy text data frame.
Table 4.16 The first ten Wizard of Oz “Joy” words.
Table 4.17 The oz.sentiment data frame with key value pairs spread across the polarity term counts.
Chapter 5: Hidden Structures: Clustering, String Distance, Text Vectors and Topic Modeling
Table 5.1 A sample corpus of documents only containing two terms.
Table 5.2 Simple term frequency for an example corpus.
Table 5.3 Top five terms for each cluster can help provide useful insights to share.
Table 5.4 The five string distance measurement methods.
Table 5.5 The distance matrix from the three fruits.
Table 5.6 A portion of the Airbnb Reviews vocabulary.
Table 5.7 A portion of the example sentence target and context words with a window of 1.
Table 5.8 A portion of the input and output relationships used in skip gram modeling.
Table 5.9 The top vector terms from good.walks.
Table 5.10 The top ten terms demonstrating the cosine distance of dirty and other nouns.
Chapter 6: Document Classification: Finding Clickbait from Headlines
Table 6.1 The confusion Table for the training set.
Table 6.2 The first six rows from headline.preds.
Table 6.3 The complete top.coef data, illustrating the negative words having a positive probability.
Chapter 7: Predictive Modeling: Using Text for Classifying and Predicting Outcomes
Table 7.1 The AUC values for the three classification models.
Table 7.2 The best model's confusion matrix for the training patient data.
Table 7.3 The confusion matrix with example variables.
Table 7.4 The test set confusion matrix.
Table 7.5 Three actual and predicted values for example movies.
Table 7.6 With error values calculated.
Table 7.8 The original train.dat data frame.
Table 7.9 The tidy format of the same table.
Chapter 8: The OpenNLP Project
Table 8.1 The five functions of the OpenNLP package.
Table 8.2 The Penn Treebank POS tag codes.
Table 8.3 The named entity models that can be used in openNLPmodels.en.
Table 8.4 Named people found in the third email.
Table 8.5 Named organizations found in the third email.
Table 8.6 Example organizations that were identified and their corresponding frequency.
Chapter 9: Text Sources
Table 9.1 Forum.posts and thread.urls using rvest.
Table 9.2 The Guardian API response data for the “text” object.
Text Mining in Practice with R
Ted Kwartler
This edition first published 2017
© 2017 John Wiley & Sons Ltd
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of Ted Kwartler to be identified as the author of this work has been asserted in accordance with law.
Registered Office
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
Editorial Offices
111 River Street, Hoboken, NJ 07030, USA
9600 Garsington Road, Oxford, OX4 2DQ, UK
The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Some content that appears in standard print versions of this book may not be available in other formats.
Limit of Liability/Disclaimer of Warranty
The publisher and the authors make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of fitness for a particular purpose. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for every situation. In view of ongoing research, equipment modifications, changes in governmental regulations, and the constant flow of information relating to the use of experimental reagents, equipment, and devices, the reader is urged to review and evaluate the information provided in the package insert or instructions for each chemical, piece of equipment, reagent, or device for, among other things, any changes in the instructions or indication of usage and for added warnings and precautions. The fact that an organization or website is referred to in this work as a citation and/or potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this works was written and when it is read. No warranty may be created or extended by any promotional statements for this work. Neither the publisher nor the author shall be liable for any damages arising here from.
Library of Congress Cataloging-in-Publication Data
Names: Kwartler, Ted, 1978- author.
Title: Text mining in practice with R / Ted Kwartler.
Description: Hoboken, NJ : John Wiley & Sons, 2017. | Includes bibliographical references and index.
Identifiers: LCCN 2017006983 (print) | LCCN 2017010584 (ebook) | ISBN 9781119282013 (cloth) | ISBN 9781119282099 (pdf) | ISBN 9781119282082 (epub)
Subjects: LCSH: Data mining. | Text processing (Computer science)
Classification: LCC QA76.9.D343 K94 2017 (print) | LCC QA76.9.D343 (ebook) | DDC 006.3/12-dc23
LC record available at https://lccn.loc.gov/2017006983
Cover Design: Wiley
Cover Image: © ChrisPole/Gettyimages
“It's the math of talking...your two favorite things!”
This book is dedicated to my beautiful wife, and best friend Meghan. Your patience, support and assurance cannot be quantified.
Additionally Nora and Brenna are my motivation, teaching me to be a better person.
Foreword
This book has been a long labor of love. When I agreed to write a book, I had no idea of the amount of work and research needed. Looking back, it was pure hubris on my part to accept a writing contract from the great people at Wiley. The six-month project extended outward to more than a year! From the outset I decided to write a book that was less technical or academic and instead focused on code explanations and case studies. I wanted to distill my years of work experience, blog reading and textbook research into a succinct and more approachable format. It is easy to copy a blog's code or state a textbook's explanation verbatim, but it is wholesale more difficult to be original, to explain technical attributes in an easy-to-understand manner and hopefully to make the journey more fun for the reader.
Each chapter demonstrates a text mining method in the context of a real case study. Generally, mathematical explanations are brief and set apart from the code snippets and visualizations. While it is still important to understand the underlying mathematical attributes of a method, this book merely gives you a glimpse. I believe it is easier to become an impassioned text miner if you get to explore and create first. Applying algorithms to interesting data should embolden you to undertake and learn more. Many of the topics covered could be expanded into a standalone book, but here they are related as a single section or chapter. This is on purpose, so you get a quick but effective glimpse at the text mining universe! So my hope is that this book will serve as a foundation as you continually add to your data science skillset.
As a writer or instructor I have always leaned on common sense and non-academic explanations. The reason for this is simple: I do not have a computer science or math degree. Instead, my MBA gives me a unique perspective on data science. It has been my observation that data scientists often enjoy the modeling and data wrangling, but very often fail to completely understand the needs of the business. Thus many data science business applications are actually months in implementation or miss a crucial aspect. This book strives to have original and thought-provoking case studies with truly messy data. In other text mining or data science books, data that perfectly describes the method is illustrated so the concept can be understood. In this book, I reverse that approach and attempt to use real data in context so you can learn how typical text mining data is modeled and what to expect. The results are less pretty but more indicative of what you should expect as a text mining practitioner.
“It takes a village to write a book.”
Throughout this journey I have had the help of many people. Thankfully, family and friends have been accommodating and understanding when I chose writing ahead of social gatherings. First and foremost thanks to my mother, Trish, who gave me the gift of gab, and qualitative understanding and to my father Yitz, who gave me quantitative and technical writing acumen. Additional thanks to Paul, MaryAnn, Holly, Rob, K, and Maureen for understanding when I had to steal away and write during visits.
Thank you to Barry Keating, Sarv Devaraj and Timothy Gilbride. The Notre Dame family, with their supportive, entertaining professors put me onto this path. Their guidance, dialogue and instructions opened my eyes to machine learning, data science and ultimately text mining. My time at Notre Dame has positively affected my life and those around me. I am forever grateful.
Multiple data scientists have helped me along the way. In fact to many to actually list. Particular thanks to Greg, Zach, Hamel, Jeremy, Tom, Dalin, Sergey, Owen, Peter, Dan, Hugo and Nick for their explanations at different points in my personal data science journey.
This book would not have been possible if it weren't for Kathy Powers. She has been a lifelong friend and supporter and amazingly stepped up to make revisions when asked. When I changed publishers and thought of giving up on the book her support and patience with my poor grammar helped me continue. My entire family owes you a debt of gratitude that is never able to be repaid.
Chapter 1What is Text Mining?
In this chapter, you will learn
the basic definition of practical text mining
why text mining is important to the modern enterprise
examples of text mining used in enterprise
the challenges facing text mining
an example workflow for processing natural language in analytical contexts
a simple text mining example
when text mining is appropriate
Learning how to perform text mining should be an interesting and exciting journey throughout this book. A fun artifact of learning text mining is that you can use the methods in this book on your own social media or online exchanges. Beyond these everyday online applications to your personal interactions, this book provides business use cases in an effort to show how text mining can improve products, customer service, marketing or human resources.
1.1 What is it?
There are many technical definitions of text mining both on the Internet and in textbooks, but as the primary goal of text mining in this book is the extraction of an output that is useful such as a visualization or structured table of outputs to be used elsewhere; this is my definition:
Text mining is the process of distilling actionable insights from text.
Text mining within the context of this book is a commitment to real world cases which impact business. Therefore, the definition and this book are aimed at meaningful distillation of text with the end goal to aid a decision-maker. While there may be some differences, the terms text mining and text analytics can be used interchangeably. Word choice is important; I use text mining because it more adequately describes the uncovering of insights and the use of specific algorithms beyond basic statistical analysis.
1.1.1 What is Text Mining in Practice?
In this book, text mining is more than an academic exercise. I hope to show that text mining has enterprise value and can contribute to various business units. Specifically, text mining can be used to identify actionable social media posts for a customer service organization. It can be used in human resources for various purposes such as understanding candidate perceptions of the organization or to match job descriptions with resumes. Text mining has marketing implications to measure campaign salience. It can even be used to identify brand evangelists and impact customer propensity modeling. Presently the state of text mining is somewhere between novelty and providing real actionable business intelligence. The book gives you not only the tools to perform text mining but also the case studies to help identify practical business applications to get your creative text mining efforts started.
1.1.2 Where Does Text Mining Fit?
Text mining fits within many disciplines. These include private and academic uses. For academics, text mining may aid in the analytical understanding of qualitatively collected transcripts or the study of language and sociology. For the private enterprise, text mining skills are often contained in a data science team. This is because text mining may yield interesting and important inputs for predictive modeling, and also because the text mining skillset has been highly technical. However, text mining can be applied beyond a data science modeling workflow. Business intelligence could benefit from the skill set by quickly reviewing internal documents such as customer satisfaction surveys. Competitive intelligence and marketers can review external text to provide insightful recommendations to the organization. As businesses are saving more textual data, they will need to break text-mining skills outside of a data science team. In the end, text mining could be used in any data driven decision where text naturally fits as an input.
1.2 Why We Care About Text Mining
We should care about textual information for a variety of reasons.
Social media continues to evolve and affect an organization's public efforts.
Online content from an organization, its competitors and outside sources, such as blogs, continues to grow.
The digitization of formerly paper records is occurring in many legacy industries, such as healthcare.
New technologies like automatic audio transcription are helping to capture customer touchpoints.
As textual sources grow in quantity, complexity and number of sources, the concurrent advance in processing power and storage has translated to vast amounts of text being stored throughout an enterprise's data lake.
Yet today's successful technology companies largely rely on numeric and categorical inputs for information gains, machine learning algorithms or operational optimization. It is illogical for an organization to study only structured information yet still devote precious resources to recording unstructured natural language. Text represents an untapped input that can further increase competitive advantage. Lastly, enterprises are transitioning from an industrial age to an information age; one could argue that the most successful companies are transitioning again to a customer-centric age. These companies realize that taking a long term view of customer wellbeing ensures long term success and helps the company to remain salient. Large companies can no longer merely create a product and forcibly market it to end-users. In an age of increasing customer expectations customers want to be heard by corporations. As a result, to be truly customer centric in a hyper competitive environment, an organization should be listening to their constituents whenever possible. Yet the amount of textual information from these interactions can be immense, so text mining offers a way to extract insights quickly.
Text mining will make an analyst's or data scientist's efforts to understand vast amounts of text easier and help ensure credibility from internal decision-makers. The alternative to text mining may mean ignoring text sources or merely sampling and manually reviewing text.
1.2.1 What Are the Consequences of Ignoring Text?
There are numerous consequences of ignoring text.
Ignoring text is not an adequate response of an analytical endeavor. Rigorous scientific and analytical exploration requires investigating sources of information that can explain phenomena.
Not performing text mining may lead an analysis to a false outcome.
Some problems are almost entirely text-based, so not using these methods would mean significant reduction in effectiveness or even not being able to perform the analysis.
Explicitly ignoring text may be a conscious analyst decision, but doing so ignores text's insightful possibilities. This is analogous to an ostrich that sticks its head in the ground when confronted. If the aim is robust investigative quantitative analysis, then ignoring text is inappropriate. Of course, there are constraints to data science or business analysis, such as strict budgets or timelines. Therefore, it is not always appropriate to use text for analytics, but if the problem being investigated has a text component, and resource constraints do not forbid it, then ignoring text is not suitable.
Wisdom of Crowds 1.1
As an alternative, some organizations will sample text and manually review it. This may mean having a single assessor or panel of readers or even outsourcing analytical efforts to human-based services like mturk or crowdflower. Often communication theory does not support these methods as a sound way to score text, or to extract meaning. Setting aside sampling biases and logistical tabulation difficulties, communication theory states that the meaning of a message relies on the recipient. Therefore a single evaluator introduces biases in meaning or numerical scoring, e.g. sentiment as a numbered scale. Additionally, the idea behind a group of people scoring text relies on Sir Francis Galton's theory of “Vox Populi” or wisdom of crowds.
To exploit the wisdom of crowds four elements must be considered:
Assessors need to exercise independent judgments.
Assessors need to possess a diverse information understanding.
Assessors need to rely on local knowledge.
There has to be a way to tabulate the assessors' results.
Sir Francis Galton's experiment exploring the wisdom of crowds met these conditions with 800 participants. At an English country fair, people were asked to guess the weight of a single ox. Participants guessed separately from each other without sharing the guess. Participants were free to look at the cow themselves yet not receive expert consultation. In this case, contestants had a diverse background. For example, there were no prerequisites stating that they needed to be a certain age, demographic or profession. Lastly, guesses were recorded on paper for tabulation by Sir Francis to study. In the end, the experiment showed the merit of the wisdom of crowds. There was not an individual correct guess. However, the median average of the group was exactly right. It was even better than the individual farming experts who guessed the weight.
If these conditions are not met explicitly, then the results of the panel are suspect. This may seem easy to do, but in practice it is hard to ensure within an organization. For example a former colleague at a major technology company in California shared a story about the company's effort to create Internet-connected eyeglasses. The eyeglasses were shared with internal employees, and feedback was then solicited. The text feedback was sampled and scored by internal employees. At first blush this seems like a fair assessment of the product's features and expected popularity. However, the conditions for the wisdom of crowds were not met. Most notably, the need for a decentralized understanding of the question was not met. As members of the same technology company, the respondents are already part of a self-selected group that understood the importance of the overall project within the company. Additionally, the panel had a similar assessment bias because they were from the same division that was working on the project. This assessing group did not satisfy the need for independent opinions when assessing the resulting surveys. Further, if a panel is creating summary text as the output of the reviews, then the effort is merely an information reduction effort similar to numerically taking an average. Thus it may not solve the problem of too much text in a reliable manner. Text mining solves all these problems. It will use all of the presented text and does so in a logical, repeatable and auditable way. There may be analyst or data scientist biases but they are documented in the effort and are therefore reviewable. In contrast, crowd-based reviewer assessments are usually not reviewable.
Despite the pitfalls of ignoring text or using a non-scientific sampling method, text mining offers benefits. Text mining technologies are evolving to meet the demands of the organization and provide benefits leading to data-driven decisions. Throughout this book, I will focus on benefits and applied applications of text mining in business.
1.2.2 What Are the Benefits of Text Mining?
There are many benefits of text mining including:
Trust is engendered among stakeholders because little to no sampling is needed to extract information.
The methodologies can be applied quickly.
Using R allows for auditable and repeatable methods.
Text mining identifies novel insights or reinforces existing perceptions based on all relevant information.
Interestingly, text mining first appears in the Gartner Hype Cycle in 2012. At that moment, it was listed in the “trough of disillusionment.” In subsequent years, it has not been listed on the cycle at all, leading me to believe that text analysis is either at a steady enterprise use state or has been abandoned by enterprises as not useful. Despite not being listed, text mining is used across industries and in various manners. It may not have exceeded the over-hyped potential of 2012's Gartner Hype Cycle, but text is showing merit. Hospitals use text mining of doctors' notes to understand readmission characteristics of patients. Financial and insurance companies use text to identify compliance risks. Retailers use customer service notes to make operational changes when failing customer expectations. Technology product companies use text mining to seek out feature requests in online reviews. Marketing is a natural fit for text analysis. For example, marketing companies monitor social media to identify brand evangelists. Human resource analytics efforts focus on resume text to match to job description text. As described here, mastering text mining is a skill set sought out across verticals and is therefore a worthwhile professional endeavor. Figure 1.1 shows possible business units that can benefit from text mining in some form.
Figure 1.1 Possible enterprise uses of text min.
1.2.3 Setting Expectations: When Text Mining Should (and Should Not) Be Used
Since text is often a large part of a company's database, it is believed that text mining will lead to ground-breaking discoveries or significant optimization. As a result, senior leaders in an organization will devote resources to text mining, expecting to yield extensive results. Often specialists are hired, and resources are explicitly devoted to text mining. Outside of text mining software, in this case R, it is best to use text mining only in cases where it naturally fits the business objective and problem definition. For example, at a previous employer, I wondered how prospective employees viewed our organization compared to peer organizations. Since these candidates were outside the organization, capturing numerical or personal information such as age or company-related perspective scoring was difficult. However, there are forums and interview reviews anonymously shared online. These are shared as text so naturally text mining was an appropriate tool. When using text mining, you should prioritize defining the problem and reviewing applicable data, not using an exotic text mining method. Text mining is not an end in itself and should be regarded as another tool in an analyst's or data scientist's toolkit.
Text mining cannot distill large amounts of text to gain an absolute view of the truth. Text mining is part art and part science. An analyst can mislead stakeholders by removing certain words or using only specific methods. Thus, it is important to be up front about the limitations of text mining. It does not reveal an absolute truth contained within the text. Just as an average reduces information for consumption of a large set of numbers, text mining will reduce information. Sometimes it confirms previously held beliefs and sometimes it provides novel insights. Similar to numeric dimension reduction techniques, text mining abridges outliers, low frequency phrases and important information. It is important to understand that language is more colorful and diverse in understanding than numerical or strict categorical data. This poses a significant problem for text miners. Stakeholders need to be wary of any text miner who knows a truth solely based on the algorithms in this book. Rather, the methods in this book can help with the narrative of the data and the problem at hand, or the outputs can even be used in supervised learning alongside numeric data to improve the predictive outcomes. If doing predictive modeling using text, a best practice when modeling alongside non-text data features is to model with and without the text in the attribute set. Text is so diverse that it may even add noise to predictive efforts. Table 1.1 refers to actual use cases where text mining may be appropriate.
Table 1.1 Example use cases and recommendations to use or not use text mining.
Example use case
Recommendation
Survey texts
Explore topics using various methods to gain a respondent's perspective.
Reviewing a small number of documents
Don't perform text mining on an extremely small corpus, as the results and conclusion can be skewed.
Human resource documents
Tread carefully; text mining may yield insights, but the data and legal barriers may make the analysis inappropriate.
Social media
Use text mining to collect (when allowed) from online sources and then apply preprocessing steps to extract information.
Data science predictive modeling
Text mining can yield structured inputs that could be useful in machine learning efforts.
Product/service reviews
Use text mining if the number of reviews is large.
Legal proceeding
Use text mining to identify individuals and specific information.
Another suggestion for effective text mining is to avoid over using a word cloud. Analysts armed with the knowledge of this book should not create a word cloud without a need for it. This is because word clouds are often used without need, and as a result they can actually diminish their impact. However, word clouds are popular and can be powerful in showing term frequency, among other things, such as the one in Figure 1.2, which runs over the text of this chapter. Throwing caution to the wind, it demonstrates a word cloud of terms in Chapter 1. It is not very insightful because, as expected, the terms text and mining are the most frequent and largest words in the cloud!
Figure 1.2 A gratuitous word cloud for Chapter 1.
In fact, word clouds are so popular that an entire chapter is devoted to various types of word clouds that can be insightful. However, many people consider word clouds a cliché, so their impact is fading. Also, word clouds represent a relatively easy way to mislead consumers of an analysis. In the end, they should be used in conjunction with other methods to confirm the correctness of a conclusion.
1.3 A Basic Workflow – How the Process Works
Text represents unstructured data that must be preprocessed into a structured manner. Features need to be defined and then extracted from the larger body of organized text known as a corpus. These extracted features are then analyzed. The chevron arrows in Figure 1.3 represent structured predefined steps that are applied to the unorganized text to reach the final output or conclusion. Overall Figure 1.3 is a high level workflow of a text mining project.
Figure 1.3 Text mining is the transition from an unstructured state to a structured understandable state.
The steps for text mining include:
1. Define the problem and specific goals.
As with other analytical endeavors, it is not prudent to start searching for answers. This will disappoint decision-makers and could lead to incorrect outputs. As the practitioner, you need to acquire subject matter expertise sufficient to define the problem and the outcome in an appropriate manner.
2. Identify the text that needs to be collected.
Text can be from within the organization or outside. Word choice varies between mediums like Twitter and print so care must be taken to explicitly select text that is appropriate to the problem definition.
Chapter 9
covers places to get text beyond reading in files. The sources covered include basic web scraping, APIs and R's specific API libraries, like “twitteR.” Sources are covered later in the book so you can focus on the tools to text mine, without the additional burden of finding text to work on.
3. Organize the text.
Once the appropriate text is identified, it is collected and organized into a corpus or collection of documents.
Chapter 2
covers two types of text mining conceptually, and then demonstrates some preparation steps used in a “bag of words” text mining method.
4. Extract features.
Creating features means preprocessing text for the specific analytical methodology being applied in the next step. Examples include making all text lowercase, or removing punctuation. The analytical technique in the next step and the problem definition dictate how the features are organized and used.
Chapters 3
and
4
work on basic extraction to be used in visualizations or in a sentiment polarity score. These chapters are not performing heavy machine learning or technical analysis, but instead rely on simple information extraction such as word frequency.
5. Analyze.
Apply the analytical technique to the prepared text. The goal of applying an analytical methodology is to gain an insight or a recommendation or to confirm existing knowledge about the problem. The analysis can be relatively simple, such as searching for a keyword, or it may be an extremely complex algorithm. Subsequent chapters require more in-depth analysis based on the prepared texts. A chapter is devoted to unsupervised machine learning to analyze possible topics. Another illustrates how to perform a supervised classification while another performs predictive modeling. Lastly you will switch from a “bag of words” method to syntactic parsing to find named entities such as people's names.
6. Reach an insight or recommendation.
The end result of the analysis is to apply the output to the problem definition or expected goal. Sometimes this can be quite novel and unexpected, or it can confirm the previously held idea. If the output does not align to the defined problem or completely satisfy the intended goal, then the process becomes repetitious and can be changed at various steps. By focusing on real case studies that I have encountered, I hope to instill a sense of practical purpose to text mining. To that end, the case studies, the use of non-academic texts and the exercises of this book are meant to lead you to an insight or narrative about the issue being investigated. As you use the tools of this book on your own, my hope is that you will remember to lead your audience to a conclusion.
The distinct steps are often specific to the particular problem definition or analytical technique being applied. For example, if one is analyzing tweets, then removing retweets may be useful but it may not be needed in other text mining exploration. Using R for text mining means the processing steps are repeatable and auditable. An analyst can customize the preprocessing steps outlined throughout the book to improve the final output. The end result is an insight, a recommendation or may be used in another analysis. The R scripts in this book follow this transition from an unorganized state to an organized state, so it is important to recall this mental map.
The rest of the book follows this workflow and adds more context and examples along the way. For example, Chapter 2 examines the two main approaches to text mining and how to organize a collection of documents into a clean corpus. From there you start to extract features of the text that are relevant to the defined problem. Subsequent chapters add visualizations, such as word clouds, so that a data scientist can tell the analytical narrative in a compelling way to stakeholders. As you progress through the book the types and methods of extracted features or information grow in complexity because the defined problems get more complex. You quickly divert to covering sentiment polarity so you can understand Airbnb reviews. Using this information you will build compelling visualizations and know what qualities are part of a good Airbnb review. Then in Chapter 5 you learn topic modeling using machine learning. Topic modeling provides a means to understand the smaller topics associated within a collection of documents without reading the documents themselves. It can be useful for tagging documents relating to a subject. The next subject, document classification, is used often. You may be familiar with document classification because it is used in email inboxes to identify spam versus legitimate emails. In this book's example you are searching for “clickbait” from online headlines. Later you examine text as it relates to patient records to model how a hospital identifies diabetic readmission. Using this method, some hospitals use text to improve patient outcomes. In the same chapter you even examine movie reviews to predict box office success. In a subsequent chapter you switch from the basic bag of words methodology to syntactic parsing using the OpenNLP library. You will identify named entities, such as people, organizations and locations within Hillary Clinton's emails. This can be useful in legal proceedings in which the volume of documentation is large and the deadlines are tight. Marketers also use named entity recognition to understand what influencers are discussing. The remaining chapters refocus your attention back to some more basic principles at the top of the workflow, namely where to get text and how to read it into R. This will let you use the scripts in this book with text that is thought provoking to your own interests.
1.4 What Tools Do I Need to Get Started with This?
To get started in text mining you need a few tools. You should have access to a laptop or workstation with at least 4GB of RAM. All of the examples in this book have been tested on a Microsoft's Windows operating systems. RAM is important because R's processing is done “in memory.” This means that the objects being analyzed must be contained in the RAM memory. Also, having a high speed internet connection will aid in downloading the scripts, R library packages and example text data and for gathering text from various webpages. Lastly, the computer needs to have an installation of R and R Studio. The operating system of the computer should not matter because R has an installation for Microsoft, Linux and Mac.
1.5 A Simple Example
Online customer reviews can be beneficial to understanding customer perspectives about a product or service. Further, reviewers can sometimes leave feedback anonymously, allowing authors to be candid and direct. While this may lead to accurate portrayals of a product it may lead to “keyboard courage” or extremely biased opinions. I consider it a form of selection bias, meaning that the people that leave feedback may have strong convictions not indicative of the overall product or service's public perception. Text mining allows an enterprise to benchmark their product reviews and develop a more accurate understanding of some public perceptions. Approaches like topic modeling and polarity (positive and negative scoring) which are covered later in this book may be applied in this context. Scoring methods can be normalizedacross different mediums such as forums or print, and when done against a competing product, the results can be compelling.
Suppose you are a Nike employee and you want to know about how consumers are viewing the Nike Men's Roshe Run Shoes. The text mining steps to follow are:
1. Define the problem and specific goals.
Using online reviews, identify overall positive or negative views. For negative reviews, identify a consistent cause of the poor review to be shared with the product manager and manufacturing personnel.
2. Identify the text that needs to be collected.
There are running websites providing expert reviews, but since the shoes are mass market, a larger collection of general use reviews would be preferable. New additions come out annually, so old reviews may not be relevant to the current release. Thus, a shopping website like Amazon could provide hundreds of reviews, and since there is a timestamp on each review, the text can be limited to a particular timeframe.
3. Organize the text.
Even though Amazon reviewers rate products with a number of stars, reviews with three or fewer stars may yield opportunities to improve. Web scraping all reviews into a simple csv with a review per row and the corresponding timestamp and number of stars in the next columns will allow the analysis to subset the corpus by these added dimensions.
4. Extract features.
Reviews will need to be cleaned so that text features can be analyzed. For this simple example, this may mean removing common words with little benefit like “shoe” or “nike,” running a spellcheck and making all text lowercase.
5. Analyze.
A very simple way to analyze clean text, discussed in an early chapter, is to scan for a specific group of keywords. The text-mining analyst may want to scan for words given their subject matter expertise. Since the analysis is about shoe problems one could scan for “fit,” “rip” or “tear,” “narrow,” “wide,” “sole,” or any other possible quality problem from reviews. Then summing each could provide an indication of the most problematic feature. Keep in mind that this is an extremely simple example and the chapters build in complexity and analytical rigor beyond this illustration.
6. Reach an insight or recommendation.
Armed with this frequency analysis, a text miner could present findings to the product manager and manufacturing personnel that the top consumer issue could be “narrow” and “fit.” In practical application, it is best to offer more methodologies beyond keyword frequency, as support for a finding.
1.6 A Real World Use Case
It is regularly the case that marketers learn best practices from each other. Unlike in other professions many marketing efforts are available outside of the enterprise, and competitors can see the efforts easily. As a result, competitive intelligence in this space is rampant. It is also another reason why novel ideas are often copied and reused, and then the novel idea quickly loses salience with its intended audience. Text mining offers a quick way to understand the basics of a competitor's text-based public efforts.
When I worked at amazon.com, creating the social customer service team, we were obsessed with how others were doing it. We regularly read and reviewed other companies' replies and learned from their missteps. This was early 2012, so customer service in social media was considered an emerging practice, let alone being at one of the largest retailers in the world. At the time, the belief was that it was fraught with risk. Amazon's legal counsel, channel marketers in charge of branding and even customer service leadership were weary of publically acknowledging any shortcomings or service issues. The legal department was involved to understand if we were going to set undeliverable expectations or cause any tax implications on a state-by-state basis. Further, each brand owner, such as Amazon Prime, Amazon Mom, Amazon MP3, Amazon Video on Demand, and Amazon Kindle had cultivated their own style of communicating through their social media properties. Lastly, customer service leadership had made multiple promises that reached all the way to Jeff Bezos, the CEO, about flawless execution and servicing in this channel demonstrating customer centricity. The mandate was clear: proceed, but do so cautiously and do not expand faster than could be reasonably handled to maintain quality set by all these internal parties. The initial channels we covered were the two “Help” forums on the site, then retail and Kindle Facebook pages, and lastly, Twitter. We had our own missteps. I remember the email from Jeff that came down through the ranks with a simple “?” concerning an inappropriate briefly posted video to the Facebook wall. That told me our efforts were constantly under review and that we had to be as good as or better than other companies.
