35,99 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Build next-generation Artificial Intelligence systems with Java
Key Features
- Implement AI techniques to build smart applications using Deeplearning4j
- Perform big data analytics to derive quality insights using Spark MLlib
- Create self-learning systems using neural networks, NLP, and reinforcement learning
Book Description
In this age of big data, companies have larger amount of consumer data than ever before, far more than what the current technologies can ever hope to keep up with. However, Artificial Intelligence closes the gap by moving past human limitations in order to analyze data.
With the help of Artificial Intelligence for big data, you will learn to use Machine Learning algorithms such as k-means, SVM, RBF, and regression to perform advanced data analysis. You will understand the current status of Machine and Deep Learning techniques to work on Genetic and Neuro-Fuzzy algorithms. In addition, you will explore how to develop Artificial Intelligence algorithms to learn from data, why they are necessary, and how they can help solve real-world problems.
By the end of this book, you'll have learned how to implement various Artificial Intelligence algorithms for your big data systems and integrate them into your product offerings such as reinforcement learning, natural language processing, image recognition, genetic algorithms, and fuzzy logic systems.
What you will learn
- Manage Artificial Intelligence techniques for big data with Java
- Build smart systems to analyze data for enhanced customer experience
- Learn to use Artificial Intelligence frameworks for big data
- Understand complex problems with algorithms and Neuro-Fuzzy systems
- Design stratagems to leverage data using Machine Learning process
- Apply Deep Learning techniques to prepare data for modeling
- Construct models that learn from data using open source tools
- Analyze big data problems using scalable Machine Learning algorithms
Who this book is for
This book is for you if you are a data scientist, big data professional, or novice who has basic knowledge of big data and wish to get proficiency in Artificial Intelligence techniques for big data. Some competence in mathematics is an added advantage in the field of elementary linear algebra and calculus.
Anand Deshpande is the Director of big data delivery at Datametica Solutions. He is responsible for partnering with clients on their data strategies and helps them become data-driven. He has extensive experience with big data ecosystem technologies. He has developed a special interest in data science, cognitive intelligence, and an algorithmic approach to data management and analytics. He is a regular speaker on data science and big data at various events. Manish Kumar is a Senior Technical Architect at Datametica Solutions. He has more than 11 years of industry experience in data management as a data, solutions, and product architect. He has extensive experience in building effective ETL pipelines, implementing security over Hadoop, implementing real-time data analytics solutions, and providing innovative and best possible solutions to data science problems. He is a regular speaker on big data and data science.Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 442
Veröffentlichungsjahr: 2018
Ähnliche
Artificial Intelligence for Big Data
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Sunith ShettyAcquisition Editor: Tushar GuptaContent Development Editor: Tejas LimkarTechnical Editor: Dinesh ChaudharyCopy Editor: Safis EditingProject Coordinator: Manthan PatelProofreader: Safis EditingIndexer: Priyanka DhadkeGraphics: Tania DuttaProduction Coordinator: Aparna Bhagat
First published: May 2018
Production reference: 1170518
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78847-217-3
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors
About the authors
Anand Deshpande is the Director of big data delivery at Datametica Solutions. He is responsible for partnering with clients on their data strategies and helps them become data-driven. He has extensive experience with big data ecosystem technologies. He has developed a special interest in data science, cognitive intelligence, and an algorithmic approach to data management and analytics. He is a regular speaker on data science and big data at various events.
Manish Kumar is a Senior Technical Architect at Datametica Solutions. He has more than 11 years of industry experience in data management as a data, solutions, and product architect. He has extensive experience in building effective ETL pipelines, implementing security over Hadoop, implementing real-time data analytics solutions, and providing innovative and best possible solutions to data science problems. He is a regular speaker on big data and data science.
About the reviewers
Albenzo Coletta is a senior software and system engineer in robotics, defense, avionics, and telecoms. He has a master's in computational robotics. He was an industrial researcher in AI, a designer for a robotic communications system for COMAU, and a business analyst. He designed a neuro-fuzzy system for financial problems (with Sannio University) and also designed a recommender system for a few key Italian editorial groups. He was also a consultant at UCID (Ministry of Economics and Finance). He developed a mobile human robotic interaction system.
Giancarlo Zaccone has more than 10 years, experience in managing research projects in scientific and industrial areas. He has worked as a researcher at the CNR, the National Research Council, in projects on parallel numerical computing, and in scientific visualization.
He is a senior software engineer at a consulting company, developing and testing software systems for space and defense applications. He holds a master's in physics from University of Naples Federico II and a 2nd-level PG master's in scientific computing from La Sapienza of Rome.
Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents
Title Page
Copyright and Credits
Artificial Intelligence for Big Data
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
About the authors
About the reviewers
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Big Data and Artificial Intelligence Systems
Results pyramid
What the human brain does best
Sensory input
Storage
Processing power
Low energy consumption
What the electronic brain does best
Speed information storage
Processing by brute force
Best of both worlds
Big Data
Evolution from dumb to intelligent machines
Intelligence
Types of intelligence
Intelligence tasks classification
Big data frameworks
Batch processing
Real-time processing
Intelligent applications with Big Data
Areas of AI
Frequently asked questions
Summary
Ontology for Big Data
Human brain and Ontology
Ontology of information science
Ontology properties
Advantages of Ontologies
Components of Ontologies
The role Ontology plays in Big Data
Ontology alignment
Goals of Ontology in big data
Challenges with Ontology in Big Data
RDF—the universal data format
RDF containers
RDF classes
RDF properties
RDF attributes
Using OWL, the Web Ontology Language
SPARQL query language
Generic structure of an SPARQL query
Additional SPARQL features
Building intelligent machines with Ontologies
Ontology learning
Ontology learning process
Frequently asked questions
Summary
Learning from Big Data
Supervised and unsupervised machine learning
The Spark programming model
The Spark MLlib library
The transformer function
The estimator algorithm
Pipeline
Regression analysis
Linear regression
Least square method
Generalized linear model
Logistic regression classification technique
Logistic regression with Spark
Polynomial regression
Stepwise regression
Forward selection
Backward elimination
Ridge regression
LASSO regression
Data clustering
The K-means algorithm
K-means implementation with Spark ML
Data dimensionality reduction
Singular value decomposition
Matrix theory and linear algebra overview
The important properties of singular value decomposition
SVD with Spark ML
The principal component analysis method
The PCA algorithm using SVD
Implementing SVD with Spark ML
Content-based recommendation systems
Frequently asked questions
Summary
Neural Network for Big Data
Fundamentals of neural networks and artificial neural networks
Perceptron and linear models
Component notations of the neural network
Mathematical representation of the simple perceptron model
Activation functions
Sigmoid function
Tanh function
ReLu
Nonlinearities model
Feed-forward neural networks
Gradient descent and backpropagation
Gradient descent pseudocode
Backpropagation model 
Overfitting
Recurrent neural networks
The need for RNNs
Structure of an RNN
Training an RNN
Frequently asked questions
Summary
Deep Big Data Analytics
Deep learning basics and the building blocks
Gradient-based learning
Backpropagation
Non-linearities
Dropout
Building data preparation pipelines
Practical approach to implementing neural net architectures
Hyperparameter tuning
Learning rate
Number of training iterations
Number of hidden units
Number of epochs
Experimenting with hyperparameters with Deeplearning4j
Distributed computing
Distributed deep learning
DL4J and Spark
API overview
TensorFlow
Keras
Frequently asked questions
Summary
Natural Language Processing
Natural language processing basics
Text preprocessing
Removing stop words
Stemming
Porter stemming
Snowball stemming
Lancaster stemming
Lovins stemming
Dawson stemming
Lemmatization
N-grams
Feature extraction
One hot encoding
TF-IDF
CountVectorizer
Word2Vec
CBOW
Skip-Gram model
Applying NLP techniques
Text classification
Introduction to Naive Bayes' algorithm
Random Forest
Naive Bayes' text classification code example
Implementing sentiment analysis
Frequently asked questions
Summary
Fuzzy Systems
Fuzzy logic fundamentals
Fuzzy sets and membership functions
Attributes and notations of crisp sets
Operations on crisp sets
Properties of crisp sets
Fuzzification
Defuzzification
Defuzzification methods
Fuzzy inference 
ANFIS network
Adaptive network
ANFIS architecture and hybrid learning algorithm
Fuzzy C-means clustering
NEFCLASS
Frequently asked questions
Summary
Genetic Programming
Genetic algorithms structure
KEEL framework
Encog machine learning framework
Encog development environment setup
Encog API structure
Introduction to the Weka framework
Weka Explorer features
Preprocess
Classify
Attribute search with genetic algorithms in Weka
Frequently asked questions
Summary
Swarm Intelligence
Swarm intelligence 
Self-organization
Stigmergy
Division of labor
Advantages of collective intelligent systems
Design principles for developing SI systems
The particle swarm optimization model
PSO implementation considerations 
Ant colony optimization model
MASON Library
MASON Layered Architecture
Opt4J library
Applications in big data analytics
Handling dynamical data
Multi-objective optimization
Frequently asked questions
Summary
Reinforcement Learning
Reinforcement learning algorithms concept
Reinforcement learning techniques
Markov decision processes
Dynamic programming and reinforcement learning
Learning in a deterministic environment with policy iteration
Q-Learning
SARSA learning
Deep reinforcement learning
Frequently asked questions
Summary
Cyber Security
Big Data for critical infrastructure protection
Data collection and analysis
Anomaly detection 
Corrective and preventive actions 
Conceptual Data Flow
Components overview
Hadoop Distributed File System
NoSQL databases
MapReduce
Apache Pig
Hive
Understanding stream processing
Stream processing semantics
Spark Streaming
Kafka
Cyber security attack types
Phishing
Lateral movement
Injection attacks
AI-based defense 
Understanding SIEM
Visualization attributes and features
Splunk
Splunk Enterprise Security
Splunk Light
ArcSight ESM
Frequently asked questions
Summary
Cognitive Computing
Cognitive science
Cognitive Systems
A brief history of Cognitive Systems
Goals of Cognitive Systems
Cognitive Systems enablers
Application in Big Data analytics
Cognitive intelligence as a service
IBM cognitive toolkit based on Watson
Watson-based cognitive apps
Developing with Watson
Setting up the prerequisites
Developing a language translator application in Java
Frequently asked questions
Summary
Other Books You May Enjoy
Leave a review - let other readers know what you think
Preface
We are at an interesting juncture in the evolution of the digital age, where there is an enormous amount of computing power and data in the hands of everyone. There has been an exponential growth in the amount of data we now have in digital form. While being associated with data-related technologies for more than 6 years, we have seen a rapid shift towards enterprises that are willing to leverage data assets initially for insights and eventually for advanced analytics. What sounded like hype initially has become a reality in a very short period of time. Most companies have realized that data is the most important asset needed to stay relevant. As practitioners in the big data analytics industry, we have seen this shift very closely by working with many clients of various sizes, across regions and functional domains. There is a common theme evolving toward open distributed open source computing to store data assets and perform advanced analytics to predict future trends and risks for businesses.
This book is an attempt to share the knowledge we have acquired over time to help new entrants in the big data space to learn from our experience. We realize that the field of artificial intelligence is vast and it is just the beginning of a revolution in the history of mankind. We are going to see AI becoming mainstream in everyone’s life and complementing human capabilities to solve some of the problems that have troubled us for a long time. This book takes a holistic approach into the theory of machine learning and AI, starting from the very basics to building applications with cognitive intelligence. We have taken a simple approach to illustrate the core concepts and theory, supplemented by illustrative diagrams and examples.
It will be encouraging for us for readers to benefit from the book and fast-track their learning and innovation into one of the most exciting fields of computing so they can create a truly intelligent system that will augment our abilities to the next level.
Who this book is for
This book is for anyone with a curious mind who is exploring the fields of machine learning, artificial intelligence, and big data analytics. This book does not assume that you have in-depth knowledge of statistics, probability, or mathematics. The concepts are illustrated with easy-to-follow examples. A basic understanding of the Java programming language and the concepts of distributed computing frameworks (Hadoop/Spark) will be an added advantage. This book will be useful for data scientists, members of technical staff in IT products and service companies, technical project managers, architects, business analysts, and anyone who deals with data assets.
What this book covers
Chapter 1, Big Data and Artificial Intelligence Systems, will set the context for the convergence of human intelligence and machine intelligence at the onset of a data revolution. We have the ability to consume and process volumes of data that were never possible before. We will understand how our quality of life is the result of our decisive power and actions and how it translates into the machine world. We will understand the paradigm of big data along with its core attributes before diving into the basics of AI. We will conceptualize the big data frameworks and see how they can be leveraged for building intelligence into machines. The chapter will end with some of the exciting applications of Big Data and AI.
Chapter 2, Ontology for Big Data, introduces semantic representation of data into knowledge assets. A semantic and standardized view of the world is essential if we want to implement artificial intelligence, which fundamentally derives knowledge from data and utilizes contextual knowledge for insights and meaningful actions in order to augment human capabilities. This semantic view of the world is expressed as ontologies.
Chapter 3, Learning from Big Data, shows broad categories of machine learning as supervised and unsupervised learning, and we understand some of the fundamental algorithms that are very widely used. In the end, we will have an overview of the Spark programming model and Spark's Machine Learning library (Spark MLlib).
Chapter 4, Neural Networks for Big Data, explores neural networks and how they have evolved with the increase in computing power with distributed computing frameworks. Neural networks get their inspiration from the human brain and help us solve some very complex problems that are not feasible with traditional mathematical models.
Chapter 5, Deep Big Data Analytics, takes our understanding of neural networks to the next level by exploring deep neural networks and the building blocks of deep learning: gradient descent and backpropagation. We will review how to build data preparation pipelines, the implementation of neural network architectures, and hyperparameter tuning. We will also explore distributed computing for deep neural networks with examples using the DL4J library.
Chapter 6, Natural Language Processing, introduces some of the fundamentals of Natural Language Processing (NLP). As we build intelligent machines, it is imperative that the interface with the machines should be as natural as possible, like day-to-day human interactions. NLP is one of the important steps towards that. We will be learning about text preprocessing, techniques for extraction of relevant features from natural language text, application of NLP techniques, and the implementation of sentiment analysis with NLP.
Chapter 7, Fuzzy Systems, explains that a level of fuzziness is essential if we want to build intelligent machines. In the real-world scenarios, we cannot depend on exact mathematical and quantitative inputs for our systems to work with, although our models (deep neural networks, for example) require actual inputs. The uncertainties are more frequent and, due to the nature of real-world scenarios, are amplified by incompleteness of contextual information, characteristic randomness, and ignorance of data. Human reasoning are capable enough to deal with these attributes of the real world. A similar level of fuzziness is essential for building intelligent machines that can complement human capabilities in a real sense. In this chapter, we are going to understand the fundamentals of fuzzy logic, its mathematical representation, and some practical implementations of fuzzy systems.
Chapter 8, Genetic Programming, big data mining tools need to be empowered by computationally efficient techniques to increase the degree of efficiency. Genetic algorithms over data mining create great, robust, computationally efficient, and adaptive systems. In fact, with the exponential explosion of data, data analytics techniques go on to take more time and inversely affect the throughput. Also due to their static nature, complex hidden patterns are often left out. In this chapter, we want to show how to use genes to mine data with great efficiency. To achieve this objective, we’ll introduce the basics of genetic programming and the fundamental algorithms.
Chapter 9, Swarm Intelligence, analyzes the potential of swarm intelligence for solving big data analytics problems. Based on the combination of swarm intelligence and data mining techniques, we can have a better understanding of the big data analytics problems and design more effective algorithms to solve real-world big data analytics problems. In this chapter, we’ll show how to use these algorithms in big data applications. The basic theory and some programming frameworks will be also explained.
Chapter 10,Reinforcement Learning, covers reinforcement learning as one of the categories of machine learning. With reinforcement learning, the intelligent agent learns the right behavior based on the reward it receives as per the actions it takes within a specific environmental context. We will understand the fundamentals of reinforcement learning, along with mathematical theory and some of the commonly used techniques for reinforcement learning.
Chapter 11,Cyber Security, analyzes the cybersecurity problem for critical infrastructure. Data centers, data base factories, and information system factories are continuously under attack. Online analysis can detect potential attacks to ensure infrastructure security. This chapter also explains Security Information and Event Management (SIEM). It emphasizes the importance of managing log files and explains how they can bring benefits. Subsequently, Splunk and ArcSight ESM systems are introduced.
Chapter 12, Cognitive Computing, introduces cognitive computing as the next level in the development of artificial intelligence. By leveraging the five primary human senses along with mind as the sixth sense, a new era of cognitive systems can begin. We will see the stages of AI and the natural progression towards strong AI, along with the key enablers for achieving strong AI. We will take a look at the history of cognitive systems and see how that growth is accelerated with the availability of big data, which brings large data volumes and processing power in a distributed computing framework.
To get the most out of this book
The chapters in this book are sequenced in such a way that the reader can progressively learn about Artificial Intelligence for Big Data starting from the fundamentals and eventually move towards cognitive intelligence. Chapter 1, Big Data and Artificial Intelligence Systems, to Chapter 5, Deep Big Data Analytics, cover the basic theory of machine learning and establish the foundation for practical approaches to AI. Starting from Chapter 6, Natural Language Processing, we conceptualize theory into practical implementations and possible use cases. To get the most out of this book, it is recommended that the first five chapters are read in order. From Chapter 6, Natural Language Processing, onward, the reader can choose any topic of interest and read in whatever sequence they prefer.
Download the example code files
You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packtpub.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub athttps://github.com/PacktPublishing/Artificial-Intelligence-for-Big-Data. We also have other code bundles from our rich catalog of books and videos available athttps://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/ArtificialIntelligenceforBigData_ColorImages.pdf.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system."
A block of code is set as follows:
StopWordsRemover
remover
new
StopWordsRemover
()
.
setInputCol
(
"raw"
)
.
setOutputCol
(
"filtered"
);
Any command-line input or output is written as follows:
$ mkdir css
$ cd css
Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel."
Get in touch
Feedback from our readers is always welcome.
General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Reviews
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.
Big Data and Artificial Intelligence Systems
The human brain is one of the most sophisticated machines in the universe. It has evolved for thousands of years to its current state. As a result of continuous evolution, we are able to make sense of nature's inherent processes and understand cause and effect relationships. Based on this understanding, we are able to learn from nature and devise similar machines and mechanisms to constantly evolve and improve our lives. For example, the video cameras we use derived from the understanding of the human eye.
Fundamentally, human intelligence works on the paradigm of sense, store, process, and act. Through the sensory organs, we gather information about our surroundings, store the information (memory), process the information to form our beliefs/patterns/links, and use the information to act based on the situational context and stimulus.
Currently, we are at a very interesting juncture of evolution where the human race has found a way to store information in an electronic format. We are also trying to devise machines that imitate the human brain to be able to sense, store, and process information to make meaningful decisions and complement human abilities.
This introductory chapter will set the context for the convergence of human intelligence and machine intelligence at the onset of a data revolution. We have the ability to consume and process volumes of data that were never possible before. We will understand how our quality of life is the result of our decisive power and actions and how it translates to the machine world. We will understand the paradigm of Big Data along with its core attributes before diving into artificial intelligence (AI) and its basic fundamentals. We will conceptualize the Big Data frameworks and how those can be leveraged for building intelligence into machines. The chapter will end with some of the exciting applications of Big Data and AI.
We will cover the following topics in the chapter:
Results pyramid
Comparing the human and the electronic brain
Overview of Big Data
Results pyramid
The quality of human life is a factor of all the decisions we make. According to Partners in Leadership, the results we get (positive, negative, good, or bad) are a result of our actions, our actions are a result of the beliefs we hold, and the beliefs we hold are a result of our experiences. This is represented as a results pyramid as follows:
At the core of the results pyramid theory is the fact that it is certain that we cannot achieve better or different results with the same actions. Take an example of an organization that is unable to meets its goals and has diverted from its vision for a few quarters. This is a result of certain actions that the management and employees are taking. If the team continues to have same beliefs, which translate to similar actions, the company cannot see noticeable changes in its outcomes. In order to achieve the set goals, there needs to be a fundamental change in day-to-day actions for the team, which is only possible with a new set of beliefs. This means a cultural overhaul for the organization.
Similarly, at the core of computing evolution, man-made machines cannot evolve to be more effective and useful with the same outcomes (actions), models (beliefs), and data (experiences) that we have access to traditionally. We can evolve for the better if human intelligence and machine power start complementing each other.
What the human brain does best
While the machines are catching up fast in the quest for intelligence, nothing can come close to some of the capabilities that the human brain has.
Sensory input
The human brain has an incredible capability to gather sensory input using all the senses in parallel. We can see, hear, touch, taste, and smell at the same time, and process the input in real time. In terms of computer terminology, these are various data sources that stream information, and the brain has the capacity to process the data and convert it into information and knowledge. There is a level of sophistication and intelligence within the human brain to generate different responses to this input based on the situational context.
For example, if the outside temperature is very high and it is sensed by the skin, the brain generates triggers within the lymphatic system to generate sweat and bring the body temperature under control. Many of these responses are triggered in real time and without the need for conscious action.
Storage
The information collected from the sensory organs is stored consciously and subconsciously. The brain is very efficient at filtering out the information that is non-critical for survival. Although there is no confirmed value of the storage capacity in the human brain, it is believed that the storage capacity is similar to terabytes in computers. The brain's information retrieval mechanism is also highly sophisticated and efficient. The brain can retrieve relevant and related information based on context. It is understood that the brain stores information in the form of linked lists, where the objects are linked to each other by a relationship, which is one of the reasons for the availability of data as information and knowledge, to be used as and when required.
Processing power
The human brain can read sensory input, use previously stored information, and make decisions within a fraction of a millisecond. This is possible due to a network of neurons and their interconnections. The human brain possesses about 100 billion neurons with one quadrillion connections known as synapses wiring these cells together. It coordinates hundreds of thousands of the body's internal and external processes in response to contextual information.
Low energy consumption
The human brain requires far less energy for sensing, storing, and processing information. The power requirement in calories (or watts) is insignificant compared to the equivalent power requirements for electronic machines. With growing amounts of data, along with the increasing requirement of processing power for artificial machines, we need to consider modeling energy utilization on the human brain. The computational model needs to fundamentally change towards quantum computing and eventually to bio-computing.
What the electronic brain does best
As the processing power increases with computers, the electronic brain—or computers—are much better when compared to the human brain in some aspects, as we will explore in the following sections.
Speed information storage
The electronic brain (computers) can read and store high volumes of information at enormous speeds. Storage capacity is exponentially increasing. The information is easily replicated and transmitted from one place to another. The more information we have at our disposal for analysis, pattern, and model formation, the more accurate our predictions will be, and the machines will be much more intelligent. Information storage speed is consistent across machines when all factors are constant. However, in the case of the human brain, storage and processing capacities vary based on individuals.
Processing by brute force
The electronic brain can process information using brute force. A distributed computing system can scan/sort/calculate and run various types of compute on very large volumes of data within milliseconds. The human brain cannot match the brute force of computers.
Computers are very easy to network and collaborate with in order to increase collective storage and processing power. The collective storage can collaborate in real time to produce intended outcomes. While human brains can collaborate, they cannot match the electronic brain in this aspect.
Best of both worlds
AI is finding and taking advantage of the best of both worlds in order to augment human capabilities. The sophistication and efficiency of the human brain and the brute force of computers combined together can result in intelligent machines that can solve some of the most challenging problems faced by human beings. At that point, the AI will complement human capabilities and will be a step closer to social inclusion and equanimity by facilitating collective intelligence. Examples include epidemic predictions, disease prevention based on DNA sampling and analysis, self driving cars, robots that work in hazardous conditions, and machine assistants for differently able people.
Taking a statistical and algorithmic approach to data in machine learning and AI has been popular for quite some time now. However, the capabilities and use cases were limited until the availability of large volumes of data along with massive processing speeds, which is called Big Data. We will understand some of the Big Data basics in the next section. The availability of Big Data has accelerated the growth and evolution of AI and machine learning applications. Here is a quick comparison of AI before and with with Big Data:
The primary goal of AI is to implement human-like intelligence in machines and to create systems that gather data, process it to create models (hypothesis), predict or influence outcomes, and ultimately improve human life. With Big Data at the core of the pyramid, we have the availability of massive datasets from heterogeneous sources in real time. This promises to be a great foundation for an AI that really augments human existence:
Big Data
- Peter Norvig, Research Director, Google
Data in dictionary terms is defined as facts and statistics collected together for reference or analysis. Storage mechanisms have greatly evolved with human evolution—sculptures, handwritten texts on leaves, punch cards, magnetic tapes, hard drives, floppy disks, CDs, DVDs, SSDs, human DNA, and more. With each new medium, we are able to store more and more data in less space; it's a transition in the right direction. With the advent of the internet and the Internet of Things (IoT), data volumes have been growing exponentially.
The term Big Data was coined to represent growing volumes of data. Along with volume, the term also incorporates three more attributes, velocity, variety, and value, as follows:
Volume
: This represents the ever increasing and exponentially growing amount of data. We are now collecting data through more and more interfaces between man-made and natural objects. For example, a patient's routine visit to a clinic now generates electronic data in the tune of megabytes. An average smartphone user generates a data footprint of at least a few GB per day. A flight traveling from one point to another generates half a terabyte of data.
Velocity
: This represents the amount of data generated with respect to time and a need to analyze that data in near-real time for some mission critical operations. There are sensors that collect data from natural phenomenon, and the data is then processed to predict hurricanes/earthquakes. Healthcare is a great example of the velocity of the data generation; analysis and action is mission critical:
Variety
: This represents variety in data formats. Historically, most electronic datasets were structured and fit into database tables (columns and rows). However, more than 80% of the electronic data we now generate is not in structured format, for example, images, video files, and voice data files. With Big Data, we are in a position to analyze the vast majority of structured/unstructured and semi-structured datasets.
Value
: This is the most important aspect of Big Data. The data is only as valuable as its utilization in the generation of actionable insight. Remember the results pyramid where actions lead to results. There is no disagreement that data holds the key to actionable insight; however, systems need to evolve quickly to be able to analyze the data, understand the patterns within the data, and, based on the contextual details, provide solutions that ultimately create value.
Evolution from dumb to intelligent machines
The machines and mechanisms that store and process these huge amounts of data have evolved greatly over a period of time. Let us briefly look at the evolution of machines (for simplicity's sake, computers). For a major portion of their evolution, computers were dumb machines instead of intelligent machines. The basic building blocks of a computer are the CPU (Central Processing Unit), the RAM (temporary memory), and the disk (persistent storage). One of the core components of a CPU is an ALU (Arithmetic and Logic Unit). This is the component that is capable of performing the basic steps of mathematical calculations along with logical operations. With these basic capabilities in place, traditional computers evolved with greater and higher processing power. However, they were still dumb machines without any inherent intelligence. These computers were extremely good at following predefined instructions by using brute force and throwing errors or exceptions for scenarios that were not predefined. These computer programs could only answer specific questions they were meant to solve.
Although these machines could process lots of data and perform computationally heavy jobs, they would be always limited to what they were programmed to do. This is extremely limiting if we take the example of a self driving car. With a computer program working on predefined instructions, it would be nearly impossible to program the car to handle all situations, and the programming would take forever if we wanted to drive the car on ALL roads and in all situations.
This limitation of traditional computers to respond to unknown or non-programmed situations leads to the question: Can a machine be developed to think and evolve as humans do? Remember, when we learn to drive a car, we just drive it in a small amount of situations and on certain roads. Our brain is very quick to learn to react to new situations and trigger various actions (apply breaks, turn, accelerate, and so on). This curiosity resulted in the evolution of traditional computers into artificially intelligent machines.
In the year 1956, the term artificial intelligence was coined. Although there were gradual steps and milestones on the way, the last decade of the 20th century marked remarkable advancements in AI techniques. In 1990, there were significant demonstrations of machine learning algorithms supported by case-based reasoning and natural language understanding and translations. Machine intelligence reached a major milestone when then World Chess Champion, Gary Kasparov, was beaten by Deep Blue in 1997. Ever since that remarkable feat, AI systems have greatly evolved to the extent that some experts have predicted that AI will beat humans at everything eventually. In this book, we are going to look at the specifics of building intelligent systems and also understand the core techniques and available technologies. Together, we are going to be part of one of the greatest revolutions in human history.
Intelligence
Fundamentally, intelligence in general, and human intelligence in particular, is a constantly evolving phenomenon. It evolves through four Ps when applied to sensory input or data assets: Perceive, Process, Persist, and Perform. In order to develop artificial intelligence, we need to also model our machines with the same cyclical approach:
Types of intelligence
Here are some of the broad categories of human intelligence:
Linguistic intelligence
: Ability to associate words to objects and use language (vocabulary and grammar) to express meaning
Logical intelligence
: Ability to calculate, quantify, and perform mathematical operations and use basic and complex logic for inference
Interpersonal and emotional intelligence
: Ability to interact with other human beings and understand feelings and emotions
Intelligence tasks classification
This is how we classify intelligence tasks:
Basic tasks:
Perception
Common sense
Reasoning
Natural language processing
Intermediate tasks:
Mathematics
Games
Expert tasks:
Financial analysis
Engineering
Scientific analysis
Medical analysis
The fundamental difference between human intelligence and machine intelligence is the handling of basic and expert tasks. For human intelligence, basic tasks are easy to master and they are hardwired at birth. However, for machine intelligence, perception, reasoning, and natural language processing are some of the most computationally challenging and complex tasks.
Big data frameworks
In order to derive value from data that is high in volume, varies in its form and structure, and is generated with ever increasing velocity, there are two primary categories of framework that have emerged over a period of time. These are based on the consideration of the differential time at which the event occurs (data origin) and the time at which the data is available for analysis and action.
Batch processing
Traditionally, the data processing pipeline within data warehousing systems consisted of Extracting, Transforming, and Loading the data for analysis and actions (ETL). With the new paradigm of file-based distributed computing, there has been a shift in the ETL process sequence. Now the data is Extracted, Loaded, and Transformed repetitively for analysis (ELTTT) a number of times:
In batch processing, the data is collected from various sources in the staging areas and loaded and transformed with defined frequencies and schedules. In most use cases with batch processing, there is no critical need to process the data in real time or in near real time. As an example, the monthly report on a student's attendance data will be generated by a process (batch) at the end of a calendar month. This process will extract the data from source systems, load it, and transform it for various views and reports. One of the most popular batch processing frameworks is Apache Hadoop. It is a highly scalable, distributed/parallel processing framework. The primary building block of Hadoop is the Hadoop Distributed File System.
As the name suggests, this is a wrapper filesystem which stores the data (structured/unstructured/semi-structured) in a distributed manner on data nodes within Hadoop. The processing that is applied on the data (instead of the data that is processed) is sent to the data on various nodes. Once the compute is performed by an individual node, the results are consolidated by the master process. In this paradigm of data-compute localization, Hadoop relies heavily on intermediate I/O operations on hard drive disks. As a result, extremely large volumes of data can be processed by Hadoop in a reliable manner at the cost of processing time. This framework is very suitable for extracting value from Big Data in batch mode.
Real-time processing
While batch processing frameworks are good for most data warehousing use cases, there is a critical need for processing the data and generating actionable insight as soon as the data is available. For example, in a credit card fraud detection system, the alert should be generated as soon as the first instance of logged malicious activity. There is no value if the actionable insight (denying the transaction) is available as a result of the end-of-month batch process. The idea of a real-time processing framework is to reduce latency between event time and processing time. In an ideal system, the expectation would be zero differential between the event time and the processing time. However, the time difference is a function of the data source input, execution engine, network bandwidth, and hardware. Real-time processing frameworks achieve low latency with minimal I/O by relying on in-memory computing in a distributed manner. Some of the most popular real-time processing frameworks are:
Apache Spark
: This is a distributed execution engine that relies on in-memory processing based on fault tolerant data abstractions named
RDDs
(
Resilient Distributed Datasets
).
Apache Storm
: This is a framework for distributed real-time computation. Storm applications are designed to easily process unbounded streams, which generate event data at a very high velocity.
Apache Flink
: This is a framework for efficient, distributed, high volume data processing. The key feature of Flink is automatic program optimization. Flink provides native support for massively iterative, compute intensive algorithms.
As the ecosystem is evolving, there are many more frameworks available for batch and real-time processing. Going back to the machine intelligence evolution cycle (Perceive, Process, Persist, Perform), we are going to leverage these frameworks to create programs that work on Big Data, take an algorithmic approach to filter relevant data, generate models based on the patterns within the data, and derive actionable insight and predictions that ultimately lead to value from the data assets.
Intelligent applications with Big Data
At this juncture of technological evolution, where we have the availability of systems that gather large volumes of data from heterogeneous sources, along with systems that store these large volumes of data at ever reducing costs, we can derive value in the form of insight into the data and build intelligent machines that can trigger actions resulting in the betterment of human life. We need to use an algorithmic approach with the massive data and compute assets we have at our disposal. Leveraging a combination of human intelligence, large volumes of data, and distributed computing power, we can create expert systems which can be used as an advantage to lead the human race to a better future.
Areas of AI
While we are in the infancy of developments in AI, here are some of the basic areas in which significant research and breakthroughs are happening:
Natural language processing
: Facilitates interactions between computers and human languages.
Fuzzy logic systems
: These are based on the degrees of truth instead of programming for all situations with IF/ELSE logic. These systems can control machines and consumer products based on acceptable reasoning.
Intelligent robotics
: These are mechanical devices that can perform mundane or hazardous repetitive tasks.
Expert systems
: These are systems or applications that solve complex problems in a specific domain. They are capable of advising, diagnosing, and predicting results based on the knowledge base and models.
Frequently asked questions
Here is a small recap of what we covered in the chapter:
Q: What is a results pyramid?
A: The results we get (man or machine) are an outcome of our experiences (data), beliefs (models), and actions. If we need to change the results, we need different (better) sets of data, models, and actions.
Q: How is this paradigm applicable to AI and Big Data?
A: In order to improve our lives, we need intelligent systems. With the advent of Big Data, there has been a boost to the theory of machine learning and AI due to the availability of huge volumes of data and increasing processing power. We are on the verge of getting better results for humanity as a result of the convergence of machine intelligence and Big Data.
Q: What are the basic categories of Big Data frameworks?
A: Based on the differentials between the event time and processing time, there are two types of framework: batch processing and real-time processing.
Q: What is the goal of AI?
A: The fundamental goal of AI is to augment and complement human life.
Q: What is the difference between machine learning and AI?
A: Machine learning is a core concept which is integral to AI. In machine learning, the conceptual models are trained based on data and the models can predict outcomes for the new datasets. AI systems try to emulate human cognitive abilities and are context sensitive. Depending on the context, AI systems can change their behaviors and outcomes to best suit the decisions and actions the human brain would take.
Have a look at the following diagram for a better understanding:
Summary
In this chapter, we understood the concept of the results pyramid, which is a model for the continuous improvement of human life and striving to get better results with an improved understanding of the world based on data (experiences), which shape our models (beliefs). With the convergence of the evolving human brain and computers, we know that the best of both worlds can really improve our lives. We have seen how computers have evolved from dumb to intelligent machines and we provided a high-level overview of intelligence and Big Data, along with types of processing frameworks.
With this introduction and context, in subsequent chapters in this book, we are going to take a deep dive into the core concepts of taking an algorithmic approach to data and the basics of machine learning with illustrative algorithms. We will implement these algorithms with available frameworks and illustrate this with code samples.
Ontology for Big Data
In the introductory chapter, we learned that big data has fueled rapid advances in the field of artificial intelligence. This is primarily because of the availability of extremely large datasets from heterogeneous sources and exponential growth in processing power due to distributed computing. It is extremely difficult to derive value from large data volumes if there is no standardization or a common language for interpreting data into information and converting information into knowledge. For example, two people who speak two different languages, and do not understand each other's languages, cannot get into a verbal conversation unless there is some translation mechanism in between. Translations and interpretations are possible only when there is a semantic meaning associated with a keyword and when grammatical rules are applied as conjunctions. As an example, here is a sentence in the English and Spanish languages:
Broadly, we can break a sentence down in the form of objects, subjects, verbs, and attributes. In this case, John and bananas are subjects. They are connected by an activity, in this case eating, and there are also attributes and contextual data—information in conjunction with the subjects and activities. Knowledge translators can be implemented in two ways:
All-inclusive mappi
ng
: Maintaining a mapping between
all
sentences in one language and translations in the other language. As you can imagine, this is impossible to achieve since there are countless ways something (object, event, attributes, context) can be expressed in a language.
Semantic view of the world
: If we associate semantic meaning with every entity that we encounter in linguistic expression, a standardized semantic view of the world can act as a centralized dictionary for all the languages.
A semantic and standardized view of the world is essential if we want to implement artificial intelligence which fundamentally derives knowledge from data and utilizes the contextual knowledge for insight and meaningful actions in order to augment human capabilities. This semantic view of the world is expressed as Ontologies. In the context of this book, Ontology is defined as: a set of concepts and categories in a subject area or domain, showing their properties and the relationships between them.
In this chapter, we are going to look at the following:
How the human brain links objects in its interpretation of the world
The role Ontology plays in the world of Big Data
Goals and challenges with Ontology in Big Data
The Resource Description Framework
The Web Ontology Language
SPARQL, the semantic query language for the RDF
Building Ontologies and using Ontologies to build intelligent machines
Ontology learning
