45,59 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Unleash the data processing and analytics capability of Apache Spark with the language of choice: Java
About This Book
- Perform big data processing with Spark—without having to learn Scala!
- Use the Spark Java API to implement efficient enterprise-grade applications for data processing and analytics
- Go beyond mainstream data processing by adding querying capability, Machine Learning, and graph processing using Spark
Who This Book Is For
If you are a Java developer interested in learning to use the popular Apache Spark framework, this book is the resource you need to get started. Apache Spark developers who are looking to build enterprise-grade applications in Java will also find this book very useful.
What You Will Learn
- Process data using different file formats such as XML, JSON, CSV, and plain and delimited text, using the Spark core Library.
- Perform analytics on data from various data sources such as Kafka, and Flume using Spark Streaming Library
- Learn SQL schema creation and the analysis of structured data using various SQL functions including Windowing functions in the Spark SQL Library
- Explore Spark Mlib APIs while implementing Machine Learning techniques to solve real-world problems
- Get to know Spark GraphX so you understand various graph-based analytics that can be performed with Spark
In Detail
Apache Spark is the buzzword in the big data industry right now, especially with the increasing need for real-time streaming and data processing. While Spark is built on Scala, the Spark Java API exposes all the Spark features available in the Scala version for Java developers. This book will show you how you can implement various functionalities of the Apache Spark framework in Java, without stepping out of your comfort zone.
The book starts with an introduction to the Apache Spark 2.x ecosystem, followed by explaining how to install and configure Spark, and refreshes the Java concepts that will be useful to you when consuming Apache Spark's APIs. You will explore RDD and its associated common Action and Transformation Java APIs, set up a production-like clustered environment, and work with Spark SQL. Moving on, you will perform near-real-time processing with Spark streaming, Machine Learning analytics with Spark MLlib, and graph processing with GraphX, all using various Java packages.
By the end of the book, you will have a solid foundation in implementing components in the Spark framework in Java to build fast, real-time applications.
Style and approach
This practical guide teaches readers the fundamentals of the Apache Spark framework and how to implement components using the Java language. It is a unique blend of theory and practical examples, and is written in a way that will gradually build your knowledge of Apache Spark.
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 346
Veröffentlichungsjahr: 2017
Ähnliche
BIRMINGHAM - MUMBAI
< html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
Apache Spark 2.x for Java Developers
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: July 2017
Production reference: 1250717
ISBN 978-1-78712-649-7
www.packtpub.com
Credits
Authors
Sourav Gulati
Sumit Kumar
Copy Editor
Safis Editing
Reviewer
Prashant Verma
Project Coordinator
Nidhi Joshi
Commissioning Editor
Amey Varangaonkar
Proofreader
Safis Editing
Acquisition Editor
Shweta Pant
Indexer
Tejal Daruwale Soni
Content Development Editor
Mayur Pawanikar
Graphics
Tania Dutta
Technical Editor
Karan Thakkar
Production Coordinator
Arvindkumar Gupta
Foreword
Sumit Kumar and Sourav Gulati are technology evangelists with deep experience in envisioning and implementing solutions, as well as complex problems dealing with large and high-velocity data. Every time I talk to them about any complex problem statement, they have provided an innovative and scalable solution.
I have over 17 years of experience in the IT industry, specializing in envisioning, architecting and implementing various enterprise solutions revolving around a variety of business domains, such as hospitality, healthcare, risk management, and insurance.
I have known Sumit and Sourav for 5 years as developers/architects who have worked closely with me implementing various complex big data solutions. From their college days, they were inclined toward exploring/implementing distributed systems. As if implementing solutions around big data systems were not enough, they also started sharing their knowledge and experience with the big data community. They have actively contributed to various blogs and tech talks, and in no circumstances do they pass up on any opportunity to help their fellow technologists.
Knowing Sumit and Sourav, I am not surprised that they have started authoring a book on Spark and I am writing foreword for their book - Apache Spark 2.x for Java Developers.
Their passion for technology has again resulted in the terrific book you now have in your hands.
This book is the product of Sumit's and Sourav's deep knowledge and extensive implementation experience in Spark for solving real problems that deal with large, fast and diverse data.
Several books on distributed systems exist, but Sumit's and Sourav's book closes a substantial gap between theory and practice. Their book offers comprehensive, detailed, and innovative techniques for leveraging Spark and its extensions/API for implementing big data solutions. This book is a precious resource for practitioners envisioning big data solutions for enterprises, as well as for undergraduate and graduate students keen to master the Spark and its extensions using its Java API.
This book starts with an introduction to Spark and then covers the overall architecture and concepts such as RDD, transformation, and partitioning. It also discuss in detail various Spark extensions, such as Spark Streaming, MLlib, Spark SQL, and GraphX.
Each chapter is dedicated to a topic and includes an illustrative case study that covers state-of-the-art Java-based tools and software. Each chapter is self-contained, providing great flexibility of usage. The accompanying website provides the source code and data. This is truly a gem for both students and big data architects/developers, who can experiment first-hand the methods just learned, or can deepen their understanding of the methods by applying them to real-world scenarios.
As I was reading the various chapters of the book, I was reminded of the passion and enthusiasm of Sumit and Sourav have for distributed frameworks. They have communicated the concepts described in the book with clarity and with the same passion. I am positive that you, as reader, will feel the same. I will certainly keep this book as a personal resource for the solutions I implement, and strongly recommend it to my fellow architects.
Sumit Gupta
Director of Engineering, Big Data, Sapient Global Markets
About the Authors
Sourav Gulati is associated with software industry for more than 7 years. He started his career with Unix/Linux and Java and then moved towards big data and NoSQL World. He has worked on various big data projects. He has recently started a technical blog called Technical Learning as well. Apart from IT world, he loves to read about mythology.
Sumit Kumar is a developer with industry insights in telecom and banking. At different junctures, he has worked as a Java and SQL developer, but it is shell scripting that he finds both challenging and satisfying at the same time. Currently, he delivers big data projects focused on batch/near-real-time analytics and the distributed indexed querying system. Besides IT, he takes a keen interest in human and ecological issues.
About the Reviewer
Prashant Verma started his IT carrier in 2011 as a Java developer in Ericsson working in telecom domain. After couple of years of JAVA EE experience, he moved into Big Data domain, and has worked on almost all the popular big data technologies, such as Hadoop, Spark, Flume, Mongo, Cassandra,etc. He has also played with Scala. Currently, He works with QA Infotech as Lead Data Enginner, working on solving e-Learning problems using analytics and machine learning.
Prashant has also worked onApache Spark 2.x for Java Developers, Packt as a Technical Reviewer.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787126498.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
Introduction to Spark
Dimensions of big data
What makes Hadoop so revolutionary?
Defining HDFS
NameNode
HDFS I/O
YARN
Processing the flow of application submission in YARN
Overview of MapReduce
Why Apache Spark?
RDD - the first citizen of Spark
Operations on RDD
Lazy evaluation
Benefits of RDD
Exploring the Spark ecosystem
What's new in Spark 2.X?
References
Summary
Revisiting Java
Why use Java for Spark?
Generics
Creating your own generic type
Interfaces
Static method in an interface
Default method in interface
What if a class implements two interfaces which have default methods with same name and signature?
Anonymous inner classes
Lambda expressions
Functional interface
Syntax of Lambda expressions
Lexical scoping
Method reference
Understanding closures
Streams
Generating streams
Intermediate operations
Working with intermediate operations
Terminal operations
Working with terminal operations
String collectors
Collection collectors
Map collectors
Groupings
Partitioning
Matching
Finding elements
Summary
Let Us Spark
Getting started with Spark
Spark REPL also known as CLI
Some basic exercises using Spark shell
Checking Spark version
Creating and filtering RDD
Word count on RDD
Finding the sum of all even numbers in an RDD of integers
Counting the number of words in a file
Spark components
Spark Driver Web UI
Jobs
Stages
Storage
Environment
Executors
SQL
Streaming
Spark job configuration and submission
Spark REST APIs
Summary
Understanding the Spark Programming Model
Hello Spark
Prerequisites
Common RDD transformations
Map
Filter
flatMap
mapToPair
flatMapToPair
union
Intersection
Distinct
Cartesian
groupByKey
reduceByKey
sortByKey
Join
CoGroup
Common RDD actions
isEmpty
collect
collectAsMap
count
countByKey
countByValue
Max
Min
First
Take
takeOrdered
takeSample
top
reduce
Fold
aggregate
forEach
saveAsTextFile
saveAsObjectFile
RDD persistence and cache
Summary
Working with Data and Storage
Interaction with external storage systems
Interaction with local filesystem
Interaction with Amazon S3
Interaction with HDFS
Interaction with Cassandra
Working with different data formats
Plain and specially formatted text
Working with CSV data
Working with JSON data
Working with XML Data
References
Summary
Spark on Cluster
Spark application in distributed-mode
Driver program
Executor program
Cluster managers
Spark standalone
Installation of Spark standalone cluster
Start master
Start slave
Stop master and slaves
Deploying applications on Spark standalone cluster
Client mode
Cluster mode
Useful job configurations
Useful cluster level configurations (Spark standalone)
Yet Another Resource Negotiator (YARN)
YARN client
YARN cluster
Useful job configuration
Summary
Spark Programming Model - Advanced
RDD partitioning
Repartitioning
How Spark calculates the partition count for transformations with shuffling (wide transformations )
Partitioner
Hash Partitioner
Range Partitioner
Custom Partitioner
Advanced transformations
mapPartitions
mapPartitionsWithIndex
mapPartitionsToPair
mapValues
flatMapValues
repartitionAndSortWithinPartitions
coalesce
foldByKey
aggregateByKey
combineByKey
Advanced actions
Approximate actions
Asynchronous actions
Miscellaneous actions
Shared variable
Broadcast variable
Properties of the broadcast variable
Lifecycle of a broadcast variable
Map-side join using broadcast variable
Accumulators
Driver program
Summary
Working with Spark SQL
SQLContext and HiveContext
Initializing SparkSession
Reading CSV using SparkSession
Dataframe and dataset
SchemaRDD
Dataframe
Dataset
Creating a dataset using encoders
Creating a dataset using StructType
Unified dataframe and dataset API
Data persistence
Spark SQL operations
Untyped dataset operation
Temporary view
Global temporary view
Spark UDF
Spark UDAF
Untyped UDAF
Type-safe UDAF:
Hive integration
Table Persistence
Summary
Near Real-Time Processing with Spark Streaming
Introducing Spark Streaming
Understanding micro batching
Getting started with Spark Streaming jobs
Streaming sources
fileStream
Kafka
Streaming transformations
Stateless transformation
Stateful transformation
Checkpointing
Windowing
Transform operation
Fault tolerance and reliability
Data receiver stage
File streams
Advanced streaming sources
Transformation stage
Output stage
Structured Streaming
Recap of the use case
Structured streaming - programming model
Built-in input sources and sinks
Input sources
Built-in Sinks
Summary
Machine Learning Analytics with Spark MLlib
Introduction to machine learning
Concepts of machine learning
Datatypes
Machine learning work flow
Pipelines
Operations on feature vectors
Feature extractors
Feature transformers
Feature selectors
Summary
Learning Spark GraphX
Introduction to GraphX
Introduction to Property Graph
Getting started with the GraphX API
Using vertex and edge RDDs
From edges
EdgeTriplet
Graph operations
mapVertices
mapEdges
mapTriplets
reverse
subgraph
aggregateMessages
outerJoinVertices
Graph algorithms
PageRank
Static PageRank
Dynamic PageRank
Triangle counting
Connected components
Summary
Preface
Apache Spark is the buzzword in the big data industry right now, especially with the increasing need for real-time streaming and data processing. While Spark is built on Scala, the Spark Java API exposes all the features of Spark available in the Scala version for Java developers. This book will show you how you can implement various functionalities of the Apache Spark framework in Java without stepping out of your comfort zone.
The book starts with an introduction to the Apache Spark 2.x ecosystem, followed by explaining how to install and configure Spark, and refreshes the concepts of Java that will be useful to you when consuming Apache Spark's APIs. You will explore RDD and its associated common Action and Transformation Java APIs, set up a production-like clustered environment, and work with Spark SQL. Moving on, you will perform near-real-time processing with Spark Streaming, machine learning analytics with Spark MLlib, and graph processing with GraphX, all using various Java packages.
By the end of the book, you will have a solid foundation in implementing the components of the Spark framework in Java to build fast, real-time applications.
What this book covers
Chapter 1, Introduction to Spark, covers the history of big data, its dimensions, and basic concepts of Hadoop and Spark.
Chapter 2, Revisiting Java, refreshes the concepts of core Java and will focus on the newer feature of Java 8 that will be leveraged while developing Spark applications.
Chapter 3, Let Us Spark, serves the purpose of providing an instruction set so that the reader becomes familiar with installing Apache Spark in standalone mode along with its dependencies.
Chapter 4, Understanding the Spark Programming Model, makes progress by explaining the word count problem in Apache Spark using Java and simultaneously setting up an IDE.
Chapter 5, Working with Data and Storage, teaches you how to read/store data in Spark from/to different storage systems.
Chapter 6, Spark on Cluster, discusses the cluster setup process and some popular cluster managers available with Spark in detail. After this chapter, you will be able to execute Spark jobs effectively in distributed mode.
Chapter 7, Spark Programming Model – Advanced, covers partitioning concepts in RDD along with advanced transformations and actions in Spark.
Chapter 8, Working with Spark SQL, discusses Spark SQL and its related concepts such as dataframe, dataset, and UDF. We will also discuss SqlContext and the newly introduced SparkSession.
Chapter 9, Near-Real-Time Processing with Spark Streaming, covers the internals of Spark Streaming, reading streams of data in Spark from various data sources with examples, and newer extensions of stream processing in Spark known as structured streaming.
Chapter 10, Machine Learning Analytics with Spark MLlib, focuses on introducing the concepts of machine learning and then moves on towards its implementation using Apache Spark Mllib libraries. We also discuss some real-world problems using Spark Mllib.
Chapter 11, Learning Spark GraphX, looks into another module of Spark, GraphX; we will discover types of GraphX RDD and various operations associated with them. We will also discuss the use cases of GraphX implementation.
What you need for this book
If you want to set up Spark on your local machine, then you can follow the instructions mentioned in Chapter 3, Let Us Spark.
Who this book is for
If you are a Java developer interested in learning to use the popular Apache Spark framework, this book is the resource you need to get started. Apache Spark developers who are looking to build enterprise-grade applications in Java will also find this book very useful.
Reader feedback
Feedback from our readers is always welcome. Let us know what you thought about this book-what you liked or disliked. Reader feedback is important for us as it helps us to develop titles that you will really get the most out of. To send us general feedback, simply email [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Downloading the example code
You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps:
Log in or register to our website using your e-mail address and password.
Hover the mouse pointer on the
SUPPORT
tab at the top.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box.
Select the book for which you're looking to download the code files.
Choose from the drop-down menu where you purchased this book from.
Click on
Code Download
.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR / 7-Zip for Windows
Zipeg / iZip / UnRarX for Mac
7-Zip / PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Apache-Spark-2x-for-Java-Developers. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Errata
Although we have taken care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us to improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately. Please contact us at [email protected] with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.
Questions
If you have a problem with any aspects of this book, you can contact us at [email protected], and we will do our best to address it.
Introduction to Spark
Arguably, the first time big data was being talked about in a context we know now was in July, 1997. MichaelCox and DavidEllsworth, scientists/researchers from NASA, described the problem they faced when processing humongous amounts of data with the traditional computers of that time. In the early 2000s, Lexis Nexis designed a proprietary system, which later went on to become the High-PerformanceComputingCluster (HPCC), to address the growing need of processing data on a cluster. It was later open sourced in 2011.
It was an era of dot coms and Google was challenging the limits of the internet by crawling and indexing the entire internet. With the rate at which the internet was expanding, Google knew it would be difficult if not impossible to scale vertically to process data of that size. Distributed computing, though still in its infancy, caught Google's attention. They not only developed a distributed fault tolerant filesystem, Google File System (GFS), but also a distributed processing engine/system called MapReduce. It was then in 2003-2004 that Google released the white paper titled The Google File System by SanjayGhemawat, HowardGobioff, and Shun-TakLeung, and shortly thereafter they released another white paper titled MapReduce: Simplified Data Processing on Large Clusters by JeffreyDean and SanjayGhemawat.
Doug Cutting, an open source contributor, around the same time was looking for ways to make an open source search engine and like Google was failing to process the data at the internet scale. By 1999, Doug Cutting had developed Lucene, a Java library with the capability of text/web searching among other things. Nutch, an open source web crawler and data indexer built by Doug Cutting along with Mike Cafarella, was not scaling well. As luck would have it, Google's white paper caught Doug Cutting's attention. He began working on similar concepts calling them Nutch Distributed File System (NDFS) and Nutch MapReduce. By 2005, he was able to scale Nutch, which could index from 100 million pages to multi-billion pages using the distributed platform.
However, it wasn't just Doug Cutting but Yahoo! too who became interested in the development of the MapReduce computing framework to serve its processing capabilities. It is here that Doug Cutting refactored the distributed computing framework of Nutch and named it after his kid's elephant toy, Hadoop. By 2008, Yahoo! was using Hadoop in its production cluster to build its search index and metadata called web map. Despite being a direct competitor to Google, one distinct strategic difference that Yahoo! took while co-developing Hadoop was the nature in which the project was to be developed: they open sourced it. And the rest, as we know is history!
In this chapter, we will cover the following topics:
What is big data?
Why Apache Spark?
RDD the first citizen of Spark
Spark ecosystem -- Spark SQL, Spark Streaming, Milb, Graphx
What's new in Spark 2.X?
Dimensions of big data
Big data can be best described by using its dimensions. Those dimensions are called the Vs of big data. To categorize a problem as a big data problem, it should lie in one or more of these dimensions.
The big data world started with three dimensions or 3Vs of big data, which are as follows:
Volume
Variety
Velocity
Let us now take a look at each one in detail:
Volume
: The amount of data being generated in the world is increasing at an exponential rate. Let's take an example of social community websites such as Facebook or Twitter. They are dealing with billions of customers all around the world. So, to analyze the amount of data being generated, they need to find a solution out of the existing RDBMS world. Moreover, not only such big giants, but also other organizations, such as banks, telecom companies, and so on, are dealing with huge numbers of customers. Performing analytics on such a humongous amount of data is a big data problem. So, according to this dimension, if you are dealing with a high volume of data, which can't be handled by traditional database systems, then it's imperative to move to big data territory.
Variety
: There was a time when only structured data was meant to be processed. But, to keep yourself ahead of your competitor, you need to analyze every sort of data which can increase value. For example, which products on a portal are more popular than others? So, you are analyzing user clicks. Now, data from these various sources that you need to use to keep yourself ahead can be structured or unstructured. It can be XML, JSON, CSV, or even plain text. So, now the data that you may need to deal with can be of different varieties. So, if you have such an issue, realize that this is a big data problem.
Velocity
: Data is not only increasing in size but the rate at which it is arriving is also increasing rapidly. Take the example of Twitter: billions of users are tweeting at a time. Twitter has to handle such a high velocity of data in almost real time. Also, you can think of YouTube. A lot of videos are being uploaded or streamed from YouTube every minute. Even look at online portals of news channels; they are being updated every second or minute to cope up with incoming data of news from all over the world. So, this dimension of big data deals with a high velocity of data and helps to provide persistence or analyze the data in near real time so as to generate real value.
Then, with time, our 3D world changed to a 7D world, with the following newer dimensions:
Veracity
: The truthfulness and completeness of the data are equally important. Take an example of a machine learning algorithm that involves automated decision making based on the data it analyzes. If the data is not accurate, this system can be disastrous. An example of such a system can be predictive analytics based on the online shopping data of end users. Using the analytics, you want to send offers to users. If the data that is fed to such a system is inaccurate or incomplete, analytics will not be meaningful or beneficial for the system. So, as per this dimension, before processing/analyzing, data should be validated. Processing high volume or high velocity data can only be meaningful if the data is accurate and complete, so before processing the data, it should be validated as well.
Variability
: This dimension of big data mainly deals with natural language processing or sentiment analytics. In language, one word can have multiple usages based on the sentiments of the user. So, to find sentiments, you should be able to comprehend the exact meaning. Let's say your favorite football team is not playing well and you posted a sarcastic tweet saying "
What a great performance today by our team!!
" Now looking at this sentence, it seems you are loving the way your team is performing but in reality it is the opposite. So to analyze the sentiments, the system should be fed with lot of other information such as the statistics of the match, and so on. Another example, the sentence
This is too good to be true
is negative but it consists of all positive words. Semantic analytics or natural language processing can only be accurate if you can understand sentiments behind the data.
Value
: There is lot of cost involved in performing big data analytics: the cost of getting the data, the cost for arranging hardware on which this data is saved and be analyzed, the cost of employees and time that goes into these analytics. All these costs are justified if the analytics provide value to the organization. Think of a healthcare company performing analytics on e-commerce data. They may be able to perform the analytics by getting data from the internet but it does not have value for them. Also, performing analytics on data which is not accurate or complete is not of any value. On the contrary, it can be harmful, as the analytics performed are misleading. So, value becomes an important dimension of big data because valuable analytics can be useful.
Visualization
: Visualization is another important aspect of the analytics. No work can be useful until it is visualized in a proper manner. Let's say engineers of your company have performed real accurate analytics but the output of them are stored in some JSON files or even in databases. The business analyst of your company, not being hard core technical, is not able to understand the outcome of the analytics thoroughly as the outcome is not visualized in a proper manner. So the analytics, even though they are correct, cannot be of much value to your organization. On the other hand, if you have created proper graphs or charts or effective visualization on the outcome, it can be much easier to understand and can be really valuable. So, visualization is a really important aspect of big data analytics because things can only be highlighted if they are visible.
What makes Hadoop so revolutionary?
In a classical sense, if we are to talk of Hadoop then it comprises of two components: a storage layer called HDFS and a processing layer called MapReduce. Resource management task prior to Hadoop 2.X was done using the MapReduce framework of Hadoop itself. However, that changed with the introduction of YARN. In Hadoop 2.0, YARN was introduced as the third component of Hadoop to manage the resources of the Hadoop cluster and make it more MapReduce agnostic.
Defining HDFS
The Hadoop Distributed File System (HDFS), as the name suggests, is a distributed filesystem based on the lines of the Google File System written in Java. In practice, HDFS resembles closely any other UNIX filesystem with support for common file operations such as ls, cp, rm, du, cat, and so on. However what makes HDFS stand out, despite its simplicity, is its mechanism to handle node failure in the Hadoop cluster without effectively changing the search time for accessing stored files. The HDFS cluster consists of two major components: DataNodes and NameNode.
HDFS has a unique way of storing data on HDFS clusters (cheap commodity networked commodity computers). It splits the regular file in smaller chunks called blocks and then makes an exact number of copies of such chunks depending on the replication factor for that file. After that, it copies such chunks to different DataNodes of the cluster.
NameNode
The NameNode is responsible for managing the metadata of the HDFS cluster, such as lists of files and folders that exist in a cluster, the number of splits each file is divided into, and their replication and storage at different DataNodes. It also maintains and manages the namespace and file permission of all the files available in the HDFS cluster. Apart from bookkeeping, NameNode also has a supervisory role that keeps a watch on the replication factor of all the files and if some block goes missing, then it issue commands to replicate the missing block of data. It also generates reports to ascertain cluster health. It is important to note that all the communication for a supervisory task happens from DataNode to NameNode; that is, DataNode sends reports (block reports) to NameNode and it is then that NameNode responds to them by issuing different commands or instructions as the need may be.
HDFS I/O
An HDFS read operation from a client involves the following:
The client requests NameNode to determine where the actual data blocks are stored for a given file.
NameNode obliges by providing the block IDs and locations of the hosts (DataNode) where the data can be found.
The client contacts DataNode with the respective block IDs to fetch the data from DataNode while preserving the order of the block files.
An HDFS write operation from a client involves the following:
The client contacts NameNode to update the namespace with the filename and verify the necessary permissions.
If the file exists, then NameNode throws an error; otherwise, it returns the client
FSDataOutputStream
which points to the data queue.
The data queue negotiates with the NameNode to allocate new blocks on suitable DataNodes.
The data is then copied to that DataNode, and, as per the replication strategy, the data is further copied from that DataNode to the rest of the DataNodes.
It's important to note that the data is never moved through the NameNode as it would caused a performance bottleneck.
YARN
The simplest way to understand YARN (YetAnotherResourceManager) is to think of it as an operating system on a cluster; provisioning resources, scheduling jobs and node maintenance. With Hadoop 2.x, the MapReduce model of processing the data and managing the cluster (Job Tracker/Task Tracker) was divided. While data processing was still left to MapReduce, the cluster's resource allocation (or rather, scheduling) task was assigned to a new component called YARN. Another objective that YARN met was that it made MapReduce one of the techniques to process the data rather than being the only technology to process data on HDFS, as was the case in Hadoop 1.x systems. This paradigm shift opened the floodgates for the development of interesting applications around Hadoop and a new ecosystem other than the classical MapReduce processing system evolved. It didn't take much time after that for Apache Spark to break the hegemony of classical MapReduce and become arguably the most popular processing framework for parallel computing as far as active development and adoption is concerned.
In order to serve multi-tenancy, fault tolerance, and resource isolation in YARN, it developed the following components to manage the cluster seamlessly:
The
ResourceManager
: This negotiates resources for different compute programs on a Hadoop cluster while guaranteeing the following: resource isolation, data locality, fault tolerance, task prioritization, and effective cluster capacity utilization. A configurable scheduler allows Resource Manager the flexibility to schedule and prioritize different applications as per the requirements.
Tasks served by the RM while serving clients
: A client or APIs user can submit or terminate an application. The user can also gather statistics on submitted applications cluster, and queue information. RM also priorities ADMIN tasks over any other task to perform a clean up or maintenance activities on a cluster, such as refreshing the node-list, the queues' configuration, and so on.
Tasks served by RM while serving cluster nodes
: Provisioning and de-provisioning of new nodes forms an important task of RM. Each node sends a heartbeat at a configured interval, the default being 10 minutes. Any failure of a node in doing so is treated as a dead node. As a clean-up activity, all the supposedly running process, including containers, are marked as dead too.
Tasks served by the RM while serving the Application Master
: The RM registers a new the AM while terminating the successfully executed ones. Just like cluster nodes, if the heartbeat of an AM is not received within a preconfigured duration, the default value being 10 minutes, then the AM is marked dead and all the associated containers are also marked dead. But since YARN is reliable as far as the application execution is concerned, a new AM is rescheduled to try another execution on a new container until it reaches the retry configurable default count of four.
Scheduling and other miscellaneous tasks served by the RM
: RM maintains a list of running, submitted and executed applications along with its statistics such as execution time, status, and so on. The privileges of the user as well as of applications are maintained and compared while serving various requests of the user per application life cycle. The RM scheduler oversees the resource allocation for the application, such as memory allocation. Two common scheduling algorithms used in YARN are fair scheduling and capacity scheduling algorithms.
NodeManager
: An NM exist per node of the cluster on a slightly similar fashion as to what slave nodes are in the master slave architecture. When an NM starts, it sends the information to RM for its availability to share its resources for upcoming jobs. Then NM sends a periodic signal, also called a heartbeat, to RM informing it of its status as being alive in the cluster. Primarily, an NM is responsible for launching containers that have been requested by an AM with certain resource requirements such as memory, disk, and so on. Once the containers are up and running, the NM keeps a watch not on the status of the container's task but on the resource utilization of the container and kills it if the container starts utilizing more resources than it has been provisioned for. Apart from managing the life cycle of the container, the NM also keeps RM informed about the node's health.
ApplicationMaster
: An AM gets launched per submitted application and manages the life cycle of the submitted application. However, the first and foremost task an AM does is to negotiate resources from RM to launch task-specific containers at different nodes. Once containers are launched, the AM keeps track of all the container's task statuses. If any node goes down or the container gets killed because of using excess resources or otherwise, in such cases the AM renegotiates resources from RM and launches those pending tasks again. The AM also keeps reporting the status of the submitted application directly to the user and other such statistics to RM. ApplicationMaster implementation is framework specific and it is because of this reason that application/framework specific code is transferred to the AM and the AM that distributes it further. This important feature also makes YARN technology agnostic, as any framework can implement its ApplicationMaster and then utilize the resources of the YARN cluster seamlessly.
Containers
: A container in an abstract sense is a set of minimal resources such as CPU, RAM, Disk I/O, disk space, and so on, that are required to run a task independently on a node. The first container after submitting the job is launched by RM to host ApplicationMaster. It is the AM which then negotiates resources from RM in the form of containers, which then gets hosted in different nodes across the Hadoop cluster.
Processing the flow of application submission in YARN
The following steps follow the flow of application submission in YARN:
Using a client or APIs, the user submits the application; let's say a Spark job jar. ResourceManager, whose primary task is to gather and report all the applications running on the entire Hadoop cluster and available resources on respective Hadoop nodes, depending on the privileges of the user submitting the job, accepts the newly submitted task.
After this RM delegates the task to a scheduler, the scheduler then searches for a container which can host the application-specific Application Master. While the scheduler does take into consideration parameters such as availability of resources, task priority, data locality, and so on, before scheduling or launching an Application Master, it has no role in monitoring or restarting a failed job. It is the responsibility of RM to keep track of an AM and restart it in a new container if it fails.
Once the ApplicationMaster gets launched it becomes the prerogative of the AM to oversee the resources negotiation with RM for launching task-specific containers. Negotiations with RM are typically over:
The priority of the tasks at hand.
The number of containers to be launched to complete the tasks.
