Learning Hunk - Dmitry Anoshin - E-Book

Learning Hunk E-Book

Dmitry Anoshin

0,0
20,39 €

-100%
Sammeln Sie Punkte in unserem Gutscheinprogramm und kaufen Sie E-Books und Hörbücher mit bis zu 100% Rabatt.
Mehr erfahren.
Beschreibung

Hunk is the big data analytics platform that lets you rapidly explore, analyse, and visualize data in Hadoop and NoSQL data stores. It provides a single, fluid user experience, designed to show you insights from your big data without the need for specialized skills, fixed schemas, or months of development. Hunk goes beyond typical data analysis methods and gives you the power to rapidly detect patterns and find anomalies across petabytes of raw data.

This book focuses on exploring, analysing, and visualizing big data in Hadoop and NoSQL data stores with this powerful full-featured big data analytics platform.
You will begin by learning the Hunk architecture and Hunk Virtual Index before moving on to how to easily analyze and visualize data using Splunk Search Language (SPL). Next you will meet Hunk Apps which can easy integrate with NoSQL data stores such as MongoDB or Sqqrl. You will also discover Hunk knowledge objects, build a semantic layer on top of Hadoop, and explore data using the friendly user-interface of Hunk Pivot. You will connect MongoDB and explore data in the data store. Finally, you will go through report acceleration techniques and analyze data in the AWS Cloud.

Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:

EPUB
MOBI

Seitenzahl: 121

Veröffentlichungsjahr: 2015

Bewertungen
0,0
0
0
0
0
0
Mehr Informationen
Mehr Informationen
Legimi prüft nicht, ob Rezensionen von Nutzern stammen, die den betreffenden Titel tatsächlich gekauft oder gelesen/gehört haben. Wir entfernen aber gefälschte Rezensionen.



Table of Contents

Learning Hunk
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Downloading the color images of this book
Errata
Piracy
Questions
1. Meet Hunk
Big data analytics
The big problem
The elegant solution
Supporting SPL
Intermediate results
Getting to know Hunk
Splunk versus Hunk
Hunk architecture
Connecting to Hadoop
Advance Hunk deployment
Native versus virtual indexes
Native indexes
Virtual index
External result provider
Computation models
Data streaming
Data reporting
Mixed mode
Hunk security
One Hunk user to one Hadoop user
Many Hunk users to one Hadoop user
Hunk user(s) to the same Hadoop user with different queues
Setting up Hadoop
Starting and using a virtual machine with CDH5
SSH user
MySQL
Starting the VM and cluster in VirtualBox
Big data use case
Importing data from RDBMS to Hadoop using Sqoop
Telecommunications – SMS, Call, and Internet dataset from dandelion.eu
Milano grid map
CDR aggregated data import process
Periodical data import from MySQL using Sqoop and Oozie
Problems to solve
Summary
2. Explore Hadoop Data with Hunk
Setting up Hunk
Extracting Hunk to a VM
Setting up Hunk variables and configuration files
Running Hunk for the first time
Setting up a data provider and virtual index for CDR data
Setting up a connection to Hadoop
Setting up a virtual index for data stored in Hadoop
Accessing data through a virtual index
Exploring data
Creating reports
The top five browsers report
Top referrers
Site errors report
Creating alerts
Creating a dashboard
Controlling security with Hunk
The default Hadoop security
One Hunk user to one Hadoop user
Summary
3. Meeting Hunk Features
Knowledge objects
Field aliases
Calculated fields
Field extractions
Tags
Event type
Workflow actions
Macros
Data model
Add auto-extracting fields
Adding GeoIP attributes
Other ways to add attributes
Introducing Pivot
Summary
4. Adding Speed to Reports
Big data performance issues
Hunk report acceleration
Creating a virtual index
Streaming mode
Creating an acceleration search
What's going on in Hadoop?
Report acceleration summaries
Reviewing summary details
Managing report accelerations
Hunk accelerations limits
Summary
5. Customizing Hunk
What we are going to do with the Splunk SDK
Supported languages
Solving problems
REST API
The implementation plan
The conclusion
Dashboard customization using Splunk Web Framework
Functionality
A description of time-series aggregated CDR data
Source data
Creating a virtual index for Milano CDR
Creating a virtual index for the Milano grid
Creating a virtual index using sample data
Implementation
Querying the visualization
Downloading the application
Custom Google Maps
Page layout
Linear gradients and bins for the activity value
Custom map components
Other components
The final result
Summary
6. Discovering Hunk Integration Apps
What is Mongo?
Installation
Installing the Mongo app
Mongo provider
Creating a virtual index
Inputting data from the recommendation engine backend
Data schemas
Data mechanics
Counting by shop in a single collection
Counting events in all collections
Counting events in shops for observed days
Summary
7. Exploring Data in the Cloud
An introduction to Amazon EMR and S3
Amazon EMR
Setting up an Amazon EMR cluster
Amazon S3
S3 as a data provider for Hunk
The advantages of EMR and S3
Integrating Hunk with EMR and S3
Method 1: BYOL
Setting up the Hunk AMI
Adding a license
Configuring the data provider
Configuring a virtual index
Setting up a provider and virtual index in the configuration file
Exploring data
Method 2: Hunk–hourly pricing
Provisioning a Hunk instance using the Cloud formation template
Provisioning a Hunk instance using the EC2 Console
Converting Hunk from an hourly rate to a license
Summary
Index

Learning Hunk

Learning Hunk

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2015

Production reference: 1181215

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78217-482-0

www.packtpub.com

Credits

Authors

Dmitry Anoshin

Sergey Sheypak

Reviewers

Jigar Bhatt

Neil Mehta

Acquisition Editors

Hemal Desai

Reshma Raman

Content Development Editor

Anish Sukumaran

Technical Editor

Shivani Kiran Mistry

Copy Editor

Stephen Copestake

Project Coordinator

Izzat Contractor

Proofreader

Safis Editing

Indexer

Hemangini Bari

Graphics

Jason Monteiro

Production Coordinator

Nilesh Mohite

Cover Work

Nilesh Mohite

About the Authors

Dmitry Anoshin is a data-centric technologist and a recognized expert in building and implementing big data and analytics solutions. He has a successful track record when it comes to implementing business and digital intelligence projects in numerous industries, including retail, finance, marketing, and e-commerce.

Dmitry possesses in-depth knowledge of digital/business intelligence, ETL, data warehousing, and big data technologies. He has extensive experience in the data integration process and is proficient in using various data warehousing methodologies. Dmitry has constantly exceeded project expectations when he has worked for financial, machine tool, and retail industries.

He has completed a number of multinational full BI/DI solution life cycle implementation projects. With expertise in data modeling, Dmitry also has a background and business experience in multiple relation databases, OLAP systems, and NoSQL databases.

In addition, he has reviewed SAP BusinessObjects Reporting Cookbook, Creating Universes with SAP BusinessObjects, and Learning SAP BusinessObjects Dashboards, all by Packt Publishing and was the author of SAP Lumira Essentials, Packt Publishing.

I would like to tell my wife Sveta how much I love her. I dedicate this book to my wife and children, Vasily and Anna. Thank you for your never-ending support that keeps me going.

Sergey Sheypak started his so-called big data practice in 2010 as a Teradata PS consultant. His was leading the Teradata Master Data Management deployment in Sberbank, Russia (which has 110 billion customers). Later Sergey switched to AsterData and Hadoop practices. Sergey joined the Research and Development team at MegaFon (one of the top three telecom companies in Russia with 70 billion customers) in 2012. While leading the Hadoop team at MegaFon, Sergey built ETL processes from existing Oracle DWH to HDFS. Automated end-to-end tests and acceptance tests were introduced as a mandatory part of the Hadoop development process. Scoring geospatial analysis systems based on specific telecom data were developed and launched. Now, Sergey works as independent consultant in Sweden.

About the Reviewer

Jigar Bhatt is a computer engineering undergraduate from the National Institute of Technology, Surat. He specializes in big data technologies and has a deep interest in data science and machine learning. He has also engineered several cloud-based Android applications. He is currently working as a full-time software developer at a renowned start-up, focusing on building and optimizing cloud platforms and ensuring profitable business intelligence round the clock.

Apart from academics, he finds adventurous sports enthralling. He can be reached at http://www.jigarbhatt.in/.

I would like to thank Dr. Dhiren Patel from the computer engineering department, NIT, Surat, who encouraged my interest in data science and guided me through the initial stages of building my career in the big data world.

Neil Mehta BSc (Hons) has 20 years of experience as a developer, analyst, and program manager and has spent the last 7 years specifically implementing business intelligence solutions to help companies leverage their corporate data. Trained in all aspects of analytics from data modeling to system architecture and reporting, Neil currently manages a large team of data architects, ETL developers, and report designers for a large insurance company.He has extensive experience with business analytics, administration, and dashboard design and has helped develop programs to establish super user communities and develop training plans. He has worked in multiple business segments, including financial, oil and gas, transportation, and retail industries.

www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at <[email protected]> for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by PacktCopy and paste, print, and bookmark contentOn demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.

Preface

This book offers a step-by-step approach to learning Hunk, diving into the technical aspects of it first. It will demonstrate the various aspects of big data analytics using the powerful capabilities of Hunk. In addition to this, it provides detailed sections on the deployment and configuration of Hunk on top of Hadoop and the NoSQL data stores. It will also teach you how to create queries using SPL, reports, and dashboards. This book covers security questions and demonstrates how to set up security for big data implementation based on Hadoop and Hunk. Moreover, it will teach you how to use the Hunk SDK and extend its default functionality. Finally, it acts as a guide to deploying Hunk on top of MongoDB and AWS Elastic MapReduce.

What this book covers

Chapter 1, Meet Hunk, covers Hunk and its basic features. Hunk is a full-featured platform to rapidly explore, analyze, and visualize data in Hadoop and the NoSQL data stores. You will learn how to install and configure Hunk. Moreover, you will learn about Hunk's architecture and Hunk Virtual Index. You will also be introduced to loading data into Hadoop in order to aid its discovery by Hunk.

Chapter 2, Explore Hadoop Data with Hunk, talks about how you can easily analyze and visualize data using the Splunk search processing language (SPL). Getting a large amount of data into Hadoop is easy but getting analytics from this data is the challenge. You will learn about the use cases of big data analytics and the security aspect of Hunk.

Chapter 3, Meet Hunk Features, teaches you about Hunk's knowledge objects. Hunk is a powerful big data analytics platform, which gives us many tools in order to explore, analyze, and visualize big data. You will learn how to build a semantic layer on top of Hadoop and discover data using the friendly user interface of Hunk Pivot.

Chapter 4, Adding Speed to Reports, covers the techniques related to report acceleration. Hunk is an extremely powerful tool and can handle a vast amount of data. However, business decisions, which depend on fresh data, can't wait.

Chapter 5, Customizing Hunk, introduces REST API, SDK, and so on. Sometimes, we want to get out of the box or need to meet business expectations and are restricted by the initial functionality. Thus, you will learn how to create customized visualization, and you will also be introduced to the Splunk Web Framework.

Chapter 6, Discovering Hunk Integration Apps, introduces you to Hunk's apps that can easily integrate with the NoSQL data stores, such as MongoDB or Sqqrl. Hunk is a universal big data analytics platform, which can easy explore data in Hadoop or the NoSQL data stores. You will learn how to connect MongoDB and explore data in its data store.

Chapter 7, Exploring Data in the Cloud, shows you how to analyze data in AWS Cloud. Some big organizations prefer to store their big data on the cloud because it gives them many benefits.

What you need for this book

In this book, you will learn how to explore, analyze, and visualize big data in Hadoop or the NoSQL data stores with the powerful, full-featured big data analytics platform, Hunk. You will discover real-world examples, dive into Hunk's architecture and capabilities, as well as learn how to build Operation Intelligence using this technology. Additionally, you will learn about report acceleration techniques, data models, and custom dashboards and views using Hunk. Moreover, this book focuses on popular use cases using powerful Hunk apps, which provide integration with the NoSQL data stores and give complete visibility into your end-to-end big data operations. Finally, you will about the Splunk web framework. We just require a laptop or PC with a 4 GB RAM (8 GB RAM recommended) and VirtualBox installed. There aren't any specific hardware requirements as VirtualBox should work everywhere.

Who this book is for

If you are big data enthusiast and want to get more business insight and build efficient, real-time Operation Intelligence Solution based on Hadoop deployments or various NoSQL data stores using Hunk, this book is for you. Aimed on big data developers, managers and consultants this is also a comprehensive reference for everyone, who want to learn how to analyze and explore big data with one of the most powerful and flexible big data analytics platform.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Rename the count field as qty."

A block of code is set as follows: