'

ENGINEERING BIG DATA WITH HADOOP

Понравилась презентация – покажи это...





Слайд 0

ENGINEERING BIG DATA WITH HADOOP BY International School of Engineering {We Are Applied Engineering} Disclaimer: Some of the Images and content have been taken from multiple online sources and this presentation is intended only for knowledge sharing but not for any commercial business intention


Слайд 1

OVERVIEW WHAT IS BIG DATA? EXPLOSION OF DATA DATA CONTRIBUTIONS DATA EXPLOSION WHO ARE THE PLAYERS? BIG DATA–BIG PICTURE– LANDSCAPE BIG DATA– ENTERPRISE ROLES WHAT IS HADOOP? EVOLUTION OF HADOOP HADOOP ECOSYSTEM HADOOP ECOSYSTEM MAP HADOOP: 30,000 FEET VIEW BIG DATA & ANALYTICS Case studies VIDEO OF HADOOP ECOSYSYTEM


Слайд 2

WHAT IS BIG DATA? High-volume, high-velocity and high- variety information assets that demand cost- effective, innovative forms of information processing for enhanced insight and decision making. -Gartner HIGH VOLUME HIGH VELOCITY HIGH VARIETY


Слайд 3

EXPLOSION OF DATA


Слайд 4

Source: http://www.emc.com/leadership/digital-universe/iview/index.htm


Слайд 5

DATA CONTRIBUTIONS


Слайд 6

DATA EXPLOSION


Слайд 7

Source: http://www.emc.com/collateral/about/news/idc-emc-digital-universe-2011-infographic.pdf


Слайд 8

Source: http://www.emc.com/collateral/about/news/idc-emc-digital-universe-2011-infographic.pdf


Слайд 9

WHO ARE THE PLAYERS?


Слайд 10


Слайд 11

BIG DATA–BIG PICTURE– LANDSCAPE


Слайд 12

BIG DATA– ENTERPRISE ROLES


Слайд 13

INTRODUCTION TO


Слайд 14

WHAT IS HADOOP? Flexible Structured/Unstructured Text/Binary Schema/Schema less 100% Open Source Scalable – Petabytes of Data – Thousands of Nodes Source: http://cloudtimes.org/2013/06/25/hadoop-as-a-service-market-growing/


Слайд 15

How does an Elephant Sneak up on you? EVOLUTION OF HADOOP


Слайд 16

HADOOP ECOSYSTEM Chukwa Sqoop Zookeeper Pig HBase Avno Mahout Flume Whirr Map Reduce Engine Hama Hive Hadoop Distributed File System Hadoop Common


Слайд 17

Source: http://indoos.wordpress.com/2010/08/16/hadoop-ecosystem-world-map/ HADOOP ECOSYSTEM MAP


Слайд 18

Hadoop Evolution – Map Explained! How did it all start- huge data on the web! Nutch built to crawl this web data Huge data had to be saved- HDFS was born! How to use this data? Map reduce framework built for coding and running analytics – java, any language-streaming (Hadoop streaming) How to get in unstructured data – Web logs, Click streams, Apache logs, Server logs  – fuse,webdav, chukwa, flume, Scribe Hiho and sqoop for loading data into HDFS – RDBMS can join the Hadoop band wagon!


Слайд 19

Continued High level interfaces required over low level map reduce programming– Pig, Hive, Jaql BI tools with advanced UI reporting- drilldown etc- Intellicus Workflow tools over Map-Reduce processes and High level languages: Oozie Monitor and manage hadoop, run jobs/hive, view HDFS – high level view- Hue, karmasphere, eclipse plugin, cacti, ganglia Support frameworks- Avro (Serialization), Zookeeper (Coordination) More High level interfaces/uses- Mahout, Elastic map Reduce OLTP- also possible – Hbase


Слайд 20

Distribute data initially Let processors / nodes work on local data Minimize data transfer over network Replicate data multiple times for increased availability Write applications at a high level Programmers should not have to worry about network programming, temporal dependencies, low level infrastructure, etc Minimize talking between nodes (share-nothing) HADOOP: 30,000 FEET VIEW


Слайд 21

BIG DATA & ANALYTICS Case Studies


Слайд 22

YAHOO - PERSONALIZATION


Слайд 23

YAHOO SEARCH ASSIST


Слайд 24

For Detailed Description of HADOOP ECOSYSTEM components checkout our video on


Слайд 25

Plot no 63/A, 1st Floor, Road No 13, Film Nagar, Jubilee Hills, Hyderabad-500033 For Individuals (+91) 9502334561/62 For Corporates (+91) 9618 483 483 Facebook: www.facebook.com/insofe Slide share: www.slideshare.net/INSOFE International School of Engineering


×

HTML:





Ссылка: