Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (Tom White - Hadoop The Definitive Guide_ 4 edition - 2015.pdf)

on editi datEd ph &U4t s e dReviHadoop: The Definitive GuideUsing Hadoop 2 exclusively, author Tom White presents new chapterson YARN and several Hadoop-related projects such as Parquet, Flume,Crunch, and Spark. You’ll learn about recent changes to Hadoop, andexplore new case studies on Hadoop’s role in healthcare systems andgenomics data processing.■■Learn fundamental components such as MapReduce, HDFS,and YARN■■Explore MapReduce in depth, including steps for developingapplications with it■■Set up and maintain a Hadoop cluster running HDFS andMapReduce on YARN■■Learn two data formats: Avro for data serialization and Parquetfor nested data■■Use data ingestion tools such as Flume (for streaming data) andSqoop (for bulk data transfer)■■Understand how high-level data processing tools like Pig, Hive,Crunch, and Spark work with Hadoop■■Learn the HBase distributed database and the ZooKeeperdistributed configuration serviceyou have the “Nowopportunityto learnabout Hadoop from amaster—not only of thetechnology, but also of common sense andplain talk.”—Doug CuttingClouderaTom White, an engineer at Cloudera and member of the Apache SoftwareFoundation, has been an Apache Hadoop committer since 2007.

He has writtennumerous articles for oreilly.com, java.net, and IBM’s developerWorks, and speaksregularly about Hadoop at industry conferences.US $49.99Twitter: @oreillymediafacebook.com/oreillyWhitePROGR AMMING L ANGUAGES/HADOOPFOURTH EDITIONHadoop:The Definitive GuideGet ready to unlock the power of your data. With the fourth edition ofthis comprehensive guide, you’ll learn how to build and maintain reliable,scalable, distributed systems with Apache Hadoop. This book is ideal forprogrammers looking to analyze datasets of any size, and for administratorswho want to set up and run Hadoop clusters.HadoopThe Definitive GuideSTORAGE AND ANALYSIS AT INTERNET SCALECAN $57.99ISBN: 978-1-491-90163-2Tom Whiteon editi datEd ph &U4t s e dReviHadoop: The Definitive GuideUsing Hadoop 2 exclusively, author Tom White presents new chapterson YARN and several Hadoop-related projects such as Parquet, Flume,Crunch, and Spark.

You’ll learn about recent changes to Hadoop, andexplore new case studies on Hadoop’s role in healthcare systems andgenomics data processing.■■Learn fundamental components such as MapReduce, HDFS,and YARN■■Explore MapReduce in depth, including steps for developingapplications with it■■Set up and maintain a Hadoop cluster running HDFS andMapReduce on YARN■■Learn two data formats: Avro for data serialization and Parquetfor nested data■■Use data ingestion tools such as Flume (for streaming data) andSqoop (for bulk data transfer)■■Understand how high-level data processing tools like Pig, Hive,Crunch, and Spark work with Hadoop■■Learn the HBase distributed database and the ZooKeeperdistributed configuration serviceyou have the “Nowopportunityto learnabout Hadoop from amaster—not only of thetechnology, but also of common sense andplain talk.”—Doug CuttingClouderaTom White, an engineer at Cloudera and member of the Apache SoftwareFoundation, has been an Apache Hadoop committer since 2007.

He has writtennumerous articles for oreilly.com, java.net, and IBM’s developerWorks, and speaksregularly about Hadoop at industry conferences.US $49.99Twitter: @oreillymediafacebook.com/oreillyWhitePROGR AMMING L ANGUAGES/HADOOPFOURTH EDITIONHadoop:The Definitive GuideGet ready to unlock the power of your data. With the fourth edition ofthis comprehensive guide, you’ll learn how to build and maintain reliable,scalable, distributed systems with Apache Hadoop. This book is ideal forprogrammers looking to analyze datasets of any size, and for administratorswho want to set up and run Hadoop clusters.HadoopThe Definitive GuideSTORAGE AND ANALYSIS AT INTERNET SCALECAN $57.99ISBN: 978-1-491-90163-2Tom WhiteFOURTH EDITIONHadoop: The Definitive GuideTom WhiteHadoop: The Definitive Guide, Fourth Editionby Tom WhiteCopyright © 2015 Tom White.

All rights reserved.Printed in the United States of America.Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions arealso available for most titles (http://safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.Editors: Mike Loukides and Meghan BlanchetteProduction Editor: Matthew HackerCopyeditor: Jasmine KwitynProofreader: Rachel HeadJune 2009:First EditionOctober 2010:Second EditionMay 2012:Third EditionApril 2015:Fourth EditionIndexer: Lucie HaskinsCover Designer: Ellie VolckhausenInterior Designer: David FutatoIllustrator: Rebecca DemarestRevision History for the Fourth Edition:2015-03-19:First release2015-04-17:Second releaseSee http://oreilly.com/catalog/errata.csp?isbn=9781491901632 for release details.The O’Reilly logo is a registered trademark of O’Reilly Media, Inc.

Hadoop: The Definitive Guide, the coverimage of an African elephant, and related trade dress are trademarks of O’Reilly Media, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademarkclaim, the designations have been printed in caps or initial caps.While the publisher and the author have used good faith efforts to ensure that the information and instruc‐tions contained in this work are accurate, the publisher and the author disclaim all responsibility for errorsor omissions, including without limitation responsibility for damages resulting from the use of or relianceon this work.

Use of the information and instructions contained in this work is at your own risk. If any codesamples or other technology this work contains or describes is subject to open source licenses or the intel‐lectual property rights of others, it is your responsibility to ensure that your use thereof complies with suchlicenses and/or rights.ISBN: 978-1-491-90163-2[LSI]For Eliane, Emilia, and LottieTable of ContentsForeword.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiPreface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . xixPart I.Hadoop Fundamentals1. Meet Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Data!Data Storage and AnalysisQuerying All Your DataBeyond BatchComparison with Other SystemsRelational Database Management SystemsGrid ComputingVolunteer ComputingA Brief History of Apache HadoopWhat’s in This Book?356688101112152.

MapReduce. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19A Weather DatasetData FormatAnalyzing the Data with Unix ToolsAnalyzing the Data with HadoopMap and ReduceJava MapReduceScaling OutData FlowCombiner FunctionsRunning a Distributed MapReduce JobHadoop Streaming1919212222243030343737vRubyPython37403.

The Hadoop Distributed Filesystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43The Design of HDFSHDFS ConceptsBlocksNamenodes and DatanodesBlock CachingHDFS FederationHDFS High AvailabilityThe Command-Line InterfaceBasic Filesystem OperationsHadoop FilesystemsInterfacesThe Java InterfaceReading Data from a Hadoop URLReading Data Using the FileSystem APIWriting DataDirectoriesQuerying the FilesystemDeleting DataData FlowAnatomy of a File ReadAnatomy of a File WriteCoherency ModelParallel Copying with distcpKeeping an HDFS Cluster Balanced4345454647484850515354565758616363686969727476774.

YARN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Anatomy of a YARN Application RunResource RequestsApplication LifespanBuilding YARN ApplicationsYARN Compared to MapReduce 1Scheduling in YARNScheduler OptionsCapacity Scheduler ConfigurationFair Scheduler ConfigurationDelay SchedulingDominant Resource FairnessFurther Readingvi| Table of Contents8081828283858688909495965.

Hadoop I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97Data IntegrityData Integrity in HDFSLocalFileSystemChecksumFileSystemCompressionCodecsCompression and Input SplitsUsing Compression in MapReduceSerializationThe Writable InterfaceWritable ClassesImplementing a Custom WritableSerialization FrameworksFile-Based Data StructuresSequenceFileMapFileOther File Formats and Column-Oriented FormatsPart II.97989999100101105107109110113121126127127135136MapReduce6.