Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (Tom White - Hadoop The Definitive Guide_ 4 edition - 2015.pdf), страница 4

New features from later releasesare occasionally mentioned in the text, however, with reference to the version that theywere introduced in.Conventions Used in This BookThe following typographical conventions are used in this book:ItalicIndicates new terms, URLs, email addresses, filenames, and file extensions.Preface|xxiConstant widthUsed for program listings, as well as within paragraphs to refer to commands andcommand-line options and to program elements such as variable or functionnames, databases, data types, environment variables, statements, and keywords.Constant width boldShows commands or other text that should be typed literally by the user.Constant width italicShows text that should be replaced with user-supplied values or by values deter‐mined by context.This icon signifies a general note.This icon signifies a tip or suggestion.This icon indicates a warning or caution.Using Code ExamplesSupplemental material (code, examples, exercise, etc.) is available for download at thisbook’s website and on GitHub.This book is here to help you get your job done.

In general, you may use the code inthis book in your programs and documentation. You do not need to contact us forpermission unless you’re reproducing a significant portion of the code. For example,writing a program that uses several chunks of code from this book does not requirepermission. Selling or distributing a CD-ROM of examples from O’Reilly books doesrequire permission. Answering a question by citing this book and quoting example codedoes not require permission. Incorporating a significant amount of example code fromthis book into your product’s documentation does require permission.xxii|PrefaceWe appreciate, but do not require, attribution.

An attribution usually includes the title,author, publisher, and ISBN. For example: “Hadoop: The Definitive Guide, Fourth Ed‐ition, by Tom White (O’Reilly). Copyright 2015 Tom White, 978-1-491-90163-2.”If you feel your use of code examples falls outside fair use or the permission given here,feel free to contact us at permissions@oreilly.com.Safari® Books OnlineSafari Books Online is an on-demand digital library thatdelivers expert content in both book and video form fromthe world’s leading authors in technology and business.Technology professionals, software developers, web designers, and business and crea‐tive professionals use Safari Books Online as their primary resource for research, prob‐lem solving, learning, and certification training.Safari Books Online offers a range of plans and pricing for enterprise, government,education, and individuals.Members have access to thousands of books, training videos, and prepublication manu‐scripts in one fully searchable database from publishers like O’Reilly Media, PrenticeHall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, PeachpitPress, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBMRedbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill,Jones & Bartlett, Course Technology, and hundreds more.

For more information aboutSafari Books Online, please visit us online.How to Contact UsPlease address comments and questions concerning this book to the publisher:O’Reilly Media, Inc.1005 Gravenstein Highway NorthSebastopol, CA 95472800-998-9938 (in the United States or Canada)707-829-0515 (international or local)707-829-0104 (fax)We have a web page for this book, where we list errata, examples, and any additionalinformation. You can access this page at http://bit.ly/hadoop_tdg_4e.To comment or ask technical questions about this book, send email tobookquestions@oreilly.com.Preface|xxiiiFor more information about our books, courses, conferences, and news, see our websiteat http://www.oreilly.com.Find us on Facebook: http://facebook.com/oreillyFollow us on Twitter: http://twitter.com/oreillymediaWatch us on YouTube: http://www.youtube.com/oreillymediaAcknowledgmentsI have relied on many people, both directly and indirectly, in writing this book. I wouldlike to thank the Hadoop community, from whom I have learned, and continue to learn,a great deal.In particular, I would like to thank Michael Stack and Jonathan Gray for writing thechapter on HBase.

Thanks also go to Adrian Woodhead, Marc de Palol, Joydeep SenSarma, Ashish Thusoo, Andrzej Białecki, Stu Hood, Chris K. Wensel, and OwenO’Malley for contributing case studies.I would like to thank the following reviewers who contributed many helpful suggestionsand improvements to my drafts: Raghu Angadi, Matt Biddulph, Christophe Bisciglia,Ryan Cox, Devaraj Das, Alex Dorman, Chris Douglas, Alan Gates, Lars George, PatrickHunt, Aaron Kimball, Peter Krey, Hairong Kuang, Simon Maxen, Olga Natkovich,Benjamin Reed, Konstantin Shvachko, Allen Wittenauer, Matei Zaharia, and PhilipZeyliger.

Ajay Anand kept the review process flowing smoothly. Philip (“flip”) Kromerkindly helped me with the NCDC weather dataset featured in the examples in this book.Special thanks to Owen O’Malley and Arun C. Murthy for explaining the intricacies ofthe MapReduce shuffle to me. Any errors that remain are, of course, to be laid at mydoor.For the second edition, I owe a debt of gratitude for the detailed reviews and feedbackfrom Jeff Bean, Doug Cutting, Glynn Durham, Alan Gates, Jeff Hammerbacher, AlexKozlov, Ken Krugler, Jimmy Lin, Todd Lipcon, Sarah Sproehnle, Vinithra Varadharajan,and Ian Wrigley, as well as all the readers who submitted errata for the first edition.

Iwould also like to thank Aaron Kimball for contributing the chapter on Sqoop, andPhilip (“flip”) Kromer for the case study on graph processing.For the third edition, thanks go to Alejandro Abdelnur, Eva Andreasson, Eli Collins,Doug Cutting, Patrick Hunt, Aaron Kimball, Aaron T. Myers, Brock Noland, ArvindPrabhakar, Ahmed Radwan, and Tom Wheeler for their feedback and suggestions.

RobWeltman kindly gave very detailed feedback for the whole book, which greatly improvedthe final manuscript. Thanks also go to all the readers who submitted errata for thesecond edition.xxiv|PrefaceFor the fourth edition, I would like to thank Jodok Batlogg, Meghan Blanchette, RyanBlue, Jarek Jarcec Cecho, Jules Damji, Dennis Dawson, Matthew Gast, Karthik Kam‐batla, Julien Le Dem, Brock Noland, Sandy Ryza, Akshai Sarma, Ben Spivey, MichaelStack, Kate Ting, Josh Walter, Josh Wills, and Adrian Woodhead for all of their invaluablereview feedback. Ryan Brush, Micah Whitacre, and Matt Massie kindly contributed newcase studies for this edition.

Thanks again to all the readers who submitted errata.I am particularly grateful to Doug Cutting for his encouragement, support, and friend‐ship, and for contributing the Foreword.Thanks also go to the many others with whom I have had conversations or emaildiscussions over the course of writing the book.Halfway through writing the first edition of this book, I joined Cloudera, and I want tothank my colleagues for being incredibly supportive in allowing me the time to writeand to get it finished promptly.I am grateful to my editors, Mike Loukides and Meghan Blanchette, and their colleaguesat O’Reilly for their help in the preparation of this book. Mike and Meghan have beenthere throughout to answer my questions, to read my first drafts, and to keep me onschedule.Finally, the writing of this book has been a great deal of work, and I couldn’t have doneit without the constant support of my family.

My wife, Eliane, not only kept the homegoing, but also stepped in to help review, edit, and chase case studies. My daughters,Emilia and Lottie, have been very understanding, and I’m looking forward to spendinglots more time with all of them.Preface|xxvPART IHadoop FundamentalsCHAPTER 1Meet HadoopIn pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log,they didn’t try to grow a larger ox.

We shouldn’t be trying for bigger computers, but formore systems of computers.—Grace HopperData!We live in the data age. It’s not easy to measure the total volume of data stored elec‐tronically, but an IDC estimate put the size of the “digital universe” at 4.4 zettabytes in2013 and is forecasting a tenfold growth by 2020 to 44 zettabytes.1 A zettabyte is 1021bytes, or equivalently one thousand exabytes, one million petabytes, or one billionterabytes.

That’s more than one disk drive for every person in the world.This flood of data is coming from many sources. Consider the following:2• The New York Stock Exchange generates about 4−5 terabytes of data per day.• Facebook hosts more than 240 billion photos, growing at 7 petabytes per month.• Ancestry.com, the genealogy site, stores around 10 petabytes of data.• The Internet Archive stores around 18.5 petabytes of data.1. These statistics were reported in a study entitled “The Digital Universe of Opportunities: Rich Data and theIncreasing Value of the Internet of Things.”2. All figures are from 2013 or 2014.

For more information, see Tom Groenfeldt, “At NYSE, The Data DelugeOverwhelms Traditional Databases”; Rich Miller, “Facebook Builds Exabyte Data Centers for Cold Stor‐age”; Ancestry.com’s “Company Facts”; Archive.org’s “Petabox”; and the Worldwide LHC Computing Gridproject’s welcome page.3• The Large Hadron Collider near Geneva, Switzerland, produces about 30 petabytesof data per year.So there’s a lot of data out there. But you are probably wondering how it affects you.Most of the data is locked up in the largest web properties (like search engines) or inscientific or financial institutions, isn’t it? Does the advent of big data affect smallerorganizations or individuals?I argue that it does. Take photos, for example.

My wife’s grandfather was an avid pho‐tographer and took photographs throughout his adult life. His entire corpus of mediumformat, slide, and 35mm film, when scanned in at high resolution, occupies around 10gigabytes. Compare this to the digital photos my family took in 2008, which take upabout 5 gigabytes of space. My family is producing photographic data at 35 times therate my wife’s grandfather’s did, and the rate is increasing every year as it becomes easierto take more and more photos.More generally, the digital streams that individuals are producing are growing apace.Microsoft Research’s MyLifeBits project gives a glimpse of the archiving of personalinformation that may become commonplace in the near future.

MyLifeBits was an ex‐periment where an individual’s interactions—phone calls, emails, documents—werecaptured electronically and stored for later access. The data gathered included a phototaken every minute, which resulted in an overall data volume of 1 gigabyte per month.When storage costs come down enough to make it feasible to store continuous audioand video, the data volume for a future MyLifeBits service will be many times that.The trend is for every individual’s data footprint to grow, but perhaps more significantly,the amount of data generated by machines as a part of the Internet of Things will beeven greater than that generated by people. Machine logs, RFID readers, sensor net‐works, vehicle GPS traces, retail transactions—all of these contribute to the growingmountain of data.The volume of data being made publicly available increases every year, too. Organiza‐tions no longer have to merely manage their own data; success in the future will bedictated to a large extent by their ability to extract value from other organizations’ data.Initiatives such as Public Data Sets on Amazon Web Services and Infochimps.org existto foster the “information commons,” where data can be freely (or for a modest price)shared for anyone to download and analyze.