Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (Tom White - Hadoop The Definitive Guide_ 4 edition - 2015.pdf), страница 3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643From CPUs to Semantic IntegrationEnter Apache CrunchBuilding a Complete PictureIntegrating Healthcare DataComposability over FrameworksMoving Forward64364464464765065123. Biological Data Science: Saving Lives with Software. . . . . . . . . . . .

. . . . . . . . . . . . . . . . 653The Structure of DNAThe Genetic Code: Turning DNA Letters into ProteinsThinking of DNA as Source CodeThe Human Genome Project and Reference GenomesSequencing and Aligning DNAADAM, A Scalable Genome Analysis PlatformLiterate programming with the Avro interface description language (IDL)Column-oriented access with ParquetA simple example: k-mer counting using Spark and ADAMFrom Personalized Ads to Personalized MedicineJoin In65565665765966066166266366566766824.

Cascading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669Fields, Tuples, and PipesOperationsTaps, Schemes, and FlowsCascading in PracticeFlexibilityHadoop and Cascading at ShareThisSummary670673675676679680684A. Installing Apache Hadoop. . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685B. Cloudera’s Distribution Including Apache Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 691C. Preparing the NCDC Weather Data. . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693D. The Old and New Java MapReduce APIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . 701xvi|Table of ContentsForewordHadoop got its start in Nutch. A few of us were attempting to build an open source websearch engine and having trouble managing computations running on even a handfulof computers. Once Google published its GFS and MapReduce papers, the route becameclear. They’d devised systems to solve precisely the problems we were having with Nutch.So we started, two of us, half-time, to try to re-create these systems as a part of Nutch.We managed to get Nutch limping along on 20 machines, but it soon became clear thatto handle the Web’s massive scale, we’d need to run it on thousands of machines, andmoreover, that the job was bigger than two half-time developers could handle.Around that time, Yahoo! got interested, and quickly put together a team that I joined.We split off the distributed computing part of Nutch, naming it Hadoop.

With the helpof Yahoo!, Hadoop soon grew into a technology that could truly scale to the Web.In 2006, Tom White started contributing to Hadoop. I already knew Tom through anexcellent article he’d written about Nutch, so I knew he could present complex ideas inclear prose.

I soon learned that he could also develop software that was as pleasant toread as his prose.From the beginning, Tom’s contributions to Hadoop showed his concern for users andfor the project. Unlike most open source contributors, Tom is not primarily interestedin tweaking the system to better meet his own needs, but rather in making it easier foranyone to use.Initially, Tom specialized in making Hadoop run well on Amazon’s EC2 and S3 services.Then he moved on to tackle a wide variety of problems, including improving the Map‐Reduce APIs, enhancing the website, and devising an object serialization framework.In all cases, Tom presented his ideas precisely.

In short order, Tom earned the role ofHadoop committer and soon thereafter became a member of the Hadoop Project Man‐agement Committee.xviiTom is now a respected senior member of the Hadoop developer community. Thoughhe’s an expert in many technical corners of the project, his specialty is making Hadoopeasier to use and understand.Given this, I was very pleased when I learned that Tom intended to write a book aboutHadoop. Who could be better qualified? Now you have the opportunity to learn aboutHadoop from a master—not only of the technology, but also of common sense andplain talk.—Doug Cutting, April 2009Shed in the Yard, Californiaxviii|ForewordPrefaceMartin Gardner, the mathematics and science writer, once said in an interview:Beyond calculus, I am lost. That was the secret of my column’s success. It took me so longto understand what I was writing about that I knew how to write in a way most readerswould understand.1In many ways, this is how I feel about Hadoop.

Its inner workings are complex, restingas they do on a mixture of distributed systems theory, practical engineering, and com‐mon sense. And to the uninitiated, Hadoop can appear alien.But it doesn’t need to be like this. Stripped to its core, the tools that Hadoop providesfor working with big data are simple. If there’s a common theme, it is about raising thelevel of abstraction—to create building blocks for programmers who have lots of datato store and analyze, and who don’t have the time, the skill, or the inclination to becomedistributed systems experts to build the infrastructure to handle it.With such a simple and generally applicable feature set, it seemed obvious to me whenI started using it that Hadoop deserved to be widely used. However, at the time (in early2006), setting up, configuring, and writing programs to use Hadoop was an art.

Thingshave certainly improved since then: there is more documentation, there are more ex‐amples, and there are thriving mailing lists to go to when you have questions. And yetthe biggest hurdle for newcomers is understanding what this technology is capable of,where it excels, and how to use it. That is why I wrote this book.The Apache Hadoop community has come a long way.

Since the publication of the firstedition of this book, the Hadoop project has blossomed. “Big data” has become a house‐hold term. 2 In this time, the software has made great leaps in adoption, performance,reliability, scalability, and manageability. The number of things being built and run onthe Hadoop platform has grown enormously. In fact, it’s difficult for one person to keep1. Alex Bellos, “The science of fun,” The Guardian, May 31, 2008.2. It was added to the Oxford English Dictionary in 2013.xixtrack.

To gain even wider adoption, I believe we need to make Hadoop even easier touse. This will involve writing more tools; integrating with even more systems; and writ‐ing new, improved APIs. I’m looking forward to being a part of this, and I hope thisbook will encourage and enable others to do so, too.Administrative NotesDuring discussion of a particular Java class in the text, I often omit its package name toreduce clutter.

If you need to know which package a class is in, you can easily look it upin the Java API documentation for Hadoop (linked to from the Apache Hadoop homepage), or the relevant project. Or if you’re using an integrated development environment(IDE), its auto-complete mechanism can help find what you’re looking for.Similarly, although it deviates from usual style guidelines, program listings that importmultiple classes from the same package may use the asterisk wildcard character to savespace (for example, import org.apache.hadoop.io.*).The sample programs in this book are available for download from the book’s website.You will also find instructions there for obtaining the datasets that are used in examplesthroughout the book, as well as further notes for running the programs in the book andlinks to updates, additional resources, and my blog.What’s New in the Fourth Edition?The fourth edition covers Hadoop 2 exclusively. The Hadoop 2 release series is thecurrent active release series and contains the most stable versions of Hadoop.There are new chapters covering YARN (Chapter 4), Parquet (Chapter 13), Flume(Chapter 14), Crunch (Chapter 18), and Spark (Chapter 19).

There’s also a new sectionto help readers navigate different pathways through the book (“What’s in This Book?”on page 15).This edition includes two new case studies (Chapters 22 and 23): one on how Hadoopis used in healthcare systems, and another on using Hadoop technologies for genomicsdata processing. Case studies from the previous editions can now be found online.Many corrections, updates, and improvements have been made to existing chapters tobring them up to date with the latest releases of Hadoop and its related projects.What’s New in the Third Edition?The third edition covers the 1.x (formerly 0.20) release series of Apache Hadoop, as wellas the newer 0.22 and 2.x (formerly 0.23) series. With a few exceptions, which are notedin the text, all the examples in this book run against these versions.xx|PrefaceThis edition uses the new MapReduce API for most of the examples.

Because the oldAPI is still in widespread use, it continues to be discussed in the text alongside the newAPI, and the equivalent code using the old API can be found on the book’s website.The major change in Hadoop 2.0 is the new MapReduce runtime, MapReduce 2, whichis built on a new distributed resource management system called YARN.

This editionincludes new sections covering MapReduce on YARN: how it works (Chapter 7) andhow to run it (Chapter 10).There is more MapReduce material, too, including development practices such as pack‐aging MapReduce jobs with Maven, setting the user’s Java classpath, and writing testswith MRUnit (all in Chapter 6). In addition, there is more depth on features such asoutput committers and the distributed cache (both in Chapter 9), as well as task memorymonitoring (Chapter 10). There is a new section on writing MapReduce jobs to processAvro data (Chapter 12), and one on running a simple MapReduce workflow in Oozie(Chapter 6).The chapter on HDFS (Chapter 3) now has introductions to high availability, federation,and the new WebHDFS and HttpFS filesystems.The chapters on Pig, Hive, Sqoop, and ZooKeeper have all been expanded to cover thenew features and changes in their latest releases.In addition, numerous corrections and improvements have been made throughout thebook.What’s New in the Second Edition?The second edition has two new chapters on Sqoop and Hive (Chapters 15 and 17,respectively), a new section covering Avro (in Chapter 12), an introduction to the newsecurity features in Hadoop (in Chapter 10), and a new case study on analyzing massivenetwork graphs using Hadoop.This edition continues to describe the 0.20 release series of Apache Hadoop, becausethis was the latest stable release at the time of writing.