Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (Tom White - Hadoop The Definitive Guide_ 4 edition - 2015.pdf), страница 7

Help was at hand with the publication of a paper in 2003 that describedthe architecture of Google’s distributed filesystem, called GFS, which was being used in8. In this book, we use the lowercase form, “namenode,” to denote the entity when it’s being referred to generally,and the CamelCase form NameNode to denote the Java class that implements it.9.

See Mike Cafarella and Doug Cutting, “Building Nutch: Open Source Search,” ACM Queue, April 2004.12|Chapter 1: Meet Hadoopproduction at Google.10 GFS, or something like it, would solve their storage needs forthe very large files generated as a part of the web crawl and indexing process. In par‐ticular, GFS would free up time being spent on administrative tasks such as managingstorage nodes.

In 2004, Nutch’s developers set about writing an open source implemen‐tation, the Nutch Distributed Filesystem (NDFS).In 2004, Google published the paper that introduced MapReduce to the world.11 Earlyin 2005, the Nutch developers had a working MapReduce implementation in Nutch,and by the middle of that year all the major Nutch algorithms had been ported to runusing MapReduce and NDFS.NDFS and the MapReduce implementation in Nutch were applicable beyond the realmof search, and in February 2006 they moved out of Nutch to form an independentsubproject of Lucene called Hadoop.

At around the same time, Doug Cutting joinedYahoo!, which provided a dedicated team and the resources to turn Hadoop into asystem that ran at web scale (see the following sidebar). This was demonstrated in Feb‐ruary 2008 when Yahoo! announced that its production search index was being gener‐ated by a 10,000-core Hadoop cluster.12Hadoop at Yahoo!Building Internet-scale search engines requires huge amounts of data and therefore largenumbers of machines to process it. Yahoo! Search consists of four primary components:the Crawler, which downloads pages from web servers; the WebMap, which builds agraph of the known Web; the Indexer, which builds a reverse index to the best pages;and the Runtime, which answers users’ queries.

The WebMap is a graph that consists ofroughly 1 trillion (1012) edges, each representing a web link, and 100 billion (1011) nodes,each representing distinct URLs. Creating and analyzing such a large graph requires alarge number of computers running for many days. In early 2005, the infrastructure forthe WebMap, named Dreadnaught, needed to be redesigned to scale up to more nodes.Dreadnaught had successfully scaled from 20 to 600 nodes, but required a completeredesign to scale out further.

Dreadnaught is similar to MapReduce in many ways, butprovides more flexibility and less structure. In particular, each fragment in a Dread‐naught job could send output to each of the fragments in the next stage of the job, butthe sort was all done in library code. In practice, most of the WebMap phases were pairsthat corresponded to MapReduce.

Therefore, the WebMap applications would not re‐quire extensive refactoring to fit into MapReduce.10. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, “The Google File System,” October 2003.11. Jeffrey Dean and Sanjay Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” December2004.12. “Yahoo! Launches World’s Largest Hadoop Production Application,” February 19, 2008.A Brief History of Apache Hadoop|13Eric Baldeschwieler (aka Eric14) created a small team, and we started designing andprototyping a new framework, written in C++ modeled and after GFS and MapReduce,to replace Dreadnaught. Although the immediate need was for a new framework forWebMap, it was clear that standardization of the batch platform across Yahoo! Searchwas critical and that by making the framework general enough to support other users,we could better leverage investment in the new platform.At the same time, we were watching Hadoop, which was part of Nutch, and its progress.In January 2006, Yahoo! hired Doug Cutting, and a month later we decided to abandonour prototype and adopt Hadoop.

The advantage of Hadoop over our prototype anddesign was that it was already working with a real application (Nutch) on 20 nodes. Thatallowed us to bring up a research cluster two months later and start helping real cus‐tomers use the new framework much sooner than we could have otherwise. Anotheradvantage, of course, was that since Hadoop was already open source, it was easier(although far from easy!) to get permission from Yahoo!’s legal department to work inopen source. So, we set up a 200-node cluster for the researchers in early 2006 and putthe WebMap conversion plans on hold while we supported and improved Hadoop forthe research users.—Owen O’Malley, 2009In January 2008, Hadoop was made its own top-level project at Apache, confirming itssuccess and its diverse, active community. By this time, Hadoop was being used by manyother companies besides Yahoo!, such as Last.fm, Facebook, and the New York Times.In one well-publicized feat, the New York Times used Amazon’s EC2 compute cloud tocrunch through 4 terabytes of scanned archives from the paper, converting them toPDFs for the Web.13 The processing took less than 24 hours to run using 100 machines,and the project probably wouldn’t have been embarked upon without the combinationof Amazon’s pay-by-the-hour model (which allowed the NYT to access a large numberof machines for a short period) and Hadoop’s easy-to-use parallel programming model.In April 2008, Hadoop broke a world record to become the fastest system to sort anentire terabyte of data.

Running on a 910-node cluster, Hadoop sorted 1 terabyte in 209seconds (just under 3.5 minutes), beating the previous year’s winner of 297 seconds.14In November of the same year, Google reported that its MapReduce implementationsorted 1 terabyte in 68 seconds.15 Then, in April 2009, it was announced that a team atYahoo! had used Hadoop to sort 1 terabyte in 62 seconds.1613.

Derek Gottfrid, “Self-Service, Prorated Super Computing Fun!” November 1, 2007.14. Owen O’Malley, “TeraByte Sort on Apache Hadoop,” May 2008.15. Grzegorz Czajkowski, “Sorting 1PB with MapReduce,” November 21, 2008.16. Owen O’Malley and Arun C. Murthy, “Winning a 60 Second Dash with a Yellow Elephant,” April 2009.14|Chapter 1: Meet HadoopThe trend since then has been to sort even larger volumes of data at ever faster rates.

Inthe 2014 competition, a team from Databricks were joint winners of the Gray Sortbenchmark. They used a 207-node Spark cluster to sort 100 terabytes of data in 1,406seconds, a rate of 4.27 terabytes per minute.17Today, Hadoop is widely used in mainstream enterprises. Hadoop’s role as a generalpurpose storage and analysis platform for big data has been recognized by the industry,and this fact is reflected in the number of products that use or incorporate Hadoop insome way. Commercial Hadoop support is available from large, established enterprisevendors, including EMC, IBM, Microsoft, and Oracle, as well as from specialist Hadoopcompanies such as Cloudera, Hortonworks, and MapR.What’s in This Book?The book is divided into five main parts: Parts I to III are about core Hadoop, Part IVcovers related projects in the Hadoop ecosystem, and Part V contains Hadoop casestudies. You can read the book from cover to cover, but there are alternative pathwaysthrough the book that allow you to skip chapters that aren’t needed to read later ones.See Figure 1-1.Part I is made up of five chapters that cover the fundamental components in Hadoopand should be read before tackling later chapters.

Chapter 1 (this chapter) is a high-levelintroduction to Hadoop. Chapter 2 provides an introduction to MapReduce. Chap‐ter 3 looks at Hadoop filesystems, and in particular HDFS, in depth. Chapter 4 discussesYARN, Hadoop’s cluster resource management system. Chapter 5 covers the I/O build‐ing blocks in Hadoop: data integrity, compression, serialization, and file-based datastructures.Part II has four chapters that cover MapReduce in depth.

They provide useful under‐standing for later chapters (such as the data processing chapters in Part IV), but couldbe skipped on a first reading. Chapter 6 goes through the practical steps needed todevelop a MapReduce application. Chapter 7 looks at how MapReduce is implementedin Hadoop, from the point of view of a user. Chapter 8 is about the MapReduce pro‐gramming model and the various data formats that MapReduce can work with. Chap‐ter 9 is on advanced MapReduce topics, including sorting and joining data.Part III concerns the administration of Hadoop: Chapters 10 and 11 describe how toset up and maintain a Hadoop cluster running HDFS and MapReduce on YARN.Part IV of the book is dedicated to projects that build on Hadoop or are closely relatedto it. Each chapter covers one project and is largely independent of the other chaptersin this part, so they can be read in any order.17.