Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 35
Текст из файла (страница 35)
(The JAR filepath may be local or an HDFS file path.)Creating a job JAR file is conveniently achieved using a build tool such as Ant or Maven.Given the POM in Example 6-3, the following Maven command will create a JAR filecalled hadoop-examples.jar in the project directory containing all of the compiledclasses:% mvn package -DskipTestsIf you have a single job per JAR, you can specify the main class to run in the JAR file’smanifest. If the main class is not in the manifest, it must be specified on the commandline (as we will see shortly when we run the job).Any dependent JAR files can be packaged in a lib subdirectory in the job JAR file, al‐though there are other ways to include dependencies, discussed later.
Similarly, resourcefiles can be packaged in a classes subdirectory. (This is analogous to a Java Web appli‐cation archive, or WAR, file, except in that case the JAR files go in a WEB-INF/libsubdirectory and classes go in a WEB-INF/classes subdirectory in the WAR file.)160|Chapter 6: Developing a MapReduce ApplicationThe client classpathThe user’s client-side classpath set by hadoop jar <jar> is made up of:• The job JAR file• Any JAR files in the lib directory of the job JAR file, and the classes directory (ifpresent)• The classpath defined by HADOOP_CLASSPATH, if setIncidentally, this explains why you have to set HADOOP_CLASSPATH to point to dependentclasses and libraries if you are running using the local job runner without a job JAR(hadoop CLASSNAME).The task classpathOn a cluster (and this includes pseudodistributed mode), map and reduce tasks run inseparate JVMs, and their classpaths are not controlled by HADOOP_CLASSPATH.HADOOP_CLASSPATH is a client-side setting and only sets the classpath for the driver JVM,which submits the job.Instead, the user’s task classpath is comprised of the following:• The job JAR file• Any JAR files contained in the lib directory of the job JAR file, and the classesdirectory (if present)• Any files added to the distributed cache using the -libjars option (see Table 6-1),or the addFileToClassPath() method on DistributedCache (old API), or Job(new API)Packaging dependenciesGiven these different ways of controlling what is on the client and task classpaths, thereare corresponding options for including library dependencies for a job:• Unpack the libraries and repackage them in the job JAR.• Package the libraries in the lib directory of the job JAR.• Keep the libraries separate from the job JAR, and add them to the client classpathvia HADOOP_CLASSPATH and to the task classpath via -libjars.The last option, using the distributed cache, is simplest from a build point of viewbecause dependencies don’t need rebundling in the job JAR.
Also, using the distributedcache can mean fewer transfers of JAR files around the cluster, since files may be cachedon a node between tasks. (You can read more about the distributed cache on page 274.)Running on a Cluster|161Task classpath precedenceUser JAR files are added to the end of both the client classpath and the task classpath,which in some cases can cause a dependency conflict with Hadoop’s built-in libraries ifHadoop uses a different, incompatible version of a library that your code uses. Some‐times you need to be able to control the task classpath order so that your classes arepicked up first.
On the client side, you can force Hadoop to put the user classpath firstin the search order by setting the HADOOP_USER_CLASSPATH_FIRST environment variableto true. For the task classpath, you can set mapreduce.job.user.classpath.first totrue. Note that by setting these options you change the class loading for Hadoopframework dependencies (but only in your job), which could potentially cause the jobsubmission or task to fail, so use these options with caution.Launching a JobTo launch the job, we need to run the driver, specifying the cluster that we want to runthe job on with the -conf option (we equally could have used the -fs and -jt options):% unset HADOOP_CLASSPATH% hadoop jar hadoop-examples.jar v2.MaxTemperatureDriver \-conf conf/hadoop-cluster.xml input/ncdc/all max-tempWe unset the HADOOP_CLASSPATH environment variable because wedon’t have any third-party dependencies for this job. If it were leftset to target/classes/ (from earlier in the chapter), Hadoopwouldn’t be able to find the job JAR; it would load the MaxTemperatureDriver class from target/classes rather than the JAR, and the jobwould fail.The waitForCompletion() method on Job launches the job and polls for progress,writing a line summarizing the map and reduce’s progress whenever either changes.Here’s the output (some lines have been removed for clarity):14/09/12 06:38:11 INFO input.FileInputFormat: Total input paths to process : 10114/09/12 06:38:11 INFO impl.YarnClientImpl: Submitted applicationapplication_1410450250506_000314/09/12 06:38:12 INFO mapreduce.Job: Running job: job_1410450250506_000314/09/12 06:38:26 INFO mapreduce.Job: map 0% reduce 0%...14/09/12 06:45:24 INFO mapreduce.Job: map 100% reduce 100%14/09/12 06:45:24 INFO mapreduce.Job: Job job_1410450250506_0003 completedsuccessfully14/09/12 06:45:24 INFO mapreduce.Job: Counters: 49File System CountersFILE: Number of bytes read=93995FILE: Number of bytes written=10273563FILE: Number of read operations=0FILE: Number of large read operations=0162|Chapter 6: Developing a MapReduce ApplicationFILE: Number of write operations=0HDFS: Number of bytes read=33485855415HDFS: Number of bytes written=904HDFS: Number of read operations=327HDFS: Number of large read operations=0HDFS: Number of write operations=16Job CountersLaunched map tasks=101Launched reduce tasks=8Data-local map tasks=101Total time spent by all maps in occupied slots (ms)=5954495Total time spent by all reduces in occupied slots (ms)=74934Total time spent by all map tasks (ms)=5954495Total time spent by all reduce tasks (ms)=74934Total vcore-seconds taken by all map tasks=5954495Total vcore-seconds taken by all reduce tasks=74934Total megabyte-seconds taken by all map tasks=6097402880Total megabyte-seconds taken by all reduce tasks=76732416Map-Reduce FrameworkMap input records=1209901509Map output records=1143764653Map output bytes=10293881877Map output materialized bytes=14193Input split bytes=14140Combine input records=1143764772Combine output records=234Reduce input groups=100Reduce shuffle bytes=14193Reduce input records=115Reduce output records=100Spilled Records=379Shuffled Maps =808Failed Shuffles=0Merged Map outputs=808GC time elapsed (ms)=101080CPU time spent (ms)=5113180Physical memory (bytes) snapshot=60509106176Virtual memory (bytes) snapshot=167657209856Total committed heap usage (bytes)=68220878848Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format CountersBytes Read=33485841275File Output Format CountersBytes Written=90Running on a Cluster|163The output includes more useful information.
Before the job starts, its ID is printed;this is needed whenever you want to refer to the job—in logfiles, for example—or wheninterrogating it via the mapred job command. When the job is complete, its statistics(known as counters) are printed out. These are very useful for confirming that the jobdid what you expected. For example, for this job, we can see that 1.2 billion records wereanalyzed (“Map input records”), read from around 34 GB of compressed files on HDFS(“HDFS: Number of bytes read”). The input was broken into 101 gzipped files of rea‐sonable size, so there was no problem with not being able to split them.You can find out more about what the counters mean in “Built-in Counters” on page 247.Job, Task, and Task Attempt IDsIn Hadoop 2, MapReduce job IDs are generated from YARN application IDs that arecreated by the YARN resource manager.
The format of an application ID is composedof the time that the resource manager (not the application) started and an incrementingcounter maintained by the resource manager to uniquely identify the application to thatinstance of the resource manager. So the application with this ID:application_1410450250506_0003is the third (0003; application IDs are 1-based) application run by the resource manager,which started at the time represented by the timestamp 1410450250506. The counter isformatted with leading zeros to make IDs sort nicely—in directory listings, for example.However, when the counter reaches 10000, it is not reset, resulting in longer applicationIDs (which don’t sort so well).The corresponding job ID is created simply by replacing the application prefix of anapplication ID with a job prefix:job_1410450250506_0003Tasks belong to a job, and their IDs are formed by replacing the job prefix of a job IDwith a task prefix and adding a suffix to identify the task within the job.
For example:task_1410450250506_0003_m_000003is the fourth (000003; task IDs are 0-based) map (m) task of the job with IDjob_1410450250506_0003. The task IDs are created for a job when it is initialized, sothey do not necessarily dictate the order in which the tasks will be executed.Tasks may be executed more than once, due to failure (see “Task Failure” on page 193) orspeculative execution (see “Speculative Execution” on page 204), so to identify differentinstances of a task execution, task attempts are given unique IDs. For example:attempt_1410450250506_0003_m_000003_0164|Chapter 6: Developing a MapReduce Applicationis the first (0; attempt IDs are 0-based) attempt at running tasktask_1410450250506_0003_m_000003. Task attempts are allocated during the job runas needed, so their ordering represents the order in which they were created to run.The MapReduce Web UIHadoop comes with a web UI for viewing information about your jobs.
It is useful forfollowing a job’s progress while it is running, as well as finding job statistics and logsafter the job has completed. You can find the UI at http://resource-manager-host:8088/.The resource manager pageA screenshot of the home page is shown in Figure 6-1. The “Cluster Metrics” sectiongives a summary of the cluster. This includes the number of applications currently run‐ning on the cluster (and in various other states), the number of resources available onthe cluster (“Memory Total”), and information about node managers.Figure 6-1.
Screenshot of the resource manager pageThe main table shows all the applications that have run or are currently running on thecluster. There is a search box that is useful for filtering the applications to find the onesyou are interested in. The main view can show up to 100 entries per page, and theresource manager will keep up to 10,000 completed applications in memory at a time(set by yarn.resourcemanager.max-completed-applications), before they are onlyavailable from the job history page.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
