Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 39
Текст из файла (страница 39)
Hint: use “Secondary Sort” on page 262.178|Chapter 6: Developing a MapReduce ApplicationFor a linear chain, the simplest approach is to run each job one after another, waitinguntil a job completes successfully before running the next:JobClient.runJob(conf1);JobClient.runJob(conf2);If a job fails, the runJob() method will throw an IOException, so later jobs in thepipeline don’t get executed. Depending on your application, you might want to catchthe exception and clean up any intermediate data that was produced by any previousjobs.The approach is similar with the new MapReduce API, except you need to examine theBoolean return value of the waitForCompletion() method on Job: true means the jobsucceeded, and false means it failed.For anything more complex than a linear chain, there are libraries that can help or‐chestrate your workflow (although they are also suited to linear chains, or even one-offjobs).
The simplest is in the org.apache.hadoop.mapreduce.jobcontrol package: theJobControl class. (There is an equivalent class in the org.apache.hadoop.mapred.jobcontrol package, too.) An instance of JobControl represents a graph of jobs to be run.You add the job configurations, then tell the JobControl instance the dependenciesbetween jobs.
You run the JobControl in a thread, and it runs the jobs in dependencyorder. You can poll for progress, and when the jobs have finished, you can query for allthe jobs’ statuses and the associated errors for any failures. If a job fails, JobControlwon’t run its dependencies.Apache OozieApache Oozie is a system for running workflows of dependent jobs. It is composed oftwo main parts: a workflow engine that stores and runs workflows composed of differenttypes of Hadoop jobs (MapReduce, Pig, Hive, and so on), and a coordinator engine thatruns workflow jobs based on predefined schedules and data availability.
Oozie has beendesigned to scale, and it can manage the timely execution of thousands of workflows ina Hadoop cluster, each composed of possibly dozens of constituent jobs.Oozie makes rerunning failed workflows more tractable, since no time is wasted runningsuccessful parts of a workflow. Anyone who has managed a complex batch system knowshow difficult it can be to catch up from jobs missed due to downtime or failure, and willappreciate this feature. (Furthermore, coordinator applications representing a singledata pipeline may be packaged into a bundle and run together as a unit.)Unlike JobControl, which runs on the client machine submitting the jobs, Oozie runsas a service in the cluster, and clients submit workflow definitions for immediate or laterexecution.
In Oozie parlance, a workflow is a DAG of action nodes and control-flownodes.MapReduce Workflows|179An action node performs a workflow task, such as moving files in HDFS; running aMapReduce, Streaming, Pig, or Hive job; performing a Sqoop import; or running anarbitrary shell script or Java program. A control-flow node governs the workflow exe‐cution between actions by allowing such constructs as conditional logic (so differentexecution branches may be followed depending on the result of an earlier action node)or parallel execution.
When the workflow completes, Oozie can make an HTTP callbackto the client to inform it of the workflow status. It is also possible to receive callbacksevery time the workflow enters or exits an action node.Defining an Oozie workflowWorkflow definitions are written in XML using the Hadoop Process Definition Lan‐guage, the specification for which can be found on the Oozie website.
Example 6-14shows a simple Oozie workflow definition for running a single MapReduce job.Example 6-14. Oozie workflow definition to run the maximum temperature MapRe‐duce job<workflow-app xmlns="uri:oozie:workflow:0.1" name="max-temp-workflow"><start to="max-temp-mr"/><action name="max-temp-mr"><map-reduce><job-tracker>${resourceManager}</job-tracker><name-node>${nameNode}</name-node><prepare><delete path="${nameNode}/user/${wf:user()}/output"/></prepare><configuration><property><name>mapred.mapper.new-api</name><value>true</value></property><property><name>mapred.reducer.new-api</name><value>true</value></property><property><name>mapreduce.job.map.class</name><value>MaxTemperatureMapper</value></property><property><name>mapreduce.job.combine.class</name><value>MaxTemperatureReducer</value></property><property><name>mapreduce.job.reduce.class</name><value>MaxTemperatureReducer</value></property><property><name>mapreduce.job.output.key.class</name>180|Chapter 6: Developing a MapReduce Application<value>org.apache.hadoop.io.Text</value></property><property><name>mapreduce.job.output.value.class</name><value>org.apache.hadoop.io.IntWritable</value></property><property><name>mapreduce.input.fileinputformat.inputdir</name><value>/user/${wf:user()}/input/ncdc/micro</value></property><property><name>mapreduce.output.fileoutputformat.outputdir</name><value>/user/${wf:user()}/output</value></property></configuration></map-reduce><ok to="end"/><error to="fail"/></action><kill name="fail"><message>MapReduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message></kill><end name="end"/></workflow-app>This workflow has three control-flow nodes and one action node: a start control node,a map-reduce action node, a kill control node, and an end control node.
The nodesand allowed transitions between them are shown in Figure 6-4.Figure 6-4. Transition diagram of an Oozie workflowAll workflows must have one start and one end node. When the workflow job starts,it transitions to the node specified by the start node (the max-temp-mr action in thisexample). A workflow job succeeds when it transitions to the end node. However, if theworkflow job transitions to a kill node, it is considered to have failed and reports theappropriate error message specified by the message element in the workflow definition.MapReduce Workflows|181The bulk of this workflow definition file specifies the map-reduce action.
The first twoelements, job-tracker and name-node, are used to specify the YARN resource manager(or jobtracker in Hadoop 1) to submit the job to and the namenode (actually a Hadoopfilesystem URI) for input and output data. Both are parameterized so that the workflowdefinition is not tied to a particular cluster (which makes it easy to test). The parametersare specified as workflow job properties at submission time, as we shall see later.Despite its name, the job-tracker element is used to specify a YARNresource manager address and port.The optional prepare element runs before the MapReduce job and is used for directorydeletion (and creation, too, if needed, although that is not shown here).
By ensuringthat the output directory is in a consistent state before running a job, Oozie can safelyrerun the action if the job fails.The MapReduce job to run is specified in the configuration element using nestedelements for specifying the Hadoop configuration name-value pairs. You can view theMapReduce configuration section as a declarative replacement for the driver classes thatwe have used elsewhere in this book for running MapReduce programs (such asExample 2-5).We have taken advantage of JSP Expression Language (EL) syntax in several places inthe workflow definition.
Oozie provides a set of functions for interacting with theworkflow. For example, ${wf:user()} returns the name of the user who started thecurrent workflow job, and we use it to specify the correct filesystem path. The Ooziespecification lists all the EL functions that Oozie supports.Packaging and deploying an Oozie workflow applicationA workflow application is made up of the workflow definition plus all the associatedresources (such as MapReduce JAR files, Pig scripts, and so on) needed to run it. Ap‐plications must adhere to a simple directory structure, and are deployed to HDFS sothat they can be accessed by Oozie. For this workflow application, we’ll put all of thefiles in a base directory called max-temp-workflow, as shown diagrammatically here:max-temp-workflow/├── lib/│└── hadoop-examples.jar└── workflow.xmlThe workflow definition file workflow.xml must appear in the top level of this directory.JAR files containing the application’s MapReduce classes are placed in the lib directory.182| Chapter 6: Developing a MapReduce ApplicationWorkflow applications that conform to this layout can be built with any suitable buildtool, such as Ant or Maven; you can find an example in the code that accompanies thisbook.
Once an application has been built, it should be copied to HDFS using regularHadoop tools. Here is the appropriate command for this application:% hadoop fs -put hadoop-examples/target/max-temp-workflow max-temp-workflowRunning an Oozie workflow jobNext, let’s see how to run a workflow job for the application we just uploaded. For thiswe use the oozie command-line tool, a client program for communicating with an Oozieserver. For convenience, we export the OOZIE_URL environment variable to tell the ooziecommand which Oozie server to use (here we’re using one running locally):% export OOZIE_URL="http://localhost:11000/oozie"There are lots of subcommands for the oozie tool (type oozie help to get a list), butwe’re going to call the job subcommand with the -run option to run the workflow job:% oozie job -config ch06-mr-dev/src/main/resources/max-temp-workflow.properties \-runjob: 0000001-140911033236814-oozie-oozi-WThe -config option specifies a local Java properties file containing definitions for theparameters in the workflow XML file (in this case, nameNode and resourceManager), aswell as oozie.wf.application.path, which tells Oozie the location of the workflowapplication in HDFS.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
