Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 34
Текст из файла (страница 34)
It just calls theparser’s parse() method, which parses the fields of interest from a line of input, checkswhether a valid temperature was found using the isValidTemperature() query meth‐od, and, if it was, retrieves the year and the temperature using the getter methods onthe parser. Notice that we check the quality status field as well as checking for missingtemperatures in isValidTemperature(), to filter out poor temperature readings.Another benefit of creating a parser class is that it makes it easy towrite related mappers for similar jobs without duplicating code. It alsogives us the opportunity to write unit tests directly against the pars‐er, for more targeted testing.Example 6-8.
A Mapper that uses a utility class to parse recordspublic class MaxTemperatureMapperextends Mapper<LongWritable, Text, Text, IntWritable> {private NcdcRecordParser parser = new NcdcRecordParser();@Overridepublic void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException {parser.parse(value);if (parser.isValidTemperature()) {Writing a Unit Test with MRUnit|155context.write(new Text(parser.getYear()),new IntWritable(parser.getAirTemperature()));}}}With the tests for the mapper now passing, we move on to writing the reducer.ReducerThe reducer has to find the maximum value for a given key.
Here’s a simple test for thisfeature, which uses a ReduceDriver:@Testpublic void returnsMaximumIntegerInValues() throws IOException,InterruptedException {new ReduceDriver<Text, IntWritable, Text, IntWritable>().withReducer(new MaxTemperatureReducer()).withInput(new Text("1950"),Arrays.asList(new IntWritable(10), new IntWritable(5))).withOutput(new Text("1950"), new IntWritable(10)).runTest();}We construct a list of some IntWritable values and then verify thatMaxTemperatureReducer picks the largest.
The code in Example 6-9 is for an imple‐mentation of MaxTemperatureReducer that passes the test.Example 6-9. Reducer for the maximum temperature examplepublic class MaxTemperatureReducerextends Reducer<Text, IntWritable, Text, IntWritable> {@Overridepublic void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException {int maxValue = Integer.MIN_VALUE;for (IntWritable value : values) {maxValue = Math.max(maxValue, value.get());}context.write(key, new IntWritable(maxValue));}}Running Locally on Test DataNow that we have the mapper and reducer working on controlled inputs, the next stepis to write a job driver and run it on some test data on a development machine.156|Chapter 6: Developing a MapReduce ApplicationRunning a Job in a Local Job RunnerUsing the Tool interface introduced earlier in the chapter, it’s easy to write a driver torun our MapReduce job for finding the maximum temperature by year (seeMaxTemperatureDriver in Example 6-10).Example 6-10.
Application to find the maximum temperaturepublic class MaxTemperatureDriver extends Configured implements Tool {@Overridepublic int run(String[] args) throws Exception {if (args.length != 2) {System.err.printf("Usage: %s [generic options] <input> <output>\n",getClass().getSimpleName());ToolRunner.printGenericCommandUsage(System.err);return -1;}Job job = new Job(getConf(), "Max temperature");job.setJarByClass(getClass());FileInputFormat.addInputPath(job, new Path(args[0]));FileOutputFormat.setOutputPath(job, new Path(args[1]));job.setMapperClass(MaxTemperatureMapper.class);job.setCombinerClass(MaxTemperatureReducer.class);job.setReducerClass(MaxTemperatureReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);return job.waitForCompletion(true) ? 0 : 1;}public static void main(String[] args) throws Exception {int exitCode = ToolRunner.run(new MaxTemperatureDriver(), args);System.exit(exitCode);}}MaxTemperatureDriver implements the Tool interface, so we get the benefit of beingable to set the options that GenericOptionsParser supports.
The run() method con‐structs a Job object based on the tool’s configuration, which it uses to launch a job.Among the possible job configuration parameters, we set the input and output file paths;the mapper, reducer, and combiner classes; and the output types (the input types aredetermined by the input format, which defaults to TextInputFormat and has LongWritable keys and Text values). It’s also a good idea to set a name for the job (Max temperature) so that you can pick it out in the job list during execution and after it hasRunning Locally on Test Data|157completed. By default, the name is the name of the JAR file, which normally is notparticularly descriptive.Now we can run this application against some local files. Hadoop comes with a localjob runner, a cut-down version of the MapReduce execution engine for running Map‐Reduce jobs in a single JVM.
It’s designed for testing and is very convenient for use inan IDE, since you can run it in a debugger to step through the code in your mapper andreducer.The local job runner is used if mapreduce.framework.name is set to local, which is thedefault.1From the command line, we can run the driver by typing:% mvn compile% export HADOOP_CLASSPATH=target/classes/% hadoop v2.MaxTemperatureDriver -conf conf/hadoop-local.xml \input/ncdc/micro outputEquivalently, we could use the -fs and -jt options provided by GenericOptionsParser:% hadoop v2.MaxTemperatureDriver -fs file:/// -jt local input/ncdc/micro outputThis command executes MaxTemperatureDriver using input from the local input/ncdc/micro directory, producing output in the local output directory.
Note that although we’veset -fs so we use the local filesystem (file:///), the local job runner will actually workfine against any filesystem, including HDFS (and it can be handy to do this if you havea few files that are on HDFS).We can examine the output on the local filesystem:% cat output/part-r-000001949111195022Testing the DriverApart from the flexible configuration options offered by making your application im‐plement Tool, you also make it more testable because it allows you to inject an arbitraryConfiguration. You can take advantage of this to write a test that uses a local job runnerto run a job against known input data, which checks that the output is as expected.There are two approaches to doing this.
The first is to use the local job runner and runthe job against a test file on the local filesystem. The code in Example 6-11 gives an ideaof how to do this.1. In Hadoop 1, mapred.job.tracker determines the means of execution: local for the local job runner, ora colon-separated host and port pair for a jobtracker address.158| Chapter 6: Developing a MapReduce ApplicationExample 6-11.
A test for MaxTemperatureDriver that uses a local, in-process jobrunner@Testpublic void test() throws Exception {Configuration conf = new Configuration();conf.set("fs.defaultFS", "file:///");conf.set("mapreduce.framework.name", "local");conf.setInt("mapreduce.task.io.sort.mb", 1);Path input = new Path("input/ncdc/micro");Path output = new Path("output");FileSystem fs = FileSystem.getLocal(conf);fs.delete(output, true); // delete old outputMaxTemperatureDriver driver = new MaxTemperatureDriver();driver.setConf(conf);int exitCode = driver.run(new String[] {input.toString(), output.toString() });assertThat(exitCode, is(0));checkOutput(conf, output);}The test explicitly sets fs.defaultFS and mapreduce.framework.name so it uses thelocal filesystem and the local job runner.
It then runs the MaxTemperatureDriver via itsTool interface against a small amount of known data. At the end of the test, the checkOutput() method is called to compare the actual output with the expected output, lineby line.The second way of testing the driver is to run it using a “mini-” cluster. Hadoop has aset of testing classes, called MiniDFSCluster, MiniMRCluster, and MiniYARNCluster,that provide a programmatic way of creating in-process clusters.
Unlike the local jobrunner, these allow testing against the full HDFS, MapReduce, and YARN machinery.Bear in mind, too, that node managers in a mini-cluster launch separate JVMs to runtasks in, which can make debugging more difficult.You can run a mini-cluster from the command line too, with thefollowing:% hadoop jar \$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-*-tests.jar \miniclusterMini-clusters are used extensively in Hadoop’s own automated test suite, but they canbe used for testing user code, too.
Hadoop’s ClusterMapReduceTestCase abstract classprovides a useful base for writing such a test, handles the details of starting and stoppingRunning Locally on Test Data|159the in-process HDFS and YARN clusters in its setUp() and tearDown() methods, andgenerates a suitable Configuration object that is set up to work with them. Subclassesneed only populate data in HDFS (perhaps by copying from a local file), run a MapRe‐duce job, and confirm the output is as expected. Refer to the MaxTemperatureDriverMiniTest class in the example code that comes with this book for the listing.Tests like this serve as regression tests, and are a useful repository of input edge casesand their expected results. As you encounter more test cases, you can simply add themto the input file and update the file of expected output accordingly.Running on a ClusterNow that we are happy with the program running on a small test dataset, we are readyto try it on the full dataset on a Hadoop cluster.
Chapter 10 covers how to set up a fullydistributed cluster, although you can also work through this section on a pseudodistributed cluster.Packaging a JobThe local job runner uses a single JVM to run a job, so as long as all the classes that yourjob needs are on its classpath, then things will just work.In a distributed setting, things are a little more complex. For a start, a job’s classes mustbe packaged into a job JAR file to send to the cluster. Hadoop will find the job JARautomatically by searching for the JAR on the driver’s classpath that contains the classset in the setJarByClass() method (on JobConf or Job). Alternatively, if you want toset an explicit JAR file by its file path, you can use the setJar() method.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
