Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 75
Текст из файла (страница 75)
For an example that uses the Specific mapping with generated classes, see the AvroSpecificMaxTemperature class in the example code.362|Chapter 12: Avro(such as SequenceFiles). See the documentation for the Avro MapReduce package fordetails.Sorting Using Avro MapReduceIn this section, we use Avro’s sort capabilities and combine them with MapReduce towrite a program to sort an Avro datafile (Example 12-3).Example 12-3. A MapReduce program to sort an Avro datafilepublic class AvroSort extends Configured implements Tool {static class SortMapper<K> extends Mapper<AvroKey<K>, NullWritable,AvroKey<K>, AvroValue<K>> {@Overrideprotected void map(AvroKey<K> key, NullWritable value,Context context) throws IOException, InterruptedException {context.write(key, new AvroValue<K>(key.datum()));}}static class SortReducer<K> extends Reducer<AvroKey<K>, AvroValue<K>,AvroKey<K>, NullWritable> {@Overrideprotected void reduce(AvroKey<K> key, Iterable<AvroValue<K>> values,Context context) throws IOException, InterruptedException {for (AvroValue<K> value : values) {context.write(new AvroKey(value.datum()), NullWritable.get());}}}@Overridepublic int run(String[] args) throws Exception {if (args.length != 3) {System.err.printf("Usage: %s [generic options] <input> <output> <schema-file>\n",getClass().getSimpleName());ToolRunner.printGenericCommandUsage(System.err);return -1;}String input = args[0];String output = args[1];String schemaFile = args[2];Job job = new Job(getConf(), "Avro sort");job.setJarByClass(getClass());job.getConfiguration().setBoolean(Sorting Using Avro MapReduce|363Job.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, true);FileInputFormat.addInputPath(job, new Path(input));FileOutputFormat.setOutputPath(job, new Path(output));AvroJob.setDataModelClass(job, GenericData.class);Schema schema = new Schema.Parser().parse(new File(schemaFile));AvroJob.setInputKeySchema(job, schema);AvroJob.setMapOutputKeySchema(job, schema);AvroJob.setMapOutputValueSchema(job, schema);AvroJob.setOutputKeySchema(job, schema);job.setInputFormatClass(AvroKeyInputFormat.class);job.setOutputFormatClass(AvroKeyOutputFormat.class);job.setOutputKeyClass(AvroKey.class);job.setOutputValueClass(NullWritable.class);job.setMapperClass(SortMapper.class);job.setReducerClass(SortReducer.class);return job.waitForCompletion(true) ? 0 : 1;}public static void main(String[] args) throws Exception {int exitCode = ToolRunner.run(new AvroSort(), args);System.exit(exitCode);}}This program (which uses the Generic Avro mapping and hence does not require anycode generation) can sort Avro records of any type, represented in Java by the generictype parameter K.
We choose a value that is the same as the key, so that when the valuesare grouped by key we can emit all of the values in the case that more than one of themshare the same key (according to the sorting function). This means we don’t lose anyrecords.7 The mapper simply emits the input key wrapped in an AvroKey and an AvroValue. The reducer acts as an identity, passing the values through as output keys, whichwill get written to an Avro datafile.The sorting happens in the MapReduce shuffle, and the sort function is determined bythe Avro schema that is passed to the program.
Let’s use the program to sort the pairs.avro file created earlier, using the SortedStringPair.avsc schema to sort by the right fieldin descending order. First, we inspect the input using the Avro tools JAR:7. If we had used the identity mapper and reducer here, the program would sort and remove duplicate keys atthe same time. We encounter this idea of duplicating information from the key in the value object again in“Secondary Sort” on page 262.364|Chapter 12: Avro% java -jar $AVRO_HOME/avro-tools-*.jar tojson input/avro/pairs.avro{"left":"a","right":"1"}{"left":"c","right":"2"}{"left":"b","right":"3"}{"left":"b","right":"2"}Then we run the sort:% hadoop jar avro-examples.jar AvroSort input/avro/pairs.avro output \ch12-avro/src/main/resources/SortedStringPair.avscFinally, we inspect the output and see that it is sorted correctly:% java -jar $AVRO_HOME/avro-tools-*.jar tojson output/part-r-00000.avro{"left":"b","right":"3"}{"left":"b","right":"2"}{"left":"c","right":"2"}{"left":"a","right":"1"}Avro in Other LanguagesFor languages and frameworks other than Java, there are a few choices for working withAvro data.AvroAsTextInputFormat is designed to allow Hadoop Streaming programs to read Avrodatafiles.
Each datum in the file is converted to a string, which is the JSON representationof the datum, or just to the raw bytes if the type is Avro bytes. Going the other way, youcan specify AvroTextOutputFormat as the output format of a Streaming job to createAvro datafiles with a bytes schema, where each datum is the tab-delimited key-valuepair written from the Streaming output. Both of these classes can be found in theorg.apache.avro.mapred package.It’s also worth considering other frameworks like Pig, Hive, Crunch, and Spark for doingAvro processing, since they can all read and write Avro datafiles by specifying the ap‐propriate storage formats.
See the relevant chapters in this book for details.Avro in Other Languages|365CHAPTER 13ParquetApache Parquet is a columnar storage format that can efficiently store nested data.Columnar formats are attractive since they enable greater efficiency, in terms of bothfile size and query performance.
File sizes are usually smaller than row-oriented equiv‐alents since in a columnar format the values from one column are stored next to eachother, which usually allows a very efficient encoding. A column storing a timestamp,for example, can be encoded by storing the first value and the differences betweensubsequent values (which tend to be small due to temporal locality: records from aroundthe same time are stored next to each other). Query performance is improved too sincea query engine can skip over columns that are not needed to answer a query. (This ideais illustrated in Figure 5-4.) This chapter looks at Parquet in more depth, but there areother columnar formats that work with Hadoop—notably ORCFile (Optimized RecordColumnar File), which is a part of the Hive project.A key strength of Parquet is its ability to store data that has a deeply nested structure intrue columnar fashion.
This is important since schemas with several levels of nestingare common in real-world systems. Parquet uses a novel technique for storing nestedstructures in a flat columnar format with little overhead, which was introduced byGoogle engineers in the Dremel paper.1 The result is that even nested fields can be readindependently of other fields, resulting in significant performance improvements.Another feature of Parquet is the large number of tools that support it as a format. Theengineers at Twitter and Cloudera who created Parquet wanted it to be easy to try newtools to process existing data, so to facilitate this they divided the project into a speci‐fication (parquet-format), which defines the file format in a language-neutral way, andimplementations of the specification for different languages (Java and C++) that made1. Sergey Melnik et al., Dremel: Interactive Analysis of Web-Scale Datasets, Proceedings of the 36th InternationalConference on Very Large Data Bases, 2010.367it easy for tools to read or write Parquet files.
In fact, most of the data processing com‐ponents covered in this book understand the Parquet format (MapReduce, Pig, Hive,Cascading, Crunch, and Spark). This flexibility also extends to the in-memory repre‐sentation: the Java implementation is not tied to a single representation, so you can usein-memory data models for Avro, Thrift, or Protocol Buffers to read your data fromand write it to Parquet files.Data ModelParquet defines a small number of primitive types, listed in Table 13-1.Table 13-1. Parquet primitive typesTypeDescriptionbooleanBinary valueint3232-bit signed integerint6464-bit signed integerint9696-bit signed integerfloatSingle-precision (32-bit) IEEE 754 floating-point numberdoubleDouble-precision (64-bit) IEEE 754 floating-point numberbinarySequence of 8-bit unsigned bytesfixed_len_byte_array Fixed number of 8-bit unsigned bytesThe data stored in a Parquet file is described by a schema, which has at its root a messagecontaining a group of fields.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
