Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 72
Текст из файла (страница 72)
For interoperability, imple‐mentations must support all Avro types.Avro’s primitive types are listed in Table 12-1. Each primitive type may also be specifiedusing a more verbose form by using the type attribute, such as:{ "type": "null" }Table 12-1. Avro primitive typesTypeDescriptionSchemanullThe absence of a value"null"boolean A binary value"boolean"int32-bit signed integer"int"long64-bit signed integer"long"floatSingle-precision (32-bit) IEEE 754 floating-point number"float"346|Chapter 12: AvroTypeDescriptionSchemadoubleDouble-precision (64-bit) IEEE 754 floating-point number "double"bytesSequence of 8-bit unsigned bytes"bytes"stringSequence of Unicode characters"string"Avro also defines the complex types listed in Table 12-2, along with a representativeexample of a schema of each type.Table 12-2.
Avro complex typesTypeDescriptionarrayAn ordered collection of objects. Allobjects in a particular array must havethe same schema.Schema example{"type": "array","items": "long"}An unordered collection of key-valuepairs. Keys must be strings and valuesmay be any type, although within aparticular map, all values must have thesame schema.{record A collection of named fields of any type.{map"type": "map","values": "string"}"type": "record","name": "WeatherRecord","doc": "A weather reading.","fields": [{"name": "year", "type": "int"},{"name": "temperature", "type": "int"},{"name": "stationId", "type": "string"}]}enumA set of named values.{"type": "enum","name": "Cutlery","doc": "An eating utensil.","symbols": ["KNIFE", "FORK", "SPOON"]}fixedA fixed number of 8-bit unsigned bytes.{"type": "fixed","name": "Md5Hash","size": 16}Avro Data Types and Schemas|347TypeDescriptionunionA union of schemas.
A union isrepresented by a JSON array, where eachelement in the array is a schema. Datarepresented by a union must match oneof the schemas in the union.Schema example["null","string",{"type": "map", "values": "string"}]Each Avro language API has a representation for each Avro type that is specific to thelanguage.
For example, Avro’s double type is represented in C, C++, and Java by adouble, in Python by a float, and in Ruby by a Float.What’s more, there may be more than one representation, or mapping, for a language.All languages support a dynamic mapping, which can be used even when the schemais not known ahead of runtime. Java calls this the Generic mapping.In addition, the Java and C++ implementations can generate code to represent the datafor an Avro schema.
Code generation, which is called the Specific mapping in Java, is anoptimization that is useful when you have a copy of the schema before you read or writedata. Generated classes also provide a more domain-oriented API for user code thanGeneric ones.Java has a third mapping, the Reflect mapping, which maps Avro types onto preexistingJava types using reflection. It is slower than the Generic and Specific mappings but canbe a convenient way of defining a type, since Avro can infer a schema automatically.Java’s type mappings are shown in Table 12-3.
As the table shows, the Specific mappingis the same as the Generic one unless otherwise noted (and the Reflect one is the sameas the Specific one unless noted). The Specific mapping differs from the Generic oneonly for record, enum, and fixed, all of which have generated classes (the names ofwhich are controlled by the name and optional namespace attributes).Table 12-3.
Avro Java type mappingsAvro type Generic Java mappingnullSpecific Java mappingReflect Java mappingnull typeboolean booleanintintbyte, short, int, or charlonglongfloatfloatdoubledoublebytesjava.nio.ByteBufferArray of bytesstringorg.apache.avro.util.Utf8 or java.lang.Stringjava.lang.Stringarrayorg.apache.avro.generic.GenericArrayArray or java.util.Collectionmapjava.util.Map348|Chapter 12: AvroAvro type Generic Java mappingSpecific Java mappingReflect Java mappingrecordGenerated classimplementingArbitrary user class with a zeroargument constructor; all inheritednontransient instance fields are usedorg.apache.avro.generic.GenericRecordorg.apache.avro.specific.SpecificRecordenumjava.lang.StringGenerated Java enumArbitrary Java enumfixedorg.apache.avro.generic.GenericFixedGenerated classimplementingorg.apache.avro.generic.GenericFixedorg.apache.avro.specific.SpecificFixedunionjava.lang.ObjectAvro string can be represented by either Java String or the AvroUtf8 Java type.
The reason to use Utf8 is efficiency: because it ismutable, a single Utf8 instance may be reused for reading or writ‐ing a series of values. Also, Java String decodes UTF-8 at objectconstruction time, whereas Avro Utf8 does it lazily, which can in‐crease performance in some cases.Utf8 implements Java’s java.lang.CharSequence interface, whichallows some interoperability with Java libraries. In other cases, it maybe necessary to convert Utf8 instances to String objects by calling itstoString() method.Utf8 is the default for Generic and Specific, but it’s possible to useString for a particular mapping. There are a couple of ways to ach‐ieve this. The first is to set the avro.java.string property in theschema to String:{ "type": "string", "avro.java.string": "String" }Alternatively, for the Specific mapping, you can generate classes thathave String-based getters and setters.
When using the Avro Mavenplug-in, this is done by setting the configuration property stringType to String (“The Specific API” on page 351 has a demonstration ofthis).Finally, note that the Java Reflect mapping always uses String ob‐jects, since it is designed for Java compatibility, not performance.In-Memory Serialization and DeserializationAvro provides APIs for serialization and deserialization that are useful when you wantto integrate Avro with an existing system, such as a messaging system where the framingformat is already defined.
In other cases, consider using Avro’s datafile format.In-Memory Serialization and Deserialization|349Let’s write a Java program to read and write Avro data from and to streams. We’ll startwith a simple Avro schema for representing a pair of strings as a record:{"type": "record","name": "StringPair","doc": "A pair of strings.","fields": [{"name": "left", "type": "string"},{"name": "right", "type": "string"}]}If this schema is saved in a file on the classpath called StringPair.avsc (.avsc is the con‐ventional extension for an Avro schema), we can load it using the following two linesof code:Schema.Parser parser = new Schema.Parser();Schema schema = parser.parse(getClass().getResourceAsStream("StringPair.avsc"));We can create an instance of an Avro record using the Generic API as follows:GenericRecord datum = new GenericData.Record(schema);datum.put("left", "L");datum.put("right", "R");Next, we serialize the record to an output stream:ByteArrayOutputStream out = new ByteArrayOutputStream();DatumWriter<GenericRecord> writer =new GenericDatumWriter<GenericRecord>(schema);Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);writer.write(datum, encoder);encoder.flush();out.close();There are two important objects here: the DatumWriter and the Encoder.
A DatumWritertranslates data objects into the types understood by an Encoder, which the latter writesto the output stream. Here we are using a GenericDatumWriter, which passes the fieldsof GenericRecord to the Encoder. We pass a null to the encoder factory because we arenot reusing a previously constructed encoder here.In this example, only one object is written to the stream, but we could call write() withmore objects before closing the stream if we wanted to.The GenericDatumWriter needs to be passed the schema because it follows the schemato determine which values from the data objects to write out.
After we have called thewriter’s write() method, we flush the encoder, then close the output stream.We can reverse the process and read the object back from the byte buffer:350|Chapter 12: AvroDatumReader<GenericRecord> reader =new GenericDatumReader<GenericRecord>(schema);Decoder decoder = DecoderFactory.get().binaryDecoder(out.toByteArray(),null);GenericRecord result = reader.read(null, decoder);assertThat(result.get("left").toString(), is("L"));assertThat(result.get("right").toString(), is("R"));We pass null to the calls to binaryDecoder() and read() because we are not reusingobjects here (the decoder or the record, respectively).The objects returned by result.get("left") and result.get("right") are of typeUtf8, so we convert them into Java String objects by calling their toString() methods.The Specific APILet’s look now at the equivalent code using the Specific API.
We can generate theStringPair class from the schema file by using Avro’s Maven plug-in for compilingschemas. The following is the relevant part of the Maven Project Object Model (POM):<project>...<build><plugins><plugin><groupId>org.apache.avro</groupId><artifactId>avro-maven-plugin</artifactId><version>${avro.version}</version><executions><execution><id>schemas</id><phase>generate-sources</phase><goals><goal>schema</goal></goals><configuration><includes><include>StringPair.avsc</include></includes><stringType>String</stringType><sourceDirectory>src/main/resources</sourceDirectory><outputDirectory>${project.build.directory}/generated-sources/java</outputDirectory></configuration></execution></executions></plugin></plugins></build>...</project>In-Memory Serialization and Deserialization|351As an alternative to Maven, you can use Avro’s Ant task, org.apache.avro.specific.SchemaTask, or the Avro command-line tools3 to generate Java code for a schema.In the code for serializing and deserializing, instead of a GenericRecord we constructa StringPair instance, which we write to the stream using a SpecificDatumWriter andread back using a SpecificDatumReader:StringPair datum = new StringPair();datum.setLeft("L");datum.setRight("R");ByteArrayOutputStream out = new ByteArrayOutputStream();DatumWriter<StringPair> writer =new SpecificDatumWriter<StringPair>(StringPair.class);Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);writer.write(datum, encoder);encoder.flush();out.close();DatumReader<StringPair> reader =new SpecificDatumReader<StringPair>(StringPair.class);Decoder decoder = DecoderFactory.get().binaryDecoder(out.toByteArray(),null);StringPair result = reader.read(null, decoder);assertThat(result.getLeft(), is("L"));assertThat(result.getRight(), is("R"));Avro DatafilesAvro’s object container file format is for storing sequences of Avro objects.
It is verysimilar in design to Hadoop’s sequence file format, described in “SequenceFile” on page127. The main difference is that Avro datafiles are designed to be portable across lan‐guages, so, for example, you can write a file in Python and read it in C (we will do exactlythis in the next section).A datafile has a header containing metadata, including the Avro schema and a syncmarker, followed by a series of (optionally compressed) blocks containing the serializedAvro objects. Blocks are separated by a sync marker that is unique to the file (the markerfor a particular file is found in the header) and that permits rapid resynchronizationwith a block boundary after seeking to an arbitrary point in the file, such as an HDFSblock boundary. Thus, Avro datafiles are splittable, which makes them amenable toefficient MapReduce processing.3. Avro can be downloaded in both source and binary forms.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
