Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 91
Текст из файла (страница 91)
You can access all ofthe Hadoop filesystem shell commands using Pig’s fs command. For example, fs -lswill show a file listing, and fs -help will show help on all the available commands.Precisely which Hadoop filesystem is used is determined by the fs.defaultFS propertyin the site file for Hadoop Core. See “The Command-Line Interface” on page 50 formore details on how to configure this property.These commands are mostly self-explanatory, except set, which is used to set optionsthat control Pig’s behavior (including arbitrary MapReduce job properties).
The debug option is used to turn debug logging on or off from within a script (you can alsocontrol the log level when launching Pig, using the -d or -debug option):grunt> set debug onAnother useful option is the job.name option, which gives a Pig job a meaningful name,making it easier to pick out your Pig MapReduce jobs when running on a shared Hadoopcluster. If Pig is running a script (rather than operating as an interactive query fromGrunt), its job name defaults to a value based on the script name.There are two commands in Table 16-4 for running a Pig script, exec and run. Thedifference is that exec runs the script in batch mode in a new Grunt shell, so any aliasesdefined in the script are not accessible to the shell after the script has completed.
Onthe other hand, when running a script with run, it is as if the contents of the script hadbeen entered manually, so the command history of the invoking shell contains all thestatements from the script. Multiquery execution, where Pig executes a batch of state‐ments in one go (see “Multiquery Execution” on page 434), is used only by exec, not run.Pig Latin|437Control FlowBy design, Pig Latin lacks native control flow statements.
The recommended approachfor writing programs that have conditional logic or loop constructs is to embed Pig Latinin another language, such as Python, JavaScript, or Java, and manage the control flowfrom there. In this model, the host script uses a compile-bind-run API to execute Pigscripts and retrieve their status. Consult the Pig documentation for details of the API.Embedded Pig programs always run in a JVM, so for Python and JavaScript you use thepig command followed by the name of your script, and the appropriate Java scriptingengine will be selected (Jython for Python, Rhino for JavaScript).ExpressionsAn expression is something that is evaluated to yield a value.
Expressions can be usedin Pig as a part of a statement containing a relational operator. Pig has a rich variety ofexpressions, many of which will be familiar from other programming languages. Theyare listed in Table 16-5, with brief descriptions and examples. We will see examples ofmany of these expressions throughout the chapter.Table 16-5. Pig Latin expressionsCategoryExpressionsDescriptionExamplesConstantLiteralConstant value (see also the “Literalexample” column in Table 16-6)1.0, 'a'Field (by position)$nField in position n (zero-based)$0Field (by name)fField named fyearField (disambiguate) r::fField named f from relation r aftergrouping or joiningA::yearProjectionc.$n, c.fField in container c (relation, bag, ortuple) by position, by namerecords.$0, records.yearMap lookupm#kValue associated with key k in map mitems#'Coat'Cast(t) fCast of field f to type t(int) yearArithmeticx + y, x - yAddition, subtraction$1 + $2, $1 - $2x * y, x / yMultiplication, division$1 * $2, $1 / $2x % yModulo, the remainder of x dividedby y$1 % $2Conditional438|+x, -xUnary positive, negation+1, –1x ? y : zBincond/ternary; y if x evaluates totrue, z otherwisequality == 0 ? 0 : 1CASEMulti-case conditionalCASE q WHEN 0 THEN 'good'ELSE 'bad' ENDChapter 16: PigCategoryExpressionsComparisonx == y, x != y Equals, does not equalx > y, x < yDescriptionGreater than, less thanx >= y, x <= y Greater than or equal to, less than orequal toquality == 0, temperature != 9999quality > 0, quality < 10quality >= 1, quality <= 9x matches yPattern matching with regularexpressionquality matches '[01459]'x is nullIs nulltemperature is nullx is not null Is not nullBooleanExamplestemperature is not nullx OR yLogical ORq == 0 OR q == 1x AND yLogical ANDq == 0 AND r == 0NOT xLogical negationNOT q matches '[01459]'IN xSet membershipq IN (0, 1, 4, 5, 9)Functionalfn(f1,f2,...) Invocation of function fn on fieldsf1, f2, etc.FlattenFLATTEN(f)Removal of a level of nesting frombags and tuplesisGood(quality)FLATTEN(group)TypesSo far you have seen some of the simple types in Pig, such as int and chararray.
Herewe will discuss Pig’s built-in types in more detail.Pig has a boolean type and six numeric types: int, long, float, double, biginteger,and bigdecimal, which are identical to their Java counterparts. There is also a bytearraytype, like Java’s byte array type for representing a blob of binary data, and chararray,which, like java.lang.String, represents textual data in UTF-16 format (although itcan be loaded or stored in UTF-8 format). The datetime type is for storing a date andtime with millisecond precision and including a time zone.Pig does not have types corresponding to Java’s byte, short, or char primitive types.These are all easily represented using Pig’s int type, or chararray for char.The Boolean, numeric, textual, binary, and temporal types are simple atomic types.
PigLatin also has three complex types for representing nested structures: tuple, bag, andmap. All of Pig Latin’s types are listed in Table 16-6.Pig Latin|439Table 16-6. Pig Latin typesCategory TypeDescriptionLiteral exampleBooleanbooleanTrue/false valuetrueNumericint32-bit signed integer1long64-bit signed integer1Lfloat32-bit floating-point number1.0Fdouble64-bit floating-point number1.0biginteger Arbitrary-precision integer'10000000000'bigdecimal Arbitrary-precision signed decimal number'0.110001000000000000000001'TextchararrayCharacter array in UTF-16 format'a'BinarybytearrayByte arrayNot supportedTemporal datetimeDate and time with time zoneNot supported, use ToDate built-in functionComplextupleSequence of fields of any type(1,'pomegranate')bagUnordered collection of tuples, possibly withduplicates{(1,'pomegranate'),(2)}mapSet of key-value pairs; keys must be characterarrays, but values may be any type['a'#'pomegranate']The complex types are usually loaded from files or constructed using relational opera‐tors.
Be aware, however, that the literal form in Table 16-6 is used when a constant valueis created from within a Pig Latin program. The raw form in a file is usually differentwhen using the standard PigStorage loader. For example, the representation in a fileof the bag in Table 16-6 would be {(1,pomegranate),(2)} (note the lack of quotationmarks), and with a suitable schema, this would be loaded as a relation with a single fieldand row, whose value was the bag.Pig provides the built-in functions TOTUPLE, TOBAG, and TOMAP, which are used for turn‐ing expressions into tuples, bags, and maps.Although relations and bags are conceptually the same (unordered collections of tuples),in practice Pig treats them slightly differently. A relation is a top-level construct, whereasa bag has to be contained in a relation. Normally you don’t have to worry about this,but there are a few restrictions that can trip up the uninitiated.
For example, it’s notpossible to create a relation from a bag literal. So, the following statement fails:A = {(1,2),(3,4)}; -- ErrorThe simplest workaround in this case is to load the data from a file using the LOADstatement.As another example, you can’t treat a relation like a bag and project a field into a newrelation ($0 refers to the first field of A, using the positional notation):B = A.$0;440|Chapter 16: PigInstead, you have to use a relational operator to turn the relation A into relation B:B = FOREACH A GENERATE $0;It’s possible that a future version of Pig Latin will remove these inconsistencies and treatrelations and bags in the same way.SchemasA relation in Pig may have an associated schema, which gives the fields in the relationnames and types. We’ve seen how an AS clause in a LOAD statement is used to attach aschema to a relation:grunt> records = LOAD 'input/ncdc/micro-tab/sample.txt'>>AS (year:int, temperature:int, quality:int);grunt> DESCRIBE records;records: {year: int,temperature: int,quality: int}This time we’ve declared the year to be an integer rather than a chararray, even thoughthe file it is being loaded from is the same.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
