Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 89
Текст из файла (страница 89)
Withthis information, we can try the fourth transformation:grunt> max_temp = FOREACH grouped_records GENERATE group,>>MAX(filtered_records.temperature);FOREACH processes every row to generate a derived set of rows, using a GENERATEclause to define the fields in each derived row.
In this example, the first field isgroup, which is just the year. The second field is a little more complex.The filtered_records.temperature reference is to the temperature field of thefiltered_records bag in the grouped_records relation. MAX is a built-in function forcalculating the maximum value of fields in a bag. In this case, it calculates the maximumtemperature for the fields in each filtered_records bag. Let’s check the result:grunt> DUMP max_temp;(1949,111)(1950,22)We’ve successfully calculated the maximum temperature for each year.Generating ExamplesIn this example, we’ve used a small sample dataset with just a handful of rows to makeit easier to follow the data flow and aid debugging.
Creating a cut-down dataset is anart, as ideally it should be rich enough to cover all the cases to exercise your queries (thecompleteness property), yet small enough to make sense to the programmer (the con‐ciseness property). Using a random sample doesn’t work well in general because joinAn Example|429and filter operations tend to remove all random data, leaving an empty result, which isnot illustrative of the general data flow.With the ILLUSTRATE operator, Pig provides a tool for generating a reasonably completeand concise sample dataset.
Here is the output from running ILLUSTRATE on our dataset(slightly reformatted to fit the page):grunt> ILLUSTRATE max_temp;------------------------------------------------------------------------------| records| year:chararray| temperature:int| quality:int|------------------------------------------------------------------------------|| 1949| 78| 1||| 1949| 111| 1||| 1949| 9999| 1|------------------------------------------------------------------------------------------------------------------------------------------------------------| filtered_records| year:chararray| temperature:int| quality:int|------------------------------------------------------------------------------|| 1949| 78| 1||| 1949| 111| 1|------------------------------------------------------------------------------------------------------------------------------------------------------------| grouped_records | group:chararray| filtered_records:bag{:tuple(|year:chararray,temperature:int,|quality:int)}|------------------------------------------------------------------------------|| 1949| {(1949, 78, 1), (1949, 111, 1)}|--------------------------------------------------------------------------------------------------------------------------------| max_temp| group:chararray| :int|--------------------------------------------------|| 1949| 111|---------------------------------------------------Notice that Pig used some of the original data (this is important to keep the generateddataset realistic), as well as creating some new data.
It noticed the special value 9999 inthe query and created a tuple containing this value to exercise the FILTER statement.In summary, the output of ILLUSTRATE is easy to follow and can help you understandwhat your query is doing.Comparison with DatabasesHaving seen Pig in action, it might seem that Pig Latin is similar to SQL. The presenceof such operators as GROUP BY and DESCRIBE reinforces this impression. However, thereare several differences between the two languages, and between Pig and relational da‐tabase management systems (RDBMSs) in general.The most significant difference is that Pig Latin is a data flow programming language,whereas SQL is a declarative programming language.
In other words, a Pig Latin pro‐430|Chapter 16: Piggram is a step-by-step set of operations on an input relation, in which each step is asingle transformation. By contrast, SQL statements are a set of constraints that, takentogether, define the output. In many ways, programming in Pig Latin is like working atthe level of an RDBMS query planner, which figures out how to turn a declarative state‐ment into a system of steps.RDBMSs store data in tables, with tightly predefined schemas. Pig is more relaxed aboutthe data that it processes: you can define a schema at runtime, but it’s optional.
Essen‐tially, it will operate on any source of tuples (although the source should support beingread in parallel, by being in multiple files, for example), where a UDF is used to readthe tuples from their raw representation.2 The most common representation is a textfile with tab-separated fields, and Pig provides a built-in load function for this format.Unlike with a traditional database, there is no data import process to load the data intothe RDBMS. The data is loaded from the filesystem (usually HDFS) as the first step inthe processing.Pig’s support for complex, nested data structures further differentiates it from SQL,which operates on flatter data structures.
Also, Pig’s ability to use UDFs and streamingoperators that are tightly integrated with the language and Pig’s nested data structuresmakes Pig Latin more customizable than most SQL dialects.RDBMSs have several features to support online, low-latency queries, such as transac‐tions and indexes, that are absent in Pig. Pig does not support random reads or querieson the order of tens of milliseconds. Nor does it support random writes to update smallportions of data; all writes are bulk streaming writes, just like with MapReduce.Hive (covered in Chapter 17) sits between Pig and conventional RDBMSs. Like Pig,Hive is designed to use HDFS for storage, but otherwise there are some significantdifferences.
Its query language, HiveQL, is based on SQL, and anyone who is familiarwith SQL will have little trouble writing queries in HiveQL. Like RDBMSs, Hive man‐dates that all data be stored in tables, with a schema under its management; however, itcan associate a schema with preexisting data in HDFS, so the load step is optional. Pigis able to work with Hive tables using HCatalog; this is discussed further in “Using Hivetables with HCatalog” on page 442.2. Or as the Pig Philosophy has it, “Pigs eat anything.”Comparison with Databases|431Pig LatinThis section gives an informal description of the syntax and semantics of the Pig Latinprogramming language.3 It is not meant to offer a complete reference to the language,4but there should be enough here for you to get a good understanding of Pig Latin’sconstructs.StructureA Pig Latin program consists of a collection of statements.
A statement can be thoughtof as an operation or a command.5 For example, a GROUP operation is a type of statement:grouped_records = GROUP records BY year;The command to list the files in a Hadoop filesystem is another example of a statement:ls /Statements are usually terminated with a semicolon, as in the example of the GROUPstatement. In fact, this is an example of a statement that must be terminated with asemicolon; it is a syntax error to omit it. The ls command, on the other hand, does nothave to be terminated with a semicolon. As a general guideline, statements or commandsfor interactive use in Grunt do not need the terminating semicolon.
This group includesthe interactive Hadoop commands, as well as the diagnostic operators such as DESCRIBE. It’s never an error to add a terminating semicolon, so if in doubt, it’s simplestto add one.Statements that have to be terminated with a semicolon can be split across multiple linesfor readability:records = LOAD 'input/ncdc/micro-tab/sample.txt'AS (year:chararray, temperature:int, quality:int);Pig Latin has two forms of comments.
Double hyphens are used for single-line com‐ments. Everything from the first hyphen to the end of the line is ignored by the Pig Latininterpreter:-- My programDUMP A; -- What's in A?3. Not to be confused with Pig Latin, the language game. English words are translated into Pig Latin by movingthe initial consonant sound to the end of the word and adding an “ay” sound. For example, “pig” becomes“ig-pay,” and “Hadoop” becomes “Adoop-hay.”4.
Pig Latin does not have a formal language definition as such, but there is a comprehensive guide to thelanguage that you can find through a link on the Pig website.5. You sometimes see these terms being used interchangeably in documentation on Pig Latin: for example,“GROUP command,” “GROUP operation,” “GROUP statement.”432|Chapter 16: PigC-style comments are more flexible since they delimit the beginning and end of thecomment block with /* and */ markers. They can span lines or be embedded in a singleline:/** Description of my program spanning* multiple lines.*/A = LOAD 'input/pig/join/A';B = LOAD 'input/pig/join/B';C = JOIN A BY $0, /* ignored */ B BY $1;DUMP C;Pig Latin has a list of keywords that have a special meaning in the language and cannotbe used as identifiers.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
