Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 88
Текст из файла (страница 88)
For example:% export PIG_HOME=~/sw/pig-x.y.z% export PATH=$PATH:$PIG_HOME/binYou also need to set the JAVA_HOME environment variable to point to a suitable Javainstallation.Try typing pig -help to get usage instructions.Execution TypesPig has two execution types or modes: local mode and MapReduce mode. Executionmodes for Apache Tez and Spark (see Chapter 19) were both under development at thetime of writing.
Both promise significant performance gains over MapReduce mode,so try them if they are available in the version of Pig you are using.424|Chapter 16: PigLocal modeIn local mode, Pig runs in a single JVM and accesses the local filesystem. This mode issuitable only for small datasets and when trying out Pig.The execution type is set using the -x or -exectype option. To run in local mode, setthe option to local:% pig -x localgrunt>This starts Grunt, the Pig interactive shell, which is discussed in more detail shortly.MapReduce modeIn MapReduce mode, Pig translates queries into MapReduce jobs and runs them on aHadoop cluster.
The cluster may be a pseudo- or fully distributed cluster. MapReducemode (with a fully distributed cluster) is what you use when you want to run Pig onlarge datasets.To use MapReduce mode, you first need to check that the version of Pig you downloadedis compatible with the version of Hadoop you are using. Pig releases will only workagainst particular versions of Hadoop; this is documented in the release notes.Pig honors the HADOOP_HOME environment variable for finding which Hadoop client torun.
However, if it is not set, Pig will use a bundled copy of the Hadoop libraries. Notethat these may not match the version of Hadoop running on your cluster, so it is bestto explicitly set HADOOP_HOME.Next, you need to point Pig at the cluster’s namenode and resource manager. If theinstallation of Hadoop at HADOOP_HOME is already configured for this, then there is noth‐ing more to do. Otherwise, you can set HADOOP_CONF_DIR to a directory containing theHadoop site file (or files) that define fs.defaultFS, yarn.resourcemanager.address,and mapreduce.framework.name (the latter should be set to yarn).Alternatively, you can set these properties in the pig.properties file in Pig’s conf directory(or the directory specified by PIG_CONF_DIR). Here’s an example for a pseudodistributed setup:fs.defaultFS=hdfs://localhost/mapreduce.framework.name=yarnyarn.resourcemanager.address=localhost:8032Once you have configured Pig to connect to a Hadoop cluster, you can launch Pig, settingthe -x option to mapreduce or omitting it entirely, as MapReduce mode is the default.We’ve used the -brief option to stop timestamps from being logged:% pig -briefLogging error messages to: /Users/tom/pig_1414246949680.logDefault bootup file /Users/tom/.pigbootup not foundInstalling and Running Pig|425Connecting to hadoop file system at: hdfs://localhost/grunt>As you can see from the output, Pig reports the filesystem (but not the YARN resourcemanager) that it has connected to.In MapReduce mode, you can optionally enable auto-local mode (by settingpig.auto.local.enabled to true), which is an optimization that runs small jobs locallyif the input is less than 100 MB (set by pig.auto.local.input.maxbytes, default100,000,000) and no more than one reducer is being used.Running Pig ProgramsThere are three ways of executing Pig programs, all of which work in both local andMapReduce mode:ScriptPig can run a script file that contains Pig commands.
For example, pig script.pigruns the commands in the local file script.pig. Alternatively, for very short scripts,you can use the -e option to run a script specified as a string on the command line.GruntGrunt is an interactive shell for running Pig commands. Grunt is started when nofile is specified for Pig to run and the -e option is not used. It is also possible to runPig scripts from within Grunt using run and exec.EmbeddedYou can run Pig programs from Java using the PigServer class, much like you canuse JDBC to run SQL programs from Java.
For programmatic access to Grunt, usePigRunner.GruntGrunt has line-editing facilities like those found in GNU Readline (used in the bashshell and many other command-line applications). For instance, the Ctrl-E key com‐bination will move the cursor to the end of the line. Grunt remembers command history,too,1 and you can recall lines in the history buffer using Ctrl-P or Ctrl-N (for previousand next), or equivalently, the up or down cursor keys.Another handy feature is Grunt’s completion mechanism, which will try to completePig Latin keywords and functions when you press the Tab key.
For example, considerthe following incomplete line:grunt> a = foreach b ge1. History is stored in a file called .pig_history in your home directory.426|Chapter 16: PigIf you press the Tab key at this point, ge will expand to generate, a Pig Latin keyword:grunt> a = foreach b generateYou can customize the completion tokens by creating a file named autocomplete andplacing it on Pig’s classpath (such as in the conf directory in Pig’s install directory) or inthe directory you invoked Grunt from. The file should have one token per line, andtokens must not contain any whitespace. Matching is case sensitive. It can be very handyto add commonly used file paths (especially because Pig does not perform filenamecompletion) or the names of any user-defined functions you have created.You can get a list of commands using the help command.
When you’ve finished yourGrunt session, you can exit with the quit command, or the equivalent shortcut \q.Pig Latin EditorsThere are Pig Latin syntax highlighters available for a variety of editors, includingEclipse, IntelliJ IDEA, Vim, Emacs, and TextMate. Details are available on the Pig wiki.Many Hadoop distributions come with the Hue web interface, which has a Pig scripteditor and launcher.An ExampleLet’s look at a simple example by writing the program to calculate the maximumrecorded temperature by year for the weather dataset in Pig Latin (just like we did usingMapReduce in Chapter 2).
The complete program is only a few lines long:-- max_temp.pig: Finds the maximum temperature by yearrecords = LOAD 'input/ncdc/micro-tab/sample.txt'AS (year:chararray, temperature:int, quality:int);filtered_records = FILTER records BY temperature != 9999 ANDquality IN (0, 1, 4, 5, 9);grouped_records = GROUP filtered_records BY year;max_temp = FOREACH grouped_records GENERATE group,MAX(filtered_records.temperature);DUMP max_temp;To explore what’s going on, we’ll use Pig’s Grunt interpreter, which allows us to enterlines and interact with the program to understand what it’s doing.
Start up Grunt inlocal mode, and then enter the first line of the Pig script:grunt> records = LOAD 'input/ncdc/micro-tab/sample.txt'>>AS (year:chararray, temperature:int, quality:int);For simplicity, the program assumes that the input is tab-delimited text, with each linehaving just year, temperature, and quality fields. (Pig actually has more flexibility thanthis with regard to the input formats it accepts, as we’ll see later.) This line describes theinput data we want to process. The year:chararray notation describes the field’s nameAn Example|427and type; chararray is like a Java String, and an int is like a Java int. The LOAD operatortakes a URI argument; here we are just using a local file, but we could refer to an HDFSURI. The AS clause (which is optional) gives the fields names to make it convenient torefer to them in subsequent statements.The result of the LOAD operator, and indeed any operator in Pig Latin, is a relation, whichis just a set of tuples.
A tuple is just like a row of data in a database table, with multiplefields in a particular order. In this example, the LOAD function produces a set of (year,temperature, quality) tuples that are present in the input file. We write a relation withone tuple per line, where tuples are represented as comma-separated items inparentheses:(1950,0,1)(1950,22,1)(1950,-11,1)(1949,111,1)Relations are given names, or aliases, so they can be referred to. This relation is giventhe records alias.
We can examine the contents of an alias using the DUMP operator:grunt> DUMP records;(1950,0,1)(1950,22,1)(1950,-11,1)(1949,111,1)(1949,78,1)We can also see the structure of a relation—the relation’s schema—using the DESCRIBEoperator on the relation’s alias:grunt> DESCRIBE records;records: {year: chararray,temperature: int,quality: int}This tells us that records has three fields, with aliases year, temperature, and quality,which are the names we gave them in the AS clause. The fields have the types given tothem in the AS clause, too. We examine types in Pig in more detail later.The second statement removes records that have a missing temperature (indicated bya value of 9999) or an unsatisfactory quality reading. For this small dataset, no recordsare filtered out:grunt> filtered_records = FILTER records BY temperature != 9999 AND>>quality IN (0, 1, 4, 5, 9);grunt> DUMP filtered_records;(1950,0,1)(1950,22,1)(1950,-11,1)(1949,111,1)(1949,78,1)428|Chapter 16: PigThe third statement uses the GROUP function to group the records relation by the yearfield.
Let’s use DUMP to see what it produces:grunt> grouped_records = GROUP filtered_records BY year;grunt> DUMP grouped_records;(1949,{(1949,78,1),(1949,111,1)})(1950,{(1950,-11,1),(1950,22,1),(1950,0,1)})We now have two rows, or tuples: one for each year in the input data. The first field ineach tuple is the field being grouped by (the year), and the second field has a bag oftuples for that year. A bag is just an unordered collection of tuples, which in Pig Latinis represented using curly braces.By grouping the data in this way, we have created a row per year, so now all that remainsis to find the maximum temperature for the tuples in each bag. Before we do this, let’sunderstand the structure of the grouped_records relation:grunt> DESCRIBE grouped_records;grouped_records: {group: chararray,filtered_records: {year: chararray,temperature: int,quality: int}}This tells us that the grouping field is given the alias group by Pig, and the second fieldis the same structure as the filtered_records relation that was being grouped.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
