Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 32
Текст из файла (страница 32)
As a matter of best practice,it’s recommended to keep these files outside Hadoop’s installation directory tree, as thismakes it easy to switch between Hadoop versions without duplicating or losing settings.For the purposes of this book, we assume the existence of a directory called conf thatcontains three configuration files: hadoop-local.xml, hadoop-localhost.xml, and hadoopcluster.xml (these are available in the example code for this book). Note that there isnothing special about the names of these files; they are just convenient ways to packageup some configuration settings. (Compare this to Table A-1 in Appendix A, which setsout the equivalent server-side configurations.)The hadoop-local.xml file contains the default Hadoop configuration for the defaultfilesystem and the local (in-JVM) framework for running MapReduce jobs:<?xml version="1.0"?><configuration><property><name>fs.defaultFS</name><value>file:///</value></property><property><name>mapreduce.framework.name</name><value>local</value></property></configuration>The settings in hadoop-localhost.xml point to a namenode and a YARN resource man‐ager both running on localhost:<?xml version="1.0"?><configuration><property><name>fs.defaultFS</name><value>hdfs://localhost/</value>146|Chapter 6: Developing a MapReduce Application</property><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>yarn.resourcemanager.address</name><value>localhost:8032</value></property></configuration>Finally, hadoop-cluster.xml contains details of the cluster’s namenode and YARN re‐source manager addresses (in practice, you would name the file after the name of thecluster, rather than “cluster” as we have here):<?xml version="1.0"?><configuration><property><name>fs.defaultFS</name><value>hdfs://namenode/</value></property><property><name>mapreduce.framework.name</name><value>yarn</value></property><property><name>yarn.resourcemanager.address</name><value>resourcemanager:8032</value></property></configuration>You can add other configuration properties to these files as needed.Setting User IdentityThe user identity that Hadoop uses for permissions in HDFS is determined by runningthe whoami command on the client system.
Similarly, the group names are derived fromthe output of running groups.If, however, your Hadoop user identity is different from the name of your user accounton your client machine, you can explicitly set your Hadoop username by setting theHADOOP_USER_NAME environment variable. You can also override user group mappingsby means of the hadoop.user.group.static.mapping.overrides configurationSetting Up the Development Environment|147property. For example, dr.who=;preston=directors,inventors means that thedr.who user is in no groups, but preston is in the directors and inventors groups.You can set the user identity that the Hadoop web interfaces run as by setting thehadoop.http.staticuser.user property.
By default, it is dr.who, which is not a su‐peruser, so system files are not accessible through the web interface.Notice that, by default, there is no authentication with this system. See “Security” onpage 309 for how to use Kerberos authentication with Hadoop.With this setup, it is easy to use any configuration with the -conf command-line switch.For example, the following command shows a directory listing on the HDFS serverrunning in pseudodistributed mode on localhost:% hadoop fs -conf conf/hadoop-localhost.xml -ls .Found 2 itemsdrwxr-xr-x- tom supergroup0 2014-09-08 10:19 inputdrwxr-xr-x- tom supergroup0 2014-09-08 10:19 outputIf you omit the -conf option, you pick up the Hadoop configuration in the etc/hadoop subdirectory under $HADOOP_HOME.
Or, if HADOOP_CONF_DIR is set, Hadoop con‐figuration files will be read from that location.Here’s an alternative way of managing configuration settings. Copythe etc/hadoop directory from your Hadoop installation to anotherlocation, place the *-site.xml configuration files there (with appropri‐ate settings), and set the HADOOP_CONF_DIR environment variable tothe alternative location.
The main advantage of this approach is thatyou don’t need to specify -conf for every command. It also allows youto isolate changes to files other than the Hadoop XML configura‐tion files (e.g., log4j.properties) since the HADOOP_CONF_DIR directoryhas a copy of all the configuration files (see “Hadoop Configura‐tion” on page 292).Tools that come with Hadoop support the -conf option, but it’s straightforward to makeyour programs (such as programs that run MapReduce jobs) support it, too, using theTool interface.GenericOptionsParser, Tool, and ToolRunnerHadoop comes with a few helper classes for making it easier to run jobs from the com‐mand line.
GenericOptionsParser is a class that interprets common Hadoopcommand-line options and sets them on a Configuration object for your applicationto use as desired. You don’t usually use GenericOptionsParser directly, as it’s more148|Chapter 6: Developing a MapReduce Applicationconvenient to implement the Tool interface and run your application with theToolRunner, which uses GenericOptionsParser internally:public interface Tool extends Configurable {int run(String [] args) throws Exception;}Example 6-4 shows a very simple implementation of Tool that prints the keys and valuesof all the properties in the Tool’s Configuration object.Example 6-4. An example Tool implementation for printing the properties in aConfigurationpublic class ConfigurationPrinter extends Configured implements Tool {static {Configuration.addDefaultResource("hdfs-default.xml");Configuration.addDefaultResource("hdfs-site.xml");Configuration.addDefaultResource("yarn-default.xml");Configuration.addDefaultResource("yarn-site.xml");Configuration.addDefaultResource("mapred-default.xml");Configuration.addDefaultResource("mapred-site.xml");}@Overridepublic int run(String[] args) throws Exception {Configuration conf = getConf();for (Entry<String, String> entry: conf) {System.out.printf("%s=%s\n", entry.getKey(), entry.getValue());}return 0;}public static void main(String[] args) throws Exception {int exitCode = ToolRunner.run(new ConfigurationPrinter(), args);System.exit(exitCode);}}We make ConfigurationPrinter a subclass of Configured, which is an implementationof the Configurable interface.
All implementations of Tool need to implementConfigurable (since Tool extends it), and subclassing Configured is often the easiestway to achieve this. The run() method obtains the Configuration using Configurable’s getConf() method and then iterates over it, printing each property to standardoutput.The static block makes sure that the HDFS, YARN, and MapReduce configurations arepicked up, in addition to the core ones (which Configuration knows about already).ConfigurationPrinter’s main() method does not invoke its own run() method di‐rectly. Instead, we call ToolRunner’s static run() method, which takes care of creatingSetting Up the Development Environment|149a Configuration object for the Tool before calling its run() method.
ToolRunner alsouses a GenericOptionsParser to pick up any standard options specified on the com‐mand line and to set them on the Configuration instance. We can see the effect ofpicking up the properties specified in conf/hadoop-localhost.xml by running the fol‐lowing commands:% mvn compile% export HADOOP_CLASSPATH=target/classes/% hadoop ConfigurationPrinter -conf conf/hadoop-localhost.xml \| grep yarn.resourcemanager.address=yarn.resourcemanager.address=localhost:8032Which Properties Can I Set?ConfigurationPrinter is a useful tool for discovering what a property is set to in yourenvironment. For a running daemon, like the namenode, you can see its configurationby viewing the /conf page on its web server. (See Table 10-6 to find port numbers.)You can also see the default settings for all the public properties in Hadoop by lookingin the share/doc directory of your Hadoop installation for files called core-default.xml,hdfs-default.xml, yarn-default.xml, and mapred-default.xml.
Each property has a descrip‐tion that explains what it is for and what values it can be set to.The default settings files’ documentation can be found online at pages linked from http://hadoop.apache.org/docs/current/ (look for the “Configuration” heading in the naviga‐tion). You can find the defaults for a particular Hadoop release by replacing current inthe preceding URL with r<version>—for example, http://hadoop.apache.org/docs/r2.5.0/.Be aware that some properties have no effect when set in the client configuration.
Forexample, if you set yarn.nodemanager.resource.memory-mb in your job submissionwith the expectation that it would change the amount of memory available to the nodemanagers running your job, you would be disappointed, because this property is hon‐ored only if set in the node manager’s yarn-site.xml file. In general, you can tell thecomponent where a property should be set by its name, so the fact thatyarn.nodemanager.resource.memory-mb starts with yarn.nodemanager gives you aclue that it can be set only for the node manager daemon. This is not a hard and fastrule, however, so in some cases you may need to resort to trial and error, or even toreading the source.Configuration property names have changed in Hadoop 2 onward, in order to give thema more regular naming structure.
For example, the HDFS properties pertaining to thenamenode have been changed to have a dfs.namenode prefix, so dfs.name.dir is nowdfs.namenode.name.dir. Similarly, MapReduce properties have the mapreduce prefixrather than the older mapred prefix, so mapred.job.name is now mapreduce.job.name.150|Chapter 6: Developing a MapReduce ApplicationThis book uses the new property names to avoid deprecation warnings.
The old propertynames still work, however, and they are often referred to in older documentation. Youcan find a table listing the deprecated property names and their replacements on theHadoop website.We discuss many of Hadoop’s most important configuration properties throughout thisbook.GenericOptionsParser also allows you to set individual properties. For example:% hadoop ConfigurationPrinter -D color=yellow | grep colorcolor=yellowHere, the -D option is used to set the configuration property with key color to the valueyellow. Options specified with -D take priority over properties from the configurationfiles.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
