Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 65
Текст из файла (страница 65)
We can get one by authen‐ticating to the KDC, using kinit:% kinitPassword for hadoop-user@LOCALDOMAIN: password% hadoop fs -put quangle.txt .% hadoop fs -stat %n quangle.txtquangle.txt7. To use Kerberos authentication with Hadoop, you need to install, configure, and run a KDC (Hadoop doesnot come with one). Your organization may already have a KDC you can use (an Active Directory installation,for example); if not, you can set up an MIT Kerberos 5 KDC.Security|311And we see that the file is successfully written to HDFS.
Notice that even though wecarried out two filesystem commands, we only needed to call kinit once, since theKerberos ticket is valid for 10 hours (use the klist command to see the expiry time ofyour tickets and kdestroy to invalidate your tickets). After we get a ticket, everythingworks just as it normally would.Delegation TokensIn a distributed system such as HDFS or MapReduce, there are many client-serverinteractions, each of which must be authenticated. For example, an HDFS read operationwill involve multiple calls to the namenode and calls to one or more datanodes. Insteadof using the three-step Kerberos ticket exchange protocol to authenticate each call,which would present a high load on the KDC on a busy cluster, Hadoop uses delegationtokens to allow later authenticated access without having to contact the KDC again.Delegation tokens are created and used transparently by Hadoop on behalf of users, sothere’s no action you need to take as a user beyond using kinit to sign in, but it’s usefulto have a basic idea of how they are used.A delegation token is generated by the server (the namenode, in this case) and can bethought of as a shared secret between the client and the server.
On the first RPC call tothe namenode, the client has no delegation token, so it uses Kerberos to authenticate.As a part of the response, it gets a delegation token from the namenode. In subsequentcalls it presents the delegation token, which the namenode can verify (since it generatedit using a secret key), and hence the client is authenticated to the server.When it wants to perform operations on HDFS blocks, the client uses a special kind ofdelegation token, called a block access token, that the namenode passes to the client inresponse to a metadata request. The client uses the block access token to authenticateitself to datanodes. This is possible only because the namenode shares its secret key usedto generate the block access token with datanodes (sending it in heartbeat messages),so that they can verify block access tokens.
Thus, an HDFS block may be accessed onlyby a client with a valid block access token from a namenode. This closes the securityhole in unsecured Hadoop where only the block ID was needed to gain access to a block.This property is enabled by setting dfs.block.access.token.enable to true.In MapReduce, job resources and metadata (such as JAR files, input splits, and config‐uration files) are shared in HDFS for the application master to access, and user coderuns on the node managers and accesses files on HDFS (the process is explained in“Anatomy of a MapReduce Job Run” on page 185). Delegation tokens are used by thesecomponents to access HDFS during the course of the job.
When the job has finished,the delegation tokens are invalidated.Delegation tokens are automatically obtained for the default HDFS instance, but if yourjob needs to access other HDFS clusters, you can load the delegation tokens for these312| Chapter 10: Setting Up a Hadoop Clusterby setting the mapreduce.job.hdfs-servers job property to a comma-separated list ofHDFS URIs.Other Security EnhancementsSecurity has been tightened throughout the Hadoop stack to protect against unauthor‐ized access to resources.
The more notable features are listed here:• Tasks can be run using the operating system account for the user who submittedthe job, rather than the user running the node manager. This means that the oper‐ating system is used to isolate running tasks, so they can’t send signals to each other(to kill another user’s tasks, for example) and so local information, such as task data,is kept private via local filesystem permissions.This feature is enabled by setting yarn.nodemanager.containerexecutor.class to org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.8 In addition, administrators need to ensure that each user is givenan account on every node in the cluster (typically using LDAP).• When tasks are run as the user who submitted the job, the distributed cache (see“Distributed Cache” on page 274) is secure. Files that are world-readable are put ina shared cache (the insecure default); otherwise, they go in a private cache, readableonly by the owner.• Users can view and modify only their own jobs, not others.
This is enabled by settingmapreduce.cluster.acls.enabled to true. There are two job configuration prop‐erties, mapreduce.job.acl-view-job and mapreduce.job.acl-modify-job,which may be set to a comma-separated list of users to control who may view ormodify a particular job.• The shuffle is secure, preventing a malicious user from requesting another user’smap outputs.• When appropriately configured, it’s no longer possible for a malicious user to runa rogue secondary namenode, datanode, or node manager that can join the clusterand potentially compromise data stored in the cluster.
This is enforced by requiringdaemons to authenticate with the master node they are connecting to.To enable this feature, you first need to configure Hadoop to use a keytab previouslygenerated with the ktutil command. For a datanode, for example, you would setthe dfs.datanode.keytab.file property to the keytab filename and dfs.datanode.kerberos.principal to the username to use for the datanode. Finally, theACL for the DataNodeProtocol (which is used by datanodes to communicate with8. LinuxTaskController uses a setuid executable called container-executor, found in the bin directory. Youshould ensure that this binary is owned by root and has the setuid bit set (with chmod +s).Security|313the namenode) must be set in hadoop-policy.xml, by restricting security.datanode.protocol.acl to the datanode’s username.• A datanode may be run on a privileged port (one lower than 1024), so a client maybe reasonably sure that it was started securely.• A task may communicate only with its parent application master, thus preventingan attacker from obtaining MapReduce data from another user’s job.• Various parts of Hadoop can be configured to encrypt network data, including RPC(hadoop.rpc.protection), HDFS block transfers (dfs.encrypt.data.transfer),the MapReduce shuffle (mapreduce.shuffle.ssl.enabled), and the web UIs(hadokop.ssl.enabled).
Work is ongoing to encrypt data “at rest,” too, so thatHDFS blocks can be stored in encrypted form, for example.Benchmarking a Hadoop ClusterIs the cluster set up correctly? The best way to answer this question is empirically: runsome jobs and confirm that you get the expected results. Benchmarks make good testsbecause you also get numbers that you can compare with other clusters as a sanity checkon whether your new cluster is performing roughly as expected. And you can tune acluster using benchmark results to squeeze the best performance out of it.
This is oftendone with monitoring systems in place (see “Monitoring” on page 330), so you can seehow resources are being used across the cluster.To get the best results, you should run benchmarks on a cluster that is not being usedby others. In practice, this will be just before it is put into service and users start relyingon it. Once users have scheduled periodic jobs on a cluster, it is generally impossible tofind a time when the cluster is not being used (unless you arrange downtime with users),so you should run benchmarks to your satisfaction before this happens.Experience has shown that most hardware failures for new systems are hard drive fail‐ures.
By running I/O-intensive benchmarks—such as the ones described next—you can“burn in” the cluster before it goes live.Hadoop BenchmarksHadoop comes with several benchmarks that you can run very easily with minimal setupcost. Benchmarks are packaged in the tests JAR file, and you can get a list of them, withdescriptions, by invoking the JAR file with no arguments:% hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-*-tests.jarMost of the benchmarks show usage instructions when invoked with no arguments.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
