Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 64
Текст из файла (страница 64)
This is a conser‐vative setting, and with modern hardware and operating systems, you will likely seeperformance benefits by increasing it; 128 KB (131,072 bytes) is a common choice. Setthe value in bytes using the io.file.buffer.size property in core-site.xml.HDFS block sizeThe HDFS block size is 128 MB by default, but many clusters use more (e.g., 256 MB,which is 268,435,456 bytes) to ease memory pressure on the namenode and to givemappers more data to work on. Use the dfs.blocksize property in hdfs-site.xml tospecify the size in bytes.Reserved storage spaceBy default, datanodes will try to use all of the space available in their storage directories.If you want to reserve some space on the storage volumes for non-HDFS use, you canset dfs.datanode.du.reserved to the amount, in bytes, of space to reserve.TrashHadoop filesystems have a trash facility, in which deleted files are not actually deletedbut rather are moved to a trash folder, where they remain for a minimum period beforebeing permanently deleted by the system.
The minimum period in minutes that a filewill remain in the trash is set using the fs.trash.interval configuration property incore-site.xml. By default, the trash interval is zero, which disables trash.Like in many operating systems, Hadoop’s trash facility is a user-level feature, meaningthat only files that are deleted using the filesystem shell are put in the trash.
Files deletedprogrammatically are deleted immediately. It is possible to use the trashHadoop Configuration|307programmatically, however, by constructing a Trash instance, then calling its moveToTrash() method with the Path of the file intended for deletion. The method returns avalue indicating success; a value of false means either that trash is not enabled or thatthe file is already in the trash.When trash is enabled, users each have their own trash directories called .Trash in theirhome directories.
File recovery is simple: you look for the file in a subdirectoryof .Trash and move it out of the trash subtree.HDFS will automatically delete files in trash folders, but other filesystems will not, soyou have to arrange for this to be done periodically. You can expunge the trash, whichwill delete files that have been in the trash longer than their minimum period, using thefilesystem shell:% hadoop fs -expungeThe Trash class exposes an expunge() method that has the same effect.Job schedulerParticularly in a multiuser setting, consider updating the job scheduler queue configu‐ration to reflect your organizational needs. For example, you can set up a queue for eachgroup using the cluster. See “Scheduling in YARN” on page 85.Reduce slow startBy default, schedulers wait until 5% of the map tasks in a job have completed beforescheduling reduce tasks for the same job.
For large jobs, this can cause problems withcluster utilization, since they take up reduce containers while waiting for the map tasksto complete. Setting mapreduce.job.reduce.slowstart.completedmaps to a highervalue, such as 0.80 (80%), can help improve throughput.Short-circuit local readsWhen reading a file from HDFS, the client contacts the datanode and the data is sentto the client via a TCP connection.
If the block being read is on the same node as theclient, then it is more efficient for the client to bypass the network and read the blockdata directly from the disk. This is termed a short-circuit local read, and can makeapplications like HBase perform better.You can enable short-circuit local reads by setting dfs.client.read.shortcircuit totrue. Short-circuit local reads are implemented using Unix domain sockets, which usea local path for client-datanode communication. The path is set using the propertydfs.domain.socket.path, and must be a path that only the datanode user (typicallyhdfs) or root can create, such as /var/run/hadoop-hdfs/dn_socket.308|Chapter 10: Setting Up a Hadoop ClusterSecurityEarly versions of Hadoop assumed that HDFS and MapReduce clusters would be usedby a group of cooperating users within a secure environment. The measures forrestricting access were designed to prevent accidental data loss, rather than to preventunauthorized access to data.
For example, the file permissions system in HDFS preventsone user from accidentally wiping out the whole filesystem because of a bug in a pro‐gram, or by mistakenly typing hadoop fs -rmr /, but it doesn’t prevent a malicioususer from assuming root’s identity to access or delete any data in the cluster.In security parlance, what was missing was a secure authentication mechanism to assureHadoop that the user seeking to perform an operation on the cluster is who he claimsto be and therefore can be trusted. HDFS file permissions provide only a mechanismfor authorization, which controls what a particular user can do to a particular file.
Forexample, a file may be readable only by a certain group of users, so anyone not in thatgroup is not authorized to read it. However, authorization is not enough by itself, becausethe system is still open to abuse via spoofing by a malicious user who can gain networkaccess to the cluster.It’s common to restrict access to data that contains personally identifiable information(such as an end user’s full name or IP address) to a small set of users (of the cluster)within the organization who are authorized to access such information. Less sensitive(or anonymized) data may be made available to a larger set of users.
It is convenient tohost a mix of datasets with different security levels on the same cluster (not least becauseit means the datasets with lower security levels can be shared). However, to meet reg‐ulatory requirements for data protection, secure authentication must be in place forshared clusters.This is the situation that Yahoo! faced in 2009, which led a team of engineers there toimplement secure authentication for Hadoop. In their design, Hadoop itself does notmanage user credentials; instead, it relies on Kerberos, a mature open-source networkauthentication protocol, to authenticate the user.
However, Kerberos doesn’t managepermissions. Kerberos says that a user is who she says she is; it’s Hadoop’s job to deter‐mine whether that user has permission to perform a given action.There’s a lot to security in Hadoop, and this section only covers the highlights. For more,readers are referred to Hadoop Security by Ben Spivey and Joey Echeverria (O’Reilly,2014).Kerberos and HadoopAt a high level, there are three steps that a client must take to access a service when usingKerberos, each of which involves a message exchange with a server:Security|3091.
Authentication. The client authenticates itself to the Authentication Server andreceives a timestamped Ticket-Granting Ticket (TGT).2. Authorization. The client uses the TGT to request a service ticket from the TicketGranting Server.3. Service request. The client uses the service ticket to authenticate itself to the serverthat is providing the service the client is using.
In the case of Hadoop, this mightbe the namenode or the resource manager.Together, the Authentication Server and the Ticket Granting Server form the Key Dis‐tribution Center (KDC). The process is shown graphically in Figure 10-2.Figure 10-2. The three-step Kerberos ticket exchange protocolThe authorization and service request steps are not user-level actions; the client per‐forms these steps on the user’s behalf. The authentication step, however, is normallycarried out explicitly by the user using the kinit command, which will prompt for apassword. However, this doesn’t mean you need to enter your password every time yourun a job or access HDFS, since TGTs last for 10 hours by default (and can be renewedfor up to a week).
It’s common to automate authentication at operating system logintime, thereby providing single sign-on to Hadoop.In cases where you don’t want to be prompted for a password (for running an unattendedMapReduce job, for example), you can create a Kerberos keytab file using the ktutilcommand.
A keytab is a file that stores passwords and may be supplied to kinit withthe -t option.310|Chapter 10: Setting Up a Hadoop ClusterAn exampleLet’s look at an example of the process in action. The first step is to enable Kerberosauthentication by setting the hadoop.security.authentication property in coresite.xml to kerberos.7 The default setting is simple, which signifies that the oldbackward-compatible (but insecure) behavior of using the operating system usernameto determine identity should be employed.We also need to enable service-level authorization by setting hadoop.security.authorization to true in the same file.
You may configure access control lists (ACLs) in thehadoop-policy.xml configuration file to control which users and groups have permissionto connect to each Hadoop service. Services are defined at the protocol level, so thereare ones for MapReduce job submission, namenode communication, and so on. Bydefault, all ACLs are set to *, which means that all users have permission to access eachservice; however, on a real cluster you should lock the ACLs down to only those usersand groups that should have access.The format for an ACL is a comma-separated list of usernames, followed by whitespace,followed by a comma-separated list of group names.
For example, the ACLpreston,howard directors,inventors would authorize access to users namedpreston or howard, or in groups directors or inventors.With Kerberos authentication turned on, let’s see what happens when we try to copy alocal file to HDFS:% hadoop fs -put quangle.txt .10/07/03 15:44:58 WARN ipc.Client: Exception encountered while connecting to theserver: javax.security.sasl.SaslException: GSS initiate failed [Caused byGSSException: No valid credentials provided (Mechanism level: Failed to findany Kerberos tgt)]Bad connection to FS. command aborted. exception: Call to localhost/127.0.0.1:8020 failed on local exception: java.io.IOException:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:No valid credentials provided(Mechanism level: Failed to find any Kerberos tgt)]The operation fails because we don’t have a Kerberos ticket.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
