Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 13
Текст из файла (страница 13)
There is no support for multiple writers orfor modifications at arbitrary offsets in the file. (These might be supported in thefuture, but they are likely to be relatively inefficient.)HDFS ConceptsBlocksA disk has a block size, which is the minimum amount of data that it can read or write.Filesystems for a single disk build on this by dealing with data in blocks, which are anintegral multiple of the disk block size. Filesystem blocks are typically a few kilobytesin size, whereas disk blocks are normally 512 bytes. This is generally transparent to thefilesystem user who is simply reading or writing a file of whatever length.
However,there are tools to perform filesystem maintenance, such as df and fsck, that operate onthe filesystem block level.HDFS, too, has the concept of a block, but it is a much larger unit—128 MB by default.Like in a filesystem for a single disk, files in HDFS are broken into block-sized chunks,which are stored as independent units. Unlike a filesystem for a single disk, a file inHDFS that is smaller than a single block does not occupy a full block’s worth of under‐lying storage. (For example, a 1 MB file stored with a block size of 128 MB uses 1 MBof disk space, not 128 MB.) When unqualified, the term “block” in this book refers to ablock in HDFS.Why Is a Block in HDFS So Large?HDFS blocks are large compared to disk blocks, and the reason is to minimize the costof seeks. If the block is large enough, the time it takes to transfer the data from the diskcan be significantly longer than the time to seek to the start of the block.
Thus, trans‐ferring a large file made of multiple blocks operates at the disk transfer rate.A quick calculation shows that if the seek time is around 10 ms and the transfer rate is100 MB/s, to make the seek time 1% of the transfer time, we need to make the block sizearound 100 MB. The default is actually 128 MB, although many HDFS installations uselarger block sizes. This figure will continue to be revised upward as transfer speeds growwith new generations of disk drives.This argument shouldn’t be taken too far, however. Map tasks in MapReduce normallyoperate on one block at a time, so if you have too few tasks (fewer than nodes in thecluster), your jobs will run slower than they could otherwise.HDFS Concepts|45Having a block abstraction for a distributed filesystem brings several benefits. The firstbenefit is the most obvious: a file can be larger than any single disk in the network.There’s nothing that requires the blocks from a file to be stored on the same disk, sothey can take advantage of any of the disks in the cluster.
In fact, it would be possible,if unusual, to store a single file on an HDFS cluster whose blocks filled all the disks inthe cluster.Second, making the unit of abstraction a block rather than a file simplifies the storagesubsystem. Simplicity is something to strive for in all systems, but it is especiallyimportant for a distributed system in which the failure modes are so varied.
The storagesubsystem deals with blocks, simplifying storage management (because blocks are afixed size, it is easy to calculate how many can be stored on a given disk) and eliminatingmetadata concerns (because blocks are just chunks of data to be stored, file metadatasuch as permissions information does not need to be stored with the blocks, so anothersystem can handle metadata separately).Furthermore, blocks fit well with replication for providing fault tolerance and availa‐bility.
To insure against corrupted blocks and disk and machine failure, each block isreplicated to a small number of physically separate machines (typically three). If a blockbecomes unavailable, a copy can be read from another location in a way that is trans‐parent to the client. A block that is no longer available due to corruption or machinefailure can be replicated from its alternative locations to other live machines to bringthe replication factor back to the normal level. (See “Data Integrity” on page 97 for moreon guarding against corrupt data.) Similarly, some applications may choose to set a highreplication factor for the blocks in a popular file to spread the read load on the cluster.Like its disk filesystem cousin, HDFS’s fsck command understands blocks.
For example,running:% hdfs fsck / -files -blockswill list the blocks that make up each file in the filesystem. (See also “Filesystem check(fsck)” on page 326.)Namenodes and DatanodesAn HDFS cluster has two types of nodes operating in a master−worker pattern: anamenode (the master) and a number of datanodes (workers). The namenode managesthe filesystem namespace.
It maintains the filesystem tree and the metadata for all thefiles and directories in the tree. This information is stored persistently on the local diskin the form of two files: the namespace image and the edit log. The namenode also knowsthe datanodes on which all the blocks for a given file are located; however, it doesnot store block locations persistently, because this information is reconstructed fromdatanodes when the system starts.46|Chapter 3: The Hadoop Distributed FilesystemA client accesses the filesystem on behalf of the user by communicating with the name‐node and datanodes.
The client presents a filesystem interface similar to a PortableOperating System Interface (POSIX), so the user code does not need to know about thenamenode and datanodes to function.Datanodes are the workhorses of the filesystem. They store and retrieve blocks whenthey are told to (by clients or the namenode), and they report back to the namenodeperiodically with lists of blocks that they are storing.Without the namenode, the filesystem cannot be used. In fact, if the machine runningthe namenode were obliterated, all the files on the filesystem would be lost since therewould be no way of knowing how to reconstruct the files from the blocks on thedatanodes. For this reason, it is important to make the namenode resilient to failure,and Hadoop provides two mechanisms for this.The first way is to back up the files that make up the persistent state of the filesystemmetadata.
Hadoop can be configured so that the namenode writes its persistent state tomultiple filesystems. These writes are synchronous and atomic. The usual configurationchoice is to write to local disk as well as a remote NFS mount.It is also possible to run a secondary namenode, which despite its name does not act asa namenode. Its main role is to periodically merge the namespace image with the editlog to prevent the edit log from becoming too large. The secondary namenode usuallyruns on a separate physical machine because it requires plenty of CPU and as muchmemory as the namenode to perform the merge. It keeps a copy of the merged name‐space image, which can be used in the event of the namenode failing.
However, the stateof the secondary namenode lags that of the primary, so in the event of total failure ofthe primary, data loss is almost certain. The usual course of action in this case is to copythe namenode’s metadata files that are on NFS to the secondary and run it as the newprimary. (Note that it is possible to run a hot standby namenode instead of a secondary,as discussed in “HDFS High Availability” on page 48.)See “The filesystem image and edit log” on page 318 for more details.Block CachingNormally a datanode reads blocks from disk, but for frequently accessed files the blocksmay be explicitly cached in the datanode’s memory, in an off-heap block cache. Bydefault, a block is cached in only one datanode’s memory, although the number is con‐figurable on a per-file basis.
Job schedulers (for MapReduce, Spark, and other frame‐works) can take advantage of cached blocks by running tasks on the datanode where ablock is cached, for increased read performance. A small lookup table used in a join isa good candidate for caching, for example.HDFS Concepts|47Users or applications instruct the namenode which files to cache (and for how long) byadding a cache directive to a cache pool. Cache pools are an administrative grouping formanaging cache permissions and resource usage.HDFS FederationThe namenode keeps a reference to every file and block in the filesystem in memory,which means that on very large clusters with many files, memory becomes the limitingfactor for scaling (see “How Much Memory Does a Namenode Need?” on page 294).HDFS federation, introduced in the 2.x release series, allows a cluster to scale by addingnamenodes, each of which manages a portion of the filesystem namespace.
For example,one namenode might manage all the files rooted under /user, say, and a second name‐node might handle files under /share.Under federation, each namenode manages a namespace volume, which is made up ofthe metadata for the namespace, and a block pool containing all the blocks for the filesin the namespace. Namespace volumes are independent of each other, which meansnamenodes do not communicate with one another, and furthermore the failure of onenamenode does not affect the availability of the namespaces managed by other namen‐odes. Block pool storage is not partitioned, however, so datanodes register with eachnamenode in the cluster and store blocks from multiple block pools.To access a federated HDFS cluster, clients use client-side mount tables to map file pathsto namenodes.
This is managed in configuration using ViewFileSystem and theviewfs:// URIs.HDFS High AvailabilityThe combination of replicating namenode metadata on multiple filesystems and usingthe secondary namenode to create checkpoints protects against data loss, but it doesnot provide high availability of the filesystem.
The namenode is still a single point offailure (SPOF). If it did fail, all clients—including MapReduce jobs—would be unableto read, write, or list files, because the namenode is the sole repository of the metadataand the file-to-block mapping. In such an event, the whole Hadoop system would ef‐fectively be out of service until a new namenode could be brought online.To recover from a failed namenode in this situation, an administrator starts a new pri‐mary namenode with one of the filesystem metadata replicas and configures datanodesand clients to use this new namenode. The new namenode is not able to serve requestsuntil it has (i) loaded its namespace image into memory, (ii) replayed its edit log, and(iii) received enough block reports from the datanodes to leave safe mode.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
