Tom White - Hadoop The Definitive Guide_ 4 edition - 2015 (811394), страница 16
Текст из файла (страница 16)
The last two arguments to the copyBytes()method are the buffer size used for copying and whether to close the streams when thecopy is complete. We close the input stream ourselves, and System.out doesn’t need tobe closed.The Java Interface|57Here’s a sample run:7% export HADOOP_CLASSPATH=hadoop-examples.jar% hadoop URLCat hdfs://localhost/user/tom/quangle.txtOn the top of the Crumpetty TreeThe Quangle Wangle sat,But his face you could not see,On account of his Beaver Hat.Reading Data Using the FileSystem APIAs the previous section explained, sometimes it is impossible to set a URLStreamHandlerFactory for your application. In this case, you will need to use the FileSystem APIto open an input stream for a file.A file in a Hadoop filesystem is represented by a Hadoop Path object (and nota java.io.File object, since its semantics are too closely tied to the local filesystem).You can think of a Path as a Hadoop filesystem URI, such as hdfs://localhost/user/tom/quangle.txt.FileSystem is a general filesystem API, so the first step is to retrieve an instance for thefilesystem we want to use—HDFS, in this case.
There are several static factory methodsfor getting a FileSystem instance:public static FileSystem get(Configuration conf) throws IOExceptionpublic static FileSystem get(URI uri, Configuration conf) throws IOExceptionpublic static FileSystem get(URI uri, Configuration conf, String user)throws IOExceptionA Configuration object encapsulates a client or server’s configuration, which is setusing configuration files read from the classpath, such as etc/hadoop/core-site.xml.
Thefirst method returns the default filesystem (as specified in core-site.xml, or the defaultlocal filesystem if not specified there). The second uses the given URI’s scheme andauthority to determine the filesystem to use, falling back to the default filesystem if noscheme is specified in the given URI. The third retrieves the filesystem as the given user,which is important in the context of security (see “Security” on page 309).In some cases, you may want to retrieve a local filesystem instance. For this, you canuse the convenience method getLocal():public static LocalFileSystem getLocal(Configuration conf) throws IOExceptionWith a FileSystem instance in hand, we invoke an open() method to get the inputstream for a file:public FSDataInputStream open(Path f) throws IOExceptionpublic abstract FSDataInputStream open(Path f, int bufferSize) throws IOException7.
The text is from The Quangle Wangle’s Hat by Edward Lear.58|Chapter 3: The Hadoop Distributed FilesystemThe first method uses a default buffer size of 4 KB.Putting this together, we can rewrite Example 3-1 as shown in Example 3-2.Example 3-2. Displaying files from a Hadoop filesystem on standard output by usingthe FileSystem directlypublic class FileSystemCat {public static void main(String[] args) throws Exception {String uri = args[0];Configuration conf = new Configuration();FileSystem fs = FileSystem.get(URI.create(uri), conf);InputStream in = null;try {in = fs.open(new Path(uri));IOUtils.copyBytes(in, System.out, 4096, false);} finally {IOUtils.closeStream(in);}}}The program runs as follows:% hadoop FileSystemCat hdfs://localhost/user/tom/quangle.txtOn the top of the Crumpetty TreeThe Quangle Wangle sat,But his face you could not see,On account of his Beaver Hat.FSDataInputStreamThe open() method on FileSystem actually returns an FSDataInputStream rather thana standard java.io class.
This class is a specialization of java.io.DataInputStreamwith support for random access, so you can read from any part of the stream:package org.apache.hadoop.fs;public class FSDataInputStream extends DataInputStreamimplements Seekable, PositionedReadable {// implementation elided}The Seekable interface permits seeking to a position in the file and provides a querymethod for the current offset from the start of the file (getPos()):public interface Seekable {void seek(long pos) throws IOException;long getPos() throws IOException;}The Java Interface|59Calling seek() with a position that is greater than the length of the file will result in anIOException.
Unlike the skip() method of java.io.InputStream, which positions thestream at a point later than the current position, seek() can move to an arbitrary,absolute position in the file.A simple extension of Example 3-2 is shown in Example 3-3, which writes a file tostandard output twice: after writing it once, it seeks to the start of the file and streamsthrough it once again.Example 3-3. Displaying files from a Hadoop filesystem on standard output twice, byusing seek()public class FileSystemDoubleCat {public static void main(String[] args) throws Exception {String uri = args[0];Configuration conf = new Configuration();FileSystem fs = FileSystem.get(URI.create(uri), conf);FSDataInputStream in = null;try {in = fs.open(new Path(uri));IOUtils.copyBytes(in, System.out, 4096, false);in.seek(0); // go back to the start of the fileIOUtils.copyBytes(in, System.out, 4096, false);} finally {IOUtils.closeStream(in);}}}Here’s the result of running it on a small file:% hadoop FileSystemDoubleCat hdfs://localhost/user/tom/quangle.txtOn the top of the Crumpetty TreeThe Quangle Wangle sat,But his face you could not see,On account of his Beaver Hat.On the top of the Crumpetty TreeThe Quangle Wangle sat,But his face you could not see,On account of his Beaver Hat.FSDataInputStream also implements the PositionedReadable interface for readingparts of a file at a given offset:public interface PositionedReadable {public int read(long position, byte[] buffer, int offset, int length)throws IOException;public void readFully(long position, byte[] buffer, int offset, int length)throws IOException;60|Chapter 3: The Hadoop Distributed Filesystempublic void readFully(long position, byte[] buffer) throws IOException;}The read() method reads up to length bytes from the given position in the file intothe buffer at the given offset in the buffer.
The return value is the number of bytesactually read; callers should check this value, as it may be less than length. The readFully() methods will read length bytes into the buffer (or buffer.length bytes forthe version that just takes a byte array buffer), unless the end of the file is reached, inwhich case an EOFException is thrown.All of these methods preserve the current offset in the file and are thread safe (althoughFSDataInputStream is not designed for concurrent access; therefore, it’s better to createmultiple instances), so they provide a convenient way to access another part of the file—metadata, perhaps—while reading the main body of the file.Finally, bear in mind that calling seek() is a relatively expensive operation and shouldbe done sparingly.
You should structure your application access patterns to rely onstreaming data (by using MapReduce, for example) rather than performing a largenumber of seeks.Writing DataThe FileSystem class has a number of methods for creating a file. The simplest is themethod that takes a Path object for the file to be created and returns an output streamto write to:public FSDataOutputStream create(Path f) throws IOExceptionThere are overloaded versions of this method that allow you to specify whether to for‐cibly overwrite existing files, the replication factor of the file, the buffer size to use whenwriting the file, the block size for the file, and file permissions.The create() methods create any parent directories of the file to bewritten that don’t already exist.
Though convenient, this behaviormay be unexpected. If you want the write to fail when the parentdirectory doesn’t exist, you should check for the existence of theparent directory first by calling the exists() method. Alternative‐ly, use FileContext, which allows you to control whether parentdirectories are created or not.There’s also an overloaded method for passing a callback interface, Progressable, soyour application can be notified of the progress of the data being written to thedatanodes:The Java Interface|61package org.apache.hadoop.util;public interface Progressable {public void progress();}As an alternative to creating a new file, you can append to an existing file using theappend() method (there are also some other overloaded versions):public FSDataOutputStream append(Path f) throws IOExceptionThe append operation allows a single writer to modify an already written file by openingit and writing data from the final offset in the file.
With this API, applications thatproduce unbounded files, such as logfiles, can write to an existing file after having closedit. The append operation is optional and not implemented by all Hadoop filesystems.For example, HDFS supports append, but S3 filesystems don’t.Example 3-4 shows how to copy a local file to a Hadoop filesystem. We illustrate progressby printing a period every time the progress() method is called by Hadoop, which isafter each 64 KB packet of data is written to the datanode pipeline.
(Note that thisparticular behavior is not specified by the API, so it is subject to change in later versionsof Hadoop. The API merely allows you to infer that “something is happening.”)Example 3-4. Copying a local file to a Hadoop filesystempublic class FileCopyWithProgress {public static void main(String[] args) throws Exception {String localSrc = args[0];String dst = args[1];InputStream in = new BufferedInputStream(new FileInputStream(localSrc));Configuration conf = new Configuration();FileSystem fs = FileSystem.get(URI.create(dst), conf);OutputStream out = fs.create(new Path(dst), new Progressable() {public void progress() {System.out.print(".");}});IOUtils.copyBytes(in, out, 4096, true);}}Typical usage:% hadoop FileCopyWithProgress input/docs/1400-8.txthdfs://localhost/user/tom/1400-8.txt.................Currently, none of the other Hadoop filesystems call progress() during writes.
Progressis important in MapReduce applications, as you will see in later chapters.62|Chapter 3: The Hadoop Distributed FilesystemFSDataOutputStreamThe create() method on FileSystem returns an FSDataOutputStream, which, likeFSDataInputStream, has a method for querying the current position in the file:package org.apache.hadoop.fs;public class FSDataOutputStream extends DataOutputStream implements Syncable {public long getPos() throws IOException {// implementation elided}// implementation elided}However, unlike FSDataInputStream, FSDataOutputStream does not permit seeking.This is because HDFS allows only sequential writes to an open file or appends to analready written file. In other words, there is no support for writing to anywhere otherthan the end of the file, so there is no value in being able to seek while writing.DirectoriesFileSystem provides a method to create a directory:public boolean mkdirs(Path f) throws IOExceptionThis method creates all of the necessary parent directories if they don’t already exist,just like the java.io.File’s mkdirs() method.
![](https://s.studizba.com/z.php?f=/templates/si/i/noavatar.png&w=100&h=100&t=1"){kind=avatar}
