Hadoop Copy Local File to HDFS – PUT Command

Hadoop Copy Local HDFS

In order to copy a file from the local file system to HDFS, use Hadoop fs -put or hdfs dfs -put, on put command, specify the local-file-path where you wanted to copy from and then HDFS-file-path where you wanted to copy to. If the file already exists on HDFS, you will get an error message saying “File already exists”. In order to overwrite the file use -f option.

Note that you can use it with either hadoop fs -put or hdfs dfs -put to upload files from the local file system to HDFS, both return the same results.

Copying files from a local file to HDFS file system, Similar to the fs -put command and copyFromLocal command both are Store files from the local file system to HDFS. Except that the source is restricted to a local file reference. For more information follow the Hadoop hdfs commands.


$ hadoop fs -put /local-file-path /hdfs-file-path
or
$ hdfs dfs -put /local-file-path /hdfs-file-path

Hadoop fs -put Command

The Hadoop fs shell command –put is used to copy the file from local file system to Hadoop HDFS file system. similarly HDFS also has –copyFromLocal. Below is the usage of -put command.

Alternatively you can also use hdfs dfs -put or hdfs dfs -copyFromLocal.


$hadoop fs -put [-f] [-p] [-l] [-d] /local-file-path /hdfs-file-path
or
$hdfs dfs -put [-f] [-p] [-l] [-d] /local-file-path /hdfs-file-path

Options:

HDFS put OptionsDescription
-pPreserves access and modification times, ownership, and permissions. (assuming the permissions can be propagated across filesystems)
-fOverwrites the destination if it already exists
-lAllow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care.
-dSkip creation of a temporary file with the suffix ._COPYING_.
Hadoop fs -put options

Hadoop fs -put Command Examples

Below are the examples of how to use hadoop hdfs put command with several options.

Example 1: Preserves Access and Modification Times

-p: Preserves access and modification time, ownership, and the mode Change the permission of a file, similar to Linux shell’s command but with a few exceptions.


$hadoop fs -put -p /local-file-path /hdfs-file-path
or
$hdfs dfs -put -p /local-file-path /hdfs-file-path

Example 2: Overwrites the Destination


$hadoop fs -put -f /local-file-path /hdfs-file-path
or
$hdfs dfs -put -f /local-file-path /hdfs-file-path

Example 3: DataNode to Lazily Persist the File to Disk


$hadoop fs -put -l /local-file-path /hdfs-file-path
or
$hdfs dfs -put -l /local-file-path /hdfs-file-path

Example 4: Skip Creation of a Temporary File

It creates a temp file by adding the suffix “.COPYING”. Once the file is successfully copied, it will remove the suffix by rename().


$hadoop fs -put -d /local-file-path /hdfs-file-path
or
$hdfs dfs -put -d /local-file-path /hdfs-file-path

Conclusion

In this article, you have learned how to copy a file from the local file system to the Hadoop HDFS file system using -put and -copyFromLocal commands. Also learned different options available with these commands with examples.

Reference

Leave a Reply