In order to copy a file from the local file system to HDFS, use Hadoop fs -put or hdfs dfs -put, on put command, specify the local-file-path where you wanted to copy from and then HDFS-file-path where you wanted to copy to. If the file already exists on HDFS, you will get an error message saying “File already exists”. In order to overwrite the file use -f option.
Note that you can use it with either hadoop fs -put or hdfs dfs -put to upload files from the local file system to HDFS, both return the same results.
Copying files from a local file to HDFS file system, Similar to the fs -put command and copyFromLocal command both are Store files from the local file system to HDFS. Except that the source is restricted to a local file reference. For more information follow the Hadoop hdfs commands.
$ hadoop fs -put /local-file-path /hdfs-file-path
or
$ hdfs dfs -put /local-file-path /hdfs-file-path
Hadoop fs -put Command
The Hadoop fs shell command –put is used to copy the file from local file system to Hadoop HDFS file system. similarly HDFS also has –copyFromLocal. Below is the usage of -put command.
Alternatively you can also use hdfs dfs -put or hdfs dfs -copyFromLocal.
$hadoop fs -put [-f] [-p] [-l] [-d] /local-file-path /hdfs-file-path
or
$hdfs dfs -put [-f] [-p] [-l] [-d] /local-file-path /hdfs-file-path
Options:
| HDFS put Options | Description |
|---|---|
| -p | Preserves access and modification times, ownership, and permissions. (assuming the permissions can be propagated across filesystems) |
| -f | Overwrites the destination if it already exists |
| -l | Allow DataNode to lazily persist the file to disk, Forces a replication factor of 1. This flag will result in reduced durability. Use with care. |
| -d | Skip creation of a temporary file with the suffix ._COPYING_. |
Hadoop fs -put Command Examples
Below are the examples of how to use hadoop hdfs put command with several options.
Example 1: Preserves Access and Modification Times
-p: Preserves access and modification time, ownership, and the mode Change the permission of a file, similar to Linux shell’s command but with a few exceptions.
$hadoop fs -put -p /local-file-path /hdfs-file-path
or
$hdfs dfs -put -p /local-file-path /hdfs-file-path
Example 2: Overwrites the Destination
$hadoop fs -put -f /local-file-path /hdfs-file-path
or
$hdfs dfs -put -f /local-file-path /hdfs-file-path
Example 3: DataNode to Lazily Persist the File to Disk
$hadoop fs -put -l /local-file-path /hdfs-file-path
or
$hdfs dfs -put -l /local-file-path /hdfs-file-path
Example 4: Skip Creation of a Temporary File
It creates a temp file by adding the suffix “.COPYING”. Once the file is successfully copied, it will remove the suffix by rename().
$hadoop fs -put -d /local-file-path /hdfs-file-path
or
$hdfs dfs -put -d /local-file-path /hdfs-file-path
Conclusion
In this article, you have learned how to copy a file from the local file system to the Hadoop HDFS file system using -put and -copyFromLocal commands. Also learned different options available with these commands with examples.
Related Articles
- Hadoop Get File From HDFS to Local
- Hadoop FS | HDFS DFS Commands with Examples
- Hadoop Count Command – Returns HDFS File Size and File Counts
- Apache Hadoop Installation on Ubuntu (multi-node cluster).
- WARNING: “HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
- Hadoop Yarn Configuration on Cluster
- Spark Step-by-Step Setup on Hadoop Yarn Cluster
- Hadoop “WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform” warning