5/24/18

Hadoop HDFS Commands



HDFS Commands
  • jps
HDFS Command to print Hadoop processes.
[root@quickstart Desktop]# jps

  • fsck
HDFS Command to check the health of the Hadoop file system.
[cloudera@quickstart training]$ hdfs fsck /

·        ls
HDFS Command to display the list of Files and Directories in HDFS.
[cloudera@quickstart training]$ hdfs dfs -ls /

  • mkdir
HDFS Command to create the directory in HDFS.
[cloudera@quickstart training]$ hdfs dfs -mkdir /bigdatatesting
[cloudera@quickstart training]$ hdfs dfs -ls /
drwxr-xr-x   - cloudera supergroup          0 2018-05-23 00:46 /bigdatatesting
Note: Here we are trying to create a directory named “Bigdatatesting” in HDFS.

  • touchz
HDFS Command to create a file in HDFS with file size 0 bytes.
[cloudera@quickstart training]$ hdfs dfs -touchz /bigdatatesting/test.dat
[cloudera@quickstart training]$ hdfs dfs -ls /bigdatatesting/
Found 1 items
-rw-r--r--   1 cloudera supergroup          0 2018-05-23 00:48 /bigdatatesting/test.tx

Note: Here we are trying to create a file named “test.dat” in the directory “bigdatatesting” of hdfs with file size 0 bytes.

  • du
HDFS Command to check the file size. 
[cloudera@quickstart training]$ hdfs dfs -du -s /bigdatatesting/test.dat
0  0  /bigdatatesting/test.dat

·        appendToFile
Appends the contents to the given destination file on HDFS. The destination file will be created if it does not exist.
[cloudera@quickstart training]$ hdfs dfs -appendToFile - /bigdatatesting/test.dat

  • cat
HDFS Command that reads a file on HDFS and prints the content of that file to the standard output.
 [cloudera@quickstart training]$ hdfs dfs -cat /bigdatatesting/test.dat

  • copyFromLocal
HDFS Command to copy the file from a Local file system to HDFS.
Step1: Create a file in Local File System.
[cloudera@quickstart training]$ cat>> test1.dat
[cloudera@quickstart training]$ ls test1.dat
test1.dat
Step2: Copy file from Local File system to HDFS
[cloudera@quickstart training]$ hdfs dfs -copyFromLocal test1.dat /bigdatatesting/
Note: Here the test is the file present in the local directory /home/cloudera/training and after the command gets executed the test file will be copied in /bigdatatesting directory of HDFS.

  • copyToLocal
HDFS Command to copy the file from HDFS to Local File System.
Step1: Check test.dat file present in local file system.
[cloudera@quickstart training]$ ls test.dat
ls: cannot access test.dat: No such file or directory
Step2: Copy test.dat file from HDFS to local file system. 
[cloudera@quickstart training]$ hdfs dfs -copyToLocal /bigdatatesting/test.dat /home/cloudera/training
Step3: Check again test.dat file present in local file system.
[cloudera@quickstart training]$ ls test.dat
test.dat
Note: Here test.dat is a file present in the bigdatatesting directory of HDFS and after the command gets executed the test.dat file will be copied to local directory /home/Cloudera/training

  • put
HDFS Command to copy single source or multiple sources from local file system to the destination file system.
Step1: Create a file in Local File System.
[cloudera@quickstart training]$ cat>> test2.dat
[cloudera@quickstart training]$ ls test2.dat
test1.dat
Step2: Copy file from Local File system to HDFS
[cloudera@quickstart training]$ hdfs dfs -put test2.dat /bigdatatesting/
Note: Here the test2.dat is the file present in the local directory /home/cloudera/training and after the command gets executed the test2.dat file will be copied in /bigdatatesting directory of HDFS.
Note:  The command put is similar to copyFromLocal command.
  • ·        get

HDFS Command to copy files from hdfs to the local file system.
Step1: Create a new file test3.dat on HDFS.
[cloudera@quickstart training]$ hdfs dfs -touchz /bigdatatesting/test3.dat

Step2: Copy test3.dat file from HDFS to local file system. 
[cloudera@quickstart training]$ hdfs dfs -get /bigdatatesting/test3.dat /home/cloudera/training

Step3: Check again test3.dat file present in local file system.
[cloudera@quickstart training]$ ls test3.dat
Test3.dat

Note1: Here test3.dat is a file present in the bigdatatesting directory of HDFS and after the command gets executed the test.dat file will be copied to local directory /home/Cloudera/training

Note2: The command get is similar to copyToLocal  command

  • cp
HDFS Command to copy files from source to destination. This command allows multiple sources as well, in which case the destination must be a directory.
[cloudera@quickstart training]$ hdfs dfs -mkdir /hadooptesting/
[cloudera@quickstart training]$ hdfs dfs -cp /bigdatatesting/test.dat /hadooptesting

  • mv
HDFS Command to move files from source to destination. This command allows multiple sources as well, in which case the destination needs to be a directory.
[cloudera@quickstart training]$ hdfs dfs -mv /bigdatatesting/test1.dat /hadooptesting/

  • rm
HDFS Command to remove the file from HDFS.
[cloudera@quickstart training]$ hdfs dfs -rm /bigdatatesting/test2.dat
Deleted /bigdatatesting/test2.dat

  • rm -r
HDFS Command to remove the entire directory and all of its content from HDFS.
[cloudera@quickstart training]$ hdfs dfs -rm -r /hadooptesting
Deleted /hadooptesting

  • rmdir
HDFS Command to remove the directory if it is empty.
[cloudera@quickstart training]$ hdfs dfs -rmdir /bigdatatesting

  • usage
HDFS Command that returns the help for an individual command.
[cloudera@quickstart training]$ hdfs dfs -usage mkdir
Note: By using usage command you can get information about any command.

  • help
HDFS Command that displays help for given command or all commands if none is specified.
[cloudera@quickstart training]$ hdfs dfs -help






0 comments:

Post a Comment