Saturday, August 22, 2015

Hadoop Basic Commands

  • List all hadoop filesystem shell commands :

$> hadoop fs

  • Help for the hadoop filesystem command :

$> hadoop fs -help

  • Print the version of Hadoop :

$> hadoop version

  • Print the content of the root directory (HDFS) :

$> hadoop fs -ls /

  • Print the amount of space used :

$> hadoop fs -df hdfs:/

  • Count the number of directory, files and bytes :

$> hadoop fs -count hdfs:/

  • Create a new directory "Subdirectory" under root :

$> hadoop fs -mkdir Root/Directory/Subdirectory

  • Upload a file A.TXT from local pathtofile to hdfs "Subdirectory" :

$> hadoop fs -put pathtofile/A.TXT /Root/Directory/Subdirectory

  • Print / show the contents of the "Subdirectory" :

$> hadoop fs -ls Root/Directory/Subdirectory

  • Delete / Remove the file A.TXT from the hdfs :

$> hadoop fs -rm Root/Directory/Subdirectory/A.TXT

  • Delete / clear / purge all files in the trash bin :

$> hadoop fs -expunge

  • Command to copy a file from local to Hadoop (Alternative to put command) :

$> hadoop fs -copyFromLocal pathtofile/A.TXT /Root/Directory/Subdirectory

  • Command to view the contents of a file on HDFS :

$> hadoop fs -cat pathtofile/A.TXT

  • Command to copy a file from Hadoop FS to Local (Alternative to get command) :

$> hadoop fs -copyToLocal /Root/Directory/Subdirectory /pathtolocal/A.TXT 

  • Copying files between 2 HDFS directories :

$> hadoop fs -cp /path/to/hadoop1/*.txt /path/to/hadoop2/

  • Download file A.TXT from HDFS to local path :

$> hadoop fs -get hadoop/A.txt /path/to/local

  • Unix Tail command on hadoop fs (HDFS)n:

$> hadoop fs -tail path/to/hadoop/A.TXT

I also keep some articles in "www.itsmadeinindia.net"

Hope this helps.

No comments:

Post a Comment

The Azure Synapse Resource Provider Error

  If you are get the error " The Azure Synapse resource provider (Microsoft.Synapse) needs to be registered with the selected subscript...