Install Hadoop in Kali Linux

In this tutorial I am going to demonstrate the steps to setup and install a single node hadoop cluster in kali linux.

Installing Hadoop:

Step 1: Install Java if it is not present on your machine.

Step 2: Download Hadoop and extract it.

Step 3: Set environment variables in ~/.bashrc.

export HADOOP_HOME=/root/Downloads/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$DERBY_HOME/bin
export JAVA_HOME=/usr/local/jdk1.7.0_79

Step 4: Apply the changes.

source ~/.bashrc

Step 5: Edit hadoop-env.sh.

export JAVA_HOME=/usr/local/jdk1.7.0_79

Step 6: Edit core-site.xml.

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://127.0.0.1:9000</value>
    </property>
</configuration>

Step 7: Edit hdfs-site.xml.

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/root/hadoopinfra/hdfs/namenode</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/root/hadoopinfra/hdfs/datanode</value>
    </property>
</configuration>

Step 8: Edit yarn-site.xml.

<configuration>

   <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
   </property>
 
</configuration>

Step 9: Copy mapreduce site template.

cp mapred-site.xml.template mapred-site.xml

Step 10: Edit mapred-site.xml.

<configuration>

   <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>

</configuration>

Step 11: Format to HDFS and start services.

hdfs namenode -format
start-dfs.sh
start-yarn.sh

Here I have used start-all.sh which is deprecated now.

Step 12: Access Hadoop from browser.

Localhost:8088
Localhost:50070