Big Data and Hadoop (170 Blogs) Become a Certified Professional

Commissioning and Decommissioning Nodes in a Hadoop Cluster

Last updated on May 05,2022 33.9K Views


One of the most attractive features of Hadoop framework is its utilization of commodity hardware. However, this leads to frequent DataNode crashes in a Hadoop cluster. Another striking feature of Hadoop Framework is the ease of scale in accordance to the rapid growth in data volume. Because of these two reasons, one of the most common task of a Hadoop administrator is to commission (Add) and decommission (Remove) Data Nodes in a Hadoop Cluster.

Commissioning and Decommissioning Nodes in a Hadoop Cluster:

Decommissioning (Removing) Data Nodes in a Hadoop Cluster

Above diagram shows a step by step process to decommission a DataNode in the cluster.

The first task is to update the ‘exclude‘ files for both HDFS (hdfs-site.xml) and MapReduce (mapred-site.xml).

The ‘exclude’ file:

  • for jobtracker contains the list of hosts that should be excluded by the jobtracker. If the value is empty, no hosts are excluded.
  • for Namenode contains a list of hosts that are not permitted to connect to the Namenode.

Here is the sample configuration for the exclude file in hdfs-site.xml and mapred-site.xml:

hdfs-site.xml

<property>
<name>dfs.hosts.exclude</name>
<value>/home/hadoop/excludes</value>
<final>true</final>
</property>

mapred-site.xml

<property>
<name>mapred.hosts.exclude</name>
<value>/home/hadoop/excludes</value>
<final>true</final>
</property>

Note: The full pathname of the files must be specified.

Similarly, we have the ‘include’ files:

  • for jobtracker containing the list of nodes that may connect to the JobTracker. If the value is empty, all hosts are permitted.
  • for Namenode containing a list of hosts that are permitted to connect to the Namenode. If the value is empty, all hosts are permitted.

The ‘dfsadmin’ and ‘mradmin’ commands refresh the configuration with the changes to make them aware of the new node.

The ‘slaves’ file on master server contains the list of all data nodes. This must also be updated to ensure any issues in future hadoop daemon start/stop.

2July 2016_Correction-01

The important step in data node commission process is to run the Cluster Balancer.

>hadoop balancer -threshold 40

Balancer attempts to provide a balance to a certain threshold among data nodes by copying block data from older nodes to newly commissioned nodes.

So, this is how you can do – Commissioning and Decommissioning Nodes in a Hadoop Cluster.

Get a better understanding of the Hadoop Cluster from this Big Data Course.

Got a question for us? Please mention it in the comments section and we will get back to you.

Related Links:

5 Reasons to Learn Hadoop

Get Started with Big Data & Hadoop

Upcoming Batches For Big Data Hadoop Certification Training Course
Course NameDate
Big Data Hadoop Certification Training Course

Class Starts on 11th February,2023

11th February

SAT&SUN (Weekend Batch)
View Details
Big Data Hadoop Certification Training Course

Class Starts on 8th April,2023

8th April

SAT&SUN (Weekend Batch)
View Details
Comments
15 Comments
  • Masked says:

    Hi, I am gonna start Hadoop cluster setup project. Could anyone help me in identifying the list of task/activities involved in this. I need to create a project plan for this.

  • dsv dinesh says:

    /home/hadoop/excludes – In this path is it EXCLUDES or EXCLUDE?

    • EdurekaSupport says:

      Hey Dinesh, thanks for checking out our blog.
      Well, dfs.hosts.exclude is just a property in hdfs-site.xml and mapred-site.xml. You can check the content of these files by going to the place where hadoop is installed and then you can check for these hdfs-site.xml and mapred-site.xml. These are configuration files so you will find these in etc folder of hadoop. Basically, the main attribute is dfs.hosts.exclude whose name is fixed and hadoop expects only “exclude” word here. And it’s value is the file which has the list. In our example our filename is excludes. It’s name can even be /home/hadoop/myCoolName and then in you will have to write
      dfs.hosts.exclude
      /home/hadoop/myCoolName
      true

      Hope this helps. Cheers!

      • dsv dinesh says:

        Thanks for reverting me back

  • Raj Kiran says:

    I got the enough information from your website.. But there is a mistake in the Flow Diagram of “Commissioning” .. In the 3rd block hadoop mraadmin -refreshNodes should come.. Please check and update so that everyone know the exact flow.

    • EdurekaSupport says:

      Thanks for taking the time out to go through our blog in detail. We are sorry you encountered an error. We have flagged this to the concerned folks and they are working on fixing it ASAP. Thanks again and do keep checking back in whenever you have the time.

  • Akshat says:

    Beautiful !! Thanks a ton !!

    • EdurekaSupport says:

      You are welcome Akshat!!

  • K Sandeep says:

    very good explanation..!

    • EdurekaSupport says:

      Thanks Sandeep. Feel free to check out our other posts as well.

  • Santosh says:

    Good blog

    • EdurekaSupport says:

      Thanks Santosh!

  • Bala Chandar says:

    Super Explanation.

    • EdurekaSupport says:

      Thanks Bala! Feel free to go through our other blog posts as well.

Join the discussion

Browse Categories

webinar REGISTER FOR FREE WEBINAR
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP

Subscribe to our Newsletter, and get personalized recommendations.

image not found!
image not found!

Commissioning and Decommissioning Nodes in a Hadoop Cluster

edureka.co