You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2010/08/20 00:04:16 UTC

[jira] Created: (HDFS-1348) DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not

DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not
--------------------------------------------------------------------------------------------------------------------------------

                 Key: HDFS-1348
                 URL: https://issues.apache.org/jira/browse/HDFS-1348
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: name-node
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.

One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900777#action_12900777 ] 

Hairong Kuang commented on HDFS-1348:
-------------------------------------

Read more code to understand how check works. Currently check iterating up to a default of 5 datanodes to see if they are done with decommission or not. It is easy to release the global fsnamesytem lock in between checking two nodes.

When checking one datanode, it transverses all its blocks to see if they are replicated during which the global lock is held. If a datanode has like 100K blocks, this is going to be a non-trivial amount of time. Any idea for breaking this into smaller chunks without sacrificing accuracy?

> DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-1348:
--------------------------------

    Attachment: decommission1.patch

This patch addressed Dmytro's review comment.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decommission.patch, decommission1.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901086#action_12901086 ] 

dhruba borthakur commented on HDFS-1348:
----------------------------------------

+1. Is it possible to do the check with fsnamesystem-readlock only.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900885#action_12900885 ] 

Hairong Kuang commented on HDFS-1348:
-------------------------------------

Here is the proposed change.

Currently the check is logic is as following:
{code}
synchronized (fsnamesystem) {
  for up to five decommission in progress datanodes
     check each datanode if all its blocks have replicated;
}
{code}

I plan to change the structure to be:
{code}
for up to five iterations {
  synchronized (fsnamesystem) {
    node = get next decommission in progress node;
  }

 do {
   synchronized (fsnamesystem) {
     fetch up to 2000 unchecked blocks from node;
   }
   for each block b
     synchronized (fsnamesystem) {
       check if block b has replicated;
    }
 } until all blocks of node have checked;
}
{code}

This proposed restructure will make the locking granularity much smaller. This should improve NameNode's responsiveness when decommissioning check occurs.
      

> DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916276#action_12916276 ] 

Dmytro Molkov commented on HDFS-1348:
-------------------------------------

The patch looks good to me. +1

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decomissionImp1.patch, decomissionImp2.patch, decommission.patch, decommission1.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-1348:
--------------------------------

        Summary: Improve NameNode reponsiveness while it is checking if datanode decommissions are complete  (was: DecommissionManager holds fsnamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not)
    Description: 
NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.

One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

  was:
NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.

One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.


> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-1348:
--------------------------------

    Attachment: decommission.patch

Here is a patch for review.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decommission.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-1348:
--------------------------------

    Attachment: decomissionImp2.patch

Change the patch to use read/write lock.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decomissionImp1.patch, decomissionImp2.patch, decommission.patch, decommission1.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901492#action_12901492 ] 

Hairong Kuang commented on HDFS-1348:
-------------------------------------

For r/w locks, the first & second cases could use the read lock and the third case has to use the write lock when it updates some stats of the decommissioning node.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-1348:
--------------------------------

    Status: Patch Available  (was: Open)

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decomissionImp1.patch, decomissionImp2.patch, decommission.patch, decommission1.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905301#action_12905301 ] 

Hairong Kuang commented on HDFS-1348:
-------------------------------------

yes, we should restrict the scanning overhead by putting an up limit on the number of block fetches. So I would propose to set the number of blocks to be fetched to
  max(2000, total # of blocks/5)
So the one check approximately scans the block list up to 5 times.

As the checkedBlock overhead, I plan to use LightweightGSet introduced by HDFS-1114.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decommission.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904882#action_12904882 ] 

Dmytro Molkov commented on HDFS-1348:
-------------------------------------

I like the patch overall. Got two questions though: why did you choose 2000 blocks to check at a time and do you think it might be worth to make it configurable? And the other one is how much memory pressure would the checkedBlocks hash set add?

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decommission.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HDFS-1348:
--------------------------------

    Attachment: decomissionImp1.patch

This new patch groups the block check in 1000 a batch so reducing the overhead of locking comparing to the previous patch.

> Improve NameNode reponsiveness while it is checking if datanode decommissions are complete
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1348
>                 URL: https://issues.apache.org/jira/browse/HDFS-1348
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: decomissionImp1.patch, decommission.patch, decommission1.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every second. But once for a while, NameNode seems to pause for more than 10 seconds without doing anything, leaving a blank in its log even though no garbage collection is happening.  All other requests to NameNode are blocked when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock during the whole process of checking if decomissioning DataNodes are finished or not, during which it checks every block of up to a default of 5 datanodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.