You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2008/01/18 00:39:34 UTC
[jira] Created: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
The ReplicationMonitor sleep period should be configurable
----------------------------------------------------------
Key: HADOOP-2649
URL: https://issues.apache.org/jira/browse/HADOOP-2649
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
Fix For: 0.16.0
The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560757#action_12560757 ]
Hadoop QA commented on HADOOP-2649:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373461/configurableReplicationPeriod.patch
against trunk revision r613446.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1657/console
This message is automatically generated.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561709#action_12561709 ]
Hudson commented on HADOOP-2649:
--------------------------------
Integrated in Hadoop-Nightly #374 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/374/])
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2649:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2649:
-------------------------------------
Attachment: configurableReplicationPeriod.patch
Make the Replication Monitor periodicity configurable. The default value of 3 seconds remains the same as before.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560569#action_12560569 ]
Hairong Kuang commented on HADOOP-2649:
---------------------------------------
+1 The patch looks good to me.
It would be nicer if the replication monitor can adjust its frequency dynamically. It slows itself down when the namenode becomes busy and replicates more aggressively when the namenode becomes idle.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560570#action_12560570 ]
dhruba borthakur commented on HADOOP-2649:
------------------------------------------
Hi hairong, thanks for ur review comments. I plan to make this period dymanic (as per your suggestion) in the jira that is linked to this issue HADOOP-2606.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2649:
-------------------------------------
Status: Patch Available (was: Open)
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561429#action_12561429 ]
Hadoop QA commented on HADOOP-2649:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373724/configurableReplicationPeriod.patch
against trunk revision r614192.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1675/console
This message is automatically generated.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-2649:
----------------------------------------
Assignee: dhruba borthakur
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2649:
-------------------------------------
Status: Patch Available (was: Open)
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-2649:
------------------------------------
Priority: Blocker (was: Major)
Promoted for consideration in 16.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2649:
-------------------------------------
Status: Open (was: Patch Available)
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2649) The ReplicationMonitor sleep period
should be configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2649:
-------------------------------------
Attachment: configurableReplicationPeriod.patch
Merged patch with latest trunk. Also, fixed the problem with Test Decommission failing because it was not progressing quickly.
> The ReplicationMonitor sleep period should be configurable
> ----------------------------------------------------------
>
> Key: HADOOP-2649
> URL: https://issues.apache.org/jira/browse/HADOOP-2649
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.16.0
>
> Attachments: configurableReplicationPeriod.patch, configurableReplicationPeriod.patch
>
>
> The HDFS Namenode computes replication work for datanodes once every 3 seconds. This should be a configurable value. On large clusters, there could be many many blocks in the neededReplication queue and computing replication work for datanodes once every 3 seconds might consume lots of CPU on namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.