You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2012/08/17 20:05:37 UTC
[jira] [Created] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Suresh Srinivas created HDFS-3814:
-------------------------------------
Summary: Make the replication monitor multipliers configurable in 1.x
Key: HDFS-3814
URL: https://issues.apache.org/jira/browse/HDFS-3814
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 1.0.0
Reporter: Suresh Srinivas
It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437237#comment-13437237 ]
Harsh J commented on HDFS-3814:
-------------------------------
That sounds good to me, thanks Suresh!
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao reassigned HDFS-3814:
-------------------------------
Assignee: Jing Zhao
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3814:
----------------------------
Target Version/s: 1.1.0 (was: 1.2.0)
Affects Version/s: (was: 1.0.0)
1.1.0
Status: Patch Available (was: Open)
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-3814:
--------------------------
Release Note:
This change adds two new configuration parameters.
# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
# {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.
Please see hdfs-default.xml for detailed description.
was:
This change adds two new configuration parameters.
# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.
Please see hdfs-default.xml for detailed description.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Fix For: 1.1.0
>
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436928#comment-13436928 ]
Suresh Srinivas commented on HDFS-3814:
---------------------------------------
We should see if we can squeeze this into 1.1.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437113#comment-13437113 ]
Hadoop QA commented on HDFS-3814:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541427/HDFS-3814.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3038//console
This message is automatically generated.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437219#comment-13437219 ]
Harsh J commented on HDFS-3814:
-------------------------------
Patch seems to deviate a bit from trunk. If we're adding in tests for increase, please do add the same tests to trunk first too? We don't want to be diverging and later regressing.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437182#comment-13437182 ]
Suresh Srinivas commented on HDFS-3814:
---------------------------------------
Minor comments:
# Instead of annotation @exception use @throws
# Also use try-invokemethod-fail-catch-and-ignore when testing exceptions. See FileSystemContractBaseTest.java for example.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-3814:
----------------------------------
Attachment: HDFS-3814.patch
Rebased patch.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437225#comment-13437225 ]
Suresh Srinivas commented on HDFS-3814:
---------------------------------------
HDFS-3475 introduced bug will be fixed in HDFS-3816. Thatis the appropriate time to add the tests from this patch.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas resolved HDFS-3814.
-----------------------------------
Resolution: Fixed
Fix Version/s: 1.1.0
Release Note:
This change adds two new configuration parameters.
# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.
Please see hdfs-default.xml for detailed description.
Hadoop Flags: Reviewed
I committed the patch to branch-1 and 1.1.0. Thank you Jing.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Fix For: 1.1.0
>
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3814:
----------------------------
Attachment: HDFS-3814.patch
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3814:
----------------------------
Attachment: HDFS-3814.patch
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437365#comment-13437365 ]
Zhihong Ted Yu commented on HDFS-3814:
--------------------------------------
The two parameters cited seem to be the same.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Fix For: 1.1.0
>
> Attachments: HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3814:
----------------------------
Attachment: HDFS-3814.patch
Added testcases, and change the value of DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION_DEFAULT from 32 to 0.32f.
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch, HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3814) Make the replication monitor
multipliers configurable in 1.x
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3814:
----------------------------
Status: Open (was: Patch Available)
> Make the replication monitor multipliers configurable in 1.x
> ------------------------------------------------------------
>
> Key: HDFS-3814
> URL: https://issues.apache.org/jira/browse/HDFS-3814
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 1.1.0
> Reporter: Suresh Srinivas
> Assignee: Jing Zhao
> Attachments: HDFS-3814.patch
>
>
> It would be good to have the equivalent change of HDFS-3475 in 1.x to allow customization of replication rate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira