You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2007/01/29 20:14:49 UTC
[jira] Created: (HADOOP-947) isReplicationInProgress() is very
heavyweight
isReplicationInProgress() is very heavyweight
---------------------------------------------
Key: HADOOP-947
URL: https://issues.apache.org/jira/browse/HADOOP-947
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-947:
------------------------------------
Attachment: (was: isReplicationInProgress.patch)
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473758 ]
dhruba borthakur commented on HADOOP-947:
-----------------------------------------
Hi wendy,
Thanks for the review. I made the change you requested. Can you pl give a
"+1" to the JIRA issue so that Nigel picks it up for the patch testing
process?
Thanks,
dhruba
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-947:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Dhruba.
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.12.0
>
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-947:
------------------------------------
Attachment: isReplicationInProgress2.patch
Incorporated review comments.
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473601 ]
Wendy Chien commented on HADOOP-947:
------------------------------------
The logic looks fine, but the new name, hasReachedReplicationFactor, implies the opposite of what the comment says and what the code does. I think keeping the old name or switching true and false would work better.
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-947:
---------------------------------------
Assignee: dhruba borthakur
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "Wendy Chien (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473786 ]
Wendy Chien commented on HADOOP-947:
------------------------------------
+1, looks good
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473766 ]
Hadoop QA commented on HADOOP-947:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12351377/isReplicationInProgress2.patch applied and successfully tested against trunk revision r508345.
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-947:
------------------------------------
Attachment: isReplicationInProgress.patch
This patch fixes the logic that determines whether a decommission is finished or not.
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Attachments: isReplicationInProgress.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-947) isReplicationInProgress() is very
heavyweight
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-947:
------------------------------------
Status: Patch Available (was: Open)
This patch should remove make TestDecommission more deterministic.
> isReplicationInProgress() is very heavyweight
> ---------------------------------------------
>
> Key: HADOOP-947
> URL: https://issues.apache.org/jira/browse/HADOOP-947
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: isReplicationInProgress2.patch
>
>
> Suppose a decommission is in progress. The namenode receives a heartbeat from the being-decommissioned node and then invokes isReplicationInProgress() to determine if the decommissioned is complete. This method is very heavyweight and chews up plenty of CPU.
> One option to fix this issue is to keep a counter to indicate how many blocks are pending replication. This could also help in showing a progress-status-indicator to display how far the decommissioning process is complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.