You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Kan Zhang (JIRA)" <ji...@apache.org> on 2009/03/12 21:50:50 UTC
[jira] Created: (HADOOP-5481) Namenode returns lease recovery
requests with other requests
Namenode returns lease recovery requests with other requests
------------------------------------------------------------
Key: HADOOP-5481
URL: https://issues.apache.org/jira/browse/HADOOP-5481
Project: Hadoop Core
Issue Type: Improvement
Reporter: Kan Zhang
Assignee: Kan Zhang
HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode should return lease
recovery request with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688394#action_12688394 ]
Kan Zhang commented on HADOOP-5481:
-----------------------------------
I think how to add a new request is a separate issue. The purpose of this JIRA is to see if we can combine existing requests and if not, why.
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5481) Namenode should return lease recovery
request with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5481:
------------------------------
Summary: Namenode should return lease recovery request with other requests (was: Namenode returns lease recovery requests with other requests)
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode should return lease
recovery request with other requests
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681586#action_12681586 ]
Hadoop QA commented on HADOOP-5481:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402087/HADOOP-5481.patch
against trunk revision 753052.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/57/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/57/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/57/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/57/console
This message is automatically generated.
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode returns lease recovery
requests with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681486#action_12681486 ]
Kan Zhang commented on HADOOP-5481:
-----------------------------------
{noformat}
//check lease recovery
cmd = nodeinfo.getLeaseRecoveryCommand(Integer.MAX_VALUE);
if (cmd != null) {
return new DatanodeCommand[] {cmd};
}
ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>(2);
//check pending replication
cmd = nodeinfo.getReplicationCommand(
maxReplicationStreams - xmitsInProgress);
if (cmd != null) {
cmds.add(cmd);
}
//check block invalidation
cmd = nodeinfo.getInvalidateBlocks(blockInvalidateLimit);
if (cmd != null) {
cmds.add(cmd);
}
{noformat}
> Namenode returns lease recovery requests with other requests
> ------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Kan Zhang
> Assignee: Kan Zhang
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5481) Namenode should return lease recovery
request with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5481:
------------------------------
Component/s: dfs
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode should return lease
recovery request with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683575#action_12683575 ]
Kan Zhang commented on HADOOP-5481:
-----------------------------------
I need to add a new type of request, which is combinable with others and preferably sent to DN at first opportunity. I could combine my new request with the lease recovery request and again with the other 2, but it looks ugly. Fundamentally, we need to be able to answer the question of whether the lease recovery request is handled independently from the other 2. If yes, we combine them. If not, we put a note in the code so that people won't get confused when adding new types of requests in the future. Otherwise, we're kind of in the dark.
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5481) Namenode should return lease
recovery request with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang reassigned HADOOP-5481:
---------------------------------
Assignee: (was: Kan Zhang)
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode should return lease
recovery request with other requests
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683591#action_12683591 ]
Hairong Kuang commented on HADOOP-5481:
---------------------------------------
Does your kind of request have to be combined with other requests? Is it Ok that the new request is sent separately with the highest priority? Initial heartbeat design prefers one single request per heartbeat. HADOOP-5034 was a cautious change to solve a deadlock situation. Currently a DataNode processes heartbeat requests inline with sending heartbeats in one thread. If too many requests get sent to DataNode in one heartbeat reply, it may delay DataNode sending next heartbeat to NameNode.
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5481) Namenode returns lease recovery
requests with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5481:
------------------------------
Status: Patch Available (was: Open)
> Namenode returns lease recovery requests with other requests
> ------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode should return lease
recovery request with other requests
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683547#action_12683547 ]
Hairong Kuang commented on HADOOP-5481:
---------------------------------------
Kan, could you please explain why you need this change? Heartbeat handling is very critical to dfs so normally we are very cautions about making any change to it. Also lease recovery is added to the system to support append/sync feature which may undergo some future change,
> Namenode should return lease recovery request with other requests
> -----------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5481) Namenode returns lease recovery
requests with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681487#action_12681487 ]
Kan Zhang commented on HADOOP-5481:
-----------------------------------
Can someone from dfs team comment on if this is desirable?
> Namenode returns lease recovery requests with other requests
> ------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Kan Zhang
> Assignee: Kan Zhang
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5481) Namenode returns lease recovery
requests with other requests
Posted by "Kan Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kan Zhang updated HADOOP-5481:
------------------------------
Attachment: HADOOP-5481.patch
> Namenode returns lease recovery requests with other requests
> ------------------------------------------------------------
>
> Key: HADOOP-5481
> URL: https://issues.apache.org/jira/browse/HADOOP-5481
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Attachments: HADOOP-5481.patch
>
>
> HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.