You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2009/10/02 00:42:23 UTC
[jira] Created: (HDFS-673) BlockReceiver#PacketResponder should not
remove a packet from the ack queue before its ack is sent
BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
--------------------------------------------------------------------------------------------------
Key: HDFS-673
URL: https://issues.apache.org/jira/browse/HDFS-673
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Priority: Blocker
Fix For: 0.21.0
After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-673) BlockReceiver#PacketResponder should
not remove a packet from the ack queue before its ack is sent
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764166#action_12764166 ]
Hadoop QA commented on HDFS-673:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421358/pkgRmOrder.patch
against trunk revision 822153.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/59/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/59/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/59/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/59/console
This message is automatically generated.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-673) BlockReceiver#PacketResponder should not
remove a packet from the ack queue before its ack is sent
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-673:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I close this jira for now. If anyone wants this fix to be in 0.20, I will do so later.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-673) BlockReceiver#PacketResponder should not
remove a packet from the ack queue before its ack is sent
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-673:
-------------------------------
Status: Patch Available (was: Open)
The existing tests sometime fail with this error. So I do not think there is a need for new test. Furthuremore, I have tested this manually too.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-673) BlockReceiver#PacketResponder should not
remove a packet from the ack queue before its ack is sent
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-673:
----------------------------------------
Hadoop Flags: [Reviewed]
+1 on the patch.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-673) BlockReceiver#PacketResponder should
not remove a packet from the ack queue before its ack is sent
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770266#action_12770266 ]
Hudson commented on HDFS-673:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #78 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/78/])
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-673) BlockReceiver#PacketResponder should
not remove a packet from the ack queue before its ack is sent
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764210#action_12764210 ]
Hairong Kuang commented on HDFS-673:
------------------------------------
There were two failed tests: org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction.testBlockCreation and org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend. The first one is a known bug. For the second one, the append write failed on "Too many open files".
This seems not related to the change in this jira. I filed HDFS-690 to track the bug.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-673) BlockReceiver#PacketResponder should not
remove a packet from the ack queue before its ack is sent
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-673:
----------------------------------------
Component/s: data-node
Patch looks good. Do we need a test?
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-673) BlockReceiver#PacketResponder should
not remove a packet from the ack queue before its ack is sent
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764905#action_12764905 ]
Hairong Kuang commented on HDFS-673:
------------------------------------
I committed this to trunk and branch 0.21.
I checked branch 0.20. It seems to have the same problem. But strangely I have not seen tests failed on this in 0.20 branch. Anyboday has seen this bug in 0.20 and wants to commit the fix to 0.20?
A consequence of this bug is that it introduces false alarms. A good datanode may fail to send an ack to the client on closedSocket exception when receiving a block.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-673) BlockReceiver#PacketResponder should not
remove a packet from the ack queue before its ack is sent
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-673:
-------------------------------
Attachment: pkgRmOrder.patch
A patch for review.
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-673) BlockReceiver#PacketResponder should
not remove a packet from the ack queue before its ack is sent
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769073#action_12769073 ]
Hudson commented on HDFS-673:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #79 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/79/])
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-673) BlockReceiver#PacketResponder should
not remove a packet from the ack queue before its ack is sent
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768404#action_12768404 ]
Hudson commented on HDFS-673:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #47 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/47/])
> BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-673
> URL: https://issues.apache.org/jira/browse/HDFS-673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: pkgRmOrder.patch
>
>
> After a BlockReceiver finish receiving the last packet of a block, it waits until the ack queue becomes empty. It them assumes that all acks have been sent and shunts down network connections. The current code removes a packet from the ack queue before its ack is sent. So there is a chance that the connection gets closed before an ack is sent.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.