You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Matt Foley (JIRA)" <ji...@apache.org> on 2011/04/05 19:28:05 UTC
[jira] [Created] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
--------------------------------------------------------------------------------------------------
Key: HDFS-1806
URL: https://issues.apache.org/jira/browse/HDFS-1806
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, name-node
Affects Versions: 0.22.0
Reporter: Matt Foley
Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-1806:
-----------------------------
Attachment: TestBlockReport.java.patch
Seems to have done the trick. Posted here for information, but I'm going to subordinate this bug to HDFS-1295 (which it was blocking) and submit a single patch to that Jira.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018952#comment-13018952 ]
Konstantin Boudnik commented on HDFS-1806:
------------------------------------------
bq. Regrettably I can't test the failure locally – it doesn't fail on any machine I've tried it on here.
:( I know. That has been a standing issue because the problem wasn't surfacing pretty much anywhere but on ASF Hudson for one reason or another. Thanks again!
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley reassigned HDFS-1806:
--------------------------------
Assignee: Matt Foley
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1806:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've committed this to branch 22 and trunk. Thanks Matt!
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0, 0.23.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Boudnik updated HDFS-1806:
-------------------------------------
Hadoop Flags: [Reviewed]
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-1806:
-----------------------------
Fix Version/s: 0.22.0
Status: Patch Available (was: Open)
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022988#comment-13022988 ]
Hudson commented on HDFS-1806:
------------------------------
Integrated in Hadoop-Hdfs-22-branch #35 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/35/])
HDFS-1806. svn merge -c 1095235 from trunk
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0, 0.23.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021853#comment-13021853 ]
Eli Collins commented on HDFS-1806:
-----------------------------------
+1 Change lgtm.
TestBlockReport no longer fails while looping it for several hours after applying this patch.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0, 0.23.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020110#comment-13020110 ]
Matt Foley commented on HDFS-1806:
----------------------------------
These failures are unrelated to the patch, which only modifies TestBlockReport.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-1806:
------------------------------
Affects Version/s: 0.23.0
Fix Version/s: 0.23.0
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0, 0.23.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-1806:
-----------------------------
Attachment: blockReport_08_failure_log.html
In the attached log excerpt, from Apache Hudson/Jenkins QA auto-test, replication starts at the datanode at 2:58:09,608.
The datanode moves the replica from tmp to finalized at 2:58:09,659-660.
Then at 2:58:09,663 we see the message "Replication state before the loop 0", which represents the START of polling -- way too late.
So both the waitTil(100) and waitTil(50) lines in waitForTempReplica() are too long.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Attachments: blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019969#comment-13019969 ]
Konstantin Boudnik commented on HDFS-1806:
------------------------------------------
+1 patch looks good.
Matt, please let me know if you need help committing it.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016576#comment-13016576 ]
Matt Foley commented on HDFS-1806:
----------------------------------
Hi Cos, I hope so. I'm going to try it with both loops polling at 5ms. If it gets through Hudson I'll tie in both HDFS-953 and HDFS-733.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Attachments: blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022954#comment-13022954 ]
Hudson commented on HDFS-1806:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #600 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/600/])
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0, 0.23.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018665#comment-13018665 ]
Matt Foley commented on HDFS-1806:
----------------------------------
Well, strictly speaking since it is an intermittent failure, we'll actually have to wait and see :-)
But yes, I see improvement on the same machine and in the same test circumstances where pretty consistent intermittent failure was previously observed:
HDFS-1295 just passed auto-test without a TestBlockReport failure; whereas it had failed previously, and indeed it has been a long time since I saw TestBlockReport NOT be on the Hudson failure list for several HDFS bugs I was interested in.
Altho this is a single instance, I think we've got rational grounds for believing that we now understand the probable cause and have addressed it. Regrettably I can't test the failure locally -- it doesn't fail on any machine I've tried it on here. The proof of the pudding will be seeing if the community stops being bothered by this false positive after it is submitted to the public code base.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020104#comment-13020104 ]
Hadoop QA commented on HDFS-1806:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12476071/TestBlockReport.java.patch
against trunk revision 1092534.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestLargeBlock
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/369//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/369//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/369//console
This message is automatically generated.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022430#comment-13022430 ]
Hudson commented on HDFS-1806:
------------------------------
Integrated in Hadoop-Hdfs-trunk #643 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/])
HDFS-1806. TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers. Contributed by Matt Foley.
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0, 0.23.0
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0, 0.23.0
>
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016506#comment-13016506 ]
Konstantin Boudnik commented on HDFS-1806:
------------------------------------------
Great analysis, Matt! Thanks for looking into this long-standing issue (I believe there's a JIRA for this already). So, if the second loop is merged with the first one and the poll time is shortened significantly (say, 5 ms?) would it solve the problem?
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Attachments: blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1806) TestBlockReport.blockReport_08() and
_09() are timing-dependent and likely to fail on fast servers
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018659#comment-13018659 ]
Konstantin Boudnik commented on HDFS-1806:
------------------------------------------
Great finding, Matt! So, you are able to see the improvement with the patch on the same machine where intermittent failure was observed before, right?
> TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-1806
> URL: https://issues.apache.org/jira/browse/HDFS-1806
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.22.0
> Reporter: Matt Foley
> Attachments: TestBlockReport.java.patch, blockReport_08_failure_log.html
>
>
> Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira