You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/01/18 23:58:43 UTC
[jira] Created: (HADOOP-7111) Several TFile tests failing when
native libraries are present
Several TFile tests failing when native libraries are present
-------------------------------------------------------------
Key: HADOOP-7111
URL: https://issues.apache.org/jira/browse/HADOOP-7111
Project: Hadoop Common
Issue Type: Bug
Components: io
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.22.0
When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-7111:
-----------------------------------
Fix Version/s: (was: 0.22.0)
0.23.0
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051227#comment-13051227 ]
Todd Lipcon commented on HADOOP-7111:
-------------------------------------
Something in trunk just changed such that the native libs are on the classpath when the tests run, so the tests are now failing:
https://builds.apache.org/job/Hadoop-Common-trunk/721/testReport/junit/org.apache.hadoop.io.file.tfile/TestTFileByteArrays/testOneBlockPlusOneEntry/
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-7111:
-----------------------------------
Status: Patch Available (was: Open)
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: hadoop-7111.0.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-7111:
-----------------------------------
Attachment: hadoop-7111.0.patch
Patch which addresses the issue. Hong's theory #2 was correct.
The only change which actually fixes the tests is the change to the initialization of {{records1stBlock}} and {{records2ndBlock}} to be dependent upon whether or not the native libs are present.
While I was in the file, I also took the opportunity to do a little cleanup by:
# Converting to the new style of test annotations.
# Refactoring some methods which had unused parameters.
# Fixing up some formatting.
# Reducing some code duplication in {{TestTFileJClassComparatorByteArrays}}
# Fixing some "unused variable" compiler warnings.
I've tested both {{TestTFileJClassComparatorByteArrays}} and {{TestTFileByteArrays}} both with and without native code being present.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: hadoop-7111.0.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068913#comment-13068913 ]
Hudson commented on HADOOP-7111:
--------------------------------
Integrated in Hadoop-Common-trunk #753 (See [https://builds.apache.org/job/Hadoop-Common-trunk/753/])
HADOOP-7111. Several TFile tests failing when native libraries are present. (atm)
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933
Files :
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch, hadoop-7111.1.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402598#comment-13402598 ]
Todd Lipcon commented on HADOOP-7111:
-------------------------------------
These seem to be failing again in some patch builds. I filed HADOOP-8537.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch, hadoop-7111.1.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-7111:
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Looking forward to a green Common build!
Thanks again for the review, Tom.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch, hadoop-7111.1.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068034#comment-13068034 ]
Hadoop QA commented on HADOOP-7111:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12487068/hadoop-7111.0.patch
against trunk revision 1147971.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//console
This message is automatically generated.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.22.0
>
> Attachments: hadoop-7111.0.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-7111:
-----------------------------------
Attachment: hadoop-7111.1.patch
Thanks a lot, Tom. Here's a patch removing the TODO and the useless assert.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch, hadoop-7111.1.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068626#comment-13068626 ]
Hadoop QA commented on HADOOP-7111:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12487211/hadoop-7111.1.patch
against trunk revision 1147971.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//console
This message is automatically generated.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch, hadoop-7111.1.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068640#comment-13068640 ]
Hudson commented on HADOOP-7111:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #695 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/695/])
HADOOP-7111. Several TFile tests failing when native libraries are present. (atm)
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933
Files :
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch, hadoop-7111.1.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985979#action_12985979 ]
Hong Tang commented on HADOOP-7111:
-----------------------------------
There could be two possibilities.
# in some of the TFile tests, we hard-coded the sizes of the compressed blocks. HADOOP-6683 may have affected the compressed sizes.
# TFile closes a block when it observes the compressed size of the block exceeds the configured minimum block size. HADOOP-6683 waits until a full-buffer if raw bytes before compressing, and thus changes the behavior.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056358#comment-13056358 ]
Eli Collins commented on HADOOP-7111:
-------------------------------------
Strange, bi-secting on my host leads to HADOOP-7206 but that was committed after these tests started failing, must be triggering something.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.22.0
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers reassigned HADOOP-7111:
--------------------------------------
Assignee: Aaron T. Myers
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.22.0
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7111) Several TFile tests failing when
native libraries are present
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068605#comment-13068605 ]
Tom White commented on HADOOP-7111:
-----------------------------------
+1
The TODO you added should be removed along with the assertNull on the following line, since the Assert.fail is sufficient.
> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
> Key: HADOOP-7111
> URL: https://issues.apache.org/jira/browse/HADOOP-7111
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7111.0.patch
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira