You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2010/11/19 04:26:14 UTC
[jira] Created: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
TestDU fails on systems with local file systems with extended attributes
------------------------------------------------------------------------
Key: HADOOP-7045
URL: https://issues.apache.org/jira/browse/HADOOP-7045
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
Fix For: 0.23.0
The test reports that the file takes an extra 4k on disk:
{noformat}
Testcase: testDU took 5.74 sec
FAILED
expected:<32768> but was:<36864>
junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
{noformat}
This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
{noformat}
common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
8+0 records in
8+0 records out
32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
common-branch-0.20 $ du data
36 data
common-branch-0.20 $ du --apparent-size data
32 data
{noformat}
We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934128#action_12934128 ]
Hudson commented on HADOOP-7045:
--------------------------------
Integrated in Hadoop-Common-trunk #520 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/520/])
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-7045:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks Cos! I've committed this.
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933702#action_12933702 ]
Hudson commented on HADOOP-7045:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #435 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/435/])
HADOOP-7045. TestDU fails on systems with local file systems with extended attributes. Contributed by Eli Collins
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-7045:
--------------------------------
Status: Patch Available (was: Open)
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Konstantin Boudnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933688#action_12933688 ]
Konstantin Boudnik commented on HADOOP-7045:
--------------------------------------------
+1 patch looks good!
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933667#action_12933667 ]
Hadoop QA commented on HADOOP-7045:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12459966/hadoop-7045-1.patch
against trunk revision 1036694.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/113//console
This message is automatically generated.
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7045) TestDU fails on systems with local
file systems with extended attributes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-7045:
--------------------------------
Attachment: hadoop-7045-1.patch
Patch attached. Modifies the test to pass if the disk usage is greater than the written size, but no greater than a given amount of slack (4k). The test now passes on local ext3 file systems with extended attributes.
> TestDU fails on systems with local file systems with extended attributes
> ------------------------------------------------------------------------
>
> Key: HADOOP-7045
> URL: https://issues.apache.org/jira/browse/HADOOP-7045
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Reporter: Eli Collins
> Assignee: Eli Collins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: hadoop-7045-1.patch
>
>
> The test reports that the file takes an extra 4k on disk:
> {noformat}
> Testcase: testDU took 5.74 sec
> FAILED
> expected:<32768> but was:<36864>
> junit.framework.AssertionFailedError: expected:<32768> but was:<36864>
> at org.apache.hadoop.fs.TestDU.testDU(TestDU.java:79)
> {noformat}
> This is because du reports 32k for the file and 4k because the file system it lives on uses extended attributes.
> {noformat}
> common-branch-0.20 $ dd if=/dev/zero of=data bs=4096 count=8
> 8+0 records in
> 8+0 records out
> 32768 bytes (33 kB) copied, 9.6e-05 seconds, 341 MB/s
> common-branch-0.20 $ du data
> 36 data
> common-branch-0.20 $ du --apparent-size data
> 32 data
> {noformat}
> We should modify the test to allow for some extra on-disk slack. The on-disk usage could also be smaller if the file data is all zeros or compression is enabled. The test currently handles the former by writing random data, we're punting on the latter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.