You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2009/03/27 00:36:50 UTC
[jira] Created: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
FileSystem statistic counters are too high when JVM reuse is enabled.
---------------------------------------------------------------------
Key: HADOOP-5585
URL: https://issues.apache.org/jira/browse/HADOOP-5585
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Priority: Blocker
Fix For: 0.20.0
When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Status: Open (was: Patch Available)
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5585) FileSystem statistic counters are
too high when JVM reuse is enabled.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696452#action_12696452 ]
Hadoop QA commented on HADOOP-5585:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12404795/h5585.patch
against trunk revision 762509.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/158/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/158/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/158/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/158/console
This message is automatically generated.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5585) FileSystem statistic counters are
too high when JVM reuse is enabled.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696751#action_12696751 ]
Chris Douglas commented on HADOOP-5585:
---------------------------------------
+1 TestSequenceFile passes with the latest patch on my machine
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Attachment: h5585.patch
Some of the unit tests were creating LocalFileSystem directly, which wasn't getting them initialized correctly. I changed their setup to call FileSystem.getLocal instead of new LocalFileSystem.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Status: Patch Available (was: Open)
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Attachment: h5585.patch
This patch fixes a javadoc warning and the broken javadoc link that Chris commented on. It also extends the testcase from HADOOP-5437 to detect the file system statistics not being cleared between tasks.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Attachment: h5585.patch
This patch is updated to current trunk.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5585) FileSystem statistic counters are
too high when JVM reuse is enabled.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696901#action_12696901 ]
Hadoop QA commented on HADOOP-5585:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12404867/h5585.patch
against trunk revision 762987.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/162/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/162/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/162/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/162/console
This message is automatically generated.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5585) FileSystem statistic counters are
too high when JVM reuse is enabled.
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694359#action_12694359 ]
Chris Douglas commented on HADOOP-5585:
---------------------------------------
This looks good. Just two nits:
* The current patch generates a javadoc warning, and has a malformed reference to the non-existent method getFileSystemClasses in the docs for {{FileSystem::getStatistics}}
* This should have a unit test, but that can be part of a separate issue (possibly once HADOOP-5437 is fixed)
+1 for the functional changes
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Attachment: h5585.patch
This patch:
1. Adds a reset method for statistics, so that when the framework switches to a new task, it can clean the counters.
2. Removes the map from scheme to statistics, since it was only used to iterate through.
3. Removes a race condition where the map was passed back out of the synchronized method, which could cause ConcurrentModificationException.
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5585) FileSystem statistic counters are too
high when JVM reuse is enabled.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-5585:
----------------------------------
Status: Patch Available (was: Open)
> FileSystem statistic counters are too high when JVM reuse is enabled.
> ---------------------------------------------------------------------
>
> Key: HADOOP-5585
> URL: https://issues.apache.org/jira/browse/HADOOP-5585
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: h5585.patch, h5585.patch, h5585.patch, h5585.patch
>
>
> When JVM reuse is enabled, the FileSystem.Statistics are not cleared between tasks. That means that the second task gets credit for its own reads and writes as well as the first. The third gets credit for all 3 tasks reads and writes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.