You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "William Slacum (JIRA)" <ji...@apache.org> on 2011/06/02 04:24:47 UTC
[jira] [Created] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Counters don't reset state when readFields() called
---------------------------------------------------
Key: MAPREDUCE-2557
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: William Slacum
Priority: Trivial
When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "William Slacum (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
William Slacum updated MAPREDUCE-2557:
--------------------------------------
Attachment: MAPREDUCE-2557-style.patch
Made the "L" next to long's capital for easier viewing.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2557:
-------------------------------------
Status: Open (was: Patch Available)
Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-2557:
-------------------------------------
Status: Patch Available (was: Open)
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "William Slacum (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045236#comment-13045236 ]
William Slacum commented on MAPREDUCE-2557:
-------------------------------------------
Hey Todd,
1) No problem on that
2) I think in terms of running a MR job, there's no real effect unless the framework is re-using the same Counters reference for different jobs, which could happen if JVM re-use is enabled. I encountered this while writing an MR job that used Counters as an input value type. I had written my own InputFormat that re-used a Counters reference and found I was getting odd values when performing operations on Counter values.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042635#comment-13042635 ]
Hadoop QA commented on MAPREDUCE-2557:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481183/MAPREDUCE-2557.patch
against trunk revision 1130393.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/331//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/331//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/331//console
This message is automatically generated.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "William Slacum (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
William Slacum updated MAPREDUCE-2557:
--------------------------------------
Attachment: MAPREDUCE-2557.patch
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "William Slacum (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212056#comment-13212056 ]
William Slacum commented on MAPREDUCE-2557:
-------------------------------------------
Five months turnaround is better than never, right? Right?!
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557-trunk.patch, MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097000#comment-13097000 ]
Harsh J commented on MAPREDUCE-2557:
------------------------------------
Hello William,
Thanks for the patch!
Can you re-up an updated patch for trunk, granting ASF permissions to use the patch? This is doable by checking the "Grant license to ASF for inclusion in ASF works" option when attaching a file via more-actions>attach-files dialog. It is required if you intend the patch to be committed into the project.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "William Slacum (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
William Slacum updated MAPREDUCE-2557:
--------------------------------------
Attachment: MAPREDUCE-2557-trunk.patch
Patch to clear Counters cache for hadoop-common trunk.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557-trunk.patch, MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045269#comment-13045269 ]
Hadoop QA commented on MAPREDUCE-2557:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481646/MAPREDUCE-2557-style.patch
against trunk revision 1132807.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/354//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/354//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/354//console
This message is automatically generated.
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557-style.patch, MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2557) Counters don't reset state when
readFields() called
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045107#comment-13045107 ]
Todd Lipcon commented on MAPREDUCE-2557:
----------------------------------------
Hi William. Thanks for the bug report. Two things:
- a silly nit: can you use "1L" and "2L" instead of "1l" and "2l" in the tests? I was staring at it for 5 minutes before I realized those were 'l' ("ell") and not '1' ("one").
- what ramifications, if any, does this have on the MR framework itself? Would this cause incorrect counters in any usual use case? How'd you come across the bug?
> Counters don't reset state when readFields() called
> ---------------------------------------------------
>
> Key: MAPREDUCE-2557
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2557
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: William Slacum
> Priority: Trivial
> Attachments: MAPREDUCE-2557.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> When calling readFields() on a Counters object, the internal state is not completely reset. The IdentityHashMap<Enum<?>, Counter> cache retains all previous mappings, even after the actual CounterGroups are changed. Using the same Counters pointer over and over again results in the cache always keeping the mapping for the first call to getCounter(Enum<?>). I've add a clear() call to the cache when readFields() is called and added a unit test to verify that it works.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira