You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/08/18 19:06:27 UTC
[jira] [Created] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
ResourceBundle lookup during counter name resolution takes a lot of time
------------------------------------------------------------------------
Key: MAPREDUCE-2855
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217810#comment-13217810 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #592 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/592/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-2855:
--------------------------------------
Attachment: MR2855.txt
Nice find Todd. These calls were taking up a good amount of time.
Attaching a patch which passes the context classloader.
For a 70MB, 11700 task history file.
ParseTime: ~4.5 seconds (remains unchanged)
Creating all Task objects: Comes down from ~11.3 seconds to ~4 seconds.
MR-3901 does defer these calls to when they're actually required.
On another note, do we really need to use ResourceBundles for the counter names, or is this something we can do away with?
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207301#comment-13207301 ]
Todd Lipcon commented on MAPREDUCE-2855:
----------------------------------------
This is taking the majority of time when loading job history for large jobs. I see the following stack:
{code}
Stack:
java.util.ResourceBundle.getClassContext(Native Method)
java.util.ResourceBundle.getLoader(ResourceBundle.java:400)
java.util.ResourceBundle.getBundle(ResourceBundle.java:705)
org.apache.hadoop.mapreduce.util.ResourceBundles.getBundle(ResourceBundles.java:36)
org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(ResourceBundles.java:54)
org.apache.hadoop.mapreduce.util.ResourceBundles.getCounterName(ResourceBundles.java:87)
org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.localizeCounterName(FileSystemCounterGroup.java:85)
org.apache.hadoop.mapreduce.counters.FileSystemCounterGroup$FSCounter.getDisplayName(FileSystemCounterGroup.java:81)
org.apache.hadoop.mapreduce.TypeConverter.toYarn(TypeConverter.java:275)
org.apache.hadoop.mapreduce.v2.hs.CompletedTask.<init>(CompletedTask.java:89)
org.apache.hadoop.mapreduce.v2.hs.CompletedJob.loadFullHistoryData(CompletedJob.java:270)
org.apache.hadoop.mapreduce.v2.hs.CompletedJob.<init>(CompletedJob.java:90)
org.apache.hadoop.mapreduce.v2.hs.JobHistory.loadJob(JobHistory.java:657)
org.apache.hadoop.mapreduce.v2.hs.JobHistory.findJob(JobHistory.java:737)
org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:903)
org.apache.hadoop.mapreduce.v2.app.webapp.AppController.requireJob(AppController.java:313)
{code}
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214296#comment-13214296 ]
Hadoop QA commented on MAPREDUCE-2855:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12515689/MR2855.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1913//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1913//console
This message is automatically generated.
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218110#comment-13218110 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #182 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/182/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217807#comment-13217807 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1791 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1791/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217834#comment-13217834 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #606 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/606/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)
Result = ABORTED
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-2855:
--------------------------------------
Labels: mrv2 (was: )
Target Version/s: 0.23.2
Status: Patch Available (was: Open)
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218116#comment-13218116 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #969 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/969/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)
Result = UNSTABLE
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217808#comment-13217808 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1865 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1865/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2855:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.2
Release Note: Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Just committed to trunk, branch-0.23 and branch-0.23.2. Thanks Sid for the patch and the validation!
Thanks Todd for filing the ticket and suggesting a fix!
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Todd Lipcon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned MAPREDUCE-2855:
--------------------------------------
Assignee: (was: Todd Lipcon)
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218205#comment-13218205 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1004 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1004/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Vinod Kumar Vavilapalli (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli reassigned MAPREDUCE-2855:
--------------------------------------------------
Assignee: Siddharth Seth
Looking at the patch for review/commit now..
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217793#comment-13217793 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2855:
----------------------------------------------------
Nothing much to review here *smile*
Looks good, +1.
bq. On another note, do we really need to use ResourceBundles for the counter names, or is this something we can do away with?
Great question. Not even sure if anybody uses this at all. I guess we should just file a ticket and see if someone responds.
Pushing this in.
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217832#comment-13217832 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1801 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1801/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth. (Revision 1294436)
Result = ABORTED
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218164#comment-13218164 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #210 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/210/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)
Result = FAILURE
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Fix For: 0.23.2
>
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2855) ResourceBundle lookup during
counter name resolution takes a lot of time
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217806#comment-13217806 ]
Hudson commented on MAPREDUCE-2855:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #605 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/605/])
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1294436 ../../trunk (Revision 1294437)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294437
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceBundles.java
> ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-2855
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2855
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Siddharth Seth
> Labels: mrv2
> Attachments: MR2855.txt
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira