You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karthik Kambatla (JIRA)" <ji...@apache.org> on 2012/08/18 21:22:37 UTC
[jira] [Created] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Karthik Kambatla created MAPREDUCE-4565:
-------------------------------------------
Summary: Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
Key: MAPREDUCE-4565
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Karthik Kambatla
Assignee: Siddharth Seth
Fix For: 0.23.2
Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439187#comment-13439187 ]
Karthik Kambatla commented on MAPREDUCE-4565:
---------------------------------------------
Sid, thanks for taking care of this.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Fix For: 1.2.0
>
> Attachments: branch1-MR-4565.patch, branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4565:
----------------------------------------
Attachment: branch1-MR-4565.patch
Thanks for reviewing it, Sid. My bad to have missed the annotation.
Uploading a patch with the annotation.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch, branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-4565:
--------------------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-1. Thanks Karthik!
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Fix For: 1.2.0
>
> Attachments: branch1-MR-4565.patch, branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439091#comment-13439091 ]
Hadoop QA commented on MAPREDUCE-4565:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541823/branch1-MR-4565.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2754//console
This message is automatically generated.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch, branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla reassigned MAPREDUCE-4565:
-------------------------------------------
Assignee: Karthik Kambatla (was: Siddharth Seth)
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439061#comment-13439061 ]
Siddharth Seth commented on MAPREDUCE-4565:
-------------------------------------------
Looks good, except for one minor nit. Since CounterGroup is a public class, getResourceBundle should be annotated as @Private, or leave things as is and update the functions in both classes.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4565:
----------------------------------------
Attachment: branch1-MR-4565.patch
Uploading a patch very similar to the one posted for MR-2855.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-4565) Backport MR-2855 to
branch-1: ResourceBundle lookup during counter name resolution takes a lot
of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-4565 started by Karthik Kambatla.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4565:
----------------------------------------
Status: Patch Available (was: In Progress)
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4565:
----------------------------------------
Target Version/s: (was: 0.23.2)
Affects Version/s: (was: 0.23.0)
1.0.3
Fix Version/s: (was: 0.23.2)
Labels: (was: mrv2)
Issue Type: Improvement (was: Bug)
Hadoop Flags: (was: Reviewed)
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Siddharth Seth
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439124#comment-13439124 ]
Siddharth Seth commented on MAPREDUCE-4565:
-------------------------------------------
+1
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch, branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4565) Backport MR-2855 to branch-1:
ResourceBundle lookup during counter name resolution takes a lot of time
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437388#comment-13437388 ]
Hadoop QA commented on MAPREDUCE-4565:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541499/branch1-MR-4565.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2749//console
This message is automatically generated.
> Backport MR-2855 to branch-1: ResourceBundle lookup during counter name resolution takes a lot of time
> ------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4565
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: branch1-MR-4565.patch
>
>
> Loading a job status page in trunk takes a lot of time, and it seems like most of the time is spent resolving counter names. Looking through the JDK source, ResourceBundle.getBundle(String) ends up calling getClassContext() which is not very efficient. I think if we pass our own classloader manually it will be faster. In Counters.incrAllCounters, we may also be able to avoid setting the counter name if one is already set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira