You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/12/06 18:47:43 UTC
[jira] Created: (HADOOP-2367) Get representative hprof information
from tasks
Get representative hprof information from tasks
-----------------------------------------------
Key: HADOOP-2367
URL: https://issues.apache.org/jira/browse/HADOOP-2367
Project: Hadoop
Issue Type: New Feature
Components: mapred
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 0.16.0
It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Attachment: profile-3.patch
Updated to trunk.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Attachment: profile.patch
This patch adds a config variable to enable profiling of the tasks. It also adds a range variable that controls which maps and reduces are profiled, which defaults to 0-2. The job client pulls down the profiles to the launching machine and they are available on the web/ui.
I also had to fix the web/ui to not put tasklogs in a table, because the browser silently truncates the table. I also fixed the task log servlet to quote html characters when being sent to the browser.
There is one problem with the current patch in that sometimes the first task is reported to the job client before the jvm is done writing the information, so the first debug log on the client side is only partial. If the user goes to the web/ui, the entire file is available.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2367) Get representative hprof
information from tasks
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560820#action_12560820 ]
Hudson commented on HADOOP-2367:
--------------------------------
Integrated in Hadoop-trunk #371 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/371/])
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile-4.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2367) Get representative hprof
information from tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559315#action_12559315 ]
Hadoop QA commented on HADOOP-2367:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373134/profile-3.patch
against trunk revision r612314.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1604/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1604/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1604/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1604/console
This message is automatically generated.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Status: Patch Available (was: Open)
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Status: Patch Available (was: Open)
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile-4.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Attachment: profile-2.patch
This patch is the previous one where the client gets the profile if the task fails or succeeds.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Status: Open (was: Patch Available)
Updating to current trunk.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2367) Get representative hprof
information from tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549342 ]
Hadoop QA commented on HADOOP-2367:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371206/profile-2.patch
against trunk revision r602002.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1291/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1291/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1291/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1291/console
This message is automatically generated.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Attachment: profile-4.patch
I addressed Chris' concerns, including create a unit test of the new IntegerRanges. (which found a bug, thanks!)
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile-4.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Status: Open (was: Patch Available)
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile-4.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2367:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Owen!
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile-4.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2367) Get representative hprof
information from tasks
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560828#action_12560828 ]
Hudson commented on HADOOP-2367:
--------------------------------
Integrated in Hadoop-Nightly #371 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/371/])
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile-4.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2367) Get representative hprof
information from tasks
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558325#action_12558325 ]
Chris Douglas commented on HADOOP-2367:
---------------------------------------
The patch no longer applies to trunk; could you update it?
Also:
* The IntegerRanges code in Configuration could use a test case
* Also in IntegerRanges, a check that start <= end is probably a good idea
* It might be useful to let the profiling string in TaskRunner be configurable
* IOUtils.copyBytes can probably replace some of downlaodProfile code, since the profile output is printable
* Speculatively executed tasks are marked as killed, so that profiling information won't be downloaded, but what of jobs killed by the user? Would an option to download all profile information regardless of the event status be useful?
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2367) Get representative hprof information
from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2367:
----------------------------------
Status: Patch Available (was: Open)
This patch is fixed with regard to trunk.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile-2.patch, profile-3.patch, profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2367) Get representative hprof
information from tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549326 ]
Owen O'Malley commented on HADOOP-2367:
---------------------------------------
The race condition between the subprocess finishing and the event being sent out seemed out of scope for this bug, so I created a new one. HADOOP-2375.
> Get representative hprof information from tasks
> -----------------------------------------------
>
> Key: HADOOP-2367
> URL: https://issues.apache.org/jira/browse/HADOOP-2367
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: profile.patch
>
>
> It would be great to get a representative (2 or 3) sample of builtin java profiler for a sample of maps and reduces. I'd store the information in the userlog directory and make it available in the local file system of the submitting application.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.