You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2009/10/22 00:46:59 UTC
[jira] Created: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Using profilers other than hprof can cause JobClient to report job failure
--------------------------------------------------------------------------
Key: MAPREDUCE-1131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Aaron Kimball
Assignee: Aaron Kimball
If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787141#action_12787141 ]
Aaron Kimball commented on MAPREDUCE-1131:
------------------------------------------
Todd, Vinod --
Do you have thoughts about this version of the patch?
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768529#action_12768529 ]
Todd Lipcon commented on MAPREDUCE-1131:
----------------------------------------
+1 lgtm
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787433#action_12787433 ]
Vinod K V commented on MAPREDUCE-1131:
--------------------------------------
Patch changes look good to me.
One issue with the testcase - because now we are passing an invalid agentlib, tasks themselves are failing in the testcase, as can be expected, and so the test takes long time to finish (the new testcase itself takes more than 3mins). Doing the following makes the test run very fast(~30secs):
{code}
job.setMaxMapAttempts(1);
job.setMaxReduceAttempts(1);
job.setJobSetupCleanupNeeded(false);
{code}
Also, w.r.t {{TestMRJobClient}}, can you please remove the unused import of FileNotFoundException and also convert the test into a junit 4 test?
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Patch Available (was: Open)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Patch Available (was: Open)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Open (was: Patch Available)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768512#action_12768512 ]
Todd Lipcon commented on MAPREDUCE-1131:
----------------------------------------
Oops - I was thrown by the job name being the same in both cases. You should change the job name for the second submission, I think.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768561#action_12768561 ]
Aaron Kimball commented on MAPREDUCE-1131:
------------------------------------------
Failure is unrelated.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1131:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Aaron!
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774528#action_12774528 ]
Todd Lipcon commented on MAPREDUCE-1131:
----------------------------------------
Sounds good to me. I'd say split on whitespace, then check part.startsWith(...) for those strings rather than straight string compare - still simple enough and a little more correct.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790531#action_12790531 ]
Hadoop QA commented on MAPREDUCE-1131:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427970/MAPREDUCE-1131.6.patch
against trunk revision 890531.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/192/console
This message is automatically generated.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774524#action_12774524 ]
Todd Lipcon commented on MAPREDUCE-1131:
----------------------------------------
The idea of parsing the params seems reasonable enough, but I'm a little nervous about it. Do we know well the specification for java command line options? Can you run multiple profilers at once with two -javaagent params? [I'm not sure, just wondering if it's going to be just a quick regex vs something more complicated]
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768546#action_12768546 ]
Hadoop QA commented on MAPREDUCE-1131:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422866/MAPREDUCE-1131.3.patch
against trunk revision 828253.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/88/console
This message is automatically generated.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Open (was: Patch Available)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768509#action_12768509 ]
Aaron Kimball commented on MAPREDUCE-1131:
------------------------------------------
Todd: the test proves both the positive and the negative forms of the issue. Only the second iteration calls {{job.setProfileFetchEnabled(false);}}.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774383#action_12774383 ]
Aaron Kimball commented on MAPREDUCE-1131:
------------------------------------------
Vinod,
Seems reasonable to me. YourKit generates a file in a separate location which developers go and collect manually. I don't have experience with any other profiling tools, so I don't know how many of them do/don't respect a {{file=}} parameter like hprof does.
Looking at its documentation, jprof seems to generate several files in the working directory, but specifying the filename of any one of them is a pain. (It's got a lot of file prefix/suffix arguments.) So it's unlikely that jprof would work with the {{downloadProfile()}} as-written anyway. I don't know what other profiling tools are commonly used, so I don't know what else to check.
So are we sold on "check for hprof and {{file=%s}}" arguments, and select {{downloadProfile()}} based solely on that?
- Aaron
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Patch Available (was: Open)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796741#action_12796741 ]
Hudson commented on MAPREDUCE-1131:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774527#action_12774527 ]
Aaron Kimball commented on MAPREDUCE-1131:
------------------------------------------
Looking at [http://java.sun.com/developer/technicalArticles/Programming/HPROF.html], I think I'm just going to parse the output of {{conf.getProfileParams()}} to see if it contains the strings {{-agentlib:hprof}} or {{-Xrunhprof}}. This is likely not a 100% solution, but would cover an awful lot of uses...
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Patch Available (was: Open)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Attachment: MAPREDUCE-1131.3.patch
Job name changed.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Patch Available (was: Open)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790723#action_12790723 ]
Vinod K V commented on MAPREDUCE-1131:
--------------------------------------
Latest patch looks good to me. +1.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775287#action_12775287 ]
Hadoop QA commented on MAPREDUCE-1131:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424426/MAPREDUCE-1131.5.patch
against trunk revision 834284.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/131/console
This message is automatically generated.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Open (was: Patch Available)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768503#action_12768503 ]
Hadoop QA commented on MAPREDUCE-1131:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422859/MAPREDUCE-1131.patch
against trunk revision 828081.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
-1 findbugs. The patch appears to cause Findbugs to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/198/console
This message is automatically generated.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Attachment: MAPREDUCE-1131.5.patch
A more correct approach occurred to me 30 seconds after posting #4. Here you go.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Status: Open (was: Patch Available)
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Attachment: MAPREDUCE-1131.2.patch
Oops. Forgot to {{ant clean}} before testing. New patch.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768507#action_12768507 ]
Todd Lipcon commented on MAPREDUCE-1131:
----------------------------------------
I'm confused by the test - it seems to submit the same job twice and expect success for one and failure for the other. Did you upload the wrong copy of the test case or something?
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Attachment: MAPREDUCE-1131.4.patch
New patch, incorporates CR comments.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Attachment: MAPREDUCE-1131.patch
This patch adds a parameter ({{mapreduce.task.profile.enable.fetch}}) which defaults to true; if set true, profiler outputs are downloaded by the client. If false, the client does not attempt to fetch the output files. Users of other profilers can disable local fetches so that profiling does not interfere with job success reporting.
This adds a testcase which runs a job on a MiniMR cluster with profiling enabled, but an empty {{agentlib}} string. This causes no output to be generated by a profiler. With fetching disabled, this still allows the job to succeed.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787439#action_12787439 ]
Vinod K V commented on MAPREDUCE-1131:
--------------------------------------
Also, now that the patch needs some update, can you also add a positive test to verify success with hprof? We can then also verify that profile files named like {{$attempt-id.profile}} are indeed download by the JobClient when hprof(by default) is used.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1131:
-------------------------------------
Attachment: MAPREDUCE-1131.6.patch
New patch with tests updated per Vinod's request.
Ran test-patch locally:
{code}
[exec] +1 overall.- -
[exec]-
[exec] +1 @author. The patch does not contain any @author tags.
[exec]-
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]-
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]-
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]-
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]-
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]-
{code}
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.4.patch, MAPREDUCE-1131.5.patch, MAPREDUCE-1131.6.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773802#action_12773802 ]
Vinod K V commented on MAPREDUCE-1131:
--------------------------------------
Sorry for coming late on this. But I think the profile file is originally intended only when hprof is in use, and that to when the 'file' argument is passed .So, instead of adding another configuration, why not downloadProfile() *only* when hprof is in use and only when the file argument is non-null? This includes parsing the profile-params configuration, of course.
Or do you expect such file to be generated with other profilers too?
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1131) Using profilers other than hprof
can cause JobClient to report job failure
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768544#action_12768544 ]
Hadoop QA commented on MAPREDUCE-1131:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422863/MAPREDUCE-1131.2.patch
against trunk revision 828081.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/199/console
This message is automatically generated.
> Using profilers other than hprof can cause JobClient to report job failure
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-1131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1131.2.patch, MAPREDUCE-1131.3.patch, MAPREDUCE-1131.patch
>
>
> If task profiling is enabled, the JobClient will download the {{profile.out}} file created by the tasks under profile. If this causes an IOException, the job is reported as a failure to the client, even though all the tasks themselves may complete successfully. The expected result files are assumed to be generated by hprof. Using the profiling system with other profilers will cause job failure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.