You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2010/07/06 22:07:52 UTC
[jira] Created: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Job.getCounters() returns null when using a cluster
---------------------------------------------------
Key: MAPREDUCE-1920
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Aaron Kimball
Priority: Critical
Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Patch Available (was: Open)
Re-running through Hudson.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897150#action_12897150 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
Tried running ant test with latest patch, but I see a lot test timeouts. The tests are timing out because of the same error: "CompletedJobStore trying to create /jobtracker/jobsInfo on the local file system" and MAPREDUCE-1366. All the tests which work on local FileSystem are timing out. Will post the list of failing tests once my ant test run finishes.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Open (was: Patch Available)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Patch Available (was: Open)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Patch Available (was: Open)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894131#action_12894131 ]
Hadoop QA commented on MAPREDUCE-1920:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448971/MAPREDUCE-1920.patch
against trunk revision 980316.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/601/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/601/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/601/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/601/console
This message is automatically generated.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886618#action_12886618 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
bq. According to a comment in JobTracker#retireJob() mapreduce.jobtracker.retirejobs is "primarily for testing" (I assume because if you set it to true on a real installation the JT eventually runs out of memory?).
Yes. The configuration is only for testing. If mapreduce.jobtracker.retirejobs is set to false, the jobs will never be retired.
Currently, JobTracker maintains a retired job cache which holds JobStatus of the retired jobs, but it does not hold counters. If we enable completed job store by default, the data(JobStatus) will be duplicated. I think we should leave the default configuration as is and let users enable completed job store if they are interested in counters. Also, we should mark MAPREDUCE-870 an incompatible change and update the release note.
Thoughts?
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Open (was: Patch Available)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Attachment: MAPREDUCE-1920.patch
The failures occur because the tests are trying to create /jobtracker/jobsInfo on the local file system. I've now fixed the tests that do this (and verified that they pass). I've also fixed TestMapredSystemDir, which was failing for a different reason.
Here are the results of test-patch:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 27 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
I think this is ready to go now. Amareshwari, would you be able to try this patch? Thanks!
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896786#action_12896786 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
I see the same errors with earlier patch also. Console output from above patch build also confirms that.
Tom, do you have any different setup on your machine?
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Patch Available (was: Open)
Assignee: Tom White
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885843#action_12885843 ]
Aaron Kimball commented on MAPREDUCE-1920:
------------------------------------------
I agree that this shouldn't break :) And yet, I configured MapReduce as a straight-up pseudo-distributed instance. I didn't set anything other than mapred.job.tracker and fs.default.name in the conf files.
My application calls job.getCounters() immediately upon return from job.waitForCompletion(). It may be possible that jobs are retiring instantaneously / "very quickly" in a manner that is racing with my application? Is there a guaranteed window of time for which a job won't be retired?
I feel like there should be a guaranteed minimum; maybe this is in time, maybe as long as the original reference to a Job object on the client is live? (Easier said than done in the latter case -- maybe the Job could be configured in such a way as to reserve the right to retrieve its Counters or other post-execution data at least once after waitForCompletion() returns?)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885817#action_12885817 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
Are you sure that the job is not retired? I strongly feel this should not break, because there are many unit tests calling this api. For example, TestMiniMRDFSSort calls this api and runs successfully on branch 0.21.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886136#action_12886136 ]
Aaron Kimball commented on MAPREDUCE-1920:
------------------------------------------
This is indeed the issue. Setting {{mapreduce.jobtracker.retirejobs}} to false allows things to run correctly.
If I remove that setting, then it fails. I think this indicates a need to do some sort of delay before retiring jobs. Otherwise the job client does not even display the counters in the stdout when the job is finished, which is an unexpected result.
What is the best option going forward? Some that I can think of:
* mapred-default.xml could enable the completed job store for 1 hr by default. Power users could override this if they need to
* we could add some code to delay job retiring for some minimum amount of time (10 minutes?)
* If the JobClient is still connected to the JT when the job finishes, the interaction could be modified to locally-cache a copy of the counters before retiring the job. Then existing references to the Job would have a guaranteed instance of the Counters available.
* At the very least, {{Job.getCounters()}} needs a javadoc comment that specifies that it may return null. I think this is an incompatible change from 0.20. This suggestion is in addition to any of the above three.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Attachment: MAPREDUCE-1920.patch
This patch (based on the first one) sets mapreduce.jobtracker.persist.jobstatus.active to false in the test mapred-site.xml. It passes all unit tests (I ran it on Linux). Here's the output of test-patch:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
{noformat}
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated MAPREDUCE-1920:
-------------------------------------
Affects Version/s: 0.21.0
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926079#action_12926079 ]
Hudson commented on MAPREDUCE-1920:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886402#action_12886402 ]
Aaron Kimball commented on MAPREDUCE-1920:
------------------------------------------
Tom,
I am using the "combined/old-style" tarball of 0.21 rc 1. I applied your patch in the {{mapred/}} directory and it applied, but I could not compile it because of:
{code}
/home/aaron/Desktop/hadoop-0.21.0/mapred/build.xml:24: Cannot find build-utils.xml imported from /home/aaron/Desktop/hadoop-0.21.0/mapred/build.xml
{code}
Is that an issue with the way you produce the combined tarball? Or a more general release bug that prevents it from self-hosting?
The patch itself looks good though: +1
I changed the two affected settings in my mapred-site.xml file and my job succeeded.
Thanks!
- Aaron
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896457#action_12896457 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
I see that patch adds conf.setUser(UserGroupInformation.getCurrentUser().getUserName()) in JobConf for many testcases. I did not understand why is that needed here. I ran TestTrackerReservation and TestClusterStatus with the attached patch, the tests are still failing.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Attachment: MAPREDUCE-1920.patch
New patch which fixes failing unit tests.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1920:
-----------------------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.21.0
Resolution: Fixed
I just committed this to trunk and branch 0.21.
Thanks Tom!
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Attachment: MAPREDUCE-1920.patch
According to a comment in JobTracker#retireJob() mapreduce.jobtracker.retirejobs is "primarily for testing" (I assume because if you set it to true on a real installation the JT eventually runs out of memory?).
I think enabling the job completed store for 1 hour is the most natural workaround. Here's a patch for enabling it by default. The patch also adds javadoc to Job#getCounters().
Aaron, does this patch fix the issue for you?
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885866#action_12885866 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
Can you bring up your cluster with mapreduce.jobtracker.retirejobs set to false and run your job? That would confirm the problem is with retire.
You can also enable completed job store, by setting mapreduce.jobtracker.persist.jobstatus.active to true and mapreduce.jobtracker.persist.jobstatus.hours to 1. Then, job details would be available for an hour.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Patch Available (was: Open)
The latest patch hasn't been run by Hudson yet, so let's see if it takes this time. I'll see if I can reproduce this error too (I ran it on a Mac when it passed).
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897330#action_12897330 ]
Tom White commented on MAPREDUCE-1920:
--------------------------------------
> Or shall we disable completed job store for the unit tests by adding conf in src/test/mapred-site.xml (similar to disabling retire jobs) as TestJobStatusPersistency anyways tests the functionality of completedJobStore?
I think this is a much better way of doing it. Thanks for the suggestion. I'll prepare a patch.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886672#action_12886672 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
I was thinking whether we should disable retited jobs cache, if we enable completed job store by default, to remove duplicate data storage. But, now i feel we can enable both, because retired job cache is served from memory, whereas completed job store is served from file system; and clients are served from retired jobs cache first, if not found in the cache, then they are served from completed job store.
Attached patch looks fine to me.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886534#action_12886534 ]
Hadoop QA commented on MAPREDUCE-1920:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448971/MAPREDUCE-1920.patch
against trunk revision 961578.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/293/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/293/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/293/console
This message is automatically generated.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896560#action_12896560 ]
Tom White commented on MAPREDUCE-1920:
--------------------------------------
Thanks for taking a look, Amareshwari. With the first patch I get the following failure for TestTrackerReservation:
{noformat}
Testcase: testTaskTrackerReservation took 0.431 sec
Caused an ERROR
null
java.lang.NullPointerException
at org.apache.hadoop.io.Text.encode(Text.java:396)
at org.apache.hadoop.io.Text.encode(Text.java:377)
at org.apache.hadoop.io.Text.writeString(Text.java:417)
at org.apache.hadoop.mapreduce.JobStatus.write(JobStatus.java:339)
at org.apache.hadoop.mapred.CompletedJobStatusStore.store(CompletedJobStatusStore.java:178)
at org.apache.hadoop.mapred.JobTracker.storeCompletedJob(JobTracker.java:3427)
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:3344)
at org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2889)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2771)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1231)
at org.apache.hadoop.mapred.FakeObjectUtilities$FakeJobInProgress.finishTask(FakeObjectUtilities.java:186)
at org.apache.hadoop.mapred.TestTrackerReservation.testTaskTrackerReservation(TestTrackerReservation.java:138)
at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
at junit.extensions.TestSetup.run(TestSetup.java:27)
{noformat}
This is caused by a null user name. The second patch sets the user for the job, and passes for me.
What failure are you getting with the second patch?
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897188#action_12897188 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
Tests that timed out till now:
TestAdminOperationsProtocolWithServiceAuthorization
TestClusterMRNotification
TestDebugScript
TestEmptyJob
TestIsolationRunner
TestJobCleanup
TestJobHistory
TestJobHistoryParsing
TestJobInProgress
TestJobInProgressListener
TestJobKillAndFail
TestJobQueueClient
TestJvmReuse
TestKillSubProcesses
TestMRWithDistributedCache
TestMapredHeartbeat
TestMiniMRBringup
Tests that failed:
TestJobTrackerStart
TestKillCompletedJob
my local ant test run is still running. So, more tests to be added to the above list.
Shall we fix MiniMRCluster to set a persist dir in local file system if fileSystem passed is local, instead of fixing these individual tests?
Or shall we disable completed job store for the unit tests by adding conf in src/test/mapred-site.xml (similar to disabling retire jobs) as TestJobStatusPersistency anyways tests the functionality of completedJobStore?
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886668#action_12886668 ]
Tom White commented on MAPREDUCE-1920:
--------------------------------------
Actually, I would do it the other way round. Users expect to be able to get counters from jobs they have just run, as witnessed by Aaron's experience that led to this bug (also http://lucene.472066.n3.nabble.com/Hadoop-0-21-job-getCounters-returns-null-td947190.html). I would rather have the default configuration work as expected, and advanced users can turn off the job store if they don't want to use it. Does that sound reasonable?
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885663#action_12885663 ]
Aaron Kimball commented on MAPREDUCE-1920:
------------------------------------------
The new API seems to have an issue w.r.t. counters. Calling Job.getCounters() after the job has completed (successfully) returns null. I can see all the counters there on the JobTracker status web page. They have the correct values. But I can't access them programmatically.
So, this is returning null:
{code}
public class Job extends JobContextImpl implements JobContext {
...
public Counters getCounters()
throws IOException, InterruptedException {
ensureState(JobState.RUNNING);
return cluster.getClient().getJobCounters(getJobID());
}
}
{code}
This seems to work fine with the LocalJobRunner.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Priority: Critical
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897134#action_12897134 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
+1
Latest patch looks fine.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Open (was: Patch Available)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-1920:
---------------------------------
Status: Open (was: Patch Available)
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1920) Job.getCounters() returns null
when using a cluster
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896764#action_12896764 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1920:
----------------------------------------------------
TestTrackerReservation failed with following exception on my machine:
{noformat}
Testcase: unknown took 0 sec
Caused an ERROR
CompletedJobStatusStore mkdirs failed to create /jobtracker/jobsInfo
java.io.IOException: CompletedJobStatusStore mkdirs failed to create /jobtracker/jobsInfo
at org.apache.hadoop.mapred.CompletedJobStatusStore.<init>(CompletedJobStatusStore.java:83)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1626)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1373)
at org.apache.hadoop.mapred.FakeObjectUtilities$FakeJobTracker.<init>(FakeObjectUtilities.java:59)
at org.apache.hadoop.mapred.TestTrackerReservation$FakeJobTracker.<init>(TestTrackerReservation.java:48)
at org.apache.hadoop.mapred.TestTrackerReservation$1.setUp(TestTrackerReservation.java:64)
at junit.extensions.TestSetup$1.protect(TestSetup.java:22)
at junit.extensions.TestSetup.run(TestSetup.java:27)
{noformat}
TestClusterStatus also fails with similar error.
> Job.getCounters() returns null when using a cluster
> ---------------------------------------------------
>
> Key: MAPREDUCE-1920
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1920
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Aaron Kimball
> Assignee: Tom White
> Priority: Critical
> Attachments: MAPREDUCE-1920.patch, MAPREDUCE-1920.patch
>
>
> Calling Job.getCounters() after the job has completed (successfully) returns null.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.