You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2010/04/14 03:05:52 UTC
[jira] Created: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
JobHistory shouldn't be disabled for any reason
-----------------------------------------------
Key: MAPREDUCE-1699
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Arun C Murthy
Fix For: 0.22.0
Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Status: Patch Available (was: Open)
I have revised the patch to address history being disabled from JobInfo.logSubmitted
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699-3.patch
Modified per Chris's comments
Removed changes related to CompletedJobStatusStore
Fix logSubmitted - will throw an error if logging fails
Fix for earlier release not for commit in the trunk
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1699:
-------------------------------------
Status: Open (was: Patch Available)
Assignee: Krishna Ramachandran
* It looks like MAPREDUCE-1778 is included in the latest patch
* {{JobHistory}} can still be disabled through {{JobInfo::logSubmitted}}
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856715#action_12856715 ]
Allen Wittenauer commented on MAPREDUCE-1699:
---------------------------------------------
We just hit this as well. Trying to build stats for capacity planning... and 'lo, we see no job stats. :(
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.22.0
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Status: Patch Available (was: Open)
Patch for Y! distribution security branch
With this change, if job history location is mis-configured JobTracker will throw an exception and fail.
A similar fix in place for trunk (part of https://issues.apache.org/jira/browse/MAPREDUCE-157)
This patch also addresses
https://issues.apache.org/jira/browse/MAPREDUCE-1778
If CompletedJobStatusStore is active & location is badly configured JobTracker will not start
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.20.3
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699.patch
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.20.3
>
> Attachments: mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876058#action_12876058 ]
Hadoop QA commented on MAPREDUCE-1699:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446441/mapred-1699-2.patch
against trunk revision 951832.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/225/console
This message is automatically generated.
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873195#action_12873195 ]
Krishna Ramachandran commented on MAPREDUCE-1699:
-------------------------------------------------
the patch is for 20.1xxS not for commit here
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.20.3
>
> Attachments: mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Status: Patch Available (was: Open)
Yes fix for 1778 was included based on discussions with Arun
oops I missed logSubmitted
In my latest revision I catch IOException and rethrow JobHistory::JobInfo::logSubmitted
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882402#action_12882402 ]
Krishna Ramachandran commented on MAPREDUCE-1699:
-------------------------------------------------
In trunk I believe, all job state loggings are handled in jobHistory.logEvent(jobId) {
/**
* Method to log the specified event
* @param event The event to log
* @param id The Job ID of the event
*/
public void logEvent(HistoryEvent event, JobID id) {
try {
final MetaInfo mi = fileMap.get(id);
if (mi != null) {
mi.writeEvent(event);
}
} catch (IOException e) {
LOG.error("Error Logging event, " + e.getMessage());
}
}
}
if logging fails just log a message. This is a lot cleaner - disableHistory is never used
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699-5.patch
Arun
I took out references to disableHistory
I do not believe we need a separate structure to store "disabled_state" for each jobId.
In jobHistory.logSubmitted, writer (entry point for history logging) is initialized for each jobID
If fileManager.addWriter(jobId, writer) fails, we log an error in catch block (just like trunk)
During subsequent writing of all log events (started/failed/killed/completed/inited/ogInfo ..) we check for
fileManager.getWriter(jobId) != null
(will be null if logSubmitted failed to initialize)
This should be sufficient and is similar to what you proposed (jobId->disabled)
I am modifying the patch accordingly
Let me know!
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699-7.patch
Revised (from Arun's review)
if logSubmitted writer fails we remove the entry from fileManager list (if not null)
Patch for an earlier release. Not for commit in trunk
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated MAPREDUCE-1699:
----------------------------------------
Affects Version/s: 0.20.2
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.22.0
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1699:
-------------------------------------
Release Note: (was: Fix committed 20.1xx)
Fix Version/s: (was: 0.20.3)
This was not fixed in the 0.20 branch. There are two options:
1) If the problem does not exist in trunk, please change resolution to "won't fix"
2) Reopen the bug, optionally attaching a patch for the Apache 0.20 branch
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699-1.patch
attached revision with test cases
not for commit in the trunk
fix for earlier release
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882320#action_12882320 ]
Krishna Ramachandran commented on MAPREDUCE-1699:
-------------------------------------------------
yes - this is good. I have added a map and logSubmitted will initialize in case of an error for that job and use that for subsequent logging (failed, killed etc.,)
Will update patch once testing is completed
Am not able to entirely take out disableHistory because of this code (incorrect javadoc - there is no param - jobID)
/**
* Logs history meta-info to the history file. This needs to be called once
* per history file.
* @param jobId job id, assigned by jobtracker.
*/
static void logMetaInfo(ArrayList<PrintWriter> writers){
if (!disableHistory){
if (null != writers){
JobHistory.log(writers, RecordTypes.Meta,
new Keys[] {Keys.VERSION},
new String[] {String.valueOf(VERSION)});
}
}
}
also there are couple of (public) methods that use this flag. though I do not see where they are called
/**
* Returns history disable status. by default history is enabled so this
* method returns false.
* @return true if history logging is disabled, false otherwise.
*/
public static boolean isDisableHistory() {
return disableHistory;
}
/**
* Enable/disable history logging. Default value is false, so history
* is enabled by default.
* @param disableHistory true if history should be disabled, false otherwise.
*/
public static void setDisableHistory(boolean disableHistory) {
JobHistory.disableHistory = disableHistory;
}
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1699:
-------------------------------------
Status: Open (was: Patch Available)
This needs more work...
# Remove JobHistory.disableHistory
# JobHistory.logSubmitted shouldn't throw an IOException, failure to log job-history shouldn't fail a job. We probably need a map from JobID -> disabled which should be checked everywhere in-lieu of JobHistory.disableHistory
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1699:
-------------------------------------
Fix Version/s: 0.20.3
(was: 0.22.0)
It would be nice if this were fixed in the 0.20 branch
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.20.3
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872019#action_12872019 ]
Hadoop QA commented on MAPREDUCE-1699:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12445602/mapred-1699.patch
against trunk revision 947758.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/207/console
This message is automatically generated.
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.20.3
>
> Attachments: mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Status: Open (was: Patch Available)
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Status: Open (was: Patch Available)
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856739#action_12856739 ]
Sharad Agarwal commented on MAPREDUCE-1699:
-------------------------------------------
This has been fixed by MAPREDUCE-157 in 0.21
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Fix For: 0.22.0
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876059#action_12876059 ]
Hadoop QA commented on MAPREDUCE-1699:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446441/mapred-1699-2.patch
against trunk revision 951832.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/226/console
This message is automatically generated.
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699-2.patch
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Status: Patch Available (was: Open)
Revised patch based on Chris review
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1699:
--------------------------------------------
Attachment: mapred-1699-8.patch
Revised
if JobHistory.log() fails for a job, the writer (PrintWriter) for that will be removed from FileManager to prevent repeated failed attempts
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran reopened MAPREDUCE-1699:
---------------------------------------------
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1699) JobHistory shouldn't be disabled
for any reason
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran resolved MAPREDUCE-1699.
---------------------------------------------
Release Note: Fix committed 20.1xx
Resolution: Fixed
> JobHistory shouldn't be disabled for any reason
> -----------------------------------------------
>
> Key: MAPREDUCE-1699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Arun C Murthy
> Assignee: Krishna Ramachandran
> Fix For: 0.20.3
>
> Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch
>
>
> Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.