[jira] Commented: (HADOOP-4974) Make jobtracker resilient to memory issues

["Amar Kamat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660276#action_12660276 ] 

Amar Kamat commented on HADOOP-4974:
------------------------------------

As of now these are the cases where the jobtracker should guard itself :
- *JOB INIT :* Consider a case when the jobtracker is running low on memory. Ideally we expect the jobtracker to queue(not expand) jobs that might potentially bring down the jobtracker. 
-  *JSP :* _loadhistory.jsp_ caches job analysis results. For larger jobs this can occupy some amount of memory. Job history parsing results are cached because these results are required for  fine-grained analysis (like task analysis and job analysis). Parsing the job history again and again for every tip will be costly. The problem occurs when the jobtracker runs low on memory and we try to analyze a job from history.
- *OLD JOBS :* Look at HADOOP-4766 for more details.

> Make jobtracker resilient to memory issues
> ------------------------------------------
>
>                 Key: HADOOP-4974
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4974
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.21.0
>            Reporter: Amar Kamat
>             Fix For: 0.21.0
>
>
> JobTracker is vulnerable to memory errors/attacks. Few of them are as follows
> - *JOB INIT :* lot of users submitting large jobs. As every jobs is expanded, the jobtracker's memory can be completely used up
> - *JSP :* jsp (jobhistory.jsp etc) can also interfere with jobtracker's memory and hence the jobtracker should be protected against such attacks
> - *OLD JOBS :* lot of completed jobs can garble up jobtracker's memory and hence should be periodically cleaned up. HADOOP-4766 addresses this.
> The main intention of this issue is to track various jira's that help jobtracker battle memory attacks. Jobtracker should always be up and available. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.