You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2009/08/13 17:38:16 UTC
[jira] Created: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
-------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-864
URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: jobtracker
Reporter: Devaraj Das
Fix For: 0.21.0
MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751445#action_12751445 ]
Sharad Agarwal commented on MAPREDUCE-864:
------------------------------------------
Had an offline discussion with Devaraj and Hemanth. Apart from the issue mentioned above, one more issue was identified with this JIRA. The issue is that of consistency. This can't be done just for Job data. To be consistent all job client apis should return data from HDFS if not found in job tracker's memory. Consider the api like getJob(Jobid). To be consistent, it should also look HDFS for completed jobs if data is not in job tracker. Looking into the HDFS completed jobs folder and building up the job structures *efficiently* is a non-trivial thing to do at this point.
So we agree that a better approach at this point would be:
Retain the contract that job clients will *only* see information which are in Jobtracker's memory. Clients will get the very basic information of the completed jobs from job tracker's retired cache (MAPREDUCE-817).
Clients which need to drill down completed jobs' *TASK* level information will need to use History parser. The assumption here is that such clients will be very few and mostly these clients want to do analysis of the completed jobs. So it is better for them to use History parser directly and keep the job client interface clean.
The only minor concern here is that many clients may just need to look at the counters which are currently not cached in the retired job info. They will have to go to the History parser path to retrieve them. There should be a easy way to get those. The proposal is to add counters to the retired job cache. The idea is to just cache the job level information and not any task level in the retired jobs cache. Some quick estimate for the memory consumption. Assuming 100 counters per job and 200 bytes per counter. For 1000 retired jobs, it comes to 100*200*1000 = 20 MB, which is quite manageable.
> Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobtracker
> Reporter: Devaraj Das
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal reassigned MAPREDUCE-864:
----------------------------------------
Assignee: Sharad Agarwal
> Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobtracker
> Reporter: Devaraj Das
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751311#action_12751311 ]
Sharad Agarwal commented on MAPREDUCE-864:
------------------------------------------
bq. Job client will read the history file from HDFS location and construct the job information using JobHistory parser (provided as part of MAPREDUCE-157). While calling api's like Job#getCounters, Job clients would be transparent to the fact that information is being served from Jobtracker or from parsed history data.
I realized that this may have issues since there are calls (such as getTaskCompletionEvents, getTaskReports etc.) which need task level information as well. One option here is to load all history data in Job, but that leads to loading of huge datastructures in memory and many clients may not at all be interested in drilling down this task level stuff. Also reading from history transparently may confuse clients as the cost and performance impact of the same call will change drastically depending on the source of information.
I am inclining to NOT have org.apache.hadoop.mapreduce.Job serve data from history. Let clients directly use JobHistory parser API to construct the info they need when job is knocked out of job tracker's memory. Thoughts ?
> Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobtracker
> Reporter: Devaraj Das
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated MAPREDUCE-864:
-------------------------------------
Fix Version/s: (was: 0.21.0)
> Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobtracker
> Reporter: Devaraj Das
> Assignee: Sharad Agarwal
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754904#action_12754904 ]
Sharad Agarwal commented on MAPREDUCE-864:
------------------------------------------
Based on the above comments, instead of job client transparently providing job information for completed jobs, it will provide a way to get the history url for a given job id - > MAPREDUCE-975. Clients will use the History apis directly to construct the information.
> Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobtracker
> Reporter: Devaraj Das
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-864) Enhance JobClient API
implementations to look at history files to get information about jobs that
are not in memory
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12750958#action_12750958 ]
Sharad Agarwal commented on MAPREDUCE-864:
------------------------------------------
An additional state COMPLETED can be added to Job#JobState {DEFINE, RUNNING} to signify that job is completed and history file is there in HDFS. In COMPLETED state, Job client will read the history file from HDFS location and construct the job information using JobHistory parser (provided as part of MAPREDUCE-157). While calling api's like Job#getCounters, Job clients would be transparent to the fact that information is being served from Jobtracker or from parsed history data.
> Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-864
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobtracker
> Reporter: Devaraj Das
> Assignee: Sharad Agarwal
> Fix For: 0.21.0
>
>
> MAPREDUCE-817 added an API to get the JobHistory URL from the JobTracker. This is useful in two ways:
> 1) Users can use this API to get the URL, copy the history files to their local disk, and, do processing on them
> 2) APIs like JobSubmissionProtocol.getJobCounters, can read a part of the history file, and then return the information to the caller (if the job is not there in JT memory). This would mimic most of the CompletedJobsStatusStore functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.