[jira] [Commented] (MAPREDUCE-7065) Improve information stored in ATSv2 for MR jobs

["Haibo Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396248#comment-16396248 ] 

Haibo Chen commented on MAPREDUCE-7065:
---------------------------------------

CC [~rohithsharma] [~vrushalic]. Is it OK at this point to change how MR data is stored in ATSv2, for example split MR_TASK into MR_MAP_TASK and MR_REDUCE_TASK?

> Improve information stored in ATSv2 for MR jobs
> -----------------------------------------------
>
>                 Key: MAPREDUCE-7065
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7065
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>            Priority: Major
>
> While exploring the possibility of retrieving every piece of information that JHS presents today through ATSv2, I found a few improvements we can make.
> 1) MR tasks are split by type in JHS, map tasks or reduce tasks. They are indistinguishably stored as entities of type MR_TASK. We can split MR_TASK into MR_REDUCE_TASK and MR_MAP_TASK. Similarly for MR_TASK_ATTEMPT
> 2) Task attempt final state are stored in the events, so we can not use infofilter to group task attempts by final state, which is what JHS does.
> 3) Display names of counters are not stored in JHS. We are currently storing (counter name, display name, value) as a metric (counter name, value). We can potentially store (counter name, display name) as an info. Similarly for sources of Job configuration properties
> 4) Job level counters and configuration properties are stored both in ApplicationTable and EntityTable. It's probably safe just to store MR specific counters in EntityTable.
>  
> One general problem I see around this area in MR:
> 1) We can precompute # of failed/killed/successful map/reduce task attempts and average map/reduce/shuffle/merge time in the AM. This would avoid iterating over all task attempts when JHS servers the Job Overview Page.
>  
> To fully replace JHS with ATSv2, three functionalities need to be supported by ATSv2
> 1) /apps/ query so that a list of all jobs can be retrieved
> 2) support streaming api to get all generic entities (YARN-5627)
> 3) support per-app data retention policy. Likely a setting in TimelineWriter that allow admins specifies how long information of a given application should be kepts, in the form of TTL in HBase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org