[jira] [Commented] (HIVE-12205) Spark: unify spark statististics aggregation between local and remote spark client

["Chengxiang Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088795#comment-15088795 ] 

Chengxiang Li commented on HIVE-12205:
--------------------------------------

[~chinnalalam], thanks working on this. 
In your patch, the statistic aggregation is still computed separately in different methods(although in same class now) for {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, i suggest you can add a initialize method in {{MetrisCollection}} with parameter {{String jobId, Map<String, List<TaskMetrics>> jobMetrics}}, so that {{LocalSparkJobStatus}} can reuse {{MetricsCollection}} to aggregate statistics as well. What do you think?
Besides, could you create a ticket on RB for this?

> Spark: unify spark statististics aggregation between local and remote spark client
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-12205
>                 URL: https://issues.apache.org/jira/browse/HIVE-12205
>             Project: Hive
>          Issue Type: Task
>          Components: Spark
>    Affects Versions: 1.1.0
>            Reporter: Xuefu Zhang
>            Assignee: Chinna Rao Lalam
>         Attachments: HIVE-12205.1.patch
>
>
> In class {{LocalSparkJobStatus}} and {{RemoteSparkJobStatus}}, spark statistics aggregation are done similar but in different code paths. Ideally, we should have a unified approach to simply maintenance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)