You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sahil Takiar (JIRA)" <ji...@apache.org> on 2018/06/01 13:58:00 UTC
[jira] [Updated] (HIVE-18690) Integrate with Spark OutputMetrics
[ https://issues.apache.org/jira/browse/HIVE-18690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sahil Takiar updated HIVE-18690:
--------------------------------
Attachment: HIVE-18690.4.patch
> Integrate with Spark OutputMetrics
> ----------------------------------
>
> Key: HIVE-18690
> URL: https://issues.apache.org/jira/browse/HIVE-18690
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-18690.1.patch, HIVE-18690.2.patch, HIVE-18690.3.patch, HIVE-18690.4.patch
>
>
> Spark has an {{OutputMetrics}} it uses to expose records / bytes written. We currently don't integrate with it and the Spark UI shows a blank value for output records / bytes. We have our own customer accumulators instead (like {{HIVE_RECORDS_OUT}}).
> Spark exposes the {{OutputMetrics}} object inside individual tasks via the {{TaskContext.get()}} method. We can use this method to access the {{OutputMetrics}} object and update it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)