[jira] [Commented] (HIVE-16295) Add support for using Hadoop's OutputCommitter

["Aaron Fabbri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273948#comment-16273948 ] 

Aaron Fabbri commented on HIVE-16295:
-------------------------------------

Just FYI for watchers: the S3 Output Committer has been merged to trunk in Hadoop Common (HADOOP-13786).

> Add support for using Hadoop's OutputCommitter
> ----------------------------------------------
>
>                 Key: HIVE-16295
>                 URL: https://issues.apache.org/jira/browse/HIVE-16295
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a {{NullOutputCommitter}} and uses its own commit logic spread across {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with S3Guard and does a safe, coordinate commit of data on S3 inside individual tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}} there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from task retries or speculative execution) should not step on each other



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)