[jira] [Comment Edited] (HIVE-15121) Last MR job in Hive should be able to write to a different scratch directory

["Sahil Takiar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634938#comment-15634938 ] 

Sahil Takiar edited comment on HIVE-15121 at 11/4/16 2:03 AM:
--------------------------------------------------------------

I think some of the issues with merge-job should be fixed by HIVE-15114

The one caveat is the merge job will now read from S3 and write to S3. For the approach described in HIVE-14269 there is probably no way to avoid this. Hive doesn't know if the merge job is necessary until the last MR job has completed. So the last MR job will always write to S3, and the merge job will conditionally read from S3 and then write back to S3.


was (Author: stakiar):
i think the issues with merge-job should be fixed by HIVE-15114

> Last MR job in Hive should be able to write to a different scratch directory
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-15121
>                 URL: https://issues.apache.org/jira/browse/HIVE-15121
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>            Reporter: Sahil Takiar
>         Attachments: HIVE-15121.WIP.1.patch, HIVE-15121.WIP.patch
>
>
> Hive should be able to configure all intermediate MR jobs to write to HDFS, but the final MR job to write to S3.
> This will be useful for implementing parallel renames on S3. The idea is that for a mutli-job query, all intermediate MR jobs write to HDFS, and then the final job writes to S3. Writing to HDFS should be faster than writing to S3, so it makes more sense to write intermediate data to HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)