You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Sahil Takiar (JIRA)" <ji...@apache.org> on 2016/11/03 21:21:58 UTC
[jira] [Created] (HIVE-15121) Last MR job in Hive should be able to
write to a different scratch directory
Sahil Takiar created HIVE-15121:
-----------------------------------
Summary: Last MR job in Hive should be able to write to a different scratch directory
Key: HIVE-15121
URL: https://issues.apache.org/jira/browse/HIVE-15121
Project: Hive
Issue Type: Sub-task
Components: Hive
Reporter: Sahil Takiar
Hive should be able to configure all intermediate MR jobs to write to HDFS, but the final MR job to write to S3.
This will be useful for implementing parallel renames on S3. The idea is that for a mutli-job query, all intermediate MR jobs write to HDFS, and then the final job writes to S3. Writing to HDFS should be faster than writing to S3, so it makes more sense to write intermediate data to HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)