You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sergio Peña (JIRA)" <ji...@apache.org> on 2015/09/24 01:06:04 UTC
[jira] [Commented] (HIVE-11940) "INSERT OVERWRITE" query is very
slow because it creates one "distcp" per file to copy data from staging
directory to target directory
[ https://issues.apache.org/jira/browse/HIVE-11940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905471#comment-14905471 ]
Sergio Peña commented on HIVE-11940:
------------------------------------
[~Ferd] Could you help me review this patch? I think you worked on this part of the code, so you might be familiar with the change.
> "INSERT OVERWRITE" query is very slow because it creates one "distcp" per file to copy data from staging directory to target directory
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-11940
> URL: https://issues.apache.org/jira/browse/HIVE-11940
> Project: Hive
> Issue Type: Bug
> Affects Versions: 1.2.1
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Attachments: HIVE-11940.1.patch
>
>
> When hive.exec.stagingdir is set to ".hive-staging", which will be placed under the target directory when running "INSERT OVERWRITE" query, Hive will grab all files under the staging directory and copy them ONE BY ONE to target directory.
> When hive exec.stagingdir is set to "/tmp/hive", Hive will simply do a RENAME operation which will be instant.
> This happens with files that are not encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)