[jira] [Commented] (HIVE-21100) Allow flattening of table subdirectories resulted when using TEZ engine and UNION clause

["Arnaud Linz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641987#comment-17641987 ] 

Arnaud Linz commented on HIVE-21100:
------------------------------------

The workaround does not always work as sometimes the merge step is skipped, despite having set hive.merge.tezfiles=true;
So to be sure we need to add a "hand made" HDFS move after each query with unions to keep the flat directory structure that is necessary for many tools (like Dataiku). Very cumbersome in my opinion.

> Allow flattening of table subdirectories resulted when using TEZ engine and UNION clause
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-21100
>                 URL: https://issues.apache.org/jira/browse/HIVE-21100
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: George Pachitariu
>            Assignee: George Pachitariu
>            Priority: Minor
>              Labels: pull-request-available
>         Attachments: HIVE-21100.1.patch, HIVE-21100.2.patch, HIVE-21100.3.patch, HIVE-21100.patch
>
>          Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Right now, when writing data into a table with Tez engine and the clause UNION ALL is the last step of the query, Hive on Tez will create a subdirectory for each branch of the UNION ALL.
> With this patch the subdirectories are removed, and the files are renamed and moved to the parent directory.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)