[jira] [Comment Edited] (HIVE-23114) Insert overwrite with dynamic partitioning is not working correctly with direct insert

["Sungwoo (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078406#comment-17078406 ] 

Sungwoo edited comment on HIVE-23114 at 4/8/20, 3:47 PM:
---------------------------------------------------------

I tested with the new commit using HDFS. The ORC database was loaded without a problem, and TPC-DS queries run okay. (I tested with both TEXT and ORC databases, and obtained the same result.) 'catalog_returns where _returned_date_sk is null' contains no row whereas 'catalog_returns where _returned_date_sk is not null' returns a non-empty list.

EDIT: with a problem --> without a problem


was (Author: glapark):
I tested with the new commit using HDFS. The ORC database was loaded with a problem, and TPC-DS queries run okay. (I tested with both TEXT and ORC databases, and obtained the same result.) 'catalog_returns where _returned_date_sk is null' contains no row whereas 'catalog_returns where _returned_date_sk is not null' returns a non-empty list.


> Insert overwrite with dynamic partitioning is not working correctly with direct insert
> --------------------------------------------------------------------------------------
>
>                 Key: HIVE-23114
>                 URL: https://issues.apache.org/jira/browse/HIVE-23114
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Marta Kuczora
>            Assignee: Marta Kuczora
>            Priority: Major
>         Attachments: HIVE-23114.1.patch, HIVE-23114.2.patch, HIVE-23114.3.patch
>
>
> This is a follow-up Jira for the [conversation|https://issues.apache.org/jira/browse/HIVE-21164?focusedCommentId=17059280&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17059280] in HIVE-21164
>  Doing an insert overwrite from a multi-insert statement with dynamic partitioning will give wrong results for ACID tables when 'hive.acid.direct.insert.enabled' is true or for insert-only tables.
> Reproduction:
> {noformat}
> set hive.acid.direct.insert.enabled=true;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.vectorized.execution.enabled=false;
> set hive.stats.autogather=false;
> create external table multiinsert_test_text (a int, b int, c int) stored as textfile;
> insert into multiinsert_test_text values (1111, 11, 1111), (2222, 22, 1111), (3333, 33, 2222), (4444, 44, NULL), (5555, 55, NULL);
> create table multiinsert_test_acid (a int, b int) partitioned by (c int) stored as orc tblproperties('transactional'='true');
> create table multiinsert_test_mm (a int, b int) partitioned by (c int) stored as orc tblproperties('transactional'='true', 'transactional_properties'='insert_only');
> from multiinsert_test_text a
> insert overwrite table multiinsert_test_acid partition (c)
> select
>  a.a,
>  a.b,
>  a.c
>  where a.c is not null
> insert overwrite table multiinsert_test_acid partition (c)
> select
>  a.a,
>  a.b,
>  a.c
> where a.c is null;
> select * from multiinsert_test_acid;
> from multiinsert_test_text a
> insert overwrite table multiinsert_test_mm partition (c)
> select
>  a.a,
>  a.b,
>  a.c
>  where a.c is not null
> insert overwrite table multiinsert_test_mm partition (c)
> select
>  a.a,
>  a.b,
>  a.c
> where a.c is null;
> select * from multiinsert_test_mm;
> {noformat}
> The result of these steps can be different, it depends on the execution order of the FileSinkOperators of the insert overwrite statements. It can happen that an error occurs due to manifest file collision, it can happen that no error occurs but the result will be incorrect.
>  Running the same insert query with an external table of with and ACID table with 'hive.acid.direct.insert.enabled=false' will give the follwing result:
> {noformat}
> 1111    11      1111
> 2222    22      1111
> 3333    33      2222
> 4444    44      NULL
> 5555    55      NULL
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)