[jira] [Comment Edited] (HIVE-20852) Compaction Initiator ignores datas inserted by Stream Data Ingest

["David G. (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-20852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17365272#comment-17365272 ] 

David G. edited comment on HIVE-20852 at 6/18/21, 6:43 AM:
-----------------------------------------------------------

We faced the same issue with the nifi processor PutHive3streaming and after analysed code and hive metastore tables we understood that the stransctions were not written into the metastore  which prevent hive compaction scanner to found our partitions. The workaround that we used is the following :

We set the table property 'NO_AUTO_COMPACTION'='true' to force Hive compaction job to ignore our tables

We Use nifi processor PutHiveQL to run scheduled compaction queries like that regarding :

alter table xxx PARTITION yyy compact 'major';

or

alter table xxx PARTITION yyy compact 'minor';


was (Author: dgu):
We faced the same issue with the nifi processor PutHive3streaming and after analysed code and hive metastore tables we understood that the stransctions were not written into the mestatore. The workaround that we used is the following :

We set the table property 'NO_AUTO_COMPACTION'='true' to force Hive compaction job to ignore our tables

We Use nifi processor PutHiveQL to run scheduled compaction queries like that regarding :

alter table xxx PARTITION yyy compact 'major';

or

alter table xxx PARTITION yyy compact 'minor';

> Compaction Initiator ignores datas inserted by Stream Data Ingest
> -----------------------------------------------------------------
>
>                 Key: HIVE-20852
>                 URL: https://issues.apache.org/jira/browse/HIVE-20852
>             Project: Hive
>          Issue Type: Bug
>          Components: API, Transactions
>    Affects Versions: 3.1.0
>            Reporter: Kei Miyauchi
>            Priority: Major
>
> HI,
> Before compaction, Initiator decides whether a table/partition is potentialCompaction by querying COMPLETE_TXN_COMPONENT.
> But I found the transactions which committed by Stream Data Ingest is not stored on COMPLETE_TXN_COMPONENT. This is because the statement "insert into COMPLETED_TXN_COMPONENTS (ctc_txnid, ctc_database, ctc_table, ctc_partition, ctc_writeid, ctc_update_delete) select tc_txnid, tc_database, tc_table, tc_partition, tc_writeid, 'N' from TXN_COMPONENTS where tc_txnid = (id)"  fails.
> I found INSERT statement to TXN_COMPONENTS isn't fired. select subquery above returns 0 row.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)