[jira] [Comment Edited] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called

["Eugene Koifman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739799#comment-16739799 ] 

Eugene Koifman edited comment on HIVE-21052 at 1/10/19 9:52 PM:
----------------------------------------------------------------

I don't think testing the manually is sufficient.
See {{HiveConf.HIVETESTMODEROLLBACKTXN}}, {{HiveConf.HIVETESTMODEFAILCOMPACTION}} etc
You can add a similar one to inject a fault during a unit test to throw at the start of {{TxnHandler.addDynamicPartitions()}} for example

Could you explain you overall strategy?
I would've thought {{TxnHandler.enqueueLockWithRetry()}} has all the relevant info about whether it's a DP write or not.

Is it not possible to check if {{lc.getOperationType()}} is I/U/D and {{lc.isIsDynamicPartitionWrite()}}, then get the (dbname,table name) from LockComonent
and create the 'p' type entry in TXN_COMPONENTS.  just make sure there is 1 per table.

The same method has logic (via SQL query) find writeID associated with this txnId.

Does this work?


was (Author: ekoifman):
I don't think testing the manually is sufficient.
See {{HiveConf.HIVETESTMODEROLLBACKTXN}}, {{HiveConf.HIVETESTMODEFAILCOMPACTION}} etc
You can add a similar one to inject a fault during a unit test to throw at the start of {{TxnHandler.addDynamicPartitions()}} for example

> Make sure transactions get cleaned if they are aborted before addPartitions is called
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-21052
>                 URL: https://issues.apache.org/jira/browse/HIVE-21052
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 3.1.1
>            Reporter: Jaume M
>            Assignee: Jaume M
>            Priority: Critical
>         Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, HIVE-21052.2.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has been written on the table the transaction manager will think it's an empty transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables. As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and when addPartitions is called remove this entry from TXN_COMPONENTS and add the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that specifies that a transaction was opened and it was aborted it must generate jobs for the worker for every possible partition available.
> cc [~ewohlstadter]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)