[jira] [Comment Edited] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called

["Denys Kuzmenko (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424366#comment-17424366 ] 

Denys Kuzmenko edited comment on HIVE-21052 at 10/5/21, 8:36 AM:
-----------------------------------------------------------------

*Warning*: could lead to data-loss.
{code}
In a situation when operation involved dynamic partitioning is aborted, Cleaner doesn't know what partition contains the aborted deltas, so it goes over all partitions and removes aborted and `obsolete` deltas below the HighWatermark (highest writeid that could be cleaned up). Those `obsolete` deltas could be in reality an `active` ones (there is no easy way to identify those as HighWatermark is defined on a table level). Current fix, in case of dynamic partition abort operation, is to skip the check for so called `obsolete` deltas and only handle 'aborted' ones.
{code}
Make sure to include HIVE-25502 as the fix for the described above issue.


was (Author: dkuzmenko):
*Warning*: could lead to data-loss.
{code}
In a situation when operation involved dynamic partitioning is aborted, Cleaner doesn't know what partition contains the aborted deltas, so it goes over all partitions and removes aborted and `obsolete` deltas below the HighWatermark (highest writeid that could be cleaned up). Those `obsolete` deltas could be in reality an `active` ones (there is no easy way to identify those as HighWatermark is defined on a table level). Current fix, in case of dynamic partition abort operation, is to skip the check for so called `obsolete` deltas and only handle 'aborted' ones.
{code}
Make sure to include HIVE-25503 as the fix for the described above issue.

> Make sure transactions get cleaned if they are aborted before addPartitions is called
> -------------------------------------------------------------------------------------
>
>                 Key: HIVE-21052
>                 URL: https://issues.apache.org/jira/browse/HIVE-21052
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 3.0.0, 3.1.1
>            Reporter: Jaume M
>            Assignee: Jaume M
>            Priority: Critical
>              Labels: pull-request-available
>         Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, HIVE-21052.8.patch, HIVE-21052.9.patch
>
>          Time Spent: 12.5h
>  Remaining Estimate: 0h
>
> If the transaction is aborted between openTxn and addPartitions and data has been written on the table the transaction manager will think it's an empty transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables. As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and when addPartitions is called remove this entry from TXN_COMPONENTS and add the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that specifies that a transaction was opened and it was aborted it must generate jobs for the worker for every possible partition available.
> cc [~ewohlstadter]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)