[jira] [Commented] (HIVE-29251) Hive ACID: HiveConf object shouldn't be shared between multiple cleanup task threads

["Denys Kuzmenko (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-29251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18029427#comment-18029427 ] 

Denys Kuzmenko commented on HIVE-29251:
---------------------------------------

Merged to master.

Thank you v much for the review, [~sbadhya] ! 

> Hive ACID: HiveConf object shouldn't be shared between multiple cleanup task threads
> ------------------------------------------------------------------------------------
>
>                 Key: HIVE-29251
>                 URL: https://issues.apache.org/jira/browse/HIVE-29251
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 4.1.0
>            Reporter: Denys Kuzmenko
>            Assignee: Denys Kuzmenko
>            Priority: Major
>              Labels: ACID, Compaction, hive-4.2.0-must, pull-request-available
>
> The {{HiveConf}} object is shared across multiple cleanup tasks. When the Cleaner runs with multiple threads, this can lead to a race condition.
> At the end of a cleanup task, the open transaction cap is lifted to check for any deltas that may become eligible for cleanup once the open transaction completes. If none are found, the cleanup request is marked complete; otherwise, it’s returned to the queue with a “ready-for-cleaning” status.
>  
> {code:java}
> conf.set(ValidTxnList.VALID_TXNS_KEY, new ValidReadTxnList().toString());
> dir = AcidUtils.getAcidState(fs, path, conf,
>   new ValidReaderWriteIdList(info.getFullTableName(), new long[0], new BitSet(), info.highestWriteId, Long.MAX_VALUE),
>   Ref.from(false), false,
>   dirSnapshots);
> List<Path> remained = subtract(CompactorUtil.getObsoleteDirs(dir, isDynPartAbort), deleted);
> if (!remained.isEmpty()) {
>   LOG.warn("Remained {} obsolete directories from {}. {}",
>   remained.size(), location, CompactorUtil.getDebugInfo(remained));
> } else {
>   LOG.debug("All cleared below the watermark: {} from {}", info.highestWriteId, location);
>   success = true;
> }{code}
> that might override the `{{{}ValidTxnList.VALID_TXNS_KEY`{}}} set in a concurrent task, leading to premature cleanup of data still in use by an active transaction.
> {code:java}
> private void cleanUsingAcidDir(CompactionInfo ci, String location, long minOpenTxn) throws Exception {
>   ValidTxnList validTxnList =
>     TxnUtils.createValidTxnListForCleaner(txnHandler.getOpenTxns(), minOpenTxn, false);
>   //save it so that getAcidState() sees it
>   conf.set(ValidTxnList.VALID_TXNS_KEY, validTxnList.writeToString());{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)