[jira] [Created] (HIVE-29644) HMS hang/deadlock during ACID replication: compaction enqueue incorrectly runs inside replTableWriteIdState transaction

["Shreenidhi (Jira)" <ji...@apache.org>]

Shreenidhi created HIVE-29644:
---------------------------------

             Summary: HMS hang/deadlock during ACID replication: compaction enqueue incorrectly runs inside replTableWriteIdState transaction
                 Key: HIVE-29644
                 URL: https://issues.apache.org/jira/browse/HIVE-29644
             Project: Hive
          Issue Type: Bug
            Reporter: Shreenidhi


h3. Problem

During large Hive ACID bootstrap replication on the target (DR) cluster, HMS can become unresponsive. Queries stall at compile time waiting to open transactions. The issue requires HMS restart to recover.

Postgres {{pg_stat_activity}} shows multiple {{idle in transaction}} connections on:
 * {{AUX_TABLE}} ({{{}SELECT ... FOR UPDATE{}}} for {{CompactionScheduler}} mutex)
 * {{COMPACTION_QUEUE}} / {{NEXT_COMPACTION_QUEUE_ID}}

HMS logs show cross-node blocking between:
 * HMS running replication ({{{}ReplTableWriteIdStateFunction{}}} / {{{}repl_tbl_writeid_state{}}})
 * HMS running compaction initiator ({{{}CompactFunction{}}} via {{{}TxnHandler.compact{}}})

----
h3. Root cause

When replication applies ACID write-ID state for tables with aborted write IDs, HMS schedules major compaction for each partition to clean aborted delta files.

Before HIVE-27481, {{TxnHandler.replTableWriteIdState}} worked correctly:
 # Apply write-ID state in one DB transaction
 # Commit
 # Call separate {{compact()}} per partition (each with its own transaction)

After HIVE-27481 ({{{}TxnHandler cleanup{}}}), logic moved to {{ReplTableWriteIdStateFunction}} inside a single "{{{}@Transactional(POOL_TX)"{}}} method. 

Compaction enqueue via {{CompactFunction}} was incorrectly inlined in the same transaction as write-ID apply:
@Transactional(POOL_TX) replTableWriteIdState()
├── apply aborted write IDs, insert NEXT_WRITE_ID
└── for each partition:
                  CompactFunction.execute() // mutex (POOL_MUTEX) + NCQ lock (POOL_TX)
└── commit (only at end)
This causes:
 * {{NEXT_COMPACTION_QUEUE_ID}} row lock held across all partition enqueues in one long transaction
 * Repeated acquisition of {{CompactionScheduler}} mutex across loop iterations
 * Cross-connection lock contention / AB-BA deadlock with concurrent {{compact()}} (initiator, another replication job, or manual compact)

Manual {{ALTER TABLE ... COMPACT 'major'}} does not exhibit this because each {{compact()}} is a separate {{@Transactional(POOL_TX)}} call that commits immediately — same as pre-HIVE-27481 behavior.
----
h3. Locking details

Compaction enqueue uses two DB connections:
||Connection||Lock||Purpose||
|POOL_MUTEX|{{AUX_TABLE}} CompactionScheduler|Serialize compaction scheduling|
|POOL_TX|{{NEXT_COMPACTION_QUEUE_ID}} FOR UPDATE|Generate unique compaction queue ID|

Deadlock/contention occurs when:
 * Thread A holds NCQ lock (long repl txn) and waits for mutex (next partition iteration)
 * Thread B holds mutex (inside {{{}CompactFunction{}}}) and waits for NCQ lock

Disabling compactor initiator on DR reduces but does not eliminate risk — concurrent replication jobs alone can trigger the same pattern.
----
h3. Regression introduced by

HIVE-27481 — {{TxnHandler cleanup}} (Dec 2023)
File: {{ReplTableWriteIdStateFunction.java}} — inlined {{CompactFunction}} loop inside {{@Transactional(POOL_TX)}} {{{}replTableWriteIdState{}}}.

Pre-HIVE-27481 code explicitly committed write-ID state first, then called {{compact()}} separately per partition.
----
h3. Proposed fix

Restore pre-HIVE-27481 behavior in the refactored code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)