[I] [Bug] ConcurrentModificationException in PublishVersionDaemon blocks publish, leading to per-table committed-txn limit exceeded [doris]

["Larborator (via GitHub)" <gi...@apache.org>]

Larborator opened a new issue, #63169:
URL: https://github.com/apache/doris/issues/63169

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   4.0/4.1 (any version with enable_parallel_publish_version enabled; default is true).
   
   ### What's Wrong?
   
     TransactionState.publishVersionTasks is a plain HashMap:
   
     private Map<Long, List<PublishVersionTask>> publishVersionTasks;
     ...
     this.publishVersionTasks = Maps.newHashMap();
   
     Before 4.0, the entire publish flow ran inside the single-threaded MasterDaemon, so the non-thread-safe HashMap was fine. In 4.0, PublishVersionDaemon introduced parallel publish via a per-db thread pool (dbExecutors, sized by Config.publish_thread_pool_num, default 128). With enable_parallel_publish_version=true (default), tryFinishOneTxn dispatches
     tryFinishTxnSync to a worker thread while the master daemon thread continues its loop. The same publishVersionTasks map is now touched by two threads concurrently:
   
     Master daemon thread
     - forEach iterate — PublishVersionDaemon.tryFinishOneTxn
     - keySet().size() / keySet() — PublishVersionDaemon.tryFinishOneTxn
     - put / computeIfAbsent — TransactionState.addPublishVersionTask
   
     PUBLISH_VERSION_EXEC-N worker thread
     - values().forEach iterate — PublishVersionDaemon.tryFinishTxnSync
     - clear — TransactionState.pruneAfterVisible
   
     The typical race: the master thread iterates publishVersionTasks for a transaction whose worker-side tryFinishTxnSync from the previous round is just calling pruneAfterVisible() → clear(). HashMap's fail-fast iterator detects the modCount change and throws ConcurrentModificationException.
   
     The exception is caught at the outer level and does not crash FE, but the affected transaction's publish does not complete and stays in COMMITTED state. Repeated occurrences cause the backlog of un-published transactions to grow until per-table or per-db txn limits are hit and writes are blocked.
   
     Sample stack:
   
     ConcurrentModificationException
         at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597)
         at java.util.HashMap$EntryIterator.next(HashMap.java:1630)
         at PublishVersionDaemon.tryFinishOneTxn(PublishVersionDaemon.java:191)
   
   ### What You Expected?
   
     Publish should not silently fail due to CME. The per-transaction publishVersionTasks map should be safe for concurrent access by the master daemon thread and the PUBLISH_VERSION_EXEC worker threads.
   
   ### How to Reproduce?
   
     The race is timing-dependent and easier to trigger with:
     - enable_parallel_publish_version=true (default in 4.0)
     - High publish volume (many concurrent stream loads across many tables)
     - Multiple ready-to-publish transactions per daemon round so that the master thread iterates publishVersionTasks of one txn while a worker is in pruneAfterVisible() of the same txn
   
     Under sustained load on a busy cluster the CME fires within hours.
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org