[PR] [Subscription] Bound pending WAL gap replay per round [iotdb]

["Caideyipi (via GitHub)" <gi...@apache.org>]

Caideyipi opened a new pull request, #18766:
URL: https://github.com/apache/iotdb/pull/18766

   ## Summary
   
   - Bound pending-entry WAL gap recovery by the configured subscription consensus per-round WAL entry budget.
   - Continue a large gap across later prefetch rounds instead of scanning it without an entry limit.
   - Add a regression test that verifies a two-entry budget advances the replay cursor across three rounds.
   
   ## Root cause
   
   After real-time pending delivery overflows, the next retained pending entry can be millions of search indexes ahead. The pending path called fillGapFromWAL, which passed Integer.MAX_VALUE to pumpFromSubscriptionWAL.
   
   For sparse partial topics, topic filtering happens after reading the region WAL. A single prefetch round could therefore scan millions of unrelated entries while holding the queue read lock. Queue close must acquire the write lock before detached cleanup starts, so unsubscribe can remain blocked and serialize later consumer-group procedures behind it.
   
   ## Field evidence
   
   The September 29, 2026 regression runs using build 48a60ca showed:
   
   - 27400: the full topic eventually consumed 8,622,154 of 8,622,154 rows, but P99 latency reached 1,133,871 ms and 3,661,849 real-time entries were dropped before WAL catch-up.
   - 27401: the partial topic consumed only 428 of 6,900 matching rows and logged 6,662,834 dropped real-time entries. DropSubscription, CreateConsumer, and DropConsumer procedures then remained stuck while subscription metadata synchronization was starved.
   - 27402: delivery stopped after 11,707,960 of 29,199,410 rows during concurrent DataNode internal RPC timeouts. That stop remains a separate investigation.
   
   ## Scope
   
   This change prevents pending-gap recovery from holding the queue lock for an unbounded WAL scan. It does not remove the region-level full WAL scan cost for sparse partial topics.
   
   ## Tests
   
   - mvn spotless:apply -pl iotdb-core/datanode
   - mvn -o test -pl iotdb-core/datanode -Dtest=ConsensusPrefetchingQueueTest#testPendingGapReplayHonorsPerRoundWalEntryLimit+testCloseDetachesBlockedPrefetchCleanup
   - Checkstyle: 0 violations
   - Spotless check: passed
   
   ## Baseline note
   
   ConsensusPrefetchingQueueTest#testWalReplayCountsOnlyUnavailableSearchIndexes currently fails identically on unmodified origin/master with expected 2 but was 0. The failure was reproduced with this commit stashed and is outside this change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [Subscription] Bound pending WAL gap replay per round [iotdb]

["jt2594838 (via GitHub)" <gi...@apache.org>]

jt2594838 merged PR #18766:
URL: https://github.com/apache/iotdb/pull/18766


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org