You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "dao-jun (via GitHub)" <gi...@apache.org> on 2026/06/23 11:35:53 UTC
[PR] [fix][broker] Guard BucketDelayedDeliveryTracker.nextDeliveryTime against empty queues [pulsar]
dao-jun opened a new pull request, #26080:
URL: https://github.com/apache/pulsar/pull/26080
### Motivation
The orphaned-message filter introduced in #25984 can leave BucketDelayedDeliveryTracker in a state where lastMutableBucket and sharedBucketPriorityQueue are both empty while numberDelayedMessages is still positive — the remaining messages live in not-yet-loaded snapshot segments of orphaned immutable buckets whose ledger
range falls entirely below the cursor's mark-deleted position.
In that state, nextDeliveryTime() falls through both existing if branches and unconditionally calls lastMutableBucket.nextDeliveryTime() → MutableBucket.nextDeliveryTime() → TripleLongPriorityQueue.peekN1(), which throws IllegalArgumentException("...") on the empty queue:
java.lang.IllegalArgumentException
at TripleLongPriorityQueue.peekN1(TripleLongPriorityQueue.java:126)
at MutableBucket.nextDeliveryTime(MutableBucket.java:222)
at BucketDelayedDeliveryTracker.nextDeliveryTime(BucketDelayedDeliveryTracker.java:621)
at AbstractDelayedDeliveryTracker.updateTimer(AbstractDelayedDeliveryTracker.java:127)
at BucketDelayedDeliveryTracker.getScheduledMessages(BucketDelayedDeliveryTracker.java:754)
The exception propagates out of getScheduledMessages() / hasMessageAvailable() and surfaces on whatever broker thread polled the tracker (e.g., a dispatcher read path). The fall-through pattern has been there since PIP-195 (#17611, 2022); the trim filter made the both-empty-but-count-positive state reachable, so this is a
regression window introduced by #25984.
### Modifications
- Added a third branch to BucketDelayedDeliveryTracker.nextDeliveryTime() that returns Long.MAX_VALUE when both lastMutableBucket and sharedBucketPriorityQueue are empty. This signals "no imminent delivery" symmetrically with how the parent AbstractDelayedDeliveryTracker.updateTimer() already treats
numberOfDelayedMessages == 0, and is interpreted correctly by both call sites:
- hasMessageAvailable(): Long.MAX_VALUE <= cutoffTime is false → returns false.
- updateTimer(): Long.MAX_VALUE - now is huge → schedules the timer far out; new addMessage() calls and a successful asyncTrimImmutableBuckets() will reschedule it. Netty's HashedWheelTimer clamps the nanos conversion at Long.MAX_VALUE (no overflow).
- Added regression test BucketDelayedDeliveryTrackerTest.testGetScheduledMessagesWhenAllOrphaned. It seals 5 orphaned messages into a single immutable bucket (5 snapshot segments, only the first one loaded into the shared queue), flushes the mutable bucket into the shared queue by advancing the clock, then calls
getScheduledMessages() and hasMessageAvailable(). Without the fix it fails with the IAE above; with the fix it asserts:
- No exception is thrown.
- scheduledMessages is empty (orphaned messages are filtered, not delivered).
- numberDelayedMessages is still positive (storage-only messages keep the counter up).
- hasMessageAvailable() returns false.
Scope note
This is a defensive fix for the crash only. The underlying inconsistency — numberDelayedMessages > 0 while no in-memory message exists — resolves itself once asyncTrimImmutableBuckets() deletes the orphaned bucket and decrements the counter, which happens as soon as immutableBuckets.size() > maxNumBuckets triggers
trim+merge again. A more thorough cleanup (e.g., probing the next snapshot segment inside the filter loop, or running trim on a schedule rather than only on bucket overflow) is intentionally out of scope for this PR and can be tracked separately if it turns into a real production concern.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Guard BucketDelayedDeliveryTracker.nextDeliveryTime against empty queues [pulsar]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #26080:
URL: https://github.com/apache/pulsar/pull/26080#discussion_r3465093876
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/delayed/bucket/BucketDelayedDeliveryTrackerTest.java:
##########
@@ -682,4 +682,49 @@ public void testMergeEarlyReturnWhenWithinLimit() throws Exception {
ts.close();
}
+
+ @Test
+ public void testGetScheduledMessagesWhenAllOrphaned() throws Exception {
+ // Reproduces IAE in nextDeliveryTime: when every delayed message lies below the
+ // mark-delete position, the filter in getScheduledMessages pops the in-memory
+ // messages without returning them. If the immutable bucket has additional messages
+ // still in storage (later snapshot segments), numberDelayedMessages stays > 0
+ // while both the mutable bucket and the shared priority queue are empty.
+ // The trailing updateTimer -> nextDeliveryTime must not throw.
+ long firstLedgerId = 50L;
+ TrackerWithStorage ts = createTrackerWithMockLedger(firstLedgerId, 50);
+
+ // Five delayed messages on the same orphaned ledger (ledgerId < firstLedgerId).
+ // They share a mutable bucket because seal requires a strictly greater ledgerId.
+ // Timestamps are 100ms apart so each lands in its own snapshot segment
+ // (timeStep=10ms); only the first segment is loaded into the shared queue at seal.
+ for (int i = 1; i <= 5; i++) {
+ ts.tracker.addMessage(1, i, i * 100);
+ }
+ // A new orphaned ledgerId triggers the seal, producing immutable bucket [1..1]
+ // with 5 messages across 5 segments; shared queue holds just the first segment.
+ ts.tracker.addMessage(2, 1, 600);
+
+ Awaitility.await().untilAsserted(() ->
+ Assert.assertTrue(ts.tracker.getImmutableBuckets().asMapOfRanges().values().stream()
+ .noneMatch(x -> x.merging)));
+
+ // Advance clock past tickTime + max deliverAt so moveScheduledMessageToSharedQueue
+ // flushes the mutable trigger message into the shared queue as well.
+ ts.clockTime.set(600 + 100000);
Review Comment:
The comment suggests the flush depends on `tickTime`, but in strict deliver-at mode `moveScheduledMessageToSharedQueue()` uses the cutoff time from `Clock.millis()` only. Consider simplifying the comment (and the time jump) to reflect that only advancing past the trigger message’s `deliverAt` is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Guard BucketDelayedDeliveryTracker.nextDeliveryTime against empty queues [pulsar]
Posted by "nodece (via GitHub)" <gi...@apache.org>.
nodece merged PR #26080:
URL: https://github.com/apache/pulsar/pull/26080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org