You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "Asthenia0412 (via GitHub)" <gi...@apache.org> on 2026/10/14 13:17:14 UTC

[PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Asthenia0412 opened a new pull request, #26899:
URL: https://github.com/apache/pulsar/pull/26899

   ## Background
   
   This PR fixes a `NullPointerException` reported in Issue #24410. The stack trace shows:
   
   ```
   java.lang.NullPointerException: Cannot read field "mbean" because "finalML" is null
       at org.apache.bookkeeper.mledger.impl.OpAddEntry.handleAddFailure(OpAddEntry.java:339)
       at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.checkAddTimeout(ManagedLedgerImpl.java:4205)
   ```
   
   PR #12103 previously attempted a fix by copying `this.ml` into a local `final` variable to prevent lambda capture issues, but it did not account for the case where `this.ml` itself is already null when `handleAddFailure()` is invoked.
   
   ## Problem
   
   The race condition is:
   
   1. `ManagedLedgerImpl.checkAddTimeout()` peeks an `OpAddEntry` from `pendingAddEntries` queue
   2. Before `handleAddFailure()` is called, another thread completes the same OpAddEntry (BK callback `addComplete` → `run()` → `recycle()`)
   3. After `recycle()`, `this.ml` is set to null
   4. Now `checkAddTimeout` calls `opAddEntry.handleAddFailure()`, which tries to access `finalMl.mbean` → **NPE**
   
   Additionally, `checkAddTimeout` passes `opAddEntry.ledger` which could also be null in some scenarios (e.g. ShadowManagedLedger), leading to a second NPE when the code tries to call `lh.asyncClose()`.
   
   ## Solution
   
   Add defensive null checks in `handleAddFailure()`:
   
   1. **Null check for `ml`**: If `finalMl == null`, the OpAddEntry has been recycled, so return immediately — there's nothing to do for an already-recycled entry.
   
   2. **Null check for `lh`**: When the code needs to close the failed ledger, first check that `finalLh != null` before calling `asyncClose()`.
   
   This approach is safe and non-intrusive — it only prevents crashes in edge cases where the OpAddEntry lifecycle has already moved on, and preserves all existing behavior for normal paths.
   
   ## Testing
   
   - The fix is a minimal defensive null-check addition following the same pattern already established elsewhere in the managed-ledger code.
   - No new test added (difficult to reproduce the exact race condition deterministically in unit tests); however, the fix makes the code resilient to this race without any behavioral change in the normal execution path.
   
   Fixes: #24410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #26899:
URL: https://github.com/apache/pulsar/pull/26899#discussion_r4374864331


##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java:
##########
@@ -366,9 +366,15 @@ void handleAddFailure(final LedgerHandle lh, Integer rc) {
         // If we get a write error, we will try to create a new ledger and re-submit the pending writes. If the
         // ledger creation fails (persistent bk failure, another instance owning the ML, ...), then the writes will
         // be marked as failed.
+
+        // The race where an entry was recycled between checkAddTimeout's peek() and handleAddFailure() no longer
+        // exists: checkAddTimeout now runs on the per-managed-ledger executor, which serializes it with OpAddEntry
+        // completion/recycling paths. So this.ml is guaranteed non-null here.
         ManagedLedgerImpl finalMl = this.ml;
         finalMl.mbean.recordAddEntryError();
 
+        final LedgerHandle finalLh = lh;

Review Comment:
   This is unnecessary since the parameter is already `final LedgerHandle lh`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #26899:
URL: https://github.com/apache/pulsar/pull/26899#issuecomment-6410647621

   already addressed by #27056


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "Asthenia0412 (via GitHub)" <gi...@apache.org>.
Asthenia0412 commented on PR #26899:
URL: https://github.com/apache/pulsar/pull/26899#issuecomment-6227944646

   Rebased onto upstream/master (no conflicts). Addressed codelipenghui's core recommendation: fix the race at the scheduling layer, not only with defensive null checks.
   
   **Root-cause fix (ManagedLedgerImpl.java):**
   
   `scheduleAddEntryTimeoutTask` now dispatches `checkAddTimeout` onto the per-managed-ledger ordered executor:
   
   ```java
   this.timeoutTask = this.scheduledExecutor.scheduleAtFixedRate(
           () -> executor.execute(this::checkAddTimeout), timeoutSec, timeoutSec, TimeUnit.SECONDS);
   ```
   
   This serializes the **entire** timeout-check work (peek from `pendingAddEntries`, expiration check, `handleAddFailure` call) with the completion/recycling paths (`addComplete` → `run()` → `recycle()`), which already run on that same executor. The race — an OpAddEntry being recycled between `peek()` and `handleAddFailure()` — is eliminated by ordering, not papered over.
   
   **Retained defensive guards (OpAddEntry.java):**
   
   - `finalMl == null` early return — now theoretically unreachable with the executor ordering, kept as belt-and-suspenders.
   - `finalLh != null` guard before `asyncClose()` — kept because ShadowManagedLedger paths can legitimately produce a null handle.
   
   Tradeoff: timeout checks now queue behind managed-ledger work, adding some latency. But recovery and other housekeeping already depend on this executor, so the ordering cost is bounded and consistent.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "Asthenia0412 (via GitHub)" <gi...@apache.org>.
Asthenia0412 commented on PR #26899:
URL: https://github.com/apache/pulsar/pull/26899#issuecomment-6169505193

   Hi @Technoboy- @nodece,
   
   This PR fixes a race-condition NPE in OpAddEntry.handleAddFailure(). When an OpAddEntry gets recycled between `pendingAddEntries.peek()` and `handleAddFailure()`, the `finalMl` reference becomes null.
   
   The fix adds defensive null checks for `finalMl` and `finalLh` at the start of `handleAddFailure()`. If the entry was already recycled, we bail out early.
   
   Added tests would be tricky since the race is timing-dependent, but the guards are conservative and only affect the error path.
   
   Would appreciate your review. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #26899:
URL: https://github.com/apache/pulsar/pull/26899#discussion_r4301067007


##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java:
##########
@@ -5052,8 +5052,13 @@ protected boolean checkAndCompleteLedgerOpTask(int rc, LedgerHandle lh, Object c
     private void scheduleAddEntryTimeoutTask() {
         if (config.getAddEntryTimeoutSeconds() > 0) {
             long timeoutSec = config.getAddEntryTimeoutSeconds();
+            // Run the whole timeout check on the per-managed-ledger executor so that
+            // pendingAddEntries.peek(), the expiration decision, and the subsequent
+            // handleAddFailure call are serialized with OpAddEntry completion/recycling
+            // paths (which also run on this executor). This eliminates the race where
+            // an entry is recycled between peek() and handleAddFailure().
             this.timeoutTask = this.scheduledExecutor.scheduleAtFixedRate(
-                    this::checkAddTimeout, timeoutSec, timeoutSec, TimeUnit.SECONDS);
+                    () -> executor.execute(this::checkAddTimeout), timeoutSec, timeoutSec, TimeUnit.SECONDS);

Review Comment:
   Could we add deterministic regression coverage for this executor handoff? A small test capturing the scheduled callback and checking the timeout-check thread fails before this change and passes afterward, without racing real timers. Completion-first and timeout-first behavior should also be covered.



##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java:
##########
@@ -366,9 +366,18 @@ void handleAddFailure(final LedgerHandle lh, Integer rc) {
         // If we get a write error, we will try to create a new ledger and re-submit the pending writes. If the
         // ledger creation fails (persistent bk failure, another instance owning the ML, ...), then the writes will
         // be marked as failed.
+
+        // Defensive check: this OpAddEntry may have been recycled (ml set to null) between the time it was
+        // peeked from the pending queue (e.g. by checkAddTimeout) and when handleAddFailure is invoked, because
+        // another thread could have already completed and recycled it. In that case, there is nothing to do.
         ManagedLedgerImpl finalMl = this.ml;
+        if (finalMl == null) {
+            return;

Review Comment:
   With the full timeout check now serialized with pending-operation completion, the race described here is no longer reachable through this path. Could we remove this guard and its stale explanation unless there is another concrete caller requiring it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "Denovo1998 (via GitHub)" <gi...@apache.org>.
Denovo1998 commented on PR #26899:
URL: https://github.com/apache/pulsar/pull/26899#issuecomment-6397946308

   Thanks for addressing the race at the executor level. The approach looks correct to me.
   
   However, #27056 has already been merged and fixes the same root cause. It also adds exception handling around the periodic timeout check and a deterministic regression test, so I think this PR is now superseded.
   
   One difference worth noting is the `finalLh != null` guard. A null ledger handle is possible for ShadowManagedLedger, but simply skipping the close also skips the recovery path. If we want to support timeouts for those operations, I think that should be investigated separately with a focused test.
   
   Given that the main fix is already in master, I think we can close this PR in favor of #27056?
   @codelipenghui @merlimat @lhotari 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari closed pull request #26899: [fix][ml] Fix NPE in OpAddEntry.handleAddFailure when OpAddEntry is already recycled
URL: https://github.com/apache/pulsar/pull/26899


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org