You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "Caideyipi (via GitHub)" <gi...@apache.org> on 2026/07/22 04:16:29 UTC

[PR] Fix query memory reservation deregistration race [iotdb]

Caideyipi opened a new pull request, #18272:
URL: https://github.com/apache/iotdb/pull/18272

   ## Description
   
   A table standalone integration test exposed a race in `MemoryPool`:
   
   ```
   SharedTsBlockQueue.remove()
     -> MemoryPool.free()
     -> NullPointerException because the fragment reservation map no longer exists
   ```
   
   `deRegisterFragmentInstanceFromQueryMemoryMap` checked whether a fragment's reservations were zero before taking the map-removal lock. A concurrent reservation could update the fragment map after that zero snapshot but before deregistration removed it. The memory block was then charged while its reservation map was detached, so the downstream consumer failed when freeing the buffered TsBlock.
   
   This is independent of the Pipe changes in #18266. That PR's head was based directly on `b8ff9eeaea0d2e5afd79446e8bb4cf654c8c8038` (`origin/master` at the time), and its diff only touched `PipeMemoryManager` and its test.
   
   ## Fix
   
   - Serialize register, reserve, rollback, free, and deregister operations on the affected fragment reservation map.
   - Revalidate the query/fragment map identities after acquiring the fragment lock so no operation updates a detached map.
   - Keep reservations for different fragments concurrent.
   
   ## Tests
   
   Added a deterministic concurrency test that pauses `Map.merge` while deregistration runs. With the `origin/master` implementation, it fails because the reservation map is removed (`expected 256 but was 0`). With this change:
   
   - `MemoryPoolTest`: 23 tests run, 0 failures
   - `mvn spotless:apply -pl iotdb-core/datanode`: success
   - Checkstyle reported 0 violations before compilation was blocked by unrelated local cross-module/generated-source state
   
   The local full reactor could not complete because the Windows Thrift 0.14 generator emitted `javax.annotation.Generated` while this branch uses Jakarta annotations; the changed production and test sources were therefore compiled directly with JDK 17 against the Maven-resolved datanode test classpath for the targeted JUnit run.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added tests that fail without the fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix query memory reservation deregistration race [iotdb]

Posted by "shuwenwei (via GitHub)" <gi...@apache.org>.
shuwenwei commented on PR #18272:
URL: https://github.com/apache/iotdb/pull/18272#issuecomment-5043783788

   Could you clarify which production interleaving this test is intended to reproduce? The failure log only proves that `SharedTsBlockQueue.remove()` called `MemoryPool.free()` after the FI/query reservation map had already been removed; it does not show that reservation was concurrent with deregistration. In the normal FI terminal path, `clearShuffleSinkHandle()`, `driver.close()`, and `releaseResourceWhenAllDriversAreClosed()` all happen before deregistration, while handle cleanup also cancels pending reservation futures using the same future lock used by `MemoryPool.free()`. The new test directly races `tryReserveForTest()` with deregistration and therefore bypasses these lifecycle guarantees. There is also an uncovered reverse interleaving in the proposed fix: if deregistration acquires the fragment lock first and removes the map, the waiting operation fails identity validation and `continue`s, but the next iteration dereferences a null `queryRelatedMemory`, resulting in another 
 NPE. Could we add a deterministic test where deregistration wins first, and either document a concrete production caller that can still reserve after FI cleanup or handle the missing mapping explicitly?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Harden query memory reservation deregistration ordering [iotdb]

Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on PR #18272:
URL: https://github.com/apache/iotdb/pull/18272#issuecomment-5044893608

   Good catch. I audited the production lifecycle and agree that the original description overstated what the failure log proves. In the normal FI terminal path, sink/driver/source cleanup happens before deregistration, and pending reservations are cancelled or completed under the future lock. I did not find a concrete normal-path caller that starts a new reservation after FI cleanup.
   
   I addressed the actionable reverse ordering in 7af40afe6a1:
   
   - added explicit missing query/fragment-map handling in `free`, `tryReserve`, and rollback, so a deregistration-first interleaving fails with a descriptive `IllegalArgumentException` instead of retrying into an NPE;
   - added a deterministic test that lets the reservation read the old fragment map, lets deregistration remove it first, and then resumes the reservation; it verifies that the operation fails explicitly and global reserved bytes remain zero;
   - kept the existing deterministic reservation-first test as defensive coverage of the `MemoryPool` API invariant.
   
   I also revised the PR title/description to distinguish this defensive hardening from the still-unproven production interleaving.
   
   Verification passed: `MemoryPoolTest` 24/24, Checkstyle, Spotless, and DataNode `test-compile` with the zh locale.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Harden query memory reservation deregistration ordering [iotdb]

Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi closed pull request #18272: Harden query memory reservation deregistration ordering
URL: https://github.com/apache/iotdb/pull/18272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org