You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/04/15 05:22:56 UTC

[PR] [codex] Clear realtime inverted index bitmaps on close [pinot]

xiangfu0 opened a new pull request, #18204:
URL: https://github.com/apache/pinot/pull/18204

   ## Summary
   - make `RealtimeInvertedIndex.close()` actively clear and trim in-memory bitmap state
   - add a regression test that verifies `close()` empties the bitmaps and returns empty results afterward
   
   ## Root cause
   `MutableSegmentImpl.destroy()` already walks mutable indexes and calls `close()`, but `RealtimeInvertedIndex.close()` was a no-op. After a consuming segment was persisted and replaced, any lingering wrapper or data-source reference could still retain the in-memory posting lists because the index never dropped its bitmap references.
   
   ## Reproduction
   1. Create a consuming segment with a dictionary-encoded column and realtime inverted index enabled.
   2. Index rows so the mutable inverted index allocates posting lists.
   3. Persist and replace the segment, then let the mutable segment destroy path run.
   4. Inspect the heap and follow a stale `indexContainer` or `DataSource` wrapper: the `RealtimeInvertedIndex` still retains its `_bitmaps` list because `close()` does not release anything.
   
   ## Impact
   This makes realtime inverted-index cleanup deterministic once the mutable segment destroy path runs, reducing retained heap after realtime segment replacement.
   
   ## Validation
   - `./mvnw spotless:apply -pl pinot-segment-local`
   - `./mvnw -pl pinot-segment-local -Dtest=RealtimeInvertedIndexReaderTest test`
   - `./mvnw checkstyle:check -pl pinot-segment-local`
   - `./mvnw license:format -pl pinot-segment-local`
   - `./mvnw license:check -pl pinot-segment-local`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [bugfix] Clear realtime inverted index bitmaps on close [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18204:
URL: https://github.com/apache/pinot/pull/18204#discussion_r3084206742


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndex.java:
##########
@@ -82,5 +82,11 @@ public MutableRoaringBitmap getDocIds(int dictId) {
 
   @Override
   public void close() {
+    try {
+      _writeLock.lock();
+      _bitmaps.clear();
+    } finally {

Review Comment:
   `close()` currently clears `_bitmaps`, which drops bitmap references but keeps the backing `ArrayList` capacity. If this index had a large cardinality, the retained `Object[]` can still hold significant heap even after close. Consider trimming the list (e.g., `trimToSize()` on the concrete `ArrayList`) so `close()` actually releases both the bitmaps and the list storage, aligning with the PR goal to clear *and* trim in-memory state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [bugfix] Clear realtime inverted index bitmaps on close [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18204:
URL: https://github.com/apache/pinot/pull/18204#issuecomment-4249964082

   > This is not really the contract of `close()`. The real issue is the dangling reference. Do you see where the dangling reference is from? We should find that as the root cause. If you want to ensure all the references to the index are removed when destroying the `MutableSegmentImpl`, you can clear the `_indexContainerMap` during the destroy.
   
   correct, this is more on the defensive side, even if the caller side hold the MutableSegment or Index reference, the leakage for memory is minimal. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [bugfix] Clear realtime inverted index bitmaps on close [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18204:
URL: https://github.com/apache/pinot/pull/18204


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [bugfix] Clear realtime inverted index bitmaps on close [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18204:
URL: https://github.com/apache/pinot/pull/18204#issuecomment-4249863409

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18204?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 55.28%. Comparing base ([`9ff14da`](https://app.codecov.io/gh/apache/pinot/commit/9ff14da6dbcb8dddbdcb7edc04b2db7f93fb70de?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6bd8541`](https://app.codecov.io/gh/apache/pinot/commit/6bd8541305ff5de82147686efc4ae21a7f5d9ddf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   > :exclamation:  There is a different number of reports uploaded between BASE (9ff14da) and HEAD (6bd8541). Click for more details.
   > 
   > <details><summary>HEAD has 32 uploads less than BASE</summary>
   >
   >| Flag | BASE (9ff14da) | HEAD (6bd8541) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests1|2|1|
   >|unittests|3|1|
   >|temurin|9|1|
   >|java-11|4|0|
   >|unittests2|1|0|
   >|integration|6|0|
   >|integration2|2|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18204      +/-   ##
   ============================================
   - Coverage     63.34%   55.28%   -8.07%     
   + Complexity     1627      837     -790     
   ============================================
     Files          3238     2543     -695     
     Lines        197003   145958   -51045     
     Branches      30464    23469    -6995     
   ============================================
   - Hits         124801    80690   -44111     
   + Misses        62202    58314    -3888     
   + Partials      10000     6954    -3046     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.28% <100.00%> (-8.05%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.28% <100.00%> (-8.07%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.28% <100.00%> (-8.07%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.28% <100.00%> (-0.01%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18204/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18204?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [bugfix] Clear realtime inverted index bitmaps on close [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #18204:
URL: https://github.com/apache/pinot/pull/18204#discussion_r3084270949


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndex.java:
##########
@@ -82,5 +82,11 @@ public MutableRoaringBitmap getDocIds(int dictId) {
 
   @Override
   public void close() {
+    try {
+      _writeLock.lock();
+      _bitmaps.clear();
+    } finally {

Review Comment:
   +1



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndex.java:
##########
@@ -82,5 +82,11 @@ public MutableRoaringBitmap getDocIds(int dictId) {
 
   @Override
   public void close() {
+    try {
+      _writeLock.lock();
+      _bitmaps.clear();
+    } finally {

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [bugfix] Clear realtime inverted index bitmaps on close [pinot]

Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on code in PR #18204:
URL: https://github.com/apache/pinot/pull/18204#discussion_r3084271979


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeInvertedIndex.java:
##########
@@ -82,5 +82,11 @@ public MutableRoaringBitmap getDocIds(int dictId) {
 
   @Override
   public void close() {
+    try {
+      _writeLock.lock();
+      _bitmaps.clear();
+    } finally {

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org