You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org> on 2026/04/24 11:19:05 UTC

[PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

rsrkpatwari1234 opened a new pull request, #18320:
URL: https://github.com/apache/pinot/pull/18320

   ## Problem
   `removeIndex()` dropped entries from` _indexBuffers` but did not close the associated `PinotDataBuffer`. Those buffers are no longer reached by `close()`, so mmap (and other) mappings could stay alive until GC/finalize behavior, which is incorrect and can leak file descriptors and native memory.
   
   ## Change
   On removal, take the buffer out of the map, call `close()` on it, then run the existing cleanup (text index, vector index, or per-file delete). I/O errors while closing are surfaced as `UncheckedIOException`.
   
   ## Tests
   - Simplify `testRemoveIndex` so it relies on `removeIndex()` to release the forward buffer.
   - Add `testRemoveIndexClosesReadBuffer` to cover the getBuffer (read) path with the class-level `PinotBuffersAfterMethodCheckRule` leak check.
   
   ## Scope note
   Text- and vector-specific cleanup still runs after the buffer is released, so we do not keep stale mapped buffers over directory deletes.
   
   Fixes #18218


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

Posted by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org>.
rsrkpatwari1234 commented on PR #18320:
URL: https://github.com/apache/pinot/pull/18320#issuecomment-4349963001

   @xiangfu0 Requesting your review on this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18320:
URL: https://github.com/apache/pinot/pull/18320#discussion_r3150988602


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/FilePerIndexDirectory.java:
##########
@@ -99,8 +100,16 @@ public void close()
 
   @Override
   public void removeIndex(String columnName, IndexType<?, ?, ?> indexType) {
-    // TODO: this leaks the removed data buffer (it's not going to be freed in close() method)
-    _indexBuffers.remove(new IndexKey(columnName, indexType));
+    IndexKey key = new IndexKey(columnName, indexType);
+    PinotDataBuffer removed = _indexBuffers.remove(key);
+    if (removed != null) {
+      try {
+        removed.close();
+      } catch (IOException e) {
+        throw new UncheckedIOException(

Review Comment:
   shall we just log error and continue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed pull request #18320: Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks
URL: https://github.com/apache/pinot/pull/18320


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

Posted by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org>.
rsrkpatwari1234 commented on code in PR #18320:
URL: https://github.com/apache/pinot/pull/18320#discussion_r3151844105


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/FilePerIndexDirectory.java:
##########
@@ -99,8 +100,16 @@ public void close()
 
   @Override
   public void removeIndex(String columnName, IndexType<?, ?, ?> indexType) {
-    // TODO: this leaks the removed data buffer (it's not going to be freed in close() method)
-    _indexBuffers.remove(new IndexKey(columnName, indexType));
+    IndexKey key = new IndexKey(columnName, indexType);
+    PinotDataBuffer removed = _indexBuffers.remove(key);
+    if (removed != null) {
+      try {
+        removed.close();
+      } catch (IOException e) {
+        throw new UncheckedIOException(

Review Comment:
   Made the changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #18320:
URL: https://github.com/apache/pinot/pull/18320#issuecomment-4736505367

   Took #18405 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Close cached PinotDataBuffer in FilePerIndexDirectory.removeIndex() to stop mmap/heap buffer leaks [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18320:
URL: https://github.com/apache/pinot/pull/18320#issuecomment-4313003889

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18320?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `8 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 55.55%. Comparing base ([`ab9ac39`](https://app.codecov.io/gh/apache/pinot/commit/ab9ac391fa16b731eb2e1498bb13cdaaf65e92da?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f29d160`](https://app.codecov.io/gh/apache/pinot/commit/f29d160805e0b5edff312297821877a4b1b89957?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18320?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...ent/local/segment/store/FilePerIndexDirectory.java](https://app.codecov.io/gh/apache/pinot/pull/18320?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fstore%2FFilePerIndexDirectory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL0ZpbGVQZXJJbmRleERpcmVjdG9yeS5qYXZh) | 0.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18320?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (ab9ac39) and HEAD (f29d160). Click for more details.
   > 
   > <details><summary>HEAD has 28 uploads less than BASE</summary>
   >
   >| Flag | BASE (ab9ac39) | HEAD (f29d160) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests1|2|1|
   >|unittests|4|1|
   >|temurin|8|1|
   >|unittests2|2|0|
   >|java-11|3|0|
   >|integration|4|0|
   >|integration1|1|0|
   >|integration2|2|0|
   >|custom-integration1|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18320      +/-   ##
   ============================================
   - Coverage     63.61%   55.55%   -8.07%     
   + Complexity     1659      837     -822     
   ============================================
     Files          3246     2536     -710     
     Lines        197514   146198   -51316     
     Branches      30578    23542    -7036     
   ============================================
   - Hits         125656    81216   -44440     
   + Misses        61813    57981    -3832     
   + Partials      10045     7001    -3044     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.55% <0.00%> (-8.04%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.55% <0.00%> (-8.07%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.55% <0.00%> (-8.07%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.55% <0.00%> (-0.04%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18320/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18320?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org