[I] Off-heap mmap buffer leaked in FilePerIndexDirectory.removeIndex() [pinot]

["dkranchii (via GitHub)" <gi...@apache.org>]

dkranchii opened a new issue, #18404:
URL: https://github.com/apache/pinot/issues/18404

   In `FilePerIndexDirectory`, `removeIndex()` drops the entry from the
   buffer cache without ever closing the underlying `PinotDataBuffer`.
   The class itself even has a `// TODO` flagging the leak, so opening
   an issue to track it.
   
   ### Where
   `pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/FilePerIndexDirectory.java:100-111`
   ```java
   @Override
   public void removeIndex(String columnName, IndexType<?, ?, ?> indexType) {
     // TODO: this leaks the removed data buffer (it's not going to be freed in close() method)
     _indexBuffers.remove(new IndexKey(columnName, indexType));
     if (indexType == StandardIndexes.text()) {
       TextIndexUtils.cleanupTextIndex(_segmentDirectory, columnName);
     } else if (indexType == StandardIndexes.vector()) {
       VectorIndexUtils.cleanupVectorIndex(_segmentDirectory, columnName);
     } else {
       getFilesFor(columnName, indexType).forEach(FileUtils::deleteQuietly);
     }
   }
   ```
   close() later iterates _indexBuffers.values() and closes whatever is still in the map — but the entry we just removed is no longer in there, so it's never closed.
   
   ### Why this is a bug
   `PinotDataBuffer` is usually mmap-backed for offline segments. Removing
   it from the cache without calling `close()` means the mmap region
   stays in the JVM's address space until the process exits (or the
   `Cleaner` happens to fire not something to rely on for deterministic
   cleanup). The on-disk file gets deleted on line 109, but the kernel
   mapping leaks.
   
   `removeIndex()` is called whenever Pinot decides an index isn't
   needed anymore segment reloads after index config changes, minion
   segment conversions, the various `*IndexHandler` drop/replace flows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [I] Off-heap mmap buffer leaked in FilePerIndexDirectory.removeIndex() [pinot]

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang closed issue #18404: Off-heap mmap buffer leaked in FilePerIndexDirectory.removeIndex()
URL: https://github.com/apache/pinot/issues/18404


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org