[jira] [Commented] (FLINK-40327) ForSt async multiGet leaks a native ReadOptions per read batch

["Yordan Pavlov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124846#comment-18124846 ] 

Yordan Pavlov commented on FLINK-40327:
---------------------------------------

[~zakelly] Kind reminder, still unmerged.

> ForSt async multiGet leaks a native ReadOptions per read batch
> --------------------------------------------------------------
>
>                 Key: FLINK-40327
>                 URL: https://issues.apache.org/jira/browse/FLINK-40327
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / State Backends
>    Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
>            Reporter: Yordan Pavlov
>            Assignee: Yordan Pavlov
>            Priority: Critical
>              Labels: pull-request-available
>         Attachments: after.png, before.png
>
>
> h4. Problem observed:
> The problem I observed unlimited memory increase of a Flink Kubernetes pod. Observed in production (Flink 2.3.0, ForSt disaggregated + async State V2, high read volume ~25k rec/s). TaskManager RSS grows ~2.6 GiB/h while heap/managed/network stay flat.  OOMKill after ~10–14 h.
> h4. Investigation:
>  jemalloc allocation profiling attributes the live-allocation 
> growth to Java_org_forstdb_ReadOptions_newReadOptions (+530 MiB live in a 12-minute steady-state window).
>  
> ForStGeneralMultiGetOperation.process() creates a new ReadOptions() inside the per-split executor lambda and no code path closes it — not the early returns, not the exception paths, not normal completion. ReadOptions is a JNI wrapper owning a native 
> rocksdb::ReadOptions; without close() the native allocation is never freed, so memory leaks proportionally to the number of executed async read batches, outside all JVM/Flink memory accounting. 
> All other ReadOptions/WriteOptions constructions in the module are lifecycle-managed (handlesToClose, toClose, closeQuietly in close()); this is the only unmanaged one.
> h4. Fix:
> Open the ReadOptions with try-with-resources spanning the lambda body. Safe because multiGetAsList is synchronous and returns byte[] copies, so nothing referencing the ReadOptions outlives the batch. We run this patched in production. Happy to open a PR.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)