[jira] [Created] (FLINK-40327) ForSt async multiGet leaks a native ReadOptions per read batch

["Yordan Pavlov (Jira)" <ji...@apache.org>]

Yordan Pavlov created FLINK-40327:
-------------------------------------

             Summary: ForSt async multiGet leaks a native ReadOptions per read batch
                 Key: FLINK-40327
                 URL: https://issues.apache.org/jira/browse/FLINK-40327
             Project: Flink
          Issue Type: Bug
          Components: Runtime / State Backends
    Affects Versions: 2.3.0, 2.2.0, 2.1.0, 2.0.0
            Reporter: Yordan Pavlov


h4. Problem observed:

The problem I observed unlimited memory increase of a Flink Kubernetes pod. Observed in production (Flink 2.3.0, ForSt disaggregated + async State V2, high read volume ~25k rec/s). TaskManager RSS grows ~2.6 GiB/h while heap/managed/network stay flat.  OOMKill after ~10–14 h.
h4. Investigation:

 jemalloc allocation profiling attributes the live-allocation 
growth to Java_org_forstdb_ReadOptions_newReadOptions (+530 MiB live in a 12-minute steady-state window).

 

ForStGeneralMultiGetOperation.process() creates a new ReadOptions() inside the per-split executor lambda and no code path closes it — not the early returns, not the exception paths, not normal completion. ReadOptions is a JNI wrapper owning a native 
rocksdb::ReadOptions; without close() the native allocation is never freed, so memory leaks proportionally to the number of executed async read batches, outside all JVM/Flink memory accounting. 

All other ReadOptions/WriteOptions constructions in the module are lifecycle-managed (handlesToClose, toClose, closeQuietly in close()); this is the only unmanaged one.
h4. Fix:

Open the ReadOptions with try-with-resources spanning the lambda body. Safe because multiGetAsList is synchronous and returns byte[] copies, so nothing referencing the ReadOptions outlives the batch. We run this patched in production. Happy to open a PR.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)