You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "YordanPavlov (via GitHub)" <gi...@apache.org> on 2026/08/17 12:48:46 UTC
[PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
YordanPavlov opened a new pull request, #28985:
URL: https://github.com/apache/flink/pull/28985
## What is the purpose of the change
`ForStGeneralMultiGetOperation.process()` allocates a `new ReadOptions()` (a JNI object owning native memory) for every executed async read batch, and no code path closes it. On a ForSt + async State V2 job the resulting native memory leak is proportional to read volume. On a production workload we measured TaskManager RSS growth of 2.2–3.9 GiB/h, leading to container OOM kills every ~10–14 hours. jemalloc allocation profiling showed `Java_org_forstdb_ReadOptions_newReadOptions` dominating the live-allocation diff.
This PR closes the per-batch `ReadOptions` with try-with-resources. This is safe because:
- `db.multiGetAsList(readOptions, ...)` is synchronous within the executor lambda and returns `byte[]` copies — nothing referencing the `ReadOptions` outlives the batch;
- all early returns and exception paths are covered by try-with-resources;
- the sync ForSt backend already treats `ReadOptions` as a managed, closeable resource (`ForStResourceContainer#getReadOptions`, registered in `handlesToClose`).
## Brief change log
- Wrap the per-batch `ReadOptions` in `ForStGeneralMultiGetOperation#process` in try-with-resources so it is closed after the batch completes.
## Verifying this change
This change is already covered by existing tests: functional behavior of `process()` for value/list/map states is exercised by `ForStGeneralMultiGetOperationTest` (full `flink-statebackend-forst` suite green locally: 979 tests, 0 failures). No new test asserts the closure itself — the `ReadOptions` is local to the executor lambda, and we preferred not to add a test-only seam to the production code; happy to add one if preferred.
Additionally, verified in production (Flink 2.3.0, ForSt async state backend, with this class patched): TaskManager RSS growth dropped from 2.2–3.9 GiB/h to ~20 MiB/h (flat over 19+ hours) with unchanged job output.
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes — async state read path; the change adds one native object close per read batch, negligible relative to the multiGet itself
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
Posted by "YordanPavlov (via GitHub)" <gi...@apache.org>.
YordanPavlov commented on PR #28985:
URL: https://github.com/apache/flink/pull/28985#issuecomment-5316339262
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
Posted by "YordanPavlov (via GitHub)" <gi...@apache.org>.
YordanPavlov commented on PR #28985:
URL: https://github.com/apache/flink/pull/28985#issuecomment-5795502853
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly merged PR #28985:
URL: https://github.com/apache/flink/pull/28985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
Posted by "YordanPavlov (via GitHub)" <gi...@apache.org>.
YordanPavlov commented on PR #28985:
URL: https://github.com/apache/flink/pull/28985#issuecomment-5801186862
Hi @Zakelly gentle ping on this one. It's rebased on current master, Azure CI is green, and it has an approval from @davidradl. It fixes a native memory leak (a ReadOptions is leaked per multiGet batch) that we hit in production with the ForSt async state backend; measurements are in FLINK-40327. Could you take a look, or merge if it looks good? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28985:
URL: https://github.com/apache/flink/pull/28985#issuecomment-5316394906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9a5fd8428222f3cfc4675efb84261c3582cdbb7d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9a5fd8428222f3cfc4675efb84261c3582cdbb7d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9a5fd8428222f3cfc4675efb84261c3582cdbb7d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40327][state/forst] Close ReadOptions in ForStGeneralMultiGetOperation to fix native memory leak [flink]
Posted by "YordanPavlov (via GitHub)" <gi...@apache.org>.
YordanPavlov commented on PR #28985:
URL: https://github.com/apache/flink/pull/28985#issuecomment-5324520347
Full Flink CI passed on [my fork](https://github.com/YordanPavlov/flink/actions/runs/32033914116). The two E2E job failures in the first attempt were runner-infra issues — minikube startup and download rate limits — green on rerun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org