[I] [Bug] PythonActionExecutor leaks Pemja PyObject references across task restarts [flink-agents]

["joeyutong (via GitHub)" <gi...@apache.org>]

joeyutong opened a new issue, #942:
URL: https://github.com/apache/flink-agents/issues/942

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
   
   ### Description
   
   `PythonActionExecutor` owns two Pemja `PyObject` handles: the Python async thread pool and the Python runner context. Closing the executor invokes their Python cleanup functions, but it does not call `PyObject.close()`; the runner-context field is only set to `null`.
   
   `PyObject.close()` performs Pemja's native `decRef`. Without it, repeated task restarts in the same TaskManager JVM can leave JNI global references to old runner contexts. A heap dump from a repeated-failover reproduction showed the following retention chain:
   
   ```text
   JNI Global Reference
     -> PythonRunnerContextImpl
     -> mailboxThreadChecker
     -> ActionExecutionOperator
     -> Task / HeapKeyedStateBackend
   ```
   
   In a local 20-restart reproduction with four Python action subtasks, the post-Full-GC heap increased by 29.2 MiB on the current close path. Explicitly closing the owned `PyObject` handles reduced the increase to 1.7 MiB and removed the old task graphs after the job terminated.
   
   The root cause is that logical Python cleanup and Pemja handle release are separate lifecycle operations, but the executor currently performs only the first one.
   
   ### How to reproduce
   
   1. Start a Flink Agents job with one Python action vertex at parallelism 4 on a TaskManager that remains alive during regional failover.
   2. Repeatedly fail and restart the action tasks in the same TaskManager JVM.
   3. Force a Full GC and sample the heap after each restart.
   4. Observe a monotonically increasing retained heap floor.
   5. Capture an HPROF and inspect JNI global roots; old `PythonRunnerContextImpl`, operator, task, and keyed-state-backend instances remain reachable.
   
   ### Version and environment
   
   - Flink Agents: release-0.3 (the same close path is present on current `main`)
   - Flink: 2.2 standalone session cluster
   - Java: 11
   - Python: 3.11
   - One TaskManager with four slots; regional failover keeps the JVM alive
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] PythonActionExecutor leaks Pemja PyObject references across task restarts [flink-agents]

["wenjin272 (via GitHub)" <gi...@apache.org>]

wenjin272 closed issue #942: [Bug] PythonActionExecutor leaks Pemja PyObject references across task restarts
URL: https://github.com/apache/flink-agents/issues/942


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org