[PR] [runtime][python] Manage Pemja references across resource bridges [flink-agents]

["joeyutong (via GitHub)" <gi...@apache.org>]

joeyutong opened a new pull request, #1058:
URL: https://github.com/apache/flink-agents/pull/1058

   Linked issue: #1057
   
   ### Purpose of change
   
   Pemja values crossing the Java/Python bridge may carry native Python references. The current resource paths do not share one ownership rule, so temporary values, nested results, long-lived resources, and partial initialization failures can retain references differently.
   
   This change introduces `PythonObjectScope` as the common ownership abstraction:
   
   - an operation scope recursively owns `PyObject` handles reachable through maps, iterables, and object arrays, deduplicates them by identity, and releases them in reverse order;
   - `release(...)` makes ownership transfer to a long-lived wrapper explicit;
   - values that must outlive a bridge call are materialized into detached Java primitives, lists, and maps before the raw references are closed;
   - resource creation and MCP discovery use transactional rollback, closing both Python lifecycle state and native handles before any failed resource can remain registered.
   
   The abstraction is applied consistently to chat and embedding models, vector stores, MCP servers/tools/prompts, Mem0 call-scoped values, Python-backed `FunctionTool` calls, and Java/Python resource-cache initialization.
   
   This complements #944, which owns executor-lifetime roots and the shutdown close chain, and #1049, which owns per-Action execution values. This PR covers the remaining operation-level resource bridges and initialization rollback paths.
   
   ### Tests
   
   - `mvn -pl runtime -am -DskipITs test`
     - plan: 294 tests, 1 skipped
     - runtime: 721 tests, 0 failures/errors
   - Final rollback-path changes: focused `PythonResourceProviderTest,ResourceCacheTest`
   - `bash tools/ut.sh -p`: 935 passed, 13 skipped, 165 deselected
   - `./tools/lint.sh -c`
   - `bash tools/check-license.sh`
   - `git diff --check`
   
   Local Flink 2.2 A/B validation ran six jobs, each with a fresh TaskManager, one probe subtask, Pemja 0.5.7, and 200 measured operations:
   
   - chat: fixed `0` live bridge values; leak variant retained `600` messages, `200` tools, `200 MiB`, and increased RSS by `160.3 MiB`;
   - vector: fixed `0` live bridge values; leak variant retained `600` documents, `200` normalized handles, about `100 MiB`, and increased RSS by `99.1 MiB`;
   - resource recreation: fixed `0` live resources with `200` logical closes; leak variant retained `200` resources and `50 MiB` with no close calls.
   
   The deterministic live-reference, close-count, and reachable-payload measurements are the primary evidence; RSS is supporting evidence because JVM/CPython startup and native allocator behavior affect short runs.
   
   ### API
   
   This adds an `@Internal` ownership utility and a default materialization hook to the existing Python resource adapter. It does not change user-facing resource contracts, and the default method is binary compatible.
   
   ### Documentation
   
   - [ ] `doc-needed`
   - [x] `doc-not-needed`
   - [ ] `doc-included`
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   - [x] Yes
   - [ ] No
   
   Generated-by: Codex (GPT-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime][python] Manage Pemja references across actions and resources [flink-agents]

["wenjin272 (via GitHub)" <gi...@apache.org>]

wenjin272 merged PR #1058:
URL: https://github.com/apache/flink-agents/pull/1058


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org