[PR] fix(python): rebuild the fallback runtime after fork [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi opened a new pull request, #832:
URL: https://github.com/apache/paimon-rust/pull/832

   ### Purpose
   
   Fix native planning hanging in forked Python workers after the parent has used a Paimon catalog or scan. A Torch DataLoader with `num_workers=2` hits this in apache/paimon#9825: the child inherits the global Tokio runtime but none of its worker threads, so catalog I/O and planning never finish.
   
   ### Brief change log
   
   - Associate the lazily created fallback runtime with its process ID and publish its state through an atomic pointer.
   - After fork, initialize a runtime for the child without acquiring a potentially inherited initialization lock or dropping the parent's runtime state. Published states remain allocated because dropping the inherited runtime could wait for threads that no longer exist.
   - Preserve the existing behavior of using a Tokio runtime already entered on the calling thread.
   
   ### Tests
   
   - Two real Python fork regressions: create a new catalog or reuse the parent's catalog, plan and serialize splits twice in the child, and confirm the parent can still plan. Both timed out before the fix and pass with it; each has bounded waiting and child cleanup.
   - Concurrent fallback-runtime reuse and preservation of an entered runtime: 2 Rust unit tests passed.
   - Python binding fork/read/table/write/catalog tests: 105 passed.
   - PyPaimon validation with the rebuilt wheel: the explicit-fork DataLoader reproduction completes; 25 contiguous-window dataset tests and 357 planner/reader regressions pass with native planning.
   - `cargo clippy --offline -p paimon-datafusion --all-targets -- -D warnings`, `cargo fmt --all -- --check`, Python test Flake8, and `git diff --check` passed.
   
   ### API and Format
   
   No public API or storage-format changes. This fixes the fallback runtime used by catalog and planning calls; explicitly entered Tokio runtimes retain their existing behavior.
   
   ### Documentation
   
   Updated runtime comments to explain process ownership and why inherited state is retained.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(python): rebuild the fallback runtime after fork [paimon-rust]

["leaves12138 (via GitHub)" <gi...@apache.org>]

leaves12138 commented on PR #832:
URL: https://github.com/apache/paimon-rust/pull/832#issuecomment-5673630743

   Fixed the failing CI dependency-policy check in `0891c50d91768af2261e015582a2317ca73553a2`.
   
   `cargo-deny` rejected rustls 0.23.42 for RUSTSEC-2026-0285. Upgraded to rustls 0.23.45 and its required AWS-LC/webpki dependencies, and regenerated the nine dependency reports. No advisory suppression or fork-runtime changes.
   
   Local validation passed: advisories/licenses against a freshly fetched RustSec database, dependency-report verification, all 21 generated release legal files, 12 release-tool tests, compilation of rustls 0.23.45, formatting, and staged-diff checks. The legal files were regenerated but needed no content changes.
   
   The fix is appended to the existing PR branch without rewriting history. CI run `34920232610` has started and is still in progress.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(python): rebuild the fallback runtime after fork [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #832:
URL: https://github.com/apache/paimon-rust/pull/832


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org