You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "shyjsarah (via GitHub)" <gi...@apache.org> on 2026/09/18 10:00:58 UTC

[PR] fix(datafusion): preserve active runtime in blocking callbacks [paimon-rust]

shyjsarah opened a new pull request, #873:
URL: https://github.com/apache/paimon-rust/pull/873

   ### Purpose
   
   Linked issue: close #872
   
   Synchronous DataFusion callbacks may invoke `block_on_with_runtime` while already running on a Tokio multi-thread runtime. The helper currently moves the future to the process fallback runtime and waits for a new OS thread, which can stall work that depends on the active runtime.
   
   ### Brief change log
   
   - Use `block_in_place` with the active runtime handle for multi-thread runtimes.
   - Keep the existing fallback path for current-thread runtimes, where `block_in_place` is unsupported.
   - Add a regression test that verifies the blocking helper preserves the active runtime.
   
   ### Tests
   
   - `cargo test -p paimon-datafusion --lib runtime::tests -- --nocapture`
   - `cargo fmt --all --check`
   - `cargo clippy -p paimon-datafusion --lib --tests -- -D warnings`
   
   The full local lib suite completed 353 tests successfully. Seven table scan tests require shared test tables that were unavailable locally and failed with `TableNotExist`.
   
   ### API and Format
   
   No public API or storage format changes.
   
   ### Documentation
   
   No documentation changes are required.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(datafusion): preserve active runtime in blocking callbacks [paimon-rust]

Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah closed pull request #873: fix(datafusion): preserve active runtime in blocking callbacks
URL: https://github.com/apache/paimon-rust/pull/873


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(datafusion): preserve active runtime in blocking callbacks [paimon-rust]

Posted by "jerry-024 (via GitHub)" <gi...@apache.org>.
jerry-024 commented on code in PR #873:
URL: https://github.com/apache/paimon-rust/pull/873#discussion_r4055636205


##########
crates/integrations/datafusion/src/runtime.rs:
##########
@@ -103,14 +103,18 @@ where
     F: Future + Send + 'static,
     F::Output: Send + 'static,
 {
-    if Handle::try_current().is_ok() {
-        let handle = global_runtime().handle().clone();
-        std::thread::spawn(move || handle.block_on(future))
+    if let Ok(handle) = Handle::try_current() {
+        if handle.runtime_flavor() == RuntimeFlavor::MultiThread {

Review Comment:
   <!-- dlf-review -->
   **[Major] Checking only `RuntimeFlavor::MultiThread` can still panic here.**
   
   `LocalSet::run_until` may run on a multi-thread runtime while explicitly disabling `block_in_place`. `Handle::try_current()` still returns that multi-thread handle, so this branch panics instead of taking the previous fallback path.
   
   Please preserve a non-panicking fallback when in-place blocking is disallowed and add a `multi_thread` + `LocalSet::run_until` regression test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(datafusion): preserve active runtime in blocking callbacks [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #873:
URL: https://github.com/apache/paimon-rust/pull/873#issuecomment-5731756967

   Could we preserve `LocalSet` compatibility here?
   
   `RuntimeFlavor::MultiThread` does not guarantee that `block_in_place` is allowed. A `LocalSet` running on a multi-thread runtime still disallows in-place blocking.
   
   I reproduced this by calling `PaimonCatalogProvider::schema_names()` inside `LocalSet::run_until(...)`. The same test passes with the previous implementation, but this change causes it to panic at `block_in_place`:
   
   ```text
   can call blocking only when running on the multi-threaded runtime
   ```
   
   The single-worker starvation fix works as intended, but this introduces a regression for callers using `LocalSet`. Could we add a regression test and retain a compatible execution path for contexts that disallow in-place blocking, rather than relying solely on `runtime_flavor()`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(datafusion): preserve active runtime in blocking callbacks [paimon-rust]

Posted by "shyjsarah (via GitHub)" <gi...@apache.org>.
shyjsarah commented on code in PR #873:
URL: https://github.com/apache/paimon-rust/pull/873#discussion_r4055941025


##########
crates/integrations/datafusion/src/runtime.rs:
##########
@@ -103,14 +103,18 @@ where
     F: Future + Send + 'static,
     F::Output: Send + 'static,
 {
-    if Handle::try_current().is_ok() {
-        let handle = global_runtime().handle().clone();
-        std::thread::spawn(move || handle.block_on(future))
+    if let Ok(handle) = Handle::try_current() {
+        if handle.runtime_flavor() == RuntimeFlavor::MultiThread {

Review Comment:
   Thanks, this is correct. `runtime_flavor()` describes the runtime implementation, but not whether the current execution context permits `block_in_place`. A `LocalSet` changes the latter, so #873 introduces the regression you described and should not be merged as-is.
   
   I reworked the fix in #881. Instead of trying to detect when in-place blocking is safe, it removes the sync-to-async bridge from the affected catalog discovery callbacks: `schema_names`, `schema`, `table_names`, and `table_exist`. Metadata is loaded or refreshed at async boundaries, while those synchronous callbacks only read an in-memory snapshot.
   
   This avoids both the worker-starvation issue from #872 and the `LocalSet` panic. #881 also tests the stronger invariant that synchronous discovery callbacks do not access the remote catalog, and preserves the last good snapshot when refresh fails.
   
   I propose superseding and closing #873 in favor of #881. Thanks for catching this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org