You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/07/31 12:50:09 UTC

[PR] Feat/ffi query planner main [datafusion]

timsaucer opened a new pull request, #24028:
URL: https://github.com/apache/datafusion/pull/24028

   ## Which issue does this PR close?
   
   - Unblocks https://github.com/apache/datafusion-python/issues/1612
   
   ## Rationale for this change
   
   This is the last in a series of PRs that would enable FFI `Session` to support a `QueryPlanner`. The prior work was in
   
   - https://github.com/apache/datafusion/pull/23649
   - https://github.com/apache/datafusion/pull/23703
   - https://github.com/apache/datafusion/pull/23842
   
   With those changes in place we now have the dependencies correct that we can expose a `FFI_QueryPlanner` on a `FFI_Session`. With this we can enable foreign libraries such as `datafusion-distributed` and `ballista` to provide a query planner in Python and connect it directly to a `datafusion-python`'s `SessionContext`.
   
   ## What changes are included in this PR?
   
   Addition only. Adds these functions to `FFI_Session` and their supporting structures:
   
   - `query_planner`
   - `optimize`
   - `physical_optimizers()`
   
   ## Are these changes tested?
   
   Unit and integration tests are provided.
   
   ## Are there any user-facing changes?
   
   This is addition, but it does break the FFI ABI, which is already evolving in DF55.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat/ffi query planner main [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3690530861


##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -297,6 +297,7 @@ async fn get_file_decryption_properties(
 }
 
 #[cfg(not(feature = "parquet_encryption"))]
+#[expect(clippy::unused_async)]

Review Comment:
   Drive by fix to when you run clippy and don't have the `parquet_encryption` feature enabled.



##########
datafusion/datasource-parquet/src/opener/encryption.rs:
##########
@@ -76,6 +76,7 @@ impl EncryptionContext {
 
 #[cfg(not(feature = "parquet_encryption"))]
 #[expect(dead_code)]
+#[expect(clippy::unused_async)]

Review Comment:
   Drive by fix to when you run clippy and don't have the `parquet_encryption` feature enabled.



##########
datafusion/datasource-parquet/src/sink.rs:
##########
@@ -239,6 +239,7 @@ async fn set_writer_encryption_properties(
 }
 
 #[cfg(not(feature = "parquet_encryption"))]
+#[expect(clippy::unused_async)]

Review Comment:
   Drive by fix to when you run clippy and don't have the `parquet_encryption` feature enabled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3721507586


##########
datafusion/ffi/tests/ffi_query_planner.rs:
##########
@@ -0,0 +1,223 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod utils;
+
+#[cfg(feature = "integration-tests")]
+mod tests {
+    use std::sync::{Arc, OnceLock, Weak};
+
+    use arrow::datatypes::SchemaRef;
+    use datafusion::execution::SessionStateBuilder;
+    use datafusion::prelude::SessionContext;
+    use datafusion_catalog::TableProvider;
+    use datafusion_common::{
+        DataFusionError, Result, TableReference, exec_err, not_impl_err,
+    };
+    use datafusion_execution::{TaskContext, TaskContextProvider};
+    use datafusion_expr::LogicalPlan;
+    use datafusion_expr::logical_plan::Extension;
+    use datafusion_ffi::execution::FFI_TaskContextProvider;
+    use datafusion_ffi::execution_plan::ForeignExecutionPlan;
+    use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+    use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+    use datafusion_ffi::query_planner::ForeignQueryPlanner;
+    use datafusion_ffi::table_provider::ForeignTableProvider;
+    use datafusion_ffi::tests::{
+        create_test_schema,
+        utils::{get_module, get_module_copy},
+    };
+    use datafusion_physical_plan::ExecutionPlan;
+    use datafusion_physical_plan::empty::EmptyExec;
+    use datafusion_physical_plan::union::UnionExec;
+    use datafusion_proto::logical_plan::LogicalExtensionCodec;
+    use datafusion_proto::physical_plan::{
+        DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+        PhysicalProtoConverterExtension,
+    };
+    use datafusion_session::QueryPlanner;
+
+    #[tokio::test]
+    async fn test_ffi_query_planner() -> Result<(), DataFusionError> {
+        let module = get_module()?;
+        let (ctx, logical_codec) = crate::utils::ctx_and_codec();
+        let task_ctx_provider = Arc::clone(&ctx) as Arc<dyn TaskContextProvider>;
+        let task_ctx_provider = FFI_TaskContextProvider::from(&task_ctx_provider);
+        let physical_codec = FFI_PhysicalExtensionCodec::new(
+            Arc::new(DefaultPhysicalExtensionCodec {}),
+            None,
+            task_ctx_provider,
+        );
+
+        let ffi_planner = (module.create_query_planner)(logical_codec, physical_codec);
+        let planner: Arc<dyn QueryPlanner + Send + Sync> = (&ffi_planner).into();
+
+        let any_ref: &dyn std::any::Any = planner.as_ref();
+        assert!(any_ref.downcast_ref::<ForeignQueryPlanner>().is_some());
+
+        let logical_plan = datafusion_expr::LogicalPlanBuilder::empty(false).build()?;
+        let state = ctx.state();
+        let physical_plan = planner.create_physical_plan(&logical_plan, &state).await?;
+
+        assert_eq!(physical_plan.name(), "EmptyExec");
+        assert!(physical_plan.is::<EmptyExec>());
+
+        Ok(())
+    }
+
+    /// Library A's logical codec stores library B's provider while the logical
+    /// plan crosses into library C. A real application would encode enough
+    /// metadata to reconstruct or locate the provider instead.

Review Comment:
   I think "metadata" was poor phrasing. Basically you need enough data to faithfully reproduce the plan on both sides of the (de)serialization. I'm updating the docstring to make it more clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat/ffi query planner main [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#issuecomment-5143322712

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24028?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `93.48659%` with `17 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 80.87%. Comparing base ([`dbcb5c0`](https://app.codecov.io/gh/apache/datafusion/commit/dbcb5c0f729e9ef6b0ab4c79253fe3b657929f48?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`74842d9`](https://app.codecov.io/gh/apache/datafusion/commit/74842d9655b1cc7821d12298b32a81bb6408750a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24028?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/ffi/src/query\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/24028?src=pr&el=tree&filepath=datafusion%2Fffi%2Fsrc%2Fquery_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mZmkvc3JjL3F1ZXJ5X3BsYW5uZXIucnM=) | 93.67% | [1 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24028?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [datafusion/ffi/src/session/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/24028?src=pr&el=tree&filepath=datafusion%2Fffi%2Fsrc%2Fsession%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mZmkvc3JjL3Nlc3Npb24vbW9kLnJz) | 93.02% | [0 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24028?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #24028      +/-   ##
   ==========================================
   + Coverage   80.85%   80.87%   +0.01%     
   ==========================================
     Files        1101     1102       +1     
     Lines      374933   375188     +255     
     Branches   374933   375188     +255     
   ==========================================
   + Hits       303166   303417     +251     
   + Misses      53671    53655      -16     
   - Partials    18096    18116      +20     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24028?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3728959224


##########
datafusion/ffi/src/session/mod.rs:
##########
@@ -350,11 +431,50 @@ impl Drop for FFI_SessionRef {
 }
 
 impl FFI_SessionRef {
-    /// Creates a new [`FFI_SessionRef`].
+    /// Creates a new [`FFI_SessionRef`] with a default physical extension codec.
+    ///
+    /// The synthesized [`DefaultPhysicalExtensionCodec`] supports built-in physical
+    /// nodes only. A query planner obtained through this session reference therefore
+    /// cannot encode or decode custom physical extension nodes. Use
+    /// [`Self::new_with_ffi_codecs`] with matching logical and physical codecs when
+    /// custom physical nodes must cross the FFI boundary.
+    ///
+    /// The physical codec wrapper requires a
+    /// [`FFI_TaskContextProvider`](crate::execution::FFI_TaskContextProvider), but this
+    /// constructor has only a session reference and a logical codec. It therefore
+    /// reuses the logical codec's provider. The provider may be owned by another
+    /// library; this is safe, but it must remain live and return the task context
+    /// intended for codec callbacks. The default physical codec does not successfully
+    /// decode extension nodes, so callers that need such callbacks must instead use
+    /// [`Self::new_with_ffi_codecs`] with an explicitly configured physical codec and
+    /// task context provider.
     pub fn new(
-        session: &(dyn Session + Send + Sync),
+        session: &dyn Session,
         runtime: Option<Handle>,
         logical_codec: FFI_LogicalExtensionCodec,
+    ) -> Self {
+        // `Session` provides a TaskContext but not the reference-counted
+        // TaskContextProvider needed by the FFI codec. Reuse the provider associated
+        // with the logical codec under the assumptions documented above.
+        let physical_codec = FFI_PhysicalExtensionCodec::new(
+            Arc::new(DefaultPhysicalExtensionCodec {}),
+            runtime.clone(),
+            logical_codec.task_ctx_provider.clone(),
+        );
+        Self::new_with_ffi_codecs(session, runtime, logical_codec, physical_codec)
+    }
+
+    /// Creates a new [`FFI_SessionRef`] using existing FFI codecs.
+    ///
+    /// The codecs must form a matching pair that can round-trip every logical and
+    /// physical extension node exposed through the session. Their task context
+    /// providers must remain live and return contexts appropriate for their decode
+    /// callbacks.
+    pub fn new_with_ffi_codecs(
+        session: &dyn Session,
+        runtime: Option<Handle>,
+        logical_codec: FFI_LogicalExtensionCodec,
+        physical_codec: FFI_PhysicalExtensionCodec,
     ) -> Self {

Review Comment:
   Similar comment than in the `query_planner.rs`. If the session is already a `ForeignSession`, this will drop the provided `logical_codec` and `physical_codec`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3696133620


##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,

Review Comment:
   One question, why `FFI_Result<FFI_ExecutionPlan>` not used instad of serialized plan?



##########
datafusion/ffi/tests/ffi_query_planner.rs:
##########
@@ -0,0 +1,223 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod utils;
+
+#[cfg(feature = "integration-tests")]
+mod tests {
+    use std::sync::{Arc, OnceLock, Weak};
+
+    use arrow::datatypes::SchemaRef;
+    use datafusion::execution::SessionStateBuilder;
+    use datafusion::prelude::SessionContext;
+    use datafusion_catalog::TableProvider;
+    use datafusion_common::{
+        DataFusionError, Result, TableReference, exec_err, not_impl_err,
+    };
+    use datafusion_execution::{TaskContext, TaskContextProvider};
+    use datafusion_expr::LogicalPlan;
+    use datafusion_expr::logical_plan::Extension;
+    use datafusion_ffi::execution::FFI_TaskContextProvider;
+    use datafusion_ffi::execution_plan::ForeignExecutionPlan;
+    use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+    use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+    use datafusion_ffi::query_planner::ForeignQueryPlanner;
+    use datafusion_ffi::table_provider::ForeignTableProvider;
+    use datafusion_ffi::tests::{
+        create_test_schema,
+        utils::{get_module, get_module_copy},
+    };
+    use datafusion_physical_plan::ExecutionPlan;
+    use datafusion_physical_plan::empty::EmptyExec;
+    use datafusion_physical_plan::union::UnionExec;
+    use datafusion_proto::logical_plan::LogicalExtensionCodec;
+    use datafusion_proto::physical_plan::{
+        DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+        PhysicalProtoConverterExtension,
+    };
+    use datafusion_session::QueryPlanner;
+
+    #[tokio::test]
+    async fn test_ffi_query_planner() -> Result<(), DataFusionError> {
+        let module = get_module()?;
+        let (ctx, logical_codec) = crate::utils::ctx_and_codec();
+        let task_ctx_provider = Arc::clone(&ctx) as Arc<dyn TaskContextProvider>;
+        let task_ctx_provider = FFI_TaskContextProvider::from(&task_ctx_provider);
+        let physical_codec = FFI_PhysicalExtensionCodec::new(
+            Arc::new(DefaultPhysicalExtensionCodec {}),
+            None,
+            task_ctx_provider,
+        );
+
+        let ffi_planner = (module.create_query_planner)(logical_codec, physical_codec);
+        let planner: Arc<dyn QueryPlanner + Send + Sync> = (&ffi_planner).into();
+
+        let any_ref: &dyn std::any::Any = planner.as_ref();
+        assert!(any_ref.downcast_ref::<ForeignQueryPlanner>().is_some());
+
+        let logical_plan = datafusion_expr::LogicalPlanBuilder::empty(false).build()?;
+        let state = ctx.state();
+        let physical_plan = planner.create_physical_plan(&logical_plan, &state).await?;
+
+        assert_eq!(physical_plan.name(), "EmptyExec");
+        assert!(physical_plan.is::<EmptyExec>());
+
+        Ok(())
+    }
+
+    /// Library A's logical codec stores library B's provider while the logical
+    /// plan crosses into library C. A real application would encode enough
+    /// metadata to reconstruct or locate the provider instead.

Review Comment:
   just for my reference, which metadata is needed? will need some help to integrate this with py ballista 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3712459895


##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,

Review Comment:
   The reason for this is the case that is illustrated in the integration test `test_three_library_query_planner_restores_type_identity`.
   
   Suppose we want one query planner to call another. Each of these planners tend to rely on downcasting the execution plans. By making the FFI query planner calls return the serialized plans, then it gets decoded within each library locally. By doing this we should get local versions of the well known execution plans that the default codec can provide.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Feat/ffi query planner main [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#issuecomment-5143112589

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-datasource-parquet v54.1.0 (current)
          Built [  56.447s] (current)
        Parsing datafusion-datasource-parquet v54.1.0 (current)
         Parsed [   0.032s] (current)
       Building datafusion-datasource-parquet v54.1.0 (baseline)
          Built [  45.502s] (baseline)
        Parsing datafusion-datasource-parquet v54.1.0 (baseline)
         Parsed [   0.032s] (baseline)
       Checking datafusion-datasource-parquet v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.155s] 223 checks: 223 pass, 30 skip
        Summary no semver update required
       Finished [ 103.713s] datafusion-datasource-parquet
       Building datafusion-ffi v54.1.0 (current)
          Built [  62.043s] (current)
        Parsing datafusion-ffi v54.1.0 (current)
         Parsed [   0.064s] (current)
       Building datafusion-ffi v54.1.0 (baseline)
          Built [  60.554s] (baseline)
        Parsing datafusion-ffi v54.1.0 (baseline)
         Parsed [   0.065s] (baseline)
       Checking datafusion-ffi v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.228s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip
   
   --- failure constructible_struct_adds_field: externally-constructible struct adds field ---
   
   Description:
   A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ForeignLibraryModule.create_query_planner in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:122
   
   --- warning repr_c_plain_struct_fields_reordered: struct fields reordered in repr(C) struct ---
   
   Description:
   A public repr(C) struct had its fields reordered. This can change the struct's memory layout, possibly breaking FFI use cases that depend on field position and order.
           ref: https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/repr_c_plain_struct_fields_reordered.ron
   
   Failed in:
     ForeignLibraryModule.version moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:127
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
        Warning produced 1 major and 0 minor level warnings
       Finished [ 124.255s] datafusion-ffi
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3712411790


##########
datafusion/ffi/src/session/mod.rs:
##########
@@ -355,6 +421,21 @@ impl FFI_SessionRef {
         session: &(dyn Session + Send + Sync),
         runtime: Option<Handle>,
         logical_codec: FFI_LogicalExtensionCodec,
+    ) -> Self {
+        let physical_codec = FFI_PhysicalExtensionCodec::new(
+            Arc::new(DefaultPhysicalExtensionCodec {}),
+            runtime.clone(),
+            logical_codec.task_ctx_provider.clone(),
+        );
+        Self::new_with_ffi_codecs(session, runtime, logical_codec, physical_codec)
+    }
+
+    /// Creates a new [`FFI_SessionRef`] using existing FFI codecs.
+    pub fn new_with_ffi_codecs(
+        session: &(dyn Session + Send + Sync),

Review Comment:
   No, updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3711145744


##########
datafusion/ffi/src/session/mod.rs:
##########
@@ -355,6 +421,21 @@ impl FFI_SessionRef {
         session: &(dyn Session + Send + Sync),
         runtime: Option<Handle>,
         logical_codec: FFI_LogicalExtensionCodec,
+    ) -> Self {
+        let physical_codec = FFI_PhysicalExtensionCodec::new(
+            Arc::new(DefaultPhysicalExtensionCodec {}),
+            runtime.clone(),
+            logical_codec.task_ctx_provider.clone(),
+        );
+        Self::new_with_ffi_codecs(session, runtime, logical_codec, physical_codec)
+    }
+
+    /// Creates a new [`FFI_SessionRef`] using existing FFI codecs.
+    pub fn new_with_ffi_codecs(
+        session: &(dyn Session + Send + Sync),

Review Comment:
   Session is already `Send` and `Sync`, do we need these extra constraints here?



##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,416 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+//!
+//! # Delegating back to library A
+//!
+//! C commonly wants A's built-in planning as a starting point, then rewrites the
+//! result. A must export its planner *before* installing C's planner on the
+//! session, and C must retain that handle: after the swap,
+//! [`Session::query_planner`] reports C's own planner, and
+//! [`Session::create_physical_plan`] dispatches to it, so either one is a
+//! self-call. Delegating to the retained handle is safe, because DataFusion's
+//! built-in physical planner never re-dispatches through [`Session`].
+//!
+//! Retain the planner rather than the session. [`FFI_QueryPlanner`] owns a
+//! reference-counted planner, so it outlives A's original session, whereas
+//! `FFI_SessionRef` borrows its session with the lifetime erased.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,
+
+    /// Codec used to encode and decode logical plans and extension nodes.
+    pub logical_codec: FFI_LogicalExtensionCodec,
+
+    /// Codec used to encode and decode physical plans and extension nodes.
+    pub physical_codec: FFI_PhysicalExtensionCodec,
+
+    /// Used to create a clone of the query planner.
+    clone: unsafe extern "C" fn(planner: &Self) -> Self,
+
+    /// Release the memory of the private data when it is no longer being used.
+    release: unsafe extern "C" fn(arg: &mut Self),
+
+    /// Return the major DataFusion version number of this planner.
+    pub version: unsafe extern "C" fn() -> u64,
+
+    /// Internal data. This is only to be accessed by the provider of the planner.
+    /// A [`ForeignQueryPlanner`] should never attempt to access this data.
+    private_data: *mut c_void,
+
+    /// Utility to identify when FFI objects are accessed locally through
+    /// the foreign interface. See [`crate::get_library_marker_id`].
+    pub library_marker_id: extern "C" fn() -> usize,
+}
+
+unsafe impl Send for FFI_QueryPlanner {}
+unsafe impl Sync for FFI_QueryPlanner {}
+
+struct QueryPlannerPrivateData {
+    planner: Arc<dyn QueryPlanner + Send + Sync>,
+}
+
+impl FFI_QueryPlanner {
+    fn inner(&self) -> &Arc<dyn QueryPlanner + Send + Sync> {
+        let private_data = self.private_data as *const QueryPlannerPrivateData;
+        unsafe { &(*private_data).planner }
+    }
+}
+
+unsafe extern "C" fn create_physical_plan_fn_wrapper(
+    planner: &FFI_QueryPlanner,
+    logical_plan_serialized: SVec<u8>,
+    session: FFI_SessionRef,
+) -> FfiFuture<FFI_Result<SVec<u8>>> {
+    let internal_planner = Arc::clone(planner.inner());
+    let logical_codec: Arc<dyn LogicalExtensionCodec> = (&planner.logical_codec).into();
+    let physical_codec: Arc<dyn PhysicalExtensionCodec> =
+        (&planner.physical_codec).into();
+
+    async move {
+        let mut foreign_session = None;
+        let session = sresult_return!(
+            session
+                .as_local()
+                .map(Ok::<&(dyn Session + Send + Sync), DataFusionError>)
+                .unwrap_or_else(|| {
+                    foreign_session = Some(ForeignSession::try_from(&session)?);
+                    Ok(foreign_session.as_ref().unwrap())
+                })
+        );
+
+        let logical_plan = sresult_return!(logical_plan_from_bytes_with_extension_codec(
+            logical_plan_serialized.as_slice(),
+            session.task_ctx().as_ref(),
+            logical_codec.as_ref(),
+        ));
+
+        let physical_plan = sresult_return!(
+            internal_planner
+                .create_physical_plan(&logical_plan, session)
+                .await
+        );
+        let physical_plan = sresult_return!(physical_plan_to_bytes_with_extension_codec(
+            physical_plan,
+            physical_codec.as_ref(),
+        ));
+
+        FFI_Result::Ok(physical_plan.iter().copied().collect())
+    }
+    .into_ffi()
+}
+
+unsafe extern "C" fn release_fn_wrapper(planner: &mut FFI_QueryPlanner) {
+    unsafe {
+        debug_assert!(!planner.private_data.is_null());
+        let private_data =
+            Box::from_raw(planner.private_data as *mut QueryPlannerPrivateData);
+        drop(private_data);
+        planner.private_data = std::ptr::null_mut();
+    }
+}
+
+unsafe extern "C" fn clone_fn_wrapper(planner: &FFI_QueryPlanner) -> FFI_QueryPlanner {
+    let old_planner = Arc::clone(planner.inner());
+
+    let private_data = Box::into_raw(Box::new(QueryPlannerPrivateData {
+        planner: old_planner,
+    })) as *mut c_void;
+
+    FFI_QueryPlanner {
+        create_physical_plan: create_physical_plan_fn_wrapper,
+        logical_codec: planner.logical_codec.clone(),
+        physical_codec: planner.physical_codec.clone(),
+        clone: clone_fn_wrapper,
+        release: release_fn_wrapper,
+        version: super::version,
+        private_data,
+        library_marker_id: crate::get_library_marker_id,
+    }
+}
+
+impl Drop for FFI_QueryPlanner {
+    fn drop(&mut self) {
+        unsafe { (self.release)(self) }
+    }
+}
+
+impl Clone for FFI_QueryPlanner {
+    fn clone(&self) -> Self {
+        unsafe { (self.clone)(self) }
+    }
+}
+
+impl FFI_QueryPlanner {
+    /// Creates an [`FFI_QueryPlanner`] with native extension codecs.
+    ///
+    /// Missing codecs use DataFusion's defaults. `runtime` and
+    /// `task_ctx_provider` support codec callbacks across the FFI boundary.
+    pub fn new(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        runtime: Option<Handle>,
+        task_ctx_provider: impl Into<FFI_TaskContextProvider>,
+        logical_codec: Option<Arc<dyn LogicalExtensionCodec>>,
+        physical_codec: Option<Arc<dyn PhysicalExtensionCodec + Send>>,
+    ) -> Self {
+        let task_ctx_provider = task_ctx_provider.into();
+        let logical_codec =
+            logical_codec.unwrap_or_else(|| Arc::new(DefaultLogicalExtensionCodec {}));
+        let logical_codec = FFI_LogicalExtensionCodec::new(
+            logical_codec,
+            runtime.clone(),
+            task_ctx_provider.clone(),
+        );
+        let physical_codec =
+            physical_codec.unwrap_or_else(|| Arc::new(DefaultPhysicalExtensionCodec {}));
+        let physical_codec =
+            FFI_PhysicalExtensionCodec::new(physical_codec, runtime, task_ctx_provider);
+        Self::new_with_ffi_codecs(planner, logical_codec, physical_codec)
+    }
+
+    /// Creates an [`FFI_QueryPlanner`] using prebuilt FFI extension codecs.
+    ///
+    /// If `planner` is already foreign, this returns its original FFI handle
+    /// rather than adding another wrapper layer.
+    pub fn new_with_ffi_codecs(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        logical_codec: FFI_LogicalExtensionCodec,
+        physical_codec: FFI_PhysicalExtensionCodec,
+    ) -> Self {
+        let any_ref: &dyn std::any::Any = planner.as_ref();
+        if let Some(planner) = any_ref.downcast_ref::<ForeignQueryPlanner>() {
+            return planner.0.clone();

Review Comment:
   Isn't this dropping the provided `logical_codec` and `physical_codec`? Isn't this actually a problem?



##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,416 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+//!
+//! # Delegating back to library A
+//!
+//! C commonly wants A's built-in planning as a starting point, then rewrites the
+//! result. A must export its planner *before* installing C's planner on the
+//! session, and C must retain that handle: after the swap,
+//! [`Session::query_planner`] reports C's own planner, and
+//! [`Session::create_physical_plan`] dispatches to it, so either one is a
+//! self-call. Delegating to the retained handle is safe, because DataFusion's
+//! built-in physical planner never re-dispatches through [`Session`].
+//!
+//! Retain the planner rather than the session. [`FFI_QueryPlanner`] owns a
+//! reference-counted planner, so it outlives A's original session, whereas
+//! `FFI_SessionRef` borrows its session with the lifetime erased.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,
+
+    /// Codec used to encode and decode logical plans and extension nodes.
+    pub logical_codec: FFI_LogicalExtensionCodec,
+
+    /// Codec used to encode and decode physical plans and extension nodes.
+    pub physical_codec: FFI_PhysicalExtensionCodec,
+
+    /// Used to create a clone of the query planner.
+    clone: unsafe extern "C" fn(planner: &Self) -> Self,
+
+    /// Release the memory of the private data when it is no longer being used.
+    release: unsafe extern "C" fn(arg: &mut Self),
+
+    /// Return the major DataFusion version number of this planner.
+    pub version: unsafe extern "C" fn() -> u64,
+
+    /// Internal data. This is only to be accessed by the provider of the planner.
+    /// A [`ForeignQueryPlanner`] should never attempt to access this data.
+    private_data: *mut c_void,
+
+    /// Utility to identify when FFI objects are accessed locally through
+    /// the foreign interface. See [`crate::get_library_marker_id`].
+    pub library_marker_id: extern "C" fn() -> usize,
+}
+
+unsafe impl Send for FFI_QueryPlanner {}
+unsafe impl Sync for FFI_QueryPlanner {}
+
+struct QueryPlannerPrivateData {
+    planner: Arc<dyn QueryPlanner + Send + Sync>,
+}
+
+impl FFI_QueryPlanner {
+    fn inner(&self) -> &Arc<dyn QueryPlanner + Send + Sync> {
+        let private_data = self.private_data as *const QueryPlannerPrivateData;
+        unsafe { &(*private_data).planner }
+    }
+}
+
+unsafe extern "C" fn create_physical_plan_fn_wrapper(
+    planner: &FFI_QueryPlanner,
+    logical_plan_serialized: SVec<u8>,
+    session: FFI_SessionRef,
+) -> FfiFuture<FFI_Result<SVec<u8>>> {
+    let internal_planner = Arc::clone(planner.inner());
+    let logical_codec: Arc<dyn LogicalExtensionCodec> = (&planner.logical_codec).into();
+    let physical_codec: Arc<dyn PhysicalExtensionCodec> =
+        (&planner.physical_codec).into();
+
+    async move {
+        let mut foreign_session = None;
+        let session = sresult_return!(
+            session
+                .as_local()
+                .map(Ok::<&(dyn Session + Send + Sync), DataFusionError>)
+                .unwrap_or_else(|| {
+                    foreign_session = Some(ForeignSession::try_from(&session)?);
+                    Ok(foreign_session.as_ref().unwrap())
+                })
+        );
+
+        let logical_plan = sresult_return!(logical_plan_from_bytes_with_extension_codec(
+            logical_plan_serialized.as_slice(),
+            session.task_ctx().as_ref(),
+            logical_codec.as_ref(),
+        ));
+
+        let physical_plan = sresult_return!(
+            internal_planner
+                .create_physical_plan(&logical_plan, session)
+                .await
+        );
+        let physical_plan = sresult_return!(physical_plan_to_bytes_with_extension_codec(
+            physical_plan,
+            physical_codec.as_ref(),
+        ));
+
+        FFI_Result::Ok(physical_plan.iter().copied().collect())
+    }
+    .into_ffi()
+}
+
+unsafe extern "C" fn release_fn_wrapper(planner: &mut FFI_QueryPlanner) {
+    unsafe {
+        debug_assert!(!planner.private_data.is_null());
+        let private_data =
+            Box::from_raw(planner.private_data as *mut QueryPlannerPrivateData);
+        drop(private_data);
+        planner.private_data = std::ptr::null_mut();
+    }
+}
+
+unsafe extern "C" fn clone_fn_wrapper(planner: &FFI_QueryPlanner) -> FFI_QueryPlanner {
+    let old_planner = Arc::clone(planner.inner());
+
+    let private_data = Box::into_raw(Box::new(QueryPlannerPrivateData {
+        planner: old_planner,
+    })) as *mut c_void;
+
+    FFI_QueryPlanner {
+        create_physical_plan: create_physical_plan_fn_wrapper,
+        logical_codec: planner.logical_codec.clone(),
+        physical_codec: planner.physical_codec.clone(),
+        clone: clone_fn_wrapper,
+        release: release_fn_wrapper,
+        version: super::version,
+        private_data,
+        library_marker_id: crate::get_library_marker_id,
+    }
+}
+
+impl Drop for FFI_QueryPlanner {
+    fn drop(&mut self) {
+        unsafe { (self.release)(self) }
+    }
+}
+
+impl Clone for FFI_QueryPlanner {
+    fn clone(&self) -> Self {
+        unsafe { (self.clone)(self) }
+    }
+}
+
+impl FFI_QueryPlanner {
+    /// Creates an [`FFI_QueryPlanner`] with native extension codecs.
+    ///
+    /// Missing codecs use DataFusion's defaults. `runtime` and
+    /// `task_ctx_provider` support codec callbacks across the FFI boundary.
+    pub fn new(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        runtime: Option<Handle>,
+        task_ctx_provider: impl Into<FFI_TaskContextProvider>,
+        logical_codec: Option<Arc<dyn LogicalExtensionCodec>>,
+        physical_codec: Option<Arc<dyn PhysicalExtensionCodec + Send>>,

Review Comment:
   This `Send` requirement should not be necessary, `PhysicalExtensionCodec` is already constrained to `Send`. It might actually give some problems to people that already have a type erased `Arc<dyn PhysicalExtensionCodec>`.



##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,416 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+//!
+//! # Delegating back to library A
+//!
+//! C commonly wants A's built-in planning as a starting point, then rewrites the
+//! result. A must export its planner *before* installing C's planner on the
+//! session, and C must retain that handle: after the swap,
+//! [`Session::query_planner`] reports C's own planner, and
+//! [`Session::create_physical_plan`] dispatches to it, so either one is a
+//! self-call. Delegating to the retained handle is safe, because DataFusion's
+//! built-in physical planner never re-dispatches through [`Session`].
+//!
+//! Retain the planner rather than the session. [`FFI_QueryPlanner`] owns a
+//! reference-counted planner, so it outlives A's original session, whereas
+//! `FFI_SessionRef` borrows its session with the lifetime erased.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,
+
+    /// Codec used to encode and decode logical plans and extension nodes.
+    pub logical_codec: FFI_LogicalExtensionCodec,
+
+    /// Codec used to encode and decode physical plans and extension nodes.
+    pub physical_codec: FFI_PhysicalExtensionCodec,
+
+    /// Used to create a clone of the query planner.
+    clone: unsafe extern "C" fn(planner: &Self) -> Self,
+
+    /// Release the memory of the private data when it is no longer being used.
+    release: unsafe extern "C" fn(arg: &mut Self),
+
+    /// Return the major DataFusion version number of this planner.
+    pub version: unsafe extern "C" fn() -> u64,
+
+    /// Internal data. This is only to be accessed by the provider of the planner.
+    /// A [`ForeignQueryPlanner`] should never attempt to access this data.
+    private_data: *mut c_void,
+
+    /// Utility to identify when FFI objects are accessed locally through
+    /// the foreign interface. See [`crate::get_library_marker_id`].
+    pub library_marker_id: extern "C" fn() -> usize,
+}
+
+unsafe impl Send for FFI_QueryPlanner {}
+unsafe impl Sync for FFI_QueryPlanner {}
+
+struct QueryPlannerPrivateData {
+    planner: Arc<dyn QueryPlanner + Send + Sync>,
+}
+
+impl FFI_QueryPlanner {
+    fn inner(&self) -> &Arc<dyn QueryPlanner + Send + Sync> {
+        let private_data = self.private_data as *const QueryPlannerPrivateData;
+        unsafe { &(*private_data).planner }
+    }
+}
+
+unsafe extern "C" fn create_physical_plan_fn_wrapper(
+    planner: &FFI_QueryPlanner,
+    logical_plan_serialized: SVec<u8>,
+    session: FFI_SessionRef,
+) -> FfiFuture<FFI_Result<SVec<u8>>> {
+    let internal_planner = Arc::clone(planner.inner());
+    let logical_codec: Arc<dyn LogicalExtensionCodec> = (&planner.logical_codec).into();
+    let physical_codec: Arc<dyn PhysicalExtensionCodec> =
+        (&planner.physical_codec).into();
+
+    async move {
+        let mut foreign_session = None;
+        let session = sresult_return!(
+            session
+                .as_local()
+                .map(Ok::<&(dyn Session + Send + Sync), DataFusionError>)
+                .unwrap_or_else(|| {
+                    foreign_session = Some(ForeignSession::try_from(&session)?);
+                    Ok(foreign_session.as_ref().unwrap())
+                })
+        );
+
+        let logical_plan = sresult_return!(logical_plan_from_bytes_with_extension_codec(
+            logical_plan_serialized.as_slice(),
+            session.task_ctx().as_ref(),
+            logical_codec.as_ref(),
+        ));
+
+        let physical_plan = sresult_return!(
+            internal_planner
+                .create_physical_plan(&logical_plan, session)
+                .await
+        );
+        let physical_plan = sresult_return!(physical_plan_to_bytes_with_extension_codec(
+            physical_plan,
+            physical_codec.as_ref(),
+        ));
+
+        FFI_Result::Ok(physical_plan.iter().copied().collect())
+    }
+    .into_ffi()
+}
+
+unsafe extern "C" fn release_fn_wrapper(planner: &mut FFI_QueryPlanner) {
+    unsafe {
+        debug_assert!(!planner.private_data.is_null());
+        let private_data =
+            Box::from_raw(planner.private_data as *mut QueryPlannerPrivateData);
+        drop(private_data);
+        planner.private_data = std::ptr::null_mut();
+    }
+}
+
+unsafe extern "C" fn clone_fn_wrapper(planner: &FFI_QueryPlanner) -> FFI_QueryPlanner {
+    let old_planner = Arc::clone(planner.inner());
+
+    let private_data = Box::into_raw(Box::new(QueryPlannerPrivateData {
+        planner: old_planner,
+    })) as *mut c_void;
+
+    FFI_QueryPlanner {
+        create_physical_plan: create_physical_plan_fn_wrapper,
+        logical_codec: planner.logical_codec.clone(),
+        physical_codec: planner.physical_codec.clone(),
+        clone: clone_fn_wrapper,
+        release: release_fn_wrapper,
+        version: super::version,
+        private_data,
+        library_marker_id: crate::get_library_marker_id,
+    }
+}
+
+impl Drop for FFI_QueryPlanner {
+    fn drop(&mut self) {
+        unsafe { (self.release)(self) }
+    }
+}
+
+impl Clone for FFI_QueryPlanner {
+    fn clone(&self) -> Self {
+        unsafe { (self.clone)(self) }
+    }
+}
+
+impl FFI_QueryPlanner {
+    /// Creates an [`FFI_QueryPlanner`] with native extension codecs.
+    ///
+    /// Missing codecs use DataFusion's defaults. `runtime` and
+    /// `task_ctx_provider` support codec callbacks across the FFI boundary.
+    pub fn new(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        runtime: Option<Handle>,
+        task_ctx_provider: impl Into<FFI_TaskContextProvider>,
+        logical_codec: Option<Arc<dyn LogicalExtensionCodec>>,
+        physical_codec: Option<Arc<dyn PhysicalExtensionCodec + Send>>,
+    ) -> Self {
+        let task_ctx_provider = task_ctx_provider.into();
+        let logical_codec =
+            logical_codec.unwrap_or_else(|| Arc::new(DefaultLogicalExtensionCodec {}));
+        let logical_codec = FFI_LogicalExtensionCodec::new(
+            logical_codec,
+            runtime.clone(),
+            task_ctx_provider.clone(),
+        );
+        let physical_codec =
+            physical_codec.unwrap_or_else(|| Arc::new(DefaultPhysicalExtensionCodec {}));
+        let physical_codec =
+            FFI_PhysicalExtensionCodec::new(physical_codec, runtime, task_ctx_provider);
+        Self::new_with_ffi_codecs(planner, logical_codec, physical_codec)

Review Comment:
   If `None` is passed to `logical_codec` or `physical_codec`, those arguments would default to `DefaultLogicalExtensionCodec` and `DefaultPhysicalExtensionCodec`, replacing any previously registered codecs even if users provide `None` as arguments.
   
   :thinking: there seems to be a couple of footguns with the `FFI_QueryPlanner::new` and `FFI_QueryPlanner::new_with_ffi_codecs` public APIs, maybe there's a cleaner constructor method we can expose publicly that makes these things irrepresentable?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #24028:
URL: https://github.com/apache/datafusion/pull/24028#discussion_r3712420044


##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,416 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+//!
+//! # Delegating back to library A
+//!
+//! C commonly wants A's built-in planning as a starting point, then rewrites the
+//! result. A must export its planner *before* installing C's planner on the
+//! session, and C must retain that handle: after the swap,
+//! [`Session::query_planner`] reports C's own planner, and
+//! [`Session::create_physical_plan`] dispatches to it, so either one is a
+//! self-call. Delegating to the retained handle is safe, because DataFusion's
+//! built-in physical planner never re-dispatches through [`Session`].
+//!
+//! Retain the planner rather than the session. [`FFI_QueryPlanner`] owns a
+//! reference-counted planner, so it outlives A's original session, whereas
+//! `FFI_SessionRef` borrows its session with the lifetime erased.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,
+
+    /// Codec used to encode and decode logical plans and extension nodes.
+    pub logical_codec: FFI_LogicalExtensionCodec,
+
+    /// Codec used to encode and decode physical plans and extension nodes.
+    pub physical_codec: FFI_PhysicalExtensionCodec,
+
+    /// Used to create a clone of the query planner.
+    clone: unsafe extern "C" fn(planner: &Self) -> Self,
+
+    /// Release the memory of the private data when it is no longer being used.
+    release: unsafe extern "C" fn(arg: &mut Self),
+
+    /// Return the major DataFusion version number of this planner.
+    pub version: unsafe extern "C" fn() -> u64,
+
+    /// Internal data. This is only to be accessed by the provider of the planner.
+    /// A [`ForeignQueryPlanner`] should never attempt to access this data.
+    private_data: *mut c_void,
+
+    /// Utility to identify when FFI objects are accessed locally through
+    /// the foreign interface. See [`crate::get_library_marker_id`].
+    pub library_marker_id: extern "C" fn() -> usize,
+}
+
+unsafe impl Send for FFI_QueryPlanner {}
+unsafe impl Sync for FFI_QueryPlanner {}
+
+struct QueryPlannerPrivateData {
+    planner: Arc<dyn QueryPlanner + Send + Sync>,
+}
+
+impl FFI_QueryPlanner {
+    fn inner(&self) -> &Arc<dyn QueryPlanner + Send + Sync> {
+        let private_data = self.private_data as *const QueryPlannerPrivateData;
+        unsafe { &(*private_data).planner }
+    }
+}
+
+unsafe extern "C" fn create_physical_plan_fn_wrapper(
+    planner: &FFI_QueryPlanner,
+    logical_plan_serialized: SVec<u8>,
+    session: FFI_SessionRef,
+) -> FfiFuture<FFI_Result<SVec<u8>>> {
+    let internal_planner = Arc::clone(planner.inner());
+    let logical_codec: Arc<dyn LogicalExtensionCodec> = (&planner.logical_codec).into();
+    let physical_codec: Arc<dyn PhysicalExtensionCodec> =
+        (&planner.physical_codec).into();
+
+    async move {
+        let mut foreign_session = None;
+        let session = sresult_return!(
+            session
+                .as_local()
+                .map(Ok::<&(dyn Session + Send + Sync), DataFusionError>)
+                .unwrap_or_else(|| {
+                    foreign_session = Some(ForeignSession::try_from(&session)?);
+                    Ok(foreign_session.as_ref().unwrap())
+                })
+        );
+
+        let logical_plan = sresult_return!(logical_plan_from_bytes_with_extension_codec(
+            logical_plan_serialized.as_slice(),
+            session.task_ctx().as_ref(),
+            logical_codec.as_ref(),
+        ));
+
+        let physical_plan = sresult_return!(
+            internal_planner
+                .create_physical_plan(&logical_plan, session)
+                .await
+        );
+        let physical_plan = sresult_return!(physical_plan_to_bytes_with_extension_codec(
+            physical_plan,
+            physical_codec.as_ref(),
+        ));
+
+        FFI_Result::Ok(physical_plan.iter().copied().collect())
+    }
+    .into_ffi()
+}
+
+unsafe extern "C" fn release_fn_wrapper(planner: &mut FFI_QueryPlanner) {
+    unsafe {
+        debug_assert!(!planner.private_data.is_null());
+        let private_data =
+            Box::from_raw(planner.private_data as *mut QueryPlannerPrivateData);
+        drop(private_data);
+        planner.private_data = std::ptr::null_mut();
+    }
+}
+
+unsafe extern "C" fn clone_fn_wrapper(planner: &FFI_QueryPlanner) -> FFI_QueryPlanner {
+    let old_planner = Arc::clone(planner.inner());
+
+    let private_data = Box::into_raw(Box::new(QueryPlannerPrivateData {
+        planner: old_planner,
+    })) as *mut c_void;
+
+    FFI_QueryPlanner {
+        create_physical_plan: create_physical_plan_fn_wrapper,
+        logical_codec: planner.logical_codec.clone(),
+        physical_codec: planner.physical_codec.clone(),
+        clone: clone_fn_wrapper,
+        release: release_fn_wrapper,
+        version: super::version,
+        private_data,
+        library_marker_id: crate::get_library_marker_id,
+    }
+}
+
+impl Drop for FFI_QueryPlanner {
+    fn drop(&mut self) {
+        unsafe { (self.release)(self) }
+    }
+}
+
+impl Clone for FFI_QueryPlanner {
+    fn clone(&self) -> Self {
+        unsafe { (self.clone)(self) }
+    }
+}
+
+impl FFI_QueryPlanner {
+    /// Creates an [`FFI_QueryPlanner`] with native extension codecs.
+    ///
+    /// Missing codecs use DataFusion's defaults. `runtime` and
+    /// `task_ctx_provider` support codec callbacks across the FFI boundary.
+    pub fn new(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        runtime: Option<Handle>,
+        task_ctx_provider: impl Into<FFI_TaskContextProvider>,
+        logical_codec: Option<Arc<dyn LogicalExtensionCodec>>,
+        physical_codec: Option<Arc<dyn PhysicalExtensionCodec + Send>>,

Review Comment:
   Fixed



##########
datafusion/ffi/src/query_planner.rs:
##########
@@ -0,0 +1,416 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! FFI support for [`QueryPlanner`].
+//!
+//! A typical deployment has three libraries. Library A (for example,
+//! `datafusion-python`) owns the [`Session`] and codec registry. Library B owns
+//! a custom table provider and its extension nodes. Library C (for example,
+//! Ballista or `datafusion-distributed`) owns the query planner. A serializes a
+//! logical plan and invokes C, while `FFI_SessionRef` lets C call session
+//! services in A. C deserializes the logical plan, creates a physical plan,
+//! serializes that result, and returns it for A to deserialize. The logical and
+//! physical extension codecs preserve nodes supplied by B.
+//!
+//! The physical result is serialized instead of returned as an
+//! [`crate::execution_plan::FFI_ExecutionPlan`]. An FFI execution-plan handle is
+//! a foreign trait-object proxy, so even a built-in plan created in C cannot be
+//! downcast to its concrete
+//! type in A. Serialization reconstructs known plan nodes with A's local Rust
+//! type identities, allowing A's optimizers and other consumers to downcast
+//! them. Extension codecs control how custom nodes are reconstructed.
+//!
+//! A node returned by B while C is planning is still foreign to C unless a
+//! codec boundary reconstructs it in C. The query-planner boundary guarantees
+//! that C-local serializable nodes, and extension nodes understood by the
+//! configured codecs, are reconstructed for A when the completed plan returns.
+//!
+//! # Delegating back to library A
+//!
+//! C commonly wants A's built-in planning as a starting point, then rewrites the
+//! result. A must export its planner *before* installing C's planner on the
+//! session, and C must retain that handle: after the swap,
+//! [`Session::query_planner`] reports C's own planner, and
+//! [`Session::create_physical_plan`] dispatches to it, so either one is a
+//! self-call. Delegating to the retained handle is safe, because DataFusion's
+//! built-in physical planner never re-dispatches through [`Session`].
+//!
+//! Retain the planner rather than the session. [`FFI_QueryPlanner`] owns a
+//! reference-counted planner, so it outlives A's original session, whereas
+//! `FFI_SessionRef` borrows its session with the lifetime erased.
+
+use std::ffi::c_void;
+use std::sync::Arc;
+
+use async_ffi::{FfiFuture, FutureExt};
+use async_trait::async_trait;
+use datafusion_common::error::{DataFusionError, Result};
+use datafusion_expr::LogicalPlan;
+use datafusion_physical_plan::ExecutionPlan;
+use datafusion_proto::bytes::{
+    logical_plan_from_bytes_with_extension_codec,
+    logical_plan_to_bytes_with_extension_codec,
+    physical_plan_from_bytes_with_extension_codec,
+    physical_plan_to_bytes_with_extension_codec,
+};
+use datafusion_proto::logical_plan::{
+    DefaultLogicalExtensionCodec, LogicalExtensionCodec,
+};
+use datafusion_proto::physical_plan::{
+    DefaultPhysicalExtensionCodec, PhysicalExtensionCodec,
+};
+use datafusion_session::{QueryPlanner, Session};
+use stabby::vec::Vec as SVec;
+use tokio::runtime::Handle;
+
+use crate::execution::FFI_TaskContextProvider;
+use crate::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use crate::proto::physical_extension_codec::FFI_PhysicalExtensionCodec;
+use crate::session::{FFI_SessionRef, ForeignSession};
+use crate::util::FFI_Result;
+use crate::{df_result, sresult_return};
+
+/// An ABI-stable handle to a [`QueryPlanner`] owned by another library.
+///
+/// The Rust-facing adapters serialize the input [`LogicalPlan`] and resulting
+/// [`ExecutionPlan`]; callers do not invoke the byte-oriented function pointer
+/// directly.
+#[repr(C)]
+#[derive(Debug)]
+pub struct FFI_QueryPlanner {
+    create_physical_plan: unsafe extern "C" fn(
+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,
+
+    /// Codec used to encode and decode logical plans and extension nodes.
+    pub logical_codec: FFI_LogicalExtensionCodec,
+
+    /// Codec used to encode and decode physical plans and extension nodes.
+    pub physical_codec: FFI_PhysicalExtensionCodec,
+
+    /// Used to create a clone of the query planner.
+    clone: unsafe extern "C" fn(planner: &Self) -> Self,
+
+    /// Release the memory of the private data when it is no longer being used.
+    release: unsafe extern "C" fn(arg: &mut Self),
+
+    /// Return the major DataFusion version number of this planner.
+    pub version: unsafe extern "C" fn() -> u64,
+
+    /// Internal data. This is only to be accessed by the provider of the planner.
+    /// A [`ForeignQueryPlanner`] should never attempt to access this data.
+    private_data: *mut c_void,
+
+    /// Utility to identify when FFI objects are accessed locally through
+    /// the foreign interface. See [`crate::get_library_marker_id`].
+    pub library_marker_id: extern "C" fn() -> usize,
+}
+
+unsafe impl Send for FFI_QueryPlanner {}
+unsafe impl Sync for FFI_QueryPlanner {}
+
+struct QueryPlannerPrivateData {
+    planner: Arc<dyn QueryPlanner + Send + Sync>,
+}
+
+impl FFI_QueryPlanner {
+    fn inner(&self) -> &Arc<dyn QueryPlanner + Send + Sync> {
+        let private_data = self.private_data as *const QueryPlannerPrivateData;
+        unsafe { &(*private_data).planner }
+    }
+}
+
+unsafe extern "C" fn create_physical_plan_fn_wrapper(
+    planner: &FFI_QueryPlanner,
+    logical_plan_serialized: SVec<u8>,
+    session: FFI_SessionRef,
+) -> FfiFuture<FFI_Result<SVec<u8>>> {
+    let internal_planner = Arc::clone(planner.inner());
+    let logical_codec: Arc<dyn LogicalExtensionCodec> = (&planner.logical_codec).into();
+    let physical_codec: Arc<dyn PhysicalExtensionCodec> =
+        (&planner.physical_codec).into();
+
+    async move {
+        let mut foreign_session = None;
+        let session = sresult_return!(
+            session
+                .as_local()
+                .map(Ok::<&(dyn Session + Send + Sync), DataFusionError>)
+                .unwrap_or_else(|| {
+                    foreign_session = Some(ForeignSession::try_from(&session)?);
+                    Ok(foreign_session.as_ref().unwrap())
+                })
+        );
+
+        let logical_plan = sresult_return!(logical_plan_from_bytes_with_extension_codec(
+            logical_plan_serialized.as_slice(),
+            session.task_ctx().as_ref(),
+            logical_codec.as_ref(),
+        ));
+
+        let physical_plan = sresult_return!(
+            internal_planner
+                .create_physical_plan(&logical_plan, session)
+                .await
+        );
+        let physical_plan = sresult_return!(physical_plan_to_bytes_with_extension_codec(
+            physical_plan,
+            physical_codec.as_ref(),
+        ));
+
+        FFI_Result::Ok(physical_plan.iter().copied().collect())
+    }
+    .into_ffi()
+}
+
+unsafe extern "C" fn release_fn_wrapper(planner: &mut FFI_QueryPlanner) {
+    unsafe {
+        debug_assert!(!planner.private_data.is_null());
+        let private_data =
+            Box::from_raw(planner.private_data as *mut QueryPlannerPrivateData);
+        drop(private_data);
+        planner.private_data = std::ptr::null_mut();
+    }
+}
+
+unsafe extern "C" fn clone_fn_wrapper(planner: &FFI_QueryPlanner) -> FFI_QueryPlanner {
+    let old_planner = Arc::clone(planner.inner());
+
+    let private_data = Box::into_raw(Box::new(QueryPlannerPrivateData {
+        planner: old_planner,
+    })) as *mut c_void;
+
+    FFI_QueryPlanner {
+        create_physical_plan: create_physical_plan_fn_wrapper,
+        logical_codec: planner.logical_codec.clone(),
+        physical_codec: planner.physical_codec.clone(),
+        clone: clone_fn_wrapper,
+        release: release_fn_wrapper,
+        version: super::version,
+        private_data,
+        library_marker_id: crate::get_library_marker_id,
+    }
+}
+
+impl Drop for FFI_QueryPlanner {
+    fn drop(&mut self) {
+        unsafe { (self.release)(self) }
+    }
+}
+
+impl Clone for FFI_QueryPlanner {
+    fn clone(&self) -> Self {
+        unsafe { (self.clone)(self) }
+    }
+}
+
+impl FFI_QueryPlanner {
+    /// Creates an [`FFI_QueryPlanner`] with native extension codecs.
+    ///
+    /// Missing codecs use DataFusion's defaults. `runtime` and
+    /// `task_ctx_provider` support codec callbacks across the FFI boundary.
+    pub fn new(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        runtime: Option<Handle>,
+        task_ctx_provider: impl Into<FFI_TaskContextProvider>,
+        logical_codec: Option<Arc<dyn LogicalExtensionCodec>>,
+        physical_codec: Option<Arc<dyn PhysicalExtensionCodec + Send>>,
+    ) -> Self {
+        let task_ctx_provider = task_ctx_provider.into();
+        let logical_codec =
+            logical_codec.unwrap_or_else(|| Arc::new(DefaultLogicalExtensionCodec {}));
+        let logical_codec = FFI_LogicalExtensionCodec::new(
+            logical_codec,
+            runtime.clone(),
+            task_ctx_provider.clone(),
+        );
+        let physical_codec =
+            physical_codec.unwrap_or_else(|| Arc::new(DefaultPhysicalExtensionCodec {}));
+        let physical_codec =
+            FFI_PhysicalExtensionCodec::new(physical_codec, runtime, task_ctx_provider);
+        Self::new_with_ffi_codecs(planner, logical_codec, physical_codec)
+    }
+
+    /// Creates an [`FFI_QueryPlanner`] using prebuilt FFI extension codecs.
+    ///
+    /// If `planner` is already foreign, this returns its original FFI handle
+    /// rather than adding another wrapper layer.
+    pub fn new_with_ffi_codecs(
+        planner: Arc<dyn QueryPlanner + Send + Sync>,
+        logical_codec: FFI_LogicalExtensionCodec,
+        physical_codec: FFI_PhysicalExtensionCodec,
+    ) -> Self {
+        let any_ref: &dyn std::any::Any = planner.as_ref();
+        if let Some(planner) = any_ref.downcast_ref::<ForeignQueryPlanner>() {
+            return planner.0.clone();

Review Comment:
   Yes, changed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Implement FFI_QueryPlanner [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer merged PR #24028:
URL: https://github.com/apache/datafusion/pull/24028


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org