You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/11 20:46:47 UTC

(datafusion) branch gh-readonly-queue/main/pr-22797-dd663e8888c93dd966fab1825688232e0fb7f1bf created (now 70c9dc81b5)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-22797-dd663e8888c93dd966fab1825688232e0fb7f1bf
in repository https://gitbox.apache.org/repos/asf/datafusion.git


      at 70c9dc81b5 FFI: plumb with_updated_config for FFI_ScalarUDF (#22797)

This branch includes the following new commits:

     new 70c9dc81b5 FFI: plumb with_updated_config for FFI_ScalarUDF (#22797)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


(datafusion) 01/01: FFI: plumb with_updated_config for FFI_ScalarUDF (#22797)

Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-22797-dd663e8888c93dd966fab1825688232e0fb7f1bf
in repository https://gitbox.apache.org/repos/asf/datafusion.git

commit 70c9dc81b5abe09c14f5091d38d753fecd63d911
Author: Amogh Ramesh <ra...@gmail.com>
AuthorDate: Wed Aug 12 02:16:36 2026 +0530

    FFI: plumb with_updated_config for FFI_ScalarUDF (#22797)
    
    ## Which issue does this PR close?
    
    Part of #22330.
    
    ## Rationale for this change
    
    `ForeignScalarUDF` inherits the default `with_updated_config`, so
    producer overrides are lost across the FFI boundary.
    
    ## What changes are included in this PR?
    
    - Forward `with_updated_config` through `FFI_ScalarUDF`.
    - Reuse the existing placement and timezone UDFs for unit and
    dynamic-library coverage.
    - Log transport failures before the infallible trait method returns
    `None`.
    - Update the FFI skill to cover error handling, owned returns, and
    fixture reuse.
    
    ## Are these changes tested?
    
    - `cargo test -p datafusion-ffi --features integration-tests`
    - `cargo clippy --all-targets --all-features -- -D warnings`
    - `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`
    
    ## Are there any user-facing changes?
    
    The FFI ABI changes. Foreign libraries must rebuild against the new
    DataFusion version.
    
    ---------
    
    Signed-off-by: Amogh Ramesh <ra...@gmail.com>
    Co-authored-by: Tim Saucer <ti...@gmail.com>
---
 .ai/skills/datafusion-ffi/SKILL.md        | 10 +++-
 datafusion/ffi/src/tests/udf_udaf_udwf.rs |  8 +++
 datafusion/ffi/src/udf/mod.rs             | 98 +++++++++++++++++++++++++++----
 datafusion/ffi/tests/ffi_udf.rs           | 27 +++++++++
 4 files changed, 128 insertions(+), 15 deletions(-)

diff --git a/.ai/skills/datafusion-ffi/SKILL.md b/.ai/skills/datafusion-ffi/SKILL.md
index c105da6536..ba02d22c09 100644
--- a/.ai/skills/datafusion-ffi/SKILL.md
+++ b/.ai/skills/datafusion-ffi/SKILL.md
@@ -1,6 +1,6 @@
 ---
 name: datafusion-ffi
-description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `Foreign<X>` impls, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
+description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
 ---
 
 # DataFusion FFI Skill
@@ -210,9 +210,11 @@ cargo test -p datafusion-ffi --features integration-tests
 To add coverage for a new wrapper:
 
 1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
-2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract — adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
+2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
 3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
 
+When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
+
 What integration tests catch that unit tests cannot:
 
 - **Real ABI layout bugs.** Two builds means the consumer's view of `FFI_X` is reconstructed from declaration, not aliased to the producer's memory. Mismatched alignment, padding, niche optimization, or accidentally non-`#[repr(C)]` types surface here.
@@ -235,7 +237,7 @@ What integration tests catch that unit tests cannot:
 - Primitives (`u8`/`u64`/`bool`/`usize`, etc.) and `#[repr(u8)]` FFI enums (`FFI_TableType`, `Volatility`, `InsertOp`, `TableProviderFilterPushDown`).
 - A `stabby::string::String` (`SString`) returned by value, with no other args or returns.
 
-Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer — those exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
+Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer. These exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
 
 If you skip the integration test for a layout change, you have effectively shipped untested ABI.
 
@@ -313,6 +315,8 @@ If a method's body is non-trivial, the consumer-side default is non-trivial too.
 - **Logical `Expr` / `LogicalPlan`**: serialize via `datafusion-proto` using the embedded `FFI_LogicalExtensionCodec`. Same for physical plans → `FFI_PhysicalExtensionCodec`.
 - **Enums** (`Volatility`, `TableType`, `InsertOp`, `TableProviderFilterPushDown`): `#[repr(u8)]`, with `From<Native> for FFI_X` and `From<&FFI_X> for Native`. Always write a round-trip unit test that exercises every variant.
 - **Errors**: every `FFI_X` method that can fail returns `FFI_Result<T>`. Use the `sresult!`, `sresult_return!`, `df_result!` macros from `src/util.rs` — do not roll your own.
+- **Infallible trait methods**: if an FFI call can fail but the native trait cannot return the error, log the transport error before returning the trait's fallback (`None`, `false`, or a default). Never discard it with `.ok()`, `.unwrap_or_default()`, or equivalent.
+- **Owned FFI returns**: consume an owned `FFI_X` with `From<FFI_X>` instead of converting through `&FFI_X` and cloning across the boundary. Keep the borrowed conversion's local marker fast path.
 
 ## Async, sessions, and task context
 
diff --git a/datafusion/ffi/src/tests/udf_udaf_udwf.rs b/datafusion/ffi/src/tests/udf_udaf_udwf.rs
index 04d6fb26c1..830c639c74 100644
--- a/datafusion/ffi/src/tests/udf_udaf_udwf.rs
+++ b/datafusion/ffi/src/tests/udf_udaf_udwf.rs
@@ -20,6 +20,7 @@ use std::sync::Arc;
 use arrow_schema::DataType;
 use datafusion_catalog::TableFunctionImpl;
 use datafusion_common::ScalarValue;
+use datafusion_common::config::ConfigOptions;
 use datafusion_expr::sort_properties::ExprProperties;
 use datafusion_expr::{
     AggregateUDF, ColumnarValue, ExpressionPlacement, ScalarFunctionArgs, ScalarUDF,
@@ -104,6 +105,13 @@ impl ScalarUDFImpl for TimeZoneUDF {
         let tz = args.config_options.execution.time_zone.clone();
         Ok(ColumnarValue::Scalar(ScalarValue::from(tz)))
     }
+
+    fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> {
+        config.execution.time_zone.as_ref()?;
+        Some(ScalarUDF::from(Self {
+            signature: self.signature.clone(),
+        }))
+    }
 }
 
 pub(crate) extern "C" fn create_timezone_func() -> FFI_ScalarUDF {
diff --git a/datafusion/ffi/src/udf/mod.rs b/datafusion/ffi/src/udf/mod.rs
index 8e96dd9013..fa08cbd042 100644
--- a/datafusion/ffi/src/udf/mod.rs
+++ b/datafusion/ffi/src/udf/mod.rs
@@ -45,7 +45,7 @@ use crate::expr::columnar_value::FFI_ColumnarValue;
 use crate::expr::expr_properties::FFI_ExprProperties;
 use crate::placement::FFI_ExpressionPlacement;
 use crate::util::{
-    FFI_Result, rvec_wrapped_to_vec_datatype, vec_datatype_to_rvec_wrapped,
+    FFI_Option, FFI_Result, rvec_wrapped_to_vec_datatype, vec_datatype_to_rvec_wrapped,
 };
 use crate::volatility::FFI_Volatility;
 use crate::{df_result, sresult, sresult_return};
@@ -123,6 +123,13 @@ pub struct FFI_ScalarUDF {
         udf: &Self,
         inputs: SVec<FFI_ExprProperties>,
     ) -> FFI_Result<bool>,
+
+    /// FFI equivalent to [`ScalarUDFImpl::with_updated_config`].
+    pub with_updated_config:
+        unsafe extern "C" fn(
+            udf: &Self,
+            config: FFI_ConfigOptions,
+        ) -> FFI_Result<FFI_Option<FFI_ScalarUDF>>,
 }
 
 unsafe impl Send for FFI_ScalarUDF {}
@@ -199,6 +206,21 @@ unsafe extern "C" fn preserves_lex_ordering_fn_wrapper(
     sresult!(result)
 }
 
+unsafe extern "C" fn with_updated_config_fn_wrapper(
+    udf: &FFI_ScalarUDF,
+    config: FFI_ConfigOptions,
+) -> FFI_Result<FFI_Option<FFI_ScalarUDF>> {
+    let config = sresult_return!(ConfigOptions::try_from(config));
+
+    let updated: Option<FFI_ScalarUDF> = udf
+        .inner()
+        .inner()
+        .with_updated_config(&config)
+        .map(|updated| Arc::new(updated).into());
+
+    FFI_Result::Ok(updated.into())
+}
+
 unsafe extern "C" fn invoke_with_args_fn_wrapper(
     udf: &FFI_ScalarUDF,
     args: SVec<WrappedArray>,
@@ -298,6 +320,7 @@ impl From<Arc<ScalarUDF>> for FFI_ScalarUDF {
             private_data: Box::into_raw(private_data) as *mut c_void,
             library_marker_id: crate::get_library_marker_id,
             preserves_lex_ordering: preserves_lex_ordering_fn_wrapper,
+            with_updated_config: with_updated_config_fn_wrapper,
         }
     }
 }
@@ -325,6 +348,21 @@ pub struct ForeignScalarUDF {
 unsafe impl Send for ForeignScalarUDF {}
 unsafe impl Sync for ForeignScalarUDF {}
 
+impl ForeignScalarUDF {
+    fn new(udf: FFI_ScalarUDF) -> Self {
+        let name = udf.name.to_string();
+        let signature = Signature::user_defined((&udf.volatility).into());
+        let aliases = udf.aliases.iter().map(|s| s.to_string()).collect();
+
+        Self {
+            name,
+            aliases,
+            udf,
+            signature,
+        }
+    }
+}
+
 impl PartialEq for ForeignScalarUDF {
     fn eq(&self, other: &Self) -> bool {
         let Self {
@@ -356,22 +394,22 @@ impl Hash for ForeignScalarUDF {
     }
 }
 
+impl From<FFI_ScalarUDF> for Arc<dyn ScalarUDFImpl> {
+    fn from(udf: FFI_ScalarUDF) -> Self {
+        if (udf.library_marker_id)() == crate::get_library_marker_id() {
+            Arc::clone(udf.inner().inner())
+        } else {
+            Arc::new(ForeignScalarUDF::new(udf))
+        }
+    }
+}
+
 impl From<&FFI_ScalarUDF> for Arc<dyn ScalarUDFImpl> {
     fn from(udf: &FFI_ScalarUDF) -> Self {
         if (udf.library_marker_id)() == crate::get_library_marker_id() {
             Arc::clone(udf.inner().inner())
         } else {
-            let name = udf.name.to_string();
-            let signature = Signature::user_defined((&udf.volatility).into());
-
-            let aliases = udf.aliases.iter().map(|s| s.to_string()).collect();
-
-            Arc::new(ForeignScalarUDF {
-                name,
-                udf: udf.clone(),
-                aliases,
-                signature,
-            })
+            Arc::new(ForeignScalarUDF::new(udf.clone()))
         }
     }
 }
@@ -494,6 +532,22 @@ impl ScalarUDFImpl for ForeignScalarUDF {
                 df_result!(result)
             })
     }
+
+    fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> {
+        let config: FFI_ConfigOptions = config.into();
+
+        let result = unsafe { (self.udf.with_updated_config)(&self.udf, config) };
+
+        let updated = match df_result!(result) {
+            Ok(updated) => updated.into_option()?,
+            Err(error) => {
+                log::warn!("Unable to update scalar UDF configuration over FFI: {error}");
+                return None;
+            }
+        };
+
+        Some(ScalarUDF::new_from_shared_impl(updated.into()))
+    }
 }
 
 #[cfg(test)]
@@ -542,6 +596,12 @@ mod tests {
 
             Ok(inputs.iter().all(|input| input.preserves_lex_ordering))
         }
+
+        fn with_updated_config(&self, _config: &ConfigOptions) -> Option<ScalarUDF> {
+            Some(ScalarUDF::from(Self {
+                signature: self.signature.clone(),
+            }))
+        }
     }
 
     #[test]
@@ -555,6 +615,11 @@ mod tests {
         let foreign_udf: Arc<dyn ScalarUDFImpl> = (&local_udf).into();
 
         assert_eq!(original_udf.name(), foreign_udf.name());
+        assert!(
+            foreign_udf
+                .with_updated_config(&ConfigOptions::default())
+                .is_none()
+        );
 
         Ok(())
     }
@@ -629,6 +694,15 @@ mod tests {
         );
         assert!(foreign_udf.preserves_lex_ordering(&[]).is_err());
 
+        let updated = foreign_udf
+            .with_updated_config(&ConfigOptions::default())
+            .expect("provider should return an updated UDF");
+        assert_eq!(
+            updated
+                .placement(&[ExpressionPlacement::Column, ExpressionPlacement::Literal]),
+            ExpressionPlacement::MoveTowardsLeafNodes
+        );
+
         Ok(())
     }
 }
diff --git a/datafusion/ffi/tests/ffi_udf.rs b/datafusion/ffi/tests/ffi_udf.rs
index d9e7263ccd..10e0bb5cc1 100644
--- a/datafusion/ffi/tests/ffi_udf.rs
+++ b/datafusion/ffi/tests/ffi_udf.rs
@@ -21,6 +21,7 @@
 mod tests {
     use arrow::array::{Array, AsArray, record_batch};
     use arrow::datatypes::DataType;
+    use datafusion::common::config::ConfigOptions;
     use datafusion::error::Result;
     use datafusion::logical_expr::{ExpressionPlacement, ScalarUDF, ScalarUDFImpl};
     use datafusion::prelude::{SessionContext, col};
@@ -157,4 +158,30 @@ mod tests {
 
         Ok(())
     }
+
+    /// Validates that a provider's `with_updated_config` override survives the
+    /// FFI boundary (the trait default returns `None`).
+    #[test]
+    fn test_with_updated_config_on_scalar_udf() -> Result<()> {
+        let module = get_module()?;
+
+        let ffi_udf = (module.create_timezone_udf)();
+        let foreign_udf: Arc<dyn ScalarUDFImpl> = (&ffi_udf).into();
+
+        assert!(
+            foreign_udf
+                .with_updated_config(&ConfigOptions::default())
+                .is_none()
+        );
+
+        let mut options = ConfigOptions::default();
+        options.execution.time_zone = Some("AEST".into());
+
+        let updated = foreign_udf
+            .with_updated_config(&options)
+            .expect("provider should return an updated UDF");
+        assert_eq!(updated.name(), "TimeZoneUDF");
+
+        Ok(())
+    }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org