You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "Copilot (via GitHub)" <gi...@apache.org> on 2026/06/29 03:38:50 UTC

Re: [PR] feat(c/sedona-extension): Add FFI table provider, exec plan, and synchronous stream exchange [sedona-db]

Copilot commented on code in PR #1004:
URL: https://github.com/apache/sedona-db/pull/1004#discussion_r3489171246


##########
python/sedonadb/src/import_from.rs:
##########
@@ -59,14 +59,21 @@ pub fn import_table_provider_from_any<'py>(
     }
 }
 
-pub fn import_ffi_table_provider(
+pub fn import_sedona_ffi_table_provider(
     obj: &Bound<PyAny>,
 ) -> Result<Arc<dyn TableProvider>, PySedonaError> {
-    let capsule = obj.getattr("__datafusion_table_provider__")?.call0()?;
+    let capsule = obj.getattr("__sedonadb_table_provider__")?.call0()?;
     let contents =
-        check_pycapsule(&capsule, "datafusion_table_provider")? as *mut FFI_TableProvider;
-    let provider = Arc::<dyn TableProvider>::from(unsafe { contents.as_ref().unwrap() });
-    Ok(provider)
+        check_pycapsule(&capsule, "sedonadb_table_provider")? as *mut SedonaCTableProvider;
+    // Move the SedonaCTableProvider out of the capsule into our ImportedTableProvider.
+    // We null out the release function to prevent double-free when the capsule is dropped.
+    let ffi_provider = unsafe {
+        let provider = std::ptr::read(contents);
+        (*contents).release = None;
+        provider
+    };

Review Comment:
   After moving the `SedonaCTableProvider` out of the PyCapsule with `ptr::read`, only setting `(*contents).release = None` still leaves the rest of the capsule memory looking like a valid provider. If the same capsule is imported twice, the second import can create an `ImportedTableProvider` that points at the same `private_data` but no longer has a `release` callback, leading to leaks and possible use-after-free depending on drop order. Clear the capsule contents after `ptr::read` (without dropping it) so subsequent imports fail fast and the capsule drop becomes a no-op.



##########
c/sedona-extension/src/sedona_extension.h:
##########
@@ -203,6 +204,156 @@ struct SedonaCScalarKernel {
   void* private_data;
 };
 
+struct SedonaCError {
+  const char* err;
+
+  uint32_t err_len;
+
+  uint32_t reserved;
+
+  /// \brief Release this instance
+  ///
+  /// Implementations of this callback must set self->release to NULL.
+  void (*release)(struct SedonaCError* self);
+};
+
+struct SedonaCExpr {
+  // Get a property of this expression (e.g., serialize, extract bbox)
+  int (*get_property_schema)(const struct SedonaCExpr* self, const char* property,
+                             struct SedonaCError* err);
+  int (*get_property)(const struct SedonaCExpr* self, const char* property,
+                      const char* args, struct ArrowArray* out, struct SedonaCError* err);
+
+  void* reserved;
+
+  /// \brief Release this instance
+  ///
+  /// Implementations of this callback must set self->release to NULL.
+  void (*release)(struct SedonaCExecutionPlan* self);
+
+  /// \brief Opaque implementation-specific data
+  void* private_data;
+};

Review Comment:
   `SedonaCExpr.release` has the wrong function signature in the public C header: it currently takes `struct SedonaCExecutionPlan*` instead of `struct SedonaCExpr*`. This will break C/C++ callers that try to implement or invoke the callback (ABI/type mismatch).



##########
rust/sedona/src/dataframe.rs:
##########
@@ -0,0 +1,288 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::RecordBatch;
+use async_trait::async_trait;
+use datafusion::{
+    dataframe::DataFrameWriteOptions,
+    datasource::file_format::format_as_file_type,
+    error::{DataFusionError, Result},
+    prelude::DataFrame,
+};
+use datafusion_common::not_impl_err;
+use datafusion_expr::{dml::InsertOp, LogicalPlan, LogicalPlanBuilder, SortExpr};
+use sedona_geoparquet::{format::GeoParquetFormatFactory, options::TableGeoParquetOptions};
+
+use crate::{
+    context::SedonaContext,
+    show::{show_batches, DisplayTableOptions},
+};
+
+/// Sedona-specific [`DataFrame`] actions
+///
+/// This trait, implemented for [`DataFrame`], extends the DataFrame API to make it
+/// ergonomic to work with dataframes that contain geometry columns. Currently these
+/// are limited to output functions, as geometry columns currently require special
+/// handling when written or exported to an external system.
+#[async_trait]

Review Comment:
   This file duplicates the existing `SedonaDataFrame` / `SedonaWriteOptions` definitions that already live in `rust/sedona/src/context.rs` (including near-identical tests). Since `lib.rs` does not `mod dataframe`, this module (and its tests) won't be compiled or run, and it risks future divergence/conflicts if it’s later wired in. Either remove this file or finish the refactor by moving the implementations out of `context.rs` and exporting them from a single place.



##########
c/sedona-extension/src/utils.rs:
##########
@@ -0,0 +1,872 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for converting between Arrow FFI streams and DataFusion streams.
+
+use std::ffi::{c_int, CString};
+use std::ptr::null_mut;
+
+use arrow_array::ffi_stream::FFI_ArrowArrayStream;
+use arrow_array::{RecordBatch, RecordBatchReader};
+use arrow_schema::ffi::FFI_ArrowSchema;
+use arrow_schema::{ArrowError, DataType, Field, SchemaRef};
+use datafusion_common::{exec_err, Result};
+use datafusion_physical_plan::SendableRecordBatchStream;
+use futures::StreamExt;
+use sedona_common::{sedona_internal_datafusion_err, sedona_internal_err};
+use serde::de::DeserializeOwned;
+use serde::Serialize;
+
+use crate::extension::{
+    SedonaCError, SedonaCExecutionPlan, SedonaCExecutionPlanArgs, SedonaCTableProvider,
+};
+
+/// Success return code for FFI functions.
+pub const ERRNO_OK: c_int = 0;
+
+/// Get the schema for a property from a [SedonaCExecutionPlan].
+///
+/// Returns the DataType describing the property's data type.
+/// If `get_property_schema` is not implemented, defaults to Binary.
+fn get_plan_property_data_type(plan: &SedonaCExecutionPlan, property: &str) -> Result<DataType> {
+    let Some(get_property_schema) = plan.get_property_schema else {
+        // Default to Binary if get_property_schema is not implemented
+        return Ok(DataType::Binary);
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_schema = FFI_ArrowSchema::empty();
+    let mut err = SedonaCError::default();
+
+    let code =
+        unsafe { get_property_schema(plan, property_cstr.as_ptr(), &mut ffi_schema, &mut err) };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get property schema for '{}': {}", property, err);
+    }
+
+    // Try to convert the FFI schema to a Field
+    let field = Field::try_from(&ffi_schema).map_err(|e| {
+        sedona_internal_datafusion_err!("Failed to parse property schema for '{}': {}", property, e)
+    })?;
+    Ok(field.data_type().clone())
+}
+
+/// Call `get_property` on a [SedonaCExecutionPlan] and deserialize the result.
+///
+/// This handles the common pattern of:
+/// 1. Extracting the get_property function pointer
+/// 2. Calling it with a property name and optional serializable args
+/// 3. Parsing the binary array result
+/// 4. Deserializing from JSON to the target type
+///
+/// # Arguments
+///
+/// * `plan` - The execution plan to query
+/// * `property` - The property name to retrieve
+/// * `args` - Optional arguments to pass (will be serialized to JSON bytes)
+///
+/// # Errors
+///
+/// Returns an error if:
+/// - The plan does not have a `get_property` callback
+/// - The FFI call fails
+/// - The result cannot be parsed as a binary array
+/// - The JSON deserialization fails
+pub fn get_plan_property<T, A>(
+    plan: &SedonaCExecutionPlan,
+    property: &str,
+    args: Option<&A>,
+) -> Result<T>
+where
+    T: DeserializeOwned,
+    A: Serialize,
+{
+    let Some(get_property) = plan.get_property else {
+        return sedona_internal_err!("SedonaCExecutionPlan does not have get_property");
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    // Serialize args if provided
+    let args_bytes = match args {
+        Some(a) => serde_json::to_vec(a)
+            .map_err(|e| sedona_internal_datafusion_err!("Failed to serialize args: {}", e))?,
+        None => Vec::new(),
+    };
+
+    let mut ffi_args = SedonaCExecutionPlanArgs {
+        args: if args_bytes.is_empty() {
+            std::ptr::null()
+        } else {
+            args_bytes.as_ptr()
+        },
+        args_len: args_bytes.len(),
+        exec_plans: std::ptr::null(),
+        num_exec_plans: 0,
+        exprs: std::ptr::null(),
+        num_exprs: 0,
+        reserved: null_mut(),
+    };
+
+    let mut ffi_array = arrow_array::ffi::FFI_ArrowArray::empty();
+    let mut err = SedonaCError::default();
+
+    let code = unsafe {
+        get_property(
+            plan,
+            property_cstr.as_ptr(),
+            &mut ffi_args,
+            &mut ffi_array,
+            &mut err,
+        )
+    };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get property '{}': {}", property, err);
+    }
+
+    // Get the property schema to know how to interpret the array
+    let data_type = get_plan_property_data_type(plan, property)?;
+
+    // Parse the array to get the JSON bytes
+    parse_ffi_array(ffi_array, &data_type)
+}
+
+/// Parse an FFI array containing JSON and deserialize to the target type.
+fn parse_ffi_array<T: DeserializeOwned>(
+    ffi_array: arrow_array::ffi::FFI_ArrowArray,
+    data_type: &DataType,
+) -> Result<T> {
+    let bytes = parse_ffi_array_to_bytes(ffi_array, data_type)?;
+    serde_json::from_slice::<T>(&bytes)
+        .map_err(|e| sedona_internal_datafusion_err!("Failed to deserialize property: {}", e))
+}
+
+/// Parse an FFI array and return the raw bytes.
+fn parse_ffi_array_to_bytes(
+    ffi_array: arrow_array::ffi::FFI_ArrowArray,
+    data_type: &DataType,
+) -> Result<Vec<u8>> {
+    let data = unsafe { arrow_array::ffi::from_ffi_and_data_type(ffi_array, data_type.clone())? };
+    let array = arrow_array::make_array(data);
+
+    // Handle different array types
+    match data_type {
+        DataType::Binary => {
+            let binary_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::BinaryArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected binary array from get_property")
+                })?;
+            Ok(binary_array.value(0).to_vec())
+        }
+        DataType::LargeBinary => {
+            let binary_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::LargeBinaryArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected large binary array from get_property")
+                })?;
+            Ok(binary_array.value(0).to_vec())
+        }
+        DataType::Utf8 => {
+            let string_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::StringArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected string array from get_property")
+                })?;
+            Ok(string_array.value(0).as_bytes().to_vec())
+        }
+        DataType::LargeUtf8 => {
+            let string_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::LargeStringArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected large string array from get_property")
+                })?;
+            Ok(string_array.value(0).as_bytes().to_vec())
+        }
+        _ => sedona_internal_err!("Unsupported data type for property: {:?}", data_type),
+    }
+}
+
+/// Get a string property from a [SedonaCExecutionPlan].
+///
+/// This is a convenience wrapper around [get_plan_property] for string values.
+pub fn get_plan_string_property(plan: &SedonaCExecutionPlan, property: &str) -> Result<String> {
+    let Some(get_property) = plan.get_property else {
+        return sedona_internal_err!("SedonaCExecutionPlan does not have get_property");
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_args = SedonaCExecutionPlanArgs {
+        args: std::ptr::null(),
+        args_len: 0,
+        exec_plans: std::ptr::null(),
+        num_exec_plans: 0,
+        exprs: std::ptr::null(),
+        num_exprs: 0,
+        reserved: null_mut(),
+    };
+
+    let mut ffi_array = arrow_array::ffi::FFI_ArrowArray::empty();
+    let mut err = SedonaCError::default();
+
+    let code = unsafe {
+        get_property(
+            plan,
+            property_cstr.as_ptr(),
+            &mut ffi_args,
+            &mut ffi_array,
+            &mut err,
+        )
+    };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get '{}': {}", property, err);
+    }
+
+    // Get the property schema to know how to interpret the array
+    let data_type = get_plan_property_data_type(plan, property)?;
+
+    let bytes = parse_ffi_array_to_bytes(ffi_array, &data_type)?;
+    String::from_utf8(bytes)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid UTF-8 in '{}': {}", property, e))
+}
+
+/// Get the schema for a property from a [SedonaCTableProvider].
+///
+/// Returns the DataType describing the property's data type.
+/// If `get_property_schema` is not implemented, defaults to Binary.
+fn get_table_provider_property_data_type(
+    provider: &SedonaCTableProvider,
+    property: &str,
+) -> Result<DataType> {
+    let Some(get_property_schema) = provider.get_property_schema else {
+        // Default to Binary if get_property_schema is not implemented
+        return Ok(DataType::Binary);
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_schema = FFI_ArrowSchema::empty();
+    let mut err = SedonaCError::default();
+
+    let code =
+        unsafe { get_property_schema(provider, property_cstr.as_ptr(), &mut ffi_schema, &mut err) };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get property schema for '{}': {}", property, err);
+    }
+
+    // Try to convert the FFI schema to a Field
+    let field = Field::try_from(&ffi_schema).map_err(|e| {
+        sedona_internal_datafusion_err!("Failed to parse property schema for '{}': {}", property, e)
+    })?;
+    Ok(field.data_type().clone())
+}
+
+/// Get a string property from a [SedonaCTableProvider].
+pub fn get_table_provider_string_property(
+    provider: &SedonaCTableProvider,
+    property: &str,
+) -> Result<String> {
+    let Some(get_property) = provider.get_property else {
+        return sedona_internal_err!("SedonaCTableProvider does not have get_property");
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_args = SedonaCExecutionPlanArgs {
+        args: std::ptr::null(),
+        args_len: 0,
+        exec_plans: std::ptr::null(),
+        num_exec_plans: 0,
+        exprs: std::ptr::null(),
+        num_exprs: 0,
+        reserved: null_mut(),
+    };
+
+    let mut ffi_array = arrow_array::ffi::FFI_ArrowArray::empty();
+    let mut err = SedonaCError::default();
+
+    let code = unsafe {
+        get_property(
+            provider,
+            property_cstr.as_ptr(),
+            &mut ffi_args,
+            &mut ffi_array,
+            &mut err,
+        )
+    };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get '{}': {}", property, err);
+    }
+
+    // Get the property schema to know how to interpret the array
+    let data_type = get_table_provider_property_data_type(provider, property)?;
+
+    let bytes = parse_ffi_array_to_bytes(ffi_array, &data_type)?;
+    String::from_utf8(bytes)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid UTF-8 in '{}': {}", property, e))
+}
+
+/// A cancellation check callback for FFI operations.
+///
+/// Returns `true` if the operation should be cancelled, `false` to continue.
+/// This allows Python, R, ADBC, and other runtimes to integrate their own
+/// cancellation mechanisms.
+pub type CancelChecker = Box<dyn Fn() -> bool + Send + Sync>;
+
+/// A RecordBatchReader that lazily polls a SendableRecordBatchStream.
+///
+/// This allows exporting a DataFusion stream over FFI without collecting all
+/// batches upfront. Used when exporting execution plans across FFI boundaries.
+pub struct StreamingRecordBatchReader {
+    schema: SchemaRef,
+    stream: SendableRecordBatchStream,
+    runtime: tokio::runtime::Handle,
+    cancel_checker: Option<CancelChecker>,
+    cancelled: bool,
+    skip_empty_batches: bool,
+    periodic_check_interval: Option<std::time::Duration>,
+}
+
+impl StreamingRecordBatchReader {
+    /// Create a new StreamingRecordBatchReader from a SendableRecordBatchStream.
+    pub fn new(stream: SendableRecordBatchStream, runtime: tokio::runtime::Handle) -> Self {
+        Self {
+            schema: stream.schema(),
+            stream,
+            runtime,
+            cancel_checker: None,
+            cancelled: false,
+            skip_empty_batches: false,
+            periodic_check_interval: None,
+        }
+    }
+
+    /// Create a new StreamingRecordBatchReader with a cancellation checker.
+    ///
+    /// The cancellation checker is called before each batch is fetched. If it
+    /// returns `true`, iteration stops with a cancellation error on the next
+    /// call and `None` on subsequent calls.
+    pub fn with_cancel_checker(
+        stream: SendableRecordBatchStream,
+        runtime: tokio::runtime::Handle,
+        cancel_checker: CancelChecker,
+    ) -> Self {
+        Self {
+            schema: stream.schema(),
+            stream,
+            runtime,
+            cancel_checker: Some(cancel_checker),
+            cancelled: false,
+            skip_empty_batches: false,
+            periodic_check_interval: None,
+        }
+    }
+
+    /// Set whether to skip empty batches (batches with 0 rows).
+    ///
+    /// When enabled, the iterator will automatically skip over any batches
+    /// that have no rows and continue to the next batch.
+    pub fn with_skip_empty_batches(mut self, skip: bool) -> Self {
+        self.skip_empty_batches = skip;
+        self
+    }
+
+    /// Set a periodic interval for checking cancellation during batch fetches.
+    ///
+    /// When set, the cancel checker will be called periodically at this interval
+    /// even while waiting for a single batch to be fetched. This is useful for
+    /// Python where we need to periodically check for signals (Ctrl+C) during
+    /// long-running operations.
+    ///
+    /// Without this, the cancel checker is only called between batch fetches.
+    pub fn with_periodic_check_interval(mut self, interval: std::time::Duration) -> Self {
+        self.periodic_check_interval = Some(interval);
+        self
+    }
+
+    fn fetch_next_batch(&mut self) -> Option<std::result::Result<RecordBatch, ArrowError>> {
+        let stream = &mut self.stream;
+        let runtime = &self.runtime;
+
+        match &self.periodic_check_interval {
+            Some(interval) => {
+                // Use tokio::select! to periodically check for cancellation
+                let interval = *interval;
+                let cancel_checker = &self.cancel_checker;
+
+                std::thread::scope(|s| {
+                    s.spawn(|| {
+                        runtime.block_on(async {
+                            use futures::StreamExt;
+                            tokio::pin!(stream);
+                            loop {
+                                tokio::select! {
+                                    res = stream.next() => {
+                                        return match res {
+                                            Some(Ok(batch)) => Some(Ok(batch)),
+                                            Some(Err(e)) => Some(Err(ArrowError::ExternalError(Box::new(e)))),
+                                            None => None,
+                                        };
+                                    }
+                                    _ = tokio::time::sleep(interval) => {
+                                        if let Some(ref checker) = cancel_checker {
+                                            if checker() {
+                                                return Some(Err(ArrowError::ExternalError(Box::new(
+                                                    std::io::Error::new(
+                                                        std::io::ErrorKind::Interrupted,
+                                                        "Operation cancelled",
+                                                    ),
+                                                ))));
+                                            }
+                                        }
+                                        // Continue waiting for the batch
+                                    }
+                                }
+                            }
+                        })
+                    })
+                    .join()
+                    .unwrap_or_else(|_| {
+                        Some(Err(ArrowError::InvalidArgumentError(
+                            "Iterator thread panicked".to_string(),
+                        )))
+                    })
+                })
+            }
+            None => {
+                // Simple blocking fetch without periodic checking
+                std::thread::scope(|s| {
+                    s.spawn(|| {
+                        runtime.block_on(async {
+                            match stream.next().await {
+                                Some(Ok(batch)) => Some(Ok(batch)),
+                                Some(Err(e)) => Some(Err(ArrowError::ExternalError(Box::new(e)))),
+                                None => None,
+                            }
+                        })
+                    })
+                    .join()
+                    .unwrap_or_else(|_| {
+                        Some(Err(ArrowError::InvalidArgumentError(
+                            "Iterator thread panicked".to_string(),
+                        )))
+                    })
+                })
+            }
+        }
+    }
+}
+
+impl Iterator for StreamingRecordBatchReader {
+    type Item = std::result::Result<RecordBatch, ArrowError>;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        // If already cancelled, return None to stop iteration
+        if self.cancelled {
+            return None;
+        }
+
+        // Check for cancellation before fetching the next batch
+        // (periodic checking during fetch is handled by fetch_next_batch if configured)
+        if let Some(ref checker) = self.cancel_checker {
+            if checker() {
+                self.cancelled = true;
+                return Some(Err(ArrowError::ExternalError(Box::new(
+                    std::io::Error::new(std::io::ErrorKind::Interrupted, "Operation cancelled"),
+                ))));
+            }
+        }
+
+        loop {
+            match self.fetch_next_batch() {
+                Some(Ok(batch)) => {
+                    if self.skip_empty_batches && batch.num_rows() == 0 {
+                        continue;
+                    }
+                    return Some(Ok(batch));
+                }
+                Some(Err(e)) => {
+                    // Check if this was a cancellation error from periodic checking
+                    if e.to_string().contains("Operation cancelled") {
+                        self.cancelled = true;
+                    }
+                    return Some(Err(e));
+                }

Review Comment:
   Cancellation detection relies on matching the substring "Operation cancelled" in the Arrow error's string. This is brittle (message changes, localization, false positives) and can cause incorrect cancellation state. Prefer checking for the concrete `std::io::ErrorKind::Interrupted` embedded in `ArrowError::ExternalError`.



##########
c/sedona-extension/src/utils.rs:
##########
@@ -0,0 +1,872 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for converting between Arrow FFI streams and DataFusion streams.
+
+use std::ffi::{c_int, CString};
+use std::ptr::null_mut;
+
+use arrow_array::ffi_stream::FFI_ArrowArrayStream;
+use arrow_array::{RecordBatch, RecordBatchReader};
+use arrow_schema::ffi::FFI_ArrowSchema;
+use arrow_schema::{ArrowError, DataType, Field, SchemaRef};
+use datafusion_common::{exec_err, Result};
+use datafusion_physical_plan::SendableRecordBatchStream;
+use futures::StreamExt;
+use sedona_common::{sedona_internal_datafusion_err, sedona_internal_err};
+use serde::de::DeserializeOwned;
+use serde::Serialize;
+
+use crate::extension::{
+    SedonaCError, SedonaCExecutionPlan, SedonaCExecutionPlanArgs, SedonaCTableProvider,
+};
+
+/// Success return code for FFI functions.
+pub const ERRNO_OK: c_int = 0;
+
+/// Get the schema for a property from a [SedonaCExecutionPlan].
+///
+/// Returns the DataType describing the property's data type.
+/// If `get_property_schema` is not implemented, defaults to Binary.
+fn get_plan_property_data_type(plan: &SedonaCExecutionPlan, property: &str) -> Result<DataType> {
+    let Some(get_property_schema) = plan.get_property_schema else {
+        // Default to Binary if get_property_schema is not implemented
+        return Ok(DataType::Binary);
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_schema = FFI_ArrowSchema::empty();
+    let mut err = SedonaCError::default();
+
+    let code =
+        unsafe { get_property_schema(plan, property_cstr.as_ptr(), &mut ffi_schema, &mut err) };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get property schema for '{}': {}", property, err);
+    }
+
+    // Try to convert the FFI schema to a Field
+    let field = Field::try_from(&ffi_schema).map_err(|e| {
+        sedona_internal_datafusion_err!("Failed to parse property schema for '{}': {}", property, e)
+    })?;
+    Ok(field.data_type().clone())
+}
+
+/// Call `get_property` on a [SedonaCExecutionPlan] and deserialize the result.
+///
+/// This handles the common pattern of:
+/// 1. Extracting the get_property function pointer
+/// 2. Calling it with a property name and optional serializable args
+/// 3. Parsing the binary array result
+/// 4. Deserializing from JSON to the target type
+///
+/// # Arguments
+///
+/// * `plan` - The execution plan to query
+/// * `property` - The property name to retrieve
+/// * `args` - Optional arguments to pass (will be serialized to JSON bytes)
+///
+/// # Errors
+///
+/// Returns an error if:
+/// - The plan does not have a `get_property` callback
+/// - The FFI call fails
+/// - The result cannot be parsed as a binary array
+/// - The JSON deserialization fails
+pub fn get_plan_property<T, A>(
+    plan: &SedonaCExecutionPlan,
+    property: &str,
+    args: Option<&A>,
+) -> Result<T>
+where
+    T: DeserializeOwned,
+    A: Serialize,
+{
+    let Some(get_property) = plan.get_property else {
+        return sedona_internal_err!("SedonaCExecutionPlan does not have get_property");
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    // Serialize args if provided
+    let args_bytes = match args {
+        Some(a) => serde_json::to_vec(a)
+            .map_err(|e| sedona_internal_datafusion_err!("Failed to serialize args: {}", e))?,
+        None => Vec::new(),
+    };
+
+    let mut ffi_args = SedonaCExecutionPlanArgs {
+        args: if args_bytes.is_empty() {
+            std::ptr::null()
+        } else {
+            args_bytes.as_ptr()
+        },
+        args_len: args_bytes.len(),
+        exec_plans: std::ptr::null(),
+        num_exec_plans: 0,
+        exprs: std::ptr::null(),
+        num_exprs: 0,
+        reserved: null_mut(),
+    };
+
+    let mut ffi_array = arrow_array::ffi::FFI_ArrowArray::empty();
+    let mut err = SedonaCError::default();
+
+    let code = unsafe {
+        get_property(
+            plan,
+            property_cstr.as_ptr(),
+            &mut ffi_args,
+            &mut ffi_array,
+            &mut err,
+        )
+    };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get property '{}': {}", property, err);
+    }
+
+    // Get the property schema to know how to interpret the array
+    let data_type = get_plan_property_data_type(plan, property)?;
+
+    // Parse the array to get the JSON bytes
+    parse_ffi_array(ffi_array, &data_type)
+}
+
+/// Parse an FFI array containing JSON and deserialize to the target type.
+fn parse_ffi_array<T: DeserializeOwned>(
+    ffi_array: arrow_array::ffi::FFI_ArrowArray,
+    data_type: &DataType,
+) -> Result<T> {
+    let bytes = parse_ffi_array_to_bytes(ffi_array, data_type)?;
+    serde_json::from_slice::<T>(&bytes)
+        .map_err(|e| sedona_internal_datafusion_err!("Failed to deserialize property: {}", e))
+}
+
+/// Parse an FFI array and return the raw bytes.
+fn parse_ffi_array_to_bytes(
+    ffi_array: arrow_array::ffi::FFI_ArrowArray,
+    data_type: &DataType,
+) -> Result<Vec<u8>> {
+    let data = unsafe { arrow_array::ffi::from_ffi_and_data_type(ffi_array, data_type.clone())? };
+    let array = arrow_array::make_array(data);
+
+    // Handle different array types
+    match data_type {
+        DataType::Binary => {
+            let binary_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::BinaryArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected binary array from get_property")
+                })?;
+            Ok(binary_array.value(0).to_vec())
+        }
+        DataType::LargeBinary => {
+            let binary_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::LargeBinaryArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected large binary array from get_property")
+                })?;
+            Ok(binary_array.value(0).to_vec())
+        }
+        DataType::Utf8 => {
+            let string_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::StringArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected string array from get_property")
+                })?;
+            Ok(string_array.value(0).as_bytes().to_vec())
+        }
+        DataType::LargeUtf8 => {
+            let string_array = array
+                .as_any()
+                .downcast_ref::<arrow_array::LargeStringArray>()
+                .ok_or_else(|| {
+                    sedona_internal_datafusion_err!("Expected large string array from get_property")
+                })?;
+            Ok(string_array.value(0).as_bytes().to_vec())
+        }
+        _ => sedona_internal_err!("Unsupported data type for property: {:?}", data_type),
+    }
+}
+
+/// Get a string property from a [SedonaCExecutionPlan].
+///
+/// This is a convenience wrapper around [get_plan_property] for string values.
+pub fn get_plan_string_property(plan: &SedonaCExecutionPlan, property: &str) -> Result<String> {
+    let Some(get_property) = plan.get_property else {
+        return sedona_internal_err!("SedonaCExecutionPlan does not have get_property");
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_args = SedonaCExecutionPlanArgs {
+        args: std::ptr::null(),
+        args_len: 0,
+        exec_plans: std::ptr::null(),
+        num_exec_plans: 0,
+        exprs: std::ptr::null(),
+        num_exprs: 0,
+        reserved: null_mut(),
+    };
+
+    let mut ffi_array = arrow_array::ffi::FFI_ArrowArray::empty();
+    let mut err = SedonaCError::default();
+
+    let code = unsafe {
+        get_property(
+            plan,
+            property_cstr.as_ptr(),
+            &mut ffi_args,
+            &mut ffi_array,
+            &mut err,
+        )
+    };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get '{}': {}", property, err);
+    }
+
+    // Get the property schema to know how to interpret the array
+    let data_type = get_plan_property_data_type(plan, property)?;
+
+    let bytes = parse_ffi_array_to_bytes(ffi_array, &data_type)?;
+    String::from_utf8(bytes)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid UTF-8 in '{}': {}", property, e))
+}
+
+/// Get the schema for a property from a [SedonaCTableProvider].
+///
+/// Returns the DataType describing the property's data type.
+/// If `get_property_schema` is not implemented, defaults to Binary.
+fn get_table_provider_property_data_type(
+    provider: &SedonaCTableProvider,
+    property: &str,
+) -> Result<DataType> {
+    let Some(get_property_schema) = provider.get_property_schema else {
+        // Default to Binary if get_property_schema is not implemented
+        return Ok(DataType::Binary);
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_schema = FFI_ArrowSchema::empty();
+    let mut err = SedonaCError::default();
+
+    let code =
+        unsafe { get_property_schema(provider, property_cstr.as_ptr(), &mut ffi_schema, &mut err) };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get property schema for '{}': {}", property, err);
+    }
+
+    // Try to convert the FFI schema to a Field
+    let field = Field::try_from(&ffi_schema).map_err(|e| {
+        sedona_internal_datafusion_err!("Failed to parse property schema for '{}': {}", property, e)
+    })?;
+    Ok(field.data_type().clone())
+}
+
+/// Get a string property from a [SedonaCTableProvider].
+pub fn get_table_provider_string_property(
+    provider: &SedonaCTableProvider,
+    property: &str,
+) -> Result<String> {
+    let Some(get_property) = provider.get_property else {
+        return sedona_internal_err!("SedonaCTableProvider does not have get_property");
+    };
+
+    let property_cstr = CString::new(property)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid property name: {}", e))?;
+
+    let mut ffi_args = SedonaCExecutionPlanArgs {
+        args: std::ptr::null(),
+        args_len: 0,
+        exec_plans: std::ptr::null(),
+        num_exec_plans: 0,
+        exprs: std::ptr::null(),
+        num_exprs: 0,
+        reserved: null_mut(),
+    };
+
+    let mut ffi_array = arrow_array::ffi::FFI_ArrowArray::empty();
+    let mut err = SedonaCError::default();
+
+    let code = unsafe {
+        get_property(
+            provider,
+            property_cstr.as_ptr(),
+            &mut ffi_args,
+            &mut ffi_array,
+            &mut err,
+        )
+    };
+
+    if code != ERRNO_OK {
+        return sedona_internal_err!("Failed to get '{}': {}", property, err);
+    }
+
+    // Get the property schema to know how to interpret the array
+    let data_type = get_table_provider_property_data_type(provider, property)?;
+
+    let bytes = parse_ffi_array_to_bytes(ffi_array, &data_type)?;
+    String::from_utf8(bytes)
+        .map_err(|e| sedona_internal_datafusion_err!("Invalid UTF-8 in '{}': {}", property, e))
+}
+
+/// A cancellation check callback for FFI operations.
+///
+/// Returns `true` if the operation should be cancelled, `false` to continue.
+/// This allows Python, R, ADBC, and other runtimes to integrate their own
+/// cancellation mechanisms.
+pub type CancelChecker = Box<dyn Fn() -> bool + Send + Sync>;
+
+/// A RecordBatchReader that lazily polls a SendableRecordBatchStream.
+///
+/// This allows exporting a DataFusion stream over FFI without collecting all
+/// batches upfront. Used when exporting execution plans across FFI boundaries.
+pub struct StreamingRecordBatchReader {
+    schema: SchemaRef,
+    stream: SendableRecordBatchStream,
+    runtime: tokio::runtime::Handle,
+    cancel_checker: Option<CancelChecker>,
+    cancelled: bool,
+    skip_empty_batches: bool,
+    periodic_check_interval: Option<std::time::Duration>,
+}
+
+impl StreamingRecordBatchReader {
+    /// Create a new StreamingRecordBatchReader from a SendableRecordBatchStream.
+    pub fn new(stream: SendableRecordBatchStream, runtime: tokio::runtime::Handle) -> Self {
+        Self {
+            schema: stream.schema(),
+            stream,
+            runtime,
+            cancel_checker: None,
+            cancelled: false,
+            skip_empty_batches: false,
+            periodic_check_interval: None,
+        }
+    }
+
+    /// Create a new StreamingRecordBatchReader with a cancellation checker.
+    ///
+    /// The cancellation checker is called before each batch is fetched. If it
+    /// returns `true`, iteration stops with a cancellation error on the next
+    /// call and `None` on subsequent calls.
+    pub fn with_cancel_checker(
+        stream: SendableRecordBatchStream,
+        runtime: tokio::runtime::Handle,
+        cancel_checker: CancelChecker,
+    ) -> Self {
+        Self {
+            schema: stream.schema(),
+            stream,
+            runtime,
+            cancel_checker: Some(cancel_checker),
+            cancelled: false,
+            skip_empty_batches: false,
+            periodic_check_interval: None,
+        }
+    }
+
+    /// Set whether to skip empty batches (batches with 0 rows).
+    ///
+    /// When enabled, the iterator will automatically skip over any batches
+    /// that have no rows and continue to the next batch.
+    pub fn with_skip_empty_batches(mut self, skip: bool) -> Self {
+        self.skip_empty_batches = skip;
+        self
+    }
+
+    /// Set a periodic interval for checking cancellation during batch fetches.
+    ///
+    /// When set, the cancel checker will be called periodically at this interval
+    /// even while waiting for a single batch to be fetched. This is useful for
+    /// Python where we need to periodically check for signals (Ctrl+C) during
+    /// long-running operations.
+    ///
+    /// Without this, the cancel checker is only called between batch fetches.
+    pub fn with_periodic_check_interval(mut self, interval: std::time::Duration) -> Self {
+        self.periodic_check_interval = Some(interval);
+        self
+    }
+
+    fn fetch_next_batch(&mut self) -> Option<std::result::Result<RecordBatch, ArrowError>> {
+        let stream = &mut self.stream;
+        let runtime = &self.runtime;
+
+        match &self.periodic_check_interval {
+            Some(interval) => {
+                // Use tokio::select! to periodically check for cancellation
+                let interval = *interval;
+                let cancel_checker = &self.cancel_checker;
+
+                std::thread::scope(|s| {
+                    s.spawn(|| {
+                        runtime.block_on(async {
+                            use futures::StreamExt;
+                            tokio::pin!(stream);
+                            loop {

Review Comment:
   `fetch_next_batch` spawns and joins a new OS thread for every batch fetch (both in the periodic and non-periodic branches). This per-batch thread churn is likely to dominate runtime for small batches and can reduce throughput significantly. Consider avoiding per-`next()` thread creation (e.g., call `Handle::block_on` directly when already outside a runtime, use `tokio::task::block_in_place` when inside, or run a dedicated worker thread for the reader).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org