You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2026/10/19 18:23:42 UTC

[PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

paleolimbot opened a new pull request, #1466:
URL: https://github.com/apache/sedona-db/pull/1466

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4390697691


##########
python/sedonadb/python/sedonadb/expr/literal.py:
##########
@@ -147,6 +148,149 @@ def _resolve_arrow_lit(obj: Any):
         ) from e
 
 
+def _resolve_arrow_array(values, type):
+    """Resolve Python values to an array, with extension-specific handling.
+
+    This is the array counterpart of `_resolve_arrow_lit`, spiritually equivalent
+    to `_resolve_arrow_lit` except (1) arrays that are resolved do not have to have
+    a length of 1 and (2) iterables resolve to their element form (e.g., lists) instead
+    of as a list of elements.
+
+    This is currently limited to UDF handling, where there is always a target type
+    available (the return type, which has already been calculated), and the values
+    list is always a Python list, and arrow arrays because they are relatively
+    straightforward.
+    """
+    import pyarrow as pa
+
+    register_pyarrow_extension_types()
+
+    if type is None:
+        raise ValueError(
+            "Resolving an array without a target type is not yet supported"
+        )
+
+    target_sedona_type = PySedonaType(type)
+    target_pyarrow_type = pa.DataType._import_from_c_capsule(
+        target_sedona_type.__arrow_c_schema__()
+    )
+
+    # Special case arrow arrays first
+    if hasattr(values, "__arrow_c_array__"):
+        schema, array = values.__arrow_c_array__()
+        source_sedona_type = PySedonaType(schema)
+
+        # Use == on the PySedonaType to handle geometry type CRS differences.
+        if target_sedona_type == source_sedona_type:
+            return pa.Array._import_from_c_capsule(
+                target_sedona_type.__arrow_c_schema__(), array
+            )
+
+        source_pyarrow_array = pa.Array._import_from_c_capsule(
+            source_sedona_type.__arrow_c_schema__(), array
+        )
+
+        # Use SedonaType's extension name to avoid pyarrow's stripping of
+        # unrecognized extensions
+        if target_sedona_type.extension_name is None:
+            return source_pyarrow_array.cast(target_pyarrow_type)
+        elif target_sedona_type.extension_name == "geoarrow.wkb":
+            return _resolve_geometry_array(source_pyarrow_array, target_pyarrow_type)
+        else:
+            raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+    # The destination type gets to handle conversion
+    if target_sedona_type is None or target_sedona_type.extension_name is None:
+        return pa.array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "geoarrow.wkb":
+        return _resolve_geometry_array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "sedona.raster":
+        return _resolve_raster_array(values, target_pyarrow_type)
+    else:
+        raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+
+def _resolve_geometry_array(values, target_pyarrow_type):
+    """Resolve WKB storage or Shapely objects to the declared geometry type."""
+    import geoarrow.pyarrow as ga
+    import pyarrow as pa
+
+    # We can't quite use ga.array() or ga.as_geoarrow() here because of some bugs
+    # https://github.com/geoarrow/geoarrow-python/pull/84
+    if isinstance(values, pa.ExtensionArray):
+        if values.type.extension_name != "geoarrow.wkb":
+            raise TypeError(
+                f"Unsupported cast from {values.type} to {target_pyarrow_type}"
+            )
+
+        # Normalize storage to compare metadata using Sedona's CRS equality.
+        source_type = ga.wkb().with_metadata(values.type.__arrow_ext_serialize__())
+        target_type = ga.wkb().with_metadata(
+            target_pyarrow_type.__arrow_ext_serialize__()
+        )
+        if PySedonaType(source_type) != PySedonaType(target_type):
+            raise TypeError(
+                f"Can't cast geometry with different CRS or edge type: "
+                f"{values.type} to {target_pyarrow_type}"
+            )
+        storage = values.storage.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, pa.Array):
+        storage = values.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, list):
+        # For lists, we find the first non-null element and choose the conversion
+        # based on that. This doesn't support mixed types in the list; however,
+        # adds significant overhead (~2x slower) for lists of small objects like
+        # Shapely points. For now, we just support a single element type in a
+        # list.
+        first_value = next((value for value in values if value is not None), None)
+        qualified_name = _qualified_type_name(first_value)
+        if qualified_name in ("builtins.bytes", "builtins.NoneType"):
+            # Empty and all-null lists also convert directly to WKB storage.
+            wkb_storage = values
+        elif qualified_name in (
+            "geoarrow.pyarrow._scalar.WkbScalar",
+            "geoarrow.pyarrow._scalar.WktScalar",
+            "geoarrow.pyarrow._scalar.GeometryExtensionScalar",
+        ):
+            wkb_storage = (
+                None if value is None or not value.is_valid else value.wkb
+                for value in values
+            )
+        else:
+            import shapely
+
+            wkb_storage = shapely.to_wkb(values, flavor="iso")
+        storage = pa.array(wkb_storage, target_pyarrow_type.storage_type)
+    else:
+        raise TypeError(f"Can't create geometry array from {type(values).__name__}")
+
+    return target_pyarrow_type.wrap_array(storage)
+
+
+def _resolve_raster_array(values, target_pyarrow_type):
+    import pyarrow as pa
+
+    from sedonadb.raster import Raster
+
+    # The equal array type should already have been handled, this is just
+    # from a list of Raster (or None) objects.
+    if not isinstance(values, list):
+        raise TypeError("Expected a list of Raster or None objects")
+
+    storage_type = target_pyarrow_type.storage_type
+    arrays = []
+    for value in values:
+        if value is None:
+            arrays.append(pa.nulls(1, storage_type))
+        elif isinstance(value, Raster):
+            arrays.append(pa.array(value).storage)
+        else:
+            raise TypeError("Expected a list of Raster or None objects")
+
+    storage = pa.concat_arrays(arrays) if arrays else pa.nulls(0, storage_type)

Review Comment:
   It is true that if a UDF returns its output with a direct reference to the input, we get this issue. Also, we have built-in native UDFs that cover most metadata transformations (i.e., I would expect most cases to allocate new pixels here). I'll open a follow-up to track further improvements.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4390946267


##########
python/sedonadb/python/sedonadb/expr/literal.py:
##########
@@ -147,6 +148,149 @@ def _resolve_arrow_lit(obj: Any):
         ) from e
 
 
+def _resolve_arrow_array(values, type):
+    """Resolve Python values to an array, with extension-specific handling.
+
+    This is the array counterpart of `_resolve_arrow_lit`, spiritually equivalent
+    to `_resolve_arrow_lit` except (1) arrays that are resolved do not have to have
+    a length of 1 and (2) iterables resolve to their element form (e.g., lists) instead
+    of as a list of elements.
+
+    This is currently limited to UDF handling, where there is always a target type
+    available (the return type, which has already been calculated), and the values
+    list is always a Python list, and arrow arrays because they are relatively
+    straightforward.
+    """
+    import pyarrow as pa
+
+    register_pyarrow_extension_types()
+
+    if type is None:
+        raise ValueError(
+            "Resolving an array without a target type is not yet supported"
+        )
+
+    target_sedona_type = PySedonaType(type)
+    target_pyarrow_type = pa.DataType._import_from_c_capsule(
+        target_sedona_type.__arrow_c_schema__()
+    )
+
+    # Special case arrow arrays first
+    if hasattr(values, "__arrow_c_array__"):
+        schema, array = values.__arrow_c_array__()
+        source_sedona_type = PySedonaType(schema)
+
+        # Use == on the PySedonaType to handle geometry type CRS differences.
+        if target_sedona_type == source_sedona_type:
+            return pa.Array._import_from_c_capsule(
+                target_sedona_type.__arrow_c_schema__(), array
+            )
+
+        source_pyarrow_array = pa.Array._import_from_c_capsule(
+            source_sedona_type.__arrow_c_schema__(), array
+        )
+
+        # Use SedonaType's extension name to avoid pyarrow's stripping of
+        # unrecognized extensions
+        if target_sedona_type.extension_name is None:
+            return source_pyarrow_array.cast(target_pyarrow_type)
+        elif target_sedona_type.extension_name == "geoarrow.wkb":
+            return _resolve_geometry_array(source_pyarrow_array, target_pyarrow_type)
+        else:
+            raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+    # The destination type gets to handle conversion
+    if target_sedona_type is None or target_sedona_type.extension_name is None:
+        return pa.array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "geoarrow.wkb":
+        return _resolve_geometry_array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "sedona.raster":
+        return _resolve_raster_array(values, target_pyarrow_type)
+    else:
+        raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+
+def _resolve_geometry_array(values, target_pyarrow_type):
+    """Resolve WKB storage or Shapely objects to the declared geometry type."""
+    import geoarrow.pyarrow as ga
+    import pyarrow as pa
+
+    # We can't quite use ga.array() or ga.as_geoarrow() here because of some bugs
+    # https://github.com/geoarrow/geoarrow-python/pull/84
+    if isinstance(values, pa.ExtensionArray):
+        if values.type.extension_name != "geoarrow.wkb":
+            raise TypeError(
+                f"Unsupported cast from {values.type} to {target_pyarrow_type}"
+            )
+
+        # Normalize storage to compare metadata using Sedona's CRS equality.
+        source_type = ga.wkb().with_metadata(values.type.__arrow_ext_serialize__())
+        target_type = ga.wkb().with_metadata(
+            target_pyarrow_type.__arrow_ext_serialize__()
+        )
+        if PySedonaType(source_type) != PySedonaType(target_type):
+            raise TypeError(
+                f"Can't cast geometry with different CRS or edge type: "
+                f"{values.type} to {target_pyarrow_type}"
+            )
+        storage = values.storage.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, pa.Array):
+        storage = values.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, list):
+        # For lists, we find the first non-null element and choose the conversion
+        # based on that. This doesn't support mixed types in the list; however,
+        # adds significant overhead (~2x slower) for lists of small objects like
+        # Shapely points. For now, we just support a single element type in a
+        # list.
+        first_value = next((value for value in values if value is not None), None)
+        qualified_name = _qualified_type_name(first_value)
+        if qualified_name in ("builtins.bytes", "builtins.NoneType"):
+            # Empty and all-null lists also convert directly to WKB storage.
+            wkb_storage = values
+        elif qualified_name in (
+            "geoarrow.pyarrow._scalar.WkbScalar",
+            "geoarrow.pyarrow._scalar.WktScalar",
+            "geoarrow.pyarrow._scalar.GeometryExtensionScalar",
+        ):
+            wkb_storage = (
+                None if value is None or not value.is_valid else value.wkb
+                for value in values
+            )
+        else:
+            import shapely
+
+            wkb_storage = shapely.to_wkb(values, flavor="iso")
+        storage = pa.array(wkb_storage, target_pyarrow_type.storage_type)
+    else:
+        raise TypeError(f"Can't create geometry array from {type(values).__name__}")
+
+    return target_pyarrow_type.wrap_array(storage)
+
+
+def _resolve_raster_array(values, target_pyarrow_type):
+    import pyarrow as pa
+
+    from sedonadb.raster import Raster
+
+    # The equal array type should already have been handled, this is just
+    # from a list of Raster (or None) objects.
+    if not isinstance(values, list):
+        raise TypeError("Expected a list of Raster or None objects")
+
+    storage_type = target_pyarrow_type.storage_type
+    arrays = []
+    for value in values:
+        if value is None:
+            arrays.append(pa.nulls(1, storage_type))
+        elif isinstance(value, Raster):
+            arrays.append(pa.array(value).storage)

Review Comment:
   Sure, tracked as https://github.com/apache/sedona-db/issues/1522



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4381719644


##########
python/sedonadb/python/sedonadb/expr/literal.py:
##########
@@ -147,6 +148,149 @@ def _resolve_arrow_lit(obj: Any):
         ) from e
 
 
+def _resolve_arrow_array(values, type):
+    """Resolve Python values to an array, with extension-specific handling.
+
+    This is the array counterpart of `_resolve_arrow_lit`, spiritually equivalent
+    to `_resolve_arrow_lit` except (1) arrays that are resolved do not have to have
+    a length of 1 and (2) iterables resolve to their element form (e.g., lists) instead
+    of as a list of elements.
+
+    This is currently limited to UDF handling, where there is always a target type
+    available (the return type, which has already been calculated), and the values
+    list is always a Python list, and arrow arrays because they are relatively
+    straightforward.
+    """
+    import pyarrow as pa
+
+    register_pyarrow_extension_types()
+
+    if type is None:
+        raise ValueError(
+            "Resolving an array without a target type is not yet supported"
+        )
+
+    target_sedona_type = PySedonaType(type)
+    target_pyarrow_type = pa.DataType._import_from_c_capsule(
+        target_sedona_type.__arrow_c_schema__()
+    )
+
+    # Special case arrow arrays first
+    if hasattr(values, "__arrow_c_array__"):
+        schema, array = values.__arrow_c_array__()
+        source_sedona_type = PySedonaType(schema)
+
+        # Use == on the PySedonaType to handle geometry type CRS differences.
+        if target_sedona_type == source_sedona_type:
+            return pa.Array._import_from_c_capsule(
+                target_sedona_type.__arrow_c_schema__(), array
+            )
+
+        source_pyarrow_array = pa.Array._import_from_c_capsule(
+            source_sedona_type.__arrow_c_schema__(), array
+        )
+
+        # Use SedonaType's extension name to avoid pyarrow's stripping of
+        # unrecognized extensions
+        if target_sedona_type.extension_name is None:
+            return source_pyarrow_array.cast(target_pyarrow_type)
+        elif target_sedona_type.extension_name == "geoarrow.wkb":
+            return _resolve_geometry_array(source_pyarrow_array, target_pyarrow_type)
+        else:
+            raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+    # The destination type gets to handle conversion
+    if target_sedona_type is None or target_sedona_type.extension_name is None:
+        return pa.array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "geoarrow.wkb":
+        return _resolve_geometry_array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "sedona.raster":
+        return _resolve_raster_array(values, target_pyarrow_type)
+    else:
+        raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+
+def _resolve_geometry_array(values, target_pyarrow_type):
+    """Resolve WKB storage or Shapely objects to the declared geometry type."""
+    import geoarrow.pyarrow as ga
+    import pyarrow as pa
+
+    # We can't quite use ga.array() or ga.as_geoarrow() here because of some bugs
+    # https://github.com/geoarrow/geoarrow-python/pull/84
+    if isinstance(values, pa.ExtensionArray):
+        if values.type.extension_name != "geoarrow.wkb":
+            raise TypeError(
+                f"Unsupported cast from {values.type} to {target_pyarrow_type}"
+            )
+
+        # Normalize storage to compare metadata using Sedona's CRS equality.
+        source_type = ga.wkb().with_metadata(values.type.__arrow_ext_serialize__())
+        target_type = ga.wkb().with_metadata(
+            target_pyarrow_type.__arrow_ext_serialize__()
+        )
+        if PySedonaType(source_type) != PySedonaType(target_type):
+            raise TypeError(
+                f"Can't cast geometry with different CRS or edge type: "
+                f"{values.type} to {target_pyarrow_type}"
+            )
+        storage = values.storage.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, pa.Array):
+        storage = values.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, list):
+        # For lists, we find the first non-null element and choose the conversion
+        # based on that. This doesn't support mixed types in the list; however,
+        # adds significant overhead (~2x slower) for lists of small objects like
+        # Shapely points. For now, we just support a single element type in a
+        # list.
+        first_value = next((value for value in values if value is not None), None)
+        qualified_name = _qualified_type_name(first_value)
+        if qualified_name in ("builtins.bytes", "builtins.NoneType"):
+            # Empty and all-null lists also convert directly to WKB storage.
+            wkb_storage = values
+        elif qualified_name in (
+            "geoarrow.pyarrow._scalar.WkbScalar",
+            "geoarrow.pyarrow._scalar.WktScalar",
+            "geoarrow.pyarrow._scalar.GeometryExtensionScalar",
+        ):
+            wkb_storage = (
+                None if value is None or not value.is_valid else value.wkb
+                for value in values
+            )
+        else:
+            import shapely
+
+            wkb_storage = shapely.to_wkb(values, flavor="iso")
+        storage = pa.array(wkb_storage, target_pyarrow_type.storage_type)
+    else:
+        raise TypeError(f"Can't create geometry array from {type(values).__name__}")
+
+    return target_pyarrow_type.wrap_array(storage)
+
+
+def _resolve_raster_array(values, target_pyarrow_type):
+    import pyarrow as pa
+
+    from sedonadb.raster import Raster
+
+    # The equal array type should already have been handled, this is just
+    # from a list of Raster (or None) objects.
+    if not isinstance(values, list):
+        raise TypeError("Expected a list of Raster or None objects")
+
+    storage_type = target_pyarrow_type.storage_type
+    arrays = []
+    for value in values:
+        if value is None:
+            arrays.append(pa.nulls(1, storage_type))
+        elif isinstance(value, Raster):
+            arrays.append(pa.array(value).storage)
+        else:
+            raise TypeError("Expected a list of Raster or None objects")
+
+    storage = pa.concat_arrays(arrays) if arrays else pa.nulls(0, storage_type)

Review Comment:
   I'm seeing the buffer-reference list grow a lot here when a UDF returns its input raster unchanged. Each row slice retains the input batch's buffer list, and concatenating those slices repeats the references. With just 32 rasters, three passes produce **32 → 1,024 → 32,768 → 1,048,576 references**. The pixel payloads remain shared; it's the reference/export/destruction work that grows.
   
   This reproduces the adapter round trip directly:
   
   ```python
   import numpy as np
   import pyarrow as pa
   from sedonadb.raster import Raster
   from sedonadb.udf import _python_values_from_raster
   from sedonadb.expr.literal import _resolve_raster_array
   from sedonadb.utility import register_pyarrow_extension_types
   
   register_pyarrow_extension_types()
   a = pa.concat_arrays([
       pa.array(Raster.from_numpy(np.full((32, 32), i, dtype=np.uint8)))
       for i in range(32)
   ])
   for depth in range(4):
       data = a.storage.field('bands').values.field('data')
       print(depth, len(data.buffers()) - 2)
       if depth < 3:
           a = _resolve_raster_array(list(_python_values_from_raster(a)), a.type)
   # 0 32
   # 1 1024
   # 2 32768
   # 3 1048576
   ```
   
   I also timed the actual query `SELECT identity(identity(identity(r))) FROM inputs`, where `identity(r)` just returns `r`, over 32 distinct 32×32 rasters. It took **about 456 ms**; a prototype that kept the source batch/row information and reused unchanged slices took **1.8 ms**, with matching output. Those are seven-run medians on `0845a640`, release build, M1 Max, CPython 3.13.7/PyArrow 21.0.0, one partition, batch size 8192, no input I/O. This is a deliberately small passthrough case, so I wouldn't generalize that speedup to other UDFs.
   
   Could we preserve the source slice for unchanged returns, and compact/remap the referenced buffers for mixed or reordered results? I'm okay with a follow-up PR if you'd prefer to tackle the adapter optimizations together, Dewey.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#issuecomment-6485485298

   Are there any further issues here before we merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4409285923


##########
python/sedonadb/python/sedonadb/udf.py:
##########
@@ -246,49 +265,210 @@ def arrow_udf(
     """
 
     def decorator(func):
+        from sedonadb.expr.literal import _resolve_arrow_array
+
         kwarg_names = _callable_kwarg_only_names(func)
         if "return_type" in kwarg_names and "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type, num_rows=num_rows)
+                result = func(*args, return_type=return_type, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         elif "return_type" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type)
+                result = func(*args, return_type=return_type)
+                return _resolve_arrow_array(result, return_type)
         elif "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, num_rows=num_rows)
+                result = func(*args, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         else:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args)
+                return _resolve_arrow_array(func(*args), return_type)
+
+        name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
+        return ScalarUdfImpl(
+            func_wrapper, return_type, input_types, volatility, name_arg, metadata
+        )
+
+    return decorator
+
+
+def python_udf(
+    return_type: Any,
+    input_types: Optional[list[Union[TypeMatcher, Any]]] = None,
+    volatility: Literal["immutable", "stable", "volatile"] = "immutable",
+    name: Optional[str] = None,
+    metadata: Optional[Mapping[str, str]] = None,
+):
+    """Python object-based user-defined scalar function decorator.
+
+    The decorated function is called once per row with ordinary Python objects.
+    Input values are converted lazily using PyArrow's scalar conversion.
+    Geometry/geography values are passed as GeoArrow scalars (or `None`).
+    Raster values are passed as [`Raster`][sedonadb.raster.Raster]
+    objects that reference the input Arrow buffers instead of materializing the
+    raster's inline pixel bytes as a Python `bytes` object.
+
+    Return conversion is selected by the declared Arrow type. Most values use
+    `pyarrow.array(values, type=return_type)` directly. Geometry/geography
+    results additionally accept GeoArrow scalars and Shapely objects, and raster
+    results preserve the Arrow-backed representation used by `sd.lit()`.
+
+    Geometry/geography results must use a consistent representation: WKB bytes,
+    GeoArrow scalars, or Shapely objects, optionally mixed with `None`. Conversion
+    is selected from the first non-null result in each batch; mixing these
+    representations within a batch is unsupported. This avoids per-row type
+    dispatch and preserves vectorized conversion, which is substantially faster
+    for small geometries such as Shapely points.
+
+    Args:
+        return_type: The return data type, or a callable return type resolver.
+        input_types: The accepted input data types or type matchers. `None`
+            accepts any number of arguments of any type.
+        volatility: `"immutable"` (default), `"stable"`, or `"volatile"`.
+        name: An optional SQL name. By default the decorated function's name is
+            used.
+        metadata: Optional key/value information used by optimizer rules.
+
+    Examples:
+
+        >>> import pyarrow as pa
+        >>> from sedonadb import udf
+        >>> @udf.python_udf(pa.int64(), [udf.STRING])
+        ... def char_count(value):
+        ...     return None if value is None else len(value)
+        ...
+        >>> sd = sedona.db.connect()
+        >>> sd.register(char_count)
+        >>> sd.sql("SELECT char_count('abcde') AS col").show()
+        ┌───────┐
+        │  col  │
+        │ int64 │
+        ╞═══════╡
+        │     5 │
+        └───────┘
+
+        Geometry results can be returned as Shapely objects by declaring a
+        geometry return type.
+
+        >>> import shapely
+        >>> @udf.python_udf(udf.GEOMETRY, [udf.NUMERIC, udf.NUMERIC])
+        ... def make_point(x, y):
+        ...     if x is None or y is None:
+        ...         return None
+        ...     return shapely.Point(x, y)
+        ...
+        >>> sd.register(make_point)
+        >>> sd.sql("SELECT make_point(1.0, 2.0) AS col").show()
+        ┌────────────┐
+        │     col    │
+        │  geometry  │
+        ╞════════════╡
+        │ POINT(1 2) │
+        └────────────┘
+
+        Raster arguments arrive as [`Raster`][sedonadb.raster.Raster] objects.
+        Set `needs_pixels` when reading pixels so that externally stored
+        raster data is loaded before the UDF runs. This example averages all
+        pixels and bands without excluding nodata values; the sample raster
+        contains a single band filled with zeros.
+
+        >>> @udf.python_udf(
+        ...     pa.float64(), [udf.RASTER], metadata={"needs_pixels": "true"}
+        ... )
+        ... def pixel_mean(value):
+        ...     if value is None:
+        ...         return None
+        ...     return float(value.to_numpy().mean())
+        ...
+        >>> sd.register(pixel_mean)
+        >>> sd.sql(
+        ...     "SELECT pixel_mean("
+        ...     "RS_MakeEmptyRaster(1, 'B', 3, 2, 0.0, 0.0, 1.0)"
+        ...     ") AS col"
+        ... ).show()
+        ┌─────────┐
+        │   col   │
+        │ float64 │
+        ╞═════════╡
+        │     0.0 │
+        └─────────┘
+    """
+
+    def decorator(func):
+        def invoke_batch(args, resolved_return_type, num_rows):
+            columns = [_python_udf_values(arg, num_rows) for arg in args]
+            if columns:
+                results = [func(*row) for row in zip(*columns)]
+            else:
+                results = [func() for _ in range(num_rows)]
+
+            return _resolve_arrow_array(results, resolved_return_type)
 
         name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
         return ScalarUdfImpl(
-            func_wrapper, return_type, input_types, volatility, name_arg
+            invoke_batch, return_type, input_types, volatility, name_arg, metadata
         )
 
     return decorator
 
 
-BINARY: TypeMatcher = "binary"
+def _python_udf_values(value, num_rows):

Review Comment:
   @paleolimbot, is the geometry case below on your radar? I can still reproduce it on merged main (`3108698d`), although the raster case passes now.
   
   Run this in a fresh Python process with GeoArrow installed but not imported:
   
   ```python
   import pyarrow as pa
   import sedonadb
   from sedonadb import udf
   
   sd = sedonadb.connect()
   
   @udf.python_udf(pa.int64(), [udf.GEOMETRY], volatility="volatile")
   def wkb_length(value):
       return len(value.wkb)
   
   sd.register(wkb_length)
   sd.sql("SELECT wkb_length(ST_Point(1, 2))").show()
   ```
   
   I get `AttributeError: 'pyarrow.lib.BinaryScalar' object has no attribute 'wkb'`. `.execute()` fails too. Importing `geoarrow.pyarrow` first, or using `.to_arrow_table()` instead, makes it pass.
   
   It looks like retaining `PySedonaType` fixes dispatch, but `pa.array(value)` still produces plain binary scalars before the extension is registered. Could we register or reconstruct the geometry type before passing values to the callback, so this doesn't depend on earlier imports or how the result is collected?
   



##########
python/sedonadb/python/sedonadb/expr/literal.py:
##########
@@ -147,6 +148,149 @@ def _resolve_arrow_lit(obj: Any):
         ) from e
 
 
+def _resolve_arrow_array(values, type):
+    """Resolve Python values to an array, with extension-specific handling.
+
+    This is the array counterpart of `_resolve_arrow_lit`, spiritually equivalent
+    to `_resolve_arrow_lit` except (1) arrays that are resolved do not have to have
+    a length of 1 and (2) iterables resolve to their element form (e.g., lists) instead
+    of as a list of elements.
+
+    This is currently limited to UDF handling, where there is always a target type
+    available (the return type, which has already been calculated), and the values
+    list is always a Python list, and arrow arrays because they are relatively
+    straightforward.
+    """
+    import pyarrow as pa
+
+    register_pyarrow_extension_types()
+
+    if type is None:
+        raise ValueError(
+            "Resolving an array without a target type is not yet supported"
+        )
+
+    target_sedona_type = PySedonaType(type)
+    target_pyarrow_type = pa.DataType._import_from_c_capsule(
+        target_sedona_type.__arrow_c_schema__()
+    )
+
+    # Special case arrow arrays first
+    if hasattr(values, "__arrow_c_array__"):
+        schema, array = values.__arrow_c_array__()
+        source_sedona_type = PySedonaType(schema)
+
+        # Use == on the PySedonaType to handle geometry type CRS differences.
+        if target_sedona_type == source_sedona_type:
+            return pa.Array._import_from_c_capsule(
+                target_sedona_type.__arrow_c_schema__(), array
+            )
+
+        source_pyarrow_array = pa.Array._import_from_c_capsule(
+            source_sedona_type.__arrow_c_schema__(), array
+        )
+
+        # Use SedonaType's extension name to avoid pyarrow's stripping of
+        # unrecognized extensions
+        if target_sedona_type.extension_name is None:
+            return source_pyarrow_array.cast(target_pyarrow_type)
+        elif target_sedona_type.extension_name == "geoarrow.wkb":
+            return _resolve_geometry_array(source_pyarrow_array, target_pyarrow_type)
+        else:
+            raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+    # The destination type gets to handle conversion
+    if target_sedona_type is None or target_sedona_type.extension_name is None:
+        return pa.array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "geoarrow.wkb":
+        return _resolve_geometry_array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "sedona.raster":
+        return _resolve_raster_array(values, target_pyarrow_type)
+    else:
+        raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+
+def _resolve_geometry_array(values, target_pyarrow_type):
+    """Resolve WKB storage or Shapely objects to the declared geometry type."""
+    import geoarrow.pyarrow as ga
+    import pyarrow as pa
+
+    # We can't quite use ga.array() or ga.as_geoarrow() here because of some bugs
+    # https://github.com/geoarrow/geoarrow-python/pull/84
+    if isinstance(values, pa.ExtensionArray):
+        if values.type.extension_name != "geoarrow.wkb":
+            raise TypeError(
+                f"Unsupported cast from {values.type} to {target_pyarrow_type}"
+            )
+
+        # Normalize storage to compare metadata using Sedona's CRS equality.
+        source_type = ga.wkb().with_metadata(values.type.__arrow_ext_serialize__())
+        target_type = ga.wkb().with_metadata(
+            target_pyarrow_type.__arrow_ext_serialize__()
+        )
+        if PySedonaType(source_type) != PySedonaType(target_type):
+            raise TypeError(
+                f"Can't cast geometry with different CRS or edge type: "
+                f"{values.type} to {target_pyarrow_type}"
+            )
+        storage = values.storage.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, pa.Array):
+        storage = values.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, list):
+        # For lists, we find the first non-null element and choose the conversion
+        # based on that. This doesn't support mixed types in the list; however,
+        # adds significant overhead (~2x slower) for lists of small objects like
+        # Shapely points. For now, we just support a single element type in a
+        # list.
+        first_value = next((value for value in values if value is not None), None)
+        qualified_name = _qualified_type_name(first_value)
+        if qualified_name in ("builtins.bytes", "builtins.NoneType"):
+            # Empty and all-null lists also convert directly to WKB storage.
+            wkb_storage = values
+        elif qualified_name in (
+            "geoarrow.pyarrow._scalar.WkbScalar",
+            "geoarrow.pyarrow._scalar.WktScalar",
+            "geoarrow.pyarrow._scalar.GeometryExtensionScalar",
+        ):
+            wkb_storage = (
+                None if value is None or not value.is_valid else value.wkb

Review Comment:
   @paleolimbot, are you aware of this scalar-return case too? On merged main (`3108698d`), a typed EPSG:3857 scalar returned from a UDF declared EPSG:4326 gets relabeled without a CRS check:
   
   ```python
   import geoarrow.pyarrow as ga
   import sedonadb
   from sedonadb import udf
   
   source = ga.wkb().with_crs("EPSG:3857").wrap_array(
       ga.as_wkb(["POINT (100000 200000)"]).storage
   )
   
   @udf.python_udf(ga.wkb().with_crs("EPSG:4326"), [])
   def mismatched_crs():
       return source[0]
   
   sd = sedonadb.connect()
   sd.register(mismatched_crs)
   print(sd.sql("""
       SELECT ST_SRID(mismatched_crs()) AS srid,
              ST_AsText(mismatched_crs()) AS wkt
   """).to_arrow_table().to_pylist())
   ```
   
   Actual: `[{'srid': 4326, 'wkt': 'POINT(100000 200000)'}]`.
   
   The coordinates are still in meters, but the result is labeled 4326. The equivalent typed-array return correctly raises `Can't cast geometry with different CRS or edge type`. Could we apply that same validation to typed scalar results before extracting `.wkb` here? I'd expect a mismatched return to be rejected, rather than silently relabeled. Matching-CRS returns work; this is specifically the conflicting metadata case.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4390939559


##########
python/sedonadb/python/sedonadb/udf.py:
##########
@@ -246,49 +265,210 @@ def arrow_udf(
     """
 
     def decorator(func):
+        from sedonadb.expr.literal import _resolve_arrow_array
+
         kwarg_names = _callable_kwarg_only_names(func)
         if "return_type" in kwarg_names and "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type, num_rows=num_rows)
+                result = func(*args, return_type=return_type, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         elif "return_type" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type)
+                result = func(*args, return_type=return_type)
+                return _resolve_arrow_array(result, return_type)
         elif "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, num_rows=num_rows)
+                result = func(*args, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         else:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args)
+                return _resolve_arrow_array(func(*args), return_type)
+
+        name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
+        return ScalarUdfImpl(
+            func_wrapper, return_type, input_types, volatility, name_arg, metadata
+        )
+
+    return decorator
+
+
+def python_udf(
+    return_type: Any,
+    input_types: Optional[list[Union[TypeMatcher, Any]]] = None,
+    volatility: Literal["immutable", "stable", "volatile"] = "immutable",
+    name: Optional[str] = None,
+    metadata: Optional[Mapping[str, str]] = None,
+):
+    """Python object-based user-defined scalar function decorator.
+
+    The decorated function is called once per row with ordinary Python objects.
+    Input values are converted lazily using PyArrow's scalar conversion.
+    Geometry/geography values are passed as GeoArrow scalars (or `None`).
+    Raster values are passed as [`Raster`][sedonadb.raster.Raster]
+    objects that reference the input Arrow buffers instead of materializing the
+    raster's inline pixel bytes as a Python `bytes` object.
+
+    Return conversion is selected by the declared Arrow type. Most values use
+    `pyarrow.array(values, type=return_type)` directly. Geometry/geography
+    results additionally accept GeoArrow scalars and Shapely objects, and raster
+    results preserve the Arrow-backed representation used by `sd.lit()`.
+
+    Geometry/geography results must use a consistent representation: WKB bytes,
+    GeoArrow scalars, or Shapely objects, optionally mixed with `None`. Conversion
+    is selected from the first non-null result in each batch; mixing these
+    representations within a batch is unsupported. This avoids per-row type
+    dispatch and preserves vectorized conversion, which is substantially faster
+    for small geometries such as Shapely points.
+
+    Args:
+        return_type: The return data type, or a callable return type resolver.
+        input_types: The accepted input data types or type matchers. `None`
+            accepts any number of arguments of any type.
+        volatility: `"immutable"` (default), `"stable"`, or `"volatile"`.
+        name: An optional SQL name. By default the decorated function's name is
+            used.
+        metadata: Optional key/value information used by optimizer rules.
+
+    Examples:
+
+        >>> import pyarrow as pa
+        >>> from sedonadb import udf
+        >>> @udf.python_udf(pa.int64(), [udf.STRING])
+        ... def char_count(value):
+        ...     return None if value is None else len(value)
+        ...
+        >>> sd = sedona.db.connect()
+        >>> sd.register(char_count)
+        >>> sd.sql("SELECT char_count('abcde') AS col").show()
+        ┌───────┐
+        │  col  │
+        │ int64 │
+        ╞═══════╡
+        │     5 │
+        └───────┘
+
+        Geometry results can be returned as Shapely objects by declaring a
+        geometry return type.
+
+        >>> import shapely
+        >>> @udf.python_udf(udf.GEOMETRY, [udf.NUMERIC, udf.NUMERIC])
+        ... def make_point(x, y):
+        ...     if x is None or y is None:
+        ...         return None
+        ...     return shapely.Point(x, y)
+        ...
+        >>> sd.register(make_point)
+        >>> sd.sql("SELECT make_point(1.0, 2.0) AS col").show()
+        ┌────────────┐
+        │     col    │
+        │  geometry  │
+        ╞════════════╡
+        │ POINT(1 2) │
+        └────────────┘
+
+        Raster arguments arrive as [`Raster`][sedonadb.raster.Raster] objects.
+        Set `needs_pixels` when reading pixels so that externally stored
+        raster data is loaded before the UDF runs. This example averages all
+        pixels and bands without excluding nodata values; the sample raster
+        contains a single band filled with zeros.
+
+        >>> @udf.python_udf(
+        ...     pa.float64(), [udf.RASTER], metadata={"needs_pixels": "true"}
+        ... )
+        ... def pixel_mean(value):
+        ...     if value is None:
+        ...         return None
+        ...     return float(value.to_numpy().mean())
+        ...
+        >>> sd.register(pixel_mean)
+        >>> sd.sql(
+        ...     "SELECT pixel_mean("
+        ...     "RS_MakeEmptyRaster(1, 'B', 3, 2, 0.0, 0.0, 1.0)"
+        ...     ") AS col"
+        ... ).show()
+        ┌─────────┐
+        │   col   │
+        │ float64 │
+        ╞═════════╡
+        │     0.0 │
+        └─────────┘
+    """
+
+    def decorator(func):
+        def invoke_batch(args, resolved_return_type, num_rows):
+            columns = [_python_udf_values(arg, num_rows) for arg in args]
+            if columns:
+                results = [func(*row) for row in zip(*columns)]
+            else:
+                results = [func() for _ in range(num_rows)]
+
+            return _resolve_arrow_array(results, resolved_return_type)
 
         name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
         return ScalarUdfImpl(
-            func_wrapper, return_type, input_types, volatility, name_arg
+            invoke_batch, return_type, input_types, volatility, name_arg, metadata
         )
 
     return decorator
 
 
-BINARY: TypeMatcher = "binary"
+def _python_udf_values(value, num_rows):

Review Comment:
   Are we assuming this will work only after the geoarrow.pyarrow is imported and sedona raster gets registered? 
   
   In a real UDF call, that might not be true, so risk in that the first batch would get wrong types of data?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4391070184


##########
python/sedonadb/python/sedonadb/udf.py:
##########
@@ -246,49 +265,210 @@ def arrow_udf(
     """
 
     def decorator(func):
+        from sedonadb.expr.literal import _resolve_arrow_array
+
         kwarg_names = _callable_kwarg_only_names(func)
         if "return_type" in kwarg_names and "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type, num_rows=num_rows)
+                result = func(*args, return_type=return_type, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         elif "return_type" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type)
+                result = func(*args, return_type=return_type)
+                return _resolve_arrow_array(result, return_type)
         elif "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, num_rows=num_rows)
+                result = func(*args, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         else:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args)
+                return _resolve_arrow_array(func(*args), return_type)
+
+        name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
+        return ScalarUdfImpl(
+            func_wrapper, return_type, input_types, volatility, name_arg, metadata
+        )
+
+    return decorator
+
+
+def python_udf(
+    return_type: Any,
+    input_types: Optional[list[Union[TypeMatcher, Any]]] = None,
+    volatility: Literal["immutable", "stable", "volatile"] = "immutable",
+    name: Optional[str] = None,
+    metadata: Optional[Mapping[str, str]] = None,
+):
+    """Python object-based user-defined scalar function decorator.
+
+    The decorated function is called once per row with ordinary Python objects.
+    Input values are converted lazily using PyArrow's scalar conversion.
+    Geometry/geography values are passed as GeoArrow scalars (or `None`).
+    Raster values are passed as [`Raster`][sedonadb.raster.Raster]
+    objects that reference the input Arrow buffers instead of materializing the
+    raster's inline pixel bytes as a Python `bytes` object.
+
+    Return conversion is selected by the declared Arrow type. Most values use
+    `pyarrow.array(values, type=return_type)` directly. Geometry/geography
+    results additionally accept GeoArrow scalars and Shapely objects, and raster
+    results preserve the Arrow-backed representation used by `sd.lit()`.
+
+    Geometry/geography results must use a consistent representation: WKB bytes,
+    GeoArrow scalars, or Shapely objects, optionally mixed with `None`. Conversion
+    is selected from the first non-null result in each batch; mixing these
+    representations within a batch is unsupported. This avoids per-row type
+    dispatch and preserves vectorized conversion, which is substantially faster
+    for small geometries such as Shapely points.
+
+    Args:
+        return_type: The return data type, or a callable return type resolver.
+        input_types: The accepted input data types or type matchers. `None`
+            accepts any number of arguments of any type.
+        volatility: `"immutable"` (default), `"stable"`, or `"volatile"`.
+        name: An optional SQL name. By default the decorated function's name is
+            used.
+        metadata: Optional key/value information used by optimizer rules.
+
+    Examples:
+
+        >>> import pyarrow as pa
+        >>> from sedonadb import udf
+        >>> @udf.python_udf(pa.int64(), [udf.STRING])
+        ... def char_count(value):
+        ...     return None if value is None else len(value)
+        ...
+        >>> sd = sedona.db.connect()
+        >>> sd.register(char_count)
+        >>> sd.sql("SELECT char_count('abcde') AS col").show()
+        ┌───────┐
+        │  col  │
+        │ int64 │
+        ╞═══════╡
+        │     5 │
+        └───────┘
+
+        Geometry results can be returned as Shapely objects by declaring a
+        geometry return type.
+
+        >>> import shapely
+        >>> @udf.python_udf(udf.GEOMETRY, [udf.NUMERIC, udf.NUMERIC])
+        ... def make_point(x, y):
+        ...     if x is None or y is None:
+        ...         return None
+        ...     return shapely.Point(x, y)
+        ...
+        >>> sd.register(make_point)
+        >>> sd.sql("SELECT make_point(1.0, 2.0) AS col").show()
+        ┌────────────┐
+        │     col    │
+        │  geometry  │
+        ╞════════════╡
+        │ POINT(1 2) │
+        └────────────┘
+
+        Raster arguments arrive as [`Raster`][sedonadb.raster.Raster] objects.
+        Set `needs_pixels` when reading pixels so that externally stored
+        raster data is loaded before the UDF runs. This example averages all
+        pixels and bands without excluding nodata values; the sample raster
+        contains a single band filled with zeros.
+
+        >>> @udf.python_udf(
+        ...     pa.float64(), [udf.RASTER], metadata={"needs_pixels": "true"}
+        ... )
+        ... def pixel_mean(value):
+        ...     if value is None:
+        ...         return None
+        ...     return float(value.to_numpy().mean())
+        ...
+        >>> sd.register(pixel_mean)
+        >>> sd.sql(
+        ...     "SELECT pixel_mean("
+        ...     "RS_MakeEmptyRaster(1, 'B', 3, 2, 0.0, 0.0, 1.0)"
+        ...     ") AS col"
+        ... ).show()
+        ┌─────────┐
+        │   col   │
+        │ float64 │
+        ╞═════════╡
+        │     0.0 │
+        └─────────┘
+    """
+
+    def decorator(func):
+        def invoke_batch(args, resolved_return_type, num_rows):
+            columns = [_python_udf_values(arg, num_rows) for arg in args]
+            if columns:
+                results = [func(*row) for row in zip(*columns)]
+            else:
+                results = [func() for _ in range(num_rows)]
+
+            return _resolve_arrow_array(results, resolved_return_type)
 
         name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
         return ScalarUdfImpl(
-            func_wrapper, return_type, input_types, volatility, name_arg
+            invoke_batch, return_type, input_types, volatility, name_arg, metadata
         )
 
     return decorator
 
 
-BINARY: TypeMatcher = "binary"
+def _python_udf_values(value, num_rows):

Review Comment:
   Good call! I updated this to pass the `PySedonaType` all the way through which should help with this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "zhangfengcdt (via GitHub)" <gi...@apache.org>.
zhangfengcdt commented on PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#issuecomment-6485562244

   > Are there any further issues here before we merge?
   
   Looks good to me @paleolimbot 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on code in PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466#discussion_r4381716690


##########
python/sedonadb/python/sedonadb/udf.py:
##########
@@ -246,49 +265,215 @@ def arrow_udf(
     """
 
     def decorator(func):
+        from sedonadb.expr.literal import _resolve_arrow_array
+
         kwarg_names = _callable_kwarg_only_names(func)
         if "return_type" in kwarg_names and "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type, num_rows=num_rows)
+                result = func(*args, return_type=return_type, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         elif "return_type" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, return_type=return_type)
+                result = func(*args, return_type=return_type)
+                return _resolve_arrow_array(result, return_type)
         elif "num_rows" in kwarg_names:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args, num_rows=num_rows)
+                result = func(*args, num_rows=num_rows)
+                return _resolve_arrow_array(result, return_type)
         else:
 
             def func_wrapper(args, return_type, num_rows):
-                return func(*args)
+                return _resolve_arrow_array(func(*args), return_type)
 
         name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
         return ScalarUdfImpl(
-            func_wrapper, return_type, input_types, volatility, name_arg
+            func_wrapper, return_type, input_types, volatility, name_arg, metadata
         )
 
     return decorator
 
 
-BINARY: TypeMatcher = "binary"
+def python_udf(
+    return_type: Any,
+    input_types: Optional[list[Union[TypeMatcher, Any]]] = None,
+    volatility: Literal["immutable", "stable", "volatile"] = "immutable",
+    name: Optional[str] = None,
+    metadata: Optional[Mapping[str, str]] = None,
+):
+    """Python object-based user-defined scalar function decorator.
+
+    The decorated function is called once per row with ordinary Python objects.
+    Input values are converted lazily using PyArrow's scalar conversion.
+    Geometry/geography values are passed as GeoArrow scalars (or `None`).
+    Raster values are passed as [`Raster`][sedonadb.raster.Raster]
+    objects that reference the input Arrow buffers instead of materializing the
+    raster's inline pixel bytes as a Python `bytes` object.
+
+    Return conversion is selected by the declared Arrow type. Most values use
+    `pyarrow.array(values, type=return_type)` directly. Geometry/geography
+    results additionally accept GeoArrow scalars and Shapely objects, and raster
+    results preserve the Arrow-backed representation used by `sd.lit()`.
+
+    Geometry/geography results must use a consistent representation: WKB bytes,
+    GeoArrow scalars, or Shapely objects, optionally mixed with `None`. Conversion
+    is selected from the first non-null result in each batch; mixing these
+    representations within a batch is unsupported. This avoids per-row type
+    dispatch and preserves vectorized conversion, which is substantially faster
+    for small geometries such as Shapely points.
+
+    Args:
+        return_type: The return data type, or a callable return type resolver.
+        input_types: The accepted input data types or type matchers. `None`
+            accepts any number of arguments of any type.
+        volatility: `"immutable"` (default), `"stable"`, or `"volatile"`.
+        name: An optional SQL name. By default the decorated function's name is
+            used.
+        metadata: Optional key/value information used by optimizer rules.
+
+    Examples:
+
+        >>> import pyarrow as pa
+        >>> from sedonadb import udf
+        >>> @udf.python_udf(pa.int64(), [udf.STRING])
+        ... def char_count(value):
+        ...     return None if value is None else len(value)
+        ...
+        >>> sd = sedona.db.connect()
+        >>> sd.register(char_count)
+        >>> sd.sql("SELECT char_count('abcde') AS col").show()
+        ┌───────┐
+        │  col  │
+        │ int64 │
+        ╞═══════╡
+        │     5 │
+        └───────┘
+
+        Geometry results can be returned as Shapely objects by declaring a
+        geometry return type.
+
+        >>> import shapely
+        >>> @udf.python_udf(udf.GEOMETRY, [udf.NUMERIC, udf.NUMERIC])
+        ... def make_point(x, y):
+        ...     if x is None or y is None:
+        ...         return None
+        ...     return shapely.Point(x, y)
+        ...
+        >>> sd.register(make_point)
+        >>> sd.sql("SELECT make_point(1.0, 2.0) AS col").show()
+        ┌────────────┐
+        │     col    │
+        │  geometry  │
+        ╞════════════╡
+        │ POINT(1 2) │
+        └────────────┘
+
+        Raster arguments arrive as [`Raster`][sedonadb.raster.Raster] objects.
+        Set `needs_pixels` when reading pixels so that externally stored
+        raster data is loaded before the UDF runs. This example averages all
+        pixels and bands without excluding nodata values; the sample raster
+        contains a single band filled with zeros.
+
+        >>> @udf.python_udf(
+        ...     pa.float64(), [udf.RASTER], metadata={"needs_pixels": "true"}
+        ... )
+        ... def pixel_mean(value):
+        ...     if value is None:
+        ...         return None
+        ...     return float(value.to_numpy().mean())
+        ...
+        >>> sd.register(pixel_mean)
+        >>> sd.sql(
+        ...     "SELECT pixel_mean("
+        ...     "RS_MakeEmptyRaster(1, 'B', 3, 2, 0.0, 0.0, 1.0)"
+        ...     ") AS col"
+        ... ).show()
+        ┌─────────┐
+        │   col   │
+        │ float64 │
+        ╞═════════╡
+        │     0.0 │
+        └─────────┘
+    """
+
+    def decorator(func):
+        def invoke_batch(args, resolved_return_type, num_rows):
+            columns = [_python_udf_values(arg, num_rows) for arg in args]
+            if columns:
+                results = [func(*row) for row in zip(*columns)]
+            else:
+                results = [func() for _ in range(num_rows)]
+
+            return _resolve_arrow_array(results, resolved_return_type)
+
+        name_arg = func.__name__ if name is None and hasattr(func, "__name__") else name
+        return ScalarUdfImpl(
+            invoke_batch, return_type, input_types, volatility, name_arg, metadata
+        )
+
+    return decorator
+
+
+def _python_udf_values(value, num_rows):
+    """Convert an input once, reusing a scalar's Python object for every row."""
+    import pyarrow as pa
+
+    type = pa.field(value.type).type
+    array = pa.array(value)
+    values = _python_values_from_arrow(array, type)
+    if value.is_scalar():
+        return itertools.repeat(next(values), num_rows)
+    else:
+        return values
+
+
+def _python_values_from_arrow(array, type):
+    """Convert an Arrow array according to its logical type."""
+    extension_name = getattr(type, "extension_name", None)
+
+    # Special case raster because its default to_pylist() will copy
+    # raster buffers under current pyarrow
+    if extension_name == "sedona.raster":
+        return _python_values_from_raster(array)
+    elif extension_name == "geoarrow.wkb":
+        return (scalar if scalar.is_valid else None for scalar in array)
+    else:
+        return (scalar.as_py() for scalar in array)
+
+
+def _python_values_from_raster(array):
+    import pyarrow as pa
+
+    from sedonadb.raster import Raster
+
+    storage = array.storage if isinstance(array, pa.ExtensionArray) else array
+    return (
+        None if not array[i].is_valid else Raster(storage, i) for i in range(len(array))

Review Comment:
   Could we cache the band fields and pixel-buffer lookup once for this batch, and let the row wrappers share them? When each row calls `r.bands[0].to_numpy()`, `_get_binary_view_buffer()` calls `.buffers()` again on an array that still carries the batch's buffer list. The pixels are shared, but the lookup/wrapper work repeats.
   
   I profiled a small NDVI example with 1,024 distinct 128×128 UInt16 red/NIR pairs. Caching the input band fields and buffers brought the median query time from **about 824 ms to 335 ms**, without changing the NumPy arithmetic or output construction. Input band/pixel access accounted for about 43% of the instrumented callback trace (that percentage includes tracing overhead).
   
   This was on `0845a640`, release build, M1 Max, CPython 3.13.7 with the GIL, NumPy 2.3.3 and PyArrow 21.0.0; one partition, batch size 8192, seven warmed-up runs with inputs already in memory. The baseline script below also ran locally at **817 ms**. Absolute timings vary, but this seems worth addressing for smaller tiles.
   
   <details>
   <summary>Baseline reproduction (no external data needed)</summary>
   
   ```python
   import gc
   from statistics import median
   from time import perf_counter
   import numpy as np
   import pyarrow as pa
   import sedonadb
   from sedonadb import udf
   from sedonadb.raster import Raster
   from sedonadb.utility import register_pyarrow_extension_types
   
   register_pyarrow_extension_types()
   sd = sedonadb.connect()
   sd.sql("SET datafusion.execution.target_partitions = 1").execute()
   sd.sql("SET datafusion.execution.batch_size = 8192").execute()
   
   red_rows, nir_rows = [], []
   for i in range(1024):
       rng = np.random.default_rng(np.random.SeedSequence([42, 128, i]))
       red = rng.integers(1, 6001, (128, 128), dtype=np.uint16)
       nir = np.clip(
           red.astype(np.int32)
           + rng.integers(-1000, 4001, red.shape, dtype=np.int32),
           1, 10000,
       ).astype(np.uint16)
       metadata = dict(
           crs="EPSG:32631",
           transform=[600000.0 + i * 10, 10, 0, 5800000.0, 0, -10],
       )
       red_rows.append(pa.array(Raster.from_numpy(red, **metadata)))
       nir_rows.append(pa.array(Raster.from_numpy(nir, **metadata)))
   sd.create_data_frame(pa.table({
       "red": pa.concat_arrays(red_rows),
       "nir": pa.concat_arrays(nir_rows),
   })).to_view("inputs")
   
   @udf.python_udf(
       udf.RASTER, [udf.RASTER, udf.RASTER],
       metadata={"needs_pixels": "true", "returns_bytes": "true"},
   )
   def ndvi(red, nir):
       r, n = red.bands[0].to_numpy(), nir.bands[0].to_numpy()
       denominator = np.add(n, r, dtype=np.float32)
       result = np.subtract(n, r, dtype=np.float32)
       invalid = denominator == 0
       with np.errstate(divide="ignore", invalid="ignore"):
           np.divide(result, denominator, out=result)
       result[invalid] = np.nan
       return Raster.from_numpy(
           result, crs=red.crs, transform=red.transform, nodata=np.nan
       )
   
   sd.register(ndvi)
   query = sd.sql("SELECT ndvi(red, nir) AS r FROM inputs")
   assert query.execute() == 1024  # warm up
   times = []
   for _ in range(7):
       gc.collect()
       start = perf_counter()
       assert query.execute() == 1024
       times.append(perf_counter() - start)
   print(f"median: {1000 * median(times):.1f} ms")
   ```
   
   The 335 ms result is from a local prototype sharing the field arrays and the result of `data.buffers()` across the batch, then resolving each row's BinaryView descriptor into a NumPy view.
   </details>
   
   Fine to put this optimization in a follow-up PR if you'd prefer, Dewey.



##########
python/sedonadb/python/sedonadb/expr/literal.py:
##########
@@ -147,6 +148,149 @@ def _resolve_arrow_lit(obj: Any):
         ) from e
 
 
+def _resolve_arrow_array(values, type):
+    """Resolve Python values to an array, with extension-specific handling.
+
+    This is the array counterpart of `_resolve_arrow_lit`, spiritually equivalent
+    to `_resolve_arrow_lit` except (1) arrays that are resolved do not have to have
+    a length of 1 and (2) iterables resolve to their element form (e.g., lists) instead
+    of as a list of elements.
+
+    This is currently limited to UDF handling, where there is always a target type
+    available (the return type, which has already been calculated), and the values
+    list is always a Python list, and arrow arrays because they are relatively
+    straightforward.
+    """
+    import pyarrow as pa
+
+    register_pyarrow_extension_types()
+
+    if type is None:
+        raise ValueError(
+            "Resolving an array without a target type is not yet supported"
+        )
+
+    target_sedona_type = PySedonaType(type)
+    target_pyarrow_type = pa.DataType._import_from_c_capsule(
+        target_sedona_type.__arrow_c_schema__()
+    )
+
+    # Special case arrow arrays first
+    if hasattr(values, "__arrow_c_array__"):
+        schema, array = values.__arrow_c_array__()
+        source_sedona_type = PySedonaType(schema)
+
+        # Use == on the PySedonaType to handle geometry type CRS differences.
+        if target_sedona_type == source_sedona_type:
+            return pa.Array._import_from_c_capsule(
+                target_sedona_type.__arrow_c_schema__(), array
+            )
+
+        source_pyarrow_array = pa.Array._import_from_c_capsule(
+            source_sedona_type.__arrow_c_schema__(), array
+        )
+
+        # Use SedonaType's extension name to avoid pyarrow's stripping of
+        # unrecognized extensions
+        if target_sedona_type.extension_name is None:
+            return source_pyarrow_array.cast(target_pyarrow_type)
+        elif target_sedona_type.extension_name == "geoarrow.wkb":
+            return _resolve_geometry_array(source_pyarrow_array, target_pyarrow_type)
+        else:
+            raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+    # The destination type gets to handle conversion
+    if target_sedona_type is None or target_sedona_type.extension_name is None:
+        return pa.array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "geoarrow.wkb":
+        return _resolve_geometry_array(values, target_pyarrow_type)
+    elif target_sedona_type.extension_name == "sedona.raster":
+        return _resolve_raster_array(values, target_pyarrow_type)
+    else:
+        raise TypeError(f"Unsupported cast to extension {target_sedona_type!r}")
+
+
+def _resolve_geometry_array(values, target_pyarrow_type):
+    """Resolve WKB storage or Shapely objects to the declared geometry type."""
+    import geoarrow.pyarrow as ga
+    import pyarrow as pa
+
+    # We can't quite use ga.array() or ga.as_geoarrow() here because of some bugs
+    # https://github.com/geoarrow/geoarrow-python/pull/84
+    if isinstance(values, pa.ExtensionArray):
+        if values.type.extension_name != "geoarrow.wkb":
+            raise TypeError(
+                f"Unsupported cast from {values.type} to {target_pyarrow_type}"
+            )
+
+        # Normalize storage to compare metadata using Sedona's CRS equality.
+        source_type = ga.wkb().with_metadata(values.type.__arrow_ext_serialize__())
+        target_type = ga.wkb().with_metadata(
+            target_pyarrow_type.__arrow_ext_serialize__()
+        )
+        if PySedonaType(source_type) != PySedonaType(target_type):
+            raise TypeError(
+                f"Can't cast geometry with different CRS or edge type: "
+                f"{values.type} to {target_pyarrow_type}"
+            )
+        storage = values.storage.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, pa.Array):
+        storage = values.cast(target_pyarrow_type.storage_type)
+    elif isinstance(values, list):
+        # For lists, we find the first non-null element and choose the conversion
+        # based on that. This doesn't support mixed types in the list; however,
+        # adds significant overhead (~2x slower) for lists of small objects like
+        # Shapely points. For now, we just support a single element type in a
+        # list.
+        first_value = next((value for value in values if value is not None), None)
+        qualified_name = _qualified_type_name(first_value)
+        if qualified_name in ("builtins.bytes", "builtins.NoneType"):
+            # Empty and all-null lists also convert directly to WKB storage.
+            wkb_storage = values
+        elif qualified_name in (
+            "geoarrow.pyarrow._scalar.WkbScalar",
+            "geoarrow.pyarrow._scalar.WktScalar",
+            "geoarrow.pyarrow._scalar.GeometryExtensionScalar",
+        ):
+            wkb_storage = (
+                None if value is None or not value.is_valid else value.wkb
+                for value in values
+            )
+        else:
+            import shapely
+
+            wkb_storage = shapely.to_wkb(values, flavor="iso")
+        storage = pa.array(wkb_storage, target_pyarrow_type.storage_type)
+    else:
+        raise TypeError(f"Can't create geometry array from {type(values).__name__}")
+
+    return target_pyarrow_type.wrap_array(storage)
+
+
+def _resolve_raster_array(values, target_pyarrow_type):
+    import pyarrow as pa
+
+    from sedonadb.raster import Raster
+
+    # The equal array type should already have been handled, this is just
+    # from a list of Raster (or None) objects.
+    if not isinstance(values, list):
+        raise TypeError("Expected a list of Raster or None objects")
+
+    storage_type = target_pyarrow_type.storage_type
+    arrays = []
+    for value in values:
+        if value is None:
+            arrays.append(pa.nulls(1, storage_type))
+        elif isinstance(value, Raster):
+            arrays.append(pa.array(value).storage)

Review Comment:
   Could canonical `Raster` results use their existing Arrow storage here? `pa.array(value)` goes through the C export/import path again for each row, even when we already have the right storage type.
   
   The NDVI example in [my input-wrapper comment](https://github.com/apache/sedona-db/pull/1466#discussion_r4381716690) also rebuilds the CRS/band metadata through `Raster.from_numpy()` for every output. That constructor predates this PR, but the new per-row UDF path makes the cost noticeable. A lighter way to return “these pixels, with this input raster's metadata” might help.
   
   For the same 1,024 × 128×128 workload, the local seven-run medians were:
   
   | Change | Query time |
   | --- | ---: |
   | Current path | 824 ms |
   | Avoid just the per-result C export/import | 749 ms |
   | Reuse output metadata only | 577 ms |
   | Shared input lookup + metadata reuse + direct returns | 114 ms |
   
   The last result combines all three changes; it isn't the gain from this line alone. The NumPy math stayed the same, and output pixels, CRS, transform, dimensions and nodata matched. Same release/M1 Max/single-partition setup as the input comment. The metadata prototype only handles this single-band Float32/NaN output, so it would need generalizing.
   
   Here's a small check of the direct-storage idea for a fresh raster and a null (with the appropriate type checks/fallback still needed in the real implementation):
   
   ```python
   import numpy as np
   import pyarrow as pa
   from sedonadb.raster import Raster, RasterType
   from sedonadb.expr.literal import _resolve_raster_array
   from sedonadb.utility import register_pyarrow_extension_types
   
   register_pyarrow_extension_types()
   values = [Raster.from_numpy(np.ones((128, 128), dtype=np.float32)), None]
   target = RasterType()
   current = _resolve_raster_array(values, target)
   direct = target.wrap_array(pa.concat_arrays([
       pa.nulls(1, target.storage_type) if r is None else r._array
       for r in values
   ]))
   assert current.equals(direct)
   print('fresh raster/null direct-storage result matched')
   ```
   
   Happy for this and the metadata work to go into a follow-up PR if you want to keep this PR focused.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python/sedonadb): Add item-wise Python UDF [sedona-db]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #1466:
URL: https://github.com/apache/sedona-db/pull/1466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org