You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/04/03 07:05:50 UTC

(spark) branch master updated: [SPARK-55902][PYTHON] Refactor SQL_ARROW_BATCHED_UDF

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new e1305252f122 [SPARK-55902][PYTHON] Refactor SQL_ARROW_BATCHED_UDF
e1305252f122 is described below

commit e1305252f12211d85674ea10d473e873d13b1a9c
Author: Yicong-Huang <17...@users.noreply.github.com>
AuthorDate: Fri Apr 3 15:05:28 2026 +0800

    [SPARK-55902][PYTHON] Refactor SQL_ARROW_BATCHED_UDF
    
    ### What changes were proposed in this pull request?
    
    Refactor `SQL_ARROW_BATCHED_UDF` (non-legacy path) to use `ArrowStreamSerializer` as pure I/O, moving Arrow-to-Python and Python-to-Arrow conversion logic from `ArrowBatchUDFSerializer` into `read_udfs()` in `worker.py`.
    
    ### Why are the changes needed?
    
    Part of SPARK-55388.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    `COLUMNS=120 asv run --python=same --bench "ArrowBatched" --attribute "repeat=(3,5,5.0)"` before and after:
    
    **ArrowBatchedUDFTimeBench** - Before (master):
    ```
    =================== ============== =============== ===============
    --                                       udf
    ------------------- ----------------------------------------------
          scenario       identity_udf   stringify_udf   nullcheck_udf
    =================== ============== =============== ===============
      sm_batch_few_col    62.1+-0.2ms      66.1+-0.8ms      61.2+-0.1ms
     sm_batch_many_col    154+-0.4ms       155+-0.4ms       154+-0.3ms
      lg_batch_few_col    148+-0.3ms       157+-0.4ms       147+-0.5ms
     lg_batch_many_col     623+-2ms         624+-2ms         620+-3ms
         pure_ints        220+-0.5ms       231+-0.7ms        220+-6ms
        pure_floats       224+-0.8ms        262+-1ms        225+-0.7ms
        pure_strings       414+-1ms        415+-0.6ms        404+-1ms
        mixed_types        311+-1ms        318+-0.8ms       308+-0.7ms
    =================== ============== =============== ===============
    ```
    
    **ArrowBatchedUDFTimeBench** - After (this PR):
    ```
    =================== ============== =============== ===============
    --                                       udf
    ------------------- ----------------------------------------------
          scenario       identity_udf   stringify_udf   nullcheck_udf
    =================== ============== =============== ===============
      sm_batch_few_col    59.9+-0.4ms     63.5+-0.06ms     59.5+-0.3ms
     sm_batch_many_col    149+-0.1ms       150+-0.3ms       149+-0.2ms
      lg_batch_few_col    144+-0.4ms       153+-0.8ms       143+-0.5ms
     lg_batch_many_col     602+-2ms         603+-2ms         598+-3ms
         pure_ints         216+-1ms       226+-0.7ms       214+-0.5ms
        pure_floats        215+-1ms       251+-0.9ms        215+-1ms
        pure_strings      391+-0.8ms        395+-2ms       385+-0.9ms
        mixed_types       299+-0.7ms        307+-1ms         297+-1ms
    =================== ============== =============== ===============
    ```
    
    **ArrowBatchedUDFPeakmemBench** - Before (master):
    ```
    =================== ============== =============== ===============
    --                                       udf
    ------------------- ----------------------------------------------
          scenario       identity_udf   stringify_udf   nullcheck_udf
    =================== ============== =============== ===============
      sm_batch_few_col       119M            119M            118M
     sm_batch_many_col       123M            123M            123M
      lg_batch_few_col       124M            124M            122M
     lg_batch_many_col       159M            160M            159M
         pure_ints           122M            123M            122M
        pure_floats          124M            125M            123M
        pure_strings         125M            125M            124M
        mixed_types          123M            124M            123M
    =================== ============== =============== ===============
    ```
    
    **ArrowBatchedUDFPeakmemBench** - After (this PR):
    ```
    =================== ============== =============== ===============
    --                                       udf
    ------------------- ----------------------------------------------
          scenario       identity_udf   stringify_udf   nullcheck_udf
    =================== ============== =============== ===============
      sm_batch_few_col       119M            119M            119M
     sm_batch_many_col       123M            123M            123M
      lg_batch_few_col       123M            124M            122M
     lg_batch_many_col       160M            161M            160M
         pure_ints           122M            123M            122M
        pure_floats          124M            125M            123M
        pure_strings         125M            125M            125M
        mixed_types          124M            124M            124M
    =================== ============== =============== ===============
    ```
    
    **Summary**: Latency improved 3-5% across all scenarios (e.g., `identity_udf` on `pure_strings` dropped from 414ms to 391ms). This is likely due to eliminating the serializer class overhead and reducing indirection layers. Peak memory is unchanged, as expected since the refactor only reorganizes logic without changing data layout or buffering strategy.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No
    
    Closes #54705 from Yicong-Huang/SPARK-55902/refactor/arrow-batch-udf.
    
    Lead-authored-by: Yicong-Huang <17...@users.noreply.github.com>
    Co-authored-by: Yicong Huang <17...@users.noreply.github.com>
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
 python/pyspark/worker.py | 364 ++++++++++++++++++++++++++---------------------
 1 file changed, 200 insertions(+), 164 deletions(-)

diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index 40489affbd15..ae044f59fce7 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -51,6 +51,7 @@ from pyspark.sql.conversion import (
     LocalDataToArrowConversion,
     ArrowTableToRowsConversion,
     ArrowBatchTransformer,
+    PandasToArrowConversion,
 )
 from pyspark.sql.functions import SkipRestOfInputTableException
 from pyspark.sql.pandas.serializers import (
@@ -68,7 +69,6 @@ from pyspark.sql.pandas.serializers import (
     TransformWithStateInPySparkRowSerializer,
     TransformWithStateInPySparkRowInitStateSerializer,
     ArrowStreamAggPandasUDFSerializer,
-    ArrowBatchUDFSerializer,
     ArrowStreamUDTFSerializer,
     ArrowStreamArrowUDTFSerializer,
 )
@@ -91,7 +91,7 @@ from pyspark.util import (
     with_faulthandler,
     start_faulthandler_periodic_traceback,
 )
-from pyspark import shuffle
+from pyspark import _NoValue, shuffle
 from pyspark.errors import PySparkRuntimeError, PySparkTypeError, PySparkValueError
 from pyspark.worker_util import (
     check_python_version,
@@ -260,6 +260,18 @@ def verify_result(expected_type: type) -> Callable[[Any], Iterator]:
     return check
 
 
+def verify_result_row_count(result_length: int, expected: int) -> None:
+    """Raise if the result row count doesn't match the expected input row count."""
+    if result_length != expected:
+        raise PySparkRuntimeError(
+            errorClass="RESULT_ROWS_MISMATCH",
+            messageParameters={
+                "output_length": str(result_length),
+                "input_length": str(expected),
+            },
+        )
+
+
 def verify_scalar_result(result: Any, num_rows: int) -> Any:
     """
     Verify a scalar UDF result is array-like and has the expected number of rows.
@@ -282,6 +294,7 @@ def verify_scalar_result(result: Any, num_rows: int) -> Any:
             },
         )
     if result_length != num_rows:
+        # TODO: change error class to RESULT_ROWS_MISMATCH
         raise PySparkRuntimeError(
             errorClass="SCHEMA_MISMATCH_FOR_PANDAS_UDF",
             messageParameters={
@@ -385,143 +398,6 @@ def wrap_scalar_pandas_udf(f, args_offsets, kwargs_offsets, return_type, runner_
     )
 
 
-def wrap_arrow_batch_udf(f, args_offsets, kwargs_offsets, return_type, runner_conf):
-    if runner_conf.use_legacy_pandas_udf_conversion:
-        return wrap_arrow_batch_udf_legacy(
-            f, args_offsets, kwargs_offsets, return_type, runner_conf
-        )
-    else:
-        return wrap_arrow_batch_udf_arrow(f, args_offsets, kwargs_offsets, return_type, runner_conf)
-
-
-def wrap_arrow_batch_udf_arrow(f, args_offsets, kwargs_offsets, return_type, runner_conf):
-    from pyspark.sql.pandas.types import to_arrow_type
-
-    func, args_kwargs_offsets = wrap_kwargs_support(f, args_offsets, kwargs_offsets)
-
-    zero_arg_exec = False
-    if len(args_kwargs_offsets) == 0:
-        args_kwargs_offsets = (0,)
-        zero_arg_exec = True
-
-    arrow_return_type = to_arrow_type(
-        return_type, timezone="UTC", prefers_large_types=runner_conf.use_large_var_types
-    )
-
-    if zero_arg_exec:
-
-        def get_args(*args: list):
-            return [() for _ in args[0]]
-
-    else:
-
-        def get_args(*args: list):
-            return zip(*args)
-
-    if runner_conf.arrow_concurrency_level > 0:
-        from concurrent.futures import ThreadPoolExecutor
-
-        @fail_on_stopiteration
-        def evaluate(*args):
-            with ThreadPoolExecutor(max_workers=runner_conf.arrow_concurrency_level) as pool:
-                """
-                Takes list of Python objects and returns tuple of
-                (results, arrow_return_type, return_type).
-                """
-                return list(pool.map(lambda row: func(*row), get_args(*args)))
-
-    else:
-
-        @fail_on_stopiteration
-        def evaluate(*args):
-            """
-            Takes list of Python objects and returns tuple of
-            (results, arrow_return_type, return_type).
-            """
-            return [func(*row) for row in get_args(*args)]
-
-    def verify_result_length(result, length):
-        if len(result) != length:
-            raise PySparkRuntimeError(
-                errorClass="SCHEMA_MISMATCH_FOR_ARROW_PYTHON_UDF",
-                messageParameters={
-                    "udf_type": "arrow_batch_udf",
-                    "expected": str(length),
-                    "actual": str(len(result)),
-                },
-            )
-        return result
-
-    return (
-        args_kwargs_offsets,
-        lambda *a: (verify_result_length(evaluate(*a), len(a[0])), arrow_return_type, return_type),
-    )
-
-
-def wrap_arrow_batch_udf_legacy(f, args_offsets, kwargs_offsets, return_type, runner_conf):
-    import pandas as pd
-
-    func, args_kwargs_offsets = wrap_kwargs_support(f, args_offsets, kwargs_offsets)
-    zero_arg_exec = False
-    if len(args_kwargs_offsets) == 0:
-        args_kwargs_offsets = (0,)  # Series([pyspark._NoValue, ...]) is used for 0-arg execution.
-        zero_arg_exec = True
-
-    # "result_func" ensures the result of a Python UDF to be consistent with/without Arrow
-    # optimization.
-    # Otherwise, an Arrow-optimized Python UDF raises "pyarrow.lib.ArrowTypeError: Expected a
-    # string or bytes dtype, got ..." whereas a non-Arrow-optimized Python UDF returns
-    # successfully.
-    result_func = lambda pdf: pdf  # noqa: E731
-    if type(return_type) == StringType:
-        result_func = lambda r: str(r) if r is not None else r  # noqa: E731
-    elif type(return_type) == BinaryType:
-        result_func = lambda r: bytes(r) if r is not None else r  # noqa: E731
-
-    if zero_arg_exec:
-
-        def get_args(*args: pd.Series):
-            return [() for _ in args[0]]
-
-    else:
-
-        def get_args(*args: pd.Series):
-            return zip(*args)
-
-    if runner_conf.arrow_concurrency_level > 0:
-        from concurrent.futures import ThreadPoolExecutor
-
-        @fail_on_stopiteration
-        def evaluate(*args: pd.Series) -> pd.Series:
-            with ThreadPoolExecutor(max_workers=runner_conf.arrow_concurrency_level) as pool:
-                return pd.Series(
-                    list(pool.map(lambda row: result_func(func(*row)), get_args(*args)))
-                )
-
-    else:
-
-        @fail_on_stopiteration
-        def evaluate(*args: pd.Series) -> pd.Series:
-            return pd.Series([result_func(func(*row)) for row in get_args(*args)])
-
-    def verify_result_length(result, length):
-        if len(result) != length:
-            raise PySparkRuntimeError(
-                errorClass="SCHEMA_MISMATCH_FOR_PANDAS_UDF",
-                messageParameters={
-                    "udf_type": "arrow_batch_udf",
-                    "expected": str(length),
-                    "actual": str(len(result)),
-                },
-            )
-        return result
-
-    return (
-        args_kwargs_offsets,
-        lambda *a: (verify_result_length(evaluate(*a), len(a[0])), return_type),
-    )
-
-
 def wrap_pandas_batch_iter_udf(f, return_type, runner_conf):
     iter_type_label = "pandas.DataFrame" if type(return_type) == StructType else "pandas.Series"
 
@@ -1296,7 +1172,7 @@ def read_single_udf(pickleSer, infile, eval_type, runner_conf, udf_index):
     elif eval_type == PythonEvalType.SQL_SCALAR_ARROW_UDF:
         return func, args_offsets, kwargs_offsets, return_type
     elif eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF:
-        return wrap_arrow_batch_udf(func, args_offsets, kwargs_offsets, return_type, runner_conf)
+        return func, args_offsets, kwargs_offsets, return_type
     elif eval_type == PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF:
         return args_offsets, wrap_pandas_batch_iter_udf(func, return_type, runner_conf)
     elif eval_type == PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF:
@@ -2655,17 +2531,9 @@ def read_udfs(pickleSer, infile, eval_type, runner_conf, eval_conf):
             PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF,
         ):
             ser = ArrowStreamSerializer(write_start_stream=True)
-        elif (
-            eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF
-            and not runner_conf.use_legacy_pandas_udf_conversion
-        ):
+        elif eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF:
             input_type = _parse_datatype_json_string(utf8_deserializer.loads(infile))
-            ser = ArrowBatchUDFSerializer(
-                safecheck=runner_conf.safecheck,
-                input_type=input_type,
-                int_to_decimal_coercion_enabled=runner_conf.int_to_decimal_coercion_enabled,
-                binary_as_bytes=runner_conf.binary_as_bytes,
-            )
+            ser = ArrowStreamSerializer(write_start_stream=True)
         else:
             # Scalar Pandas UDF handles struct type arguments as pandas DataFrames instead of
             # pandas Series. See SPARK-27240.
@@ -2674,28 +2542,17 @@ def read_udfs(pickleSer, infile, eval_type, runner_conf, eval_conf):
                 or eval_type == PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
                 or eval_type == PythonEvalType.SQL_MAP_PANDAS_ITER_UDF
             )
-            # Arrow-optimized Python UDF takes a struct type argument as a Row
-            struct_in_pandas = (
-                "row" if eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF else "dict"
-            )
-            ndarray_as_list = eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF
-            # Arrow-optimized Python UDF takes input types
-            input_type = (
-                _parse_datatype_json_string(utf8_deserializer.loads(infile))
-                if eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF
-                else None
-            )
 
             ser = ArrowStreamPandasUDFSerializer(
                 timezone=runner_conf.timezone,
                 safecheck=runner_conf.safecheck,
                 assign_cols_by_name=runner_conf.assign_cols_by_name,
                 df_for_struct=df_for_struct,
-                struct_in_pandas=struct_in_pandas,
-                ndarray_as_list=ndarray_as_list,
+                struct_in_pandas="dict",
+                ndarray_as_list=False,
                 prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,
                 arrow_cast=True,
-                input_type=input_type,
+                input_type=None,
                 int_to_decimal_coercion_enabled=runner_conf.int_to_decimal_coercion_enabled,
                 prefers_large_types=runner_conf.use_large_var_types,
             )
@@ -2962,6 +2819,185 @@ def read_udfs(pickleSer, infile, eval_type, runner_conf, eval_conf):
         # profiling is not supported for UDF
         return func, None, ser, ser
 
+    if (
+        eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF
+        and not runner_conf.use_legacy_pandas_udf_conversion
+    ):
+        import pyarrow as pa
+
+        # --- UDF preparation ---
+        udf_infos = []
+        for udf_func, udf_args_offsets, udf_kwargs_offsets, udf_return_type in udfs:
+            wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
+                udf_func, udf_args_offsets, udf_kwargs_offsets
+            )
+            zero_arg = len(args_kwargs_offsets) == 0
+            udf_infos.append(
+                (
+                    wrapped_func,
+                    args_kwargs_offsets or (0,),
+                    zero_arg,
+                    to_arrow_type(
+                        udf_return_type,
+                        timezone="UTC",
+                        prefers_large_types=runner_conf.use_large_var_types,
+                    ),
+                    LocalDataToArrowConversion._create_converter(
+                        udf_return_type,
+                        none_on_identity=True,
+                        int_to_decimal_coercion_enabled=runner_conf.int_to_decimal_coercion_enabled,
+                    ),
+                )
+            )
+        col_names = [f"_{i}" for i in range(len(udfs))]
+
+        # --- Input preparation ---
+        arrow_to_py_converters = [
+            ArrowTableToRowsConversion._create_converter(
+                f.dataType, none_on_identity=True, binary_as_bytes=runner_conf.binary_as_bytes
+            )
+            for f in input_type
+        ]
+
+        @fail_on_stopiteration
+        def _evaluate_batch_udf(udf_func, rows):
+            if runner_conf.arrow_concurrency_level <= 0:
+                return [udf_func(*row) for row in rows]
+            from concurrent.futures import ThreadPoolExecutor
+
+            with ThreadPoolExecutor(max_workers=runner_conf.arrow_concurrency_level) as pool:
+                return list(pool.map(lambda row: udf_func(*row), rows))
+
+        def func(split_index: int, batches: Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]:
+            for input_batch in batches:
+                num_rows = input_batch.num_rows
+
+                # --- Input: Arrow -> Python columns ---
+                columns = [
+                    [conv(v) for v in col.to_pylist()] if conv is not None else col.to_pylist()
+                    for col, conv in zip(input_batch.itercolumns(), arrow_to_py_converters)
+                ]
+                if not columns:
+                    columns = [[_NoValue] * num_rows]
+
+                # --- Process: evaluate each UDF row-by-row ---
+                output_arrays = []
+                for udf_func, offsets, zero_arg, arrow_return_type, result_conv in udf_infos:
+                    rows = (
+                        [() for _ in range(num_rows)]
+                        if zero_arg
+                        else list(zip(*[columns[o] for o in offsets]))
+                    )
+                    results = _evaluate_batch_udf(udf_func, rows)
+                    verify_result_row_count(len(results), num_rows)
+
+                    # --- Output: Python -> Arrow ---
+                    converted = (
+                        [result_conv(r) for r in results] if result_conv is not None else results
+                    )
+                    try:
+                        arr = pa.array(converted, type=arrow_return_type)
+                    except pa.lib.ArrowInvalid:
+                        arr = pa.array(converted).cast(
+                            target_type=arrow_return_type, safe=runner_conf.safecheck
+                        )
+                    output_arrays.append(arr)
+
+                yield pa.RecordBatch.from_arrays(output_arrays, col_names)
+
+        # profiling is not supported for UDF
+        return func, None, ser, ser
+
+    if (
+        eval_type == PythonEvalType.SQL_ARROW_BATCHED_UDF
+        and runner_conf.use_legacy_pandas_udf_conversion
+    ):
+        import pandas as pd
+        import pyarrow as pa
+
+        # --- UDF preparation ---
+        udf_infos = []
+        for udf_func, udf_args_offsets, udf_kwargs_offsets, udf_return_type in udfs:
+            wrapped_func, args_kwargs_offsets = wrap_kwargs_support(
+                udf_func, udf_args_offsets, udf_kwargs_offsets
+            )
+            zero_arg = len(args_kwargs_offsets) == 0
+            # Legacy coerces String/Binary for Arrow compatibility
+            coerce = (
+                str
+                if type(udf_return_type) == StringType
+                else bytes
+                if type(udf_return_type) == BinaryType
+                else None
+            )
+            udf_infos.append(
+                (
+                    wrapped_func,
+                    args_kwargs_offsets or (0,),
+                    zero_arg,
+                    udf_return_type,
+                    coerce,
+                )
+            )
+        col_names = [f"_{i}" for i in range(len(udfs))]
+        return_schema = StructType(
+            [StructField(name, info[3]) for name, info in zip(col_names, udf_infos)]
+        )
+
+        @fail_on_stopiteration
+        def _evaluate_batch_udf_legacy(udf_func, rows):
+            if runner_conf.arrow_concurrency_level <= 0:
+                return [udf_func(*row) for row in rows]
+            from concurrent.futures import ThreadPoolExecutor
+
+            with ThreadPoolExecutor(max_workers=runner_conf.arrow_concurrency_level) as pool:
+                return list(pool.map(lambda row: udf_func(*row), rows))
+
+        def func(split_index: int, batches: Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]:
+            for input_batch in batches:
+                # --- Input: Arrow -> pandas columns ---
+                pandas_columns = ArrowBatchTransformer.to_pandas(
+                    input_batch,
+                    timezone=runner_conf.timezone,
+                    schema=input_type,
+                    struct_in_pandas="row",
+                    ndarray_as_list=True,
+                    prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,
+                    df_for_struct=False,
+                )
+                num_rows = len(pandas_columns[0]) if pandas_columns else input_batch.num_rows
+                if not pandas_columns:
+                    pandas_columns = [pd.Series([_NoValue] * num_rows)]
+
+                # --- Process: evaluate each UDF row-by-row ---
+                result_series = []
+                for udf_func, offsets, zero_arg, _, coerce in udf_infos:
+                    rows = (
+                        [() for _ in range(num_rows)]
+                        if zero_arg
+                        else list(zip(*[pandas_columns[o].tolist() for o in offsets]))
+                    )
+                    results = _evaluate_batch_udf_legacy(udf_func, rows)
+                    verify_result_row_count(len(results), num_rows)
+                    if coerce:
+                        results = [coerce(v) if v is not None else v for v in results]
+                    result_series.append(pd.Series(results))
+
+                # --- Output: pandas -> Arrow ---
+                yield PandasToArrowConversion.convert(
+                    result_series,
+                    return_schema,
+                    timezone=runner_conf.timezone,
+                    safecheck=runner_conf.safecheck,
+                    arrow_cast=True,
+                    prefers_large_types=runner_conf.use_large_var_types,
+                    assign_cols_by_name=runner_conf.assign_cols_by_name,
+                    int_to_decimal_coercion_enabled=runner_conf.int_to_decimal_coercion_enabled,
+                )
+
+        # profiling is not supported for UDF
+        return func, None, ser, ser
+
     is_scalar_iter = eval_type == PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
     is_map_pandas_iter = eval_type == PythonEvalType.SQL_MAP_PANDAS_ITER_UDF
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org