You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/06/12 01:17:43 UTC

[PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Yicong-Huang opened a new pull request, #56464:
URL: https://github.com/apache/spark/pull/56464

   ### What changes were proposed in this pull request?
   
   This PR refactors `SQL_TRANSFORM_WITH_STATE_PANDAS_UDF` so that the worker uses the plain `ArrowStreamSerializer` for pure Arrow stream I/O, moving the per-eval-type logic (regrouping rows by grouping key, re-chunking into pandas DataFrames bounded by `arrow_max_records_per_batch`/`arrow_max_bytes_per_batch`, and converting result DataFrames back to Arrow) from `TransformWithStateInPandasSerializer` into `read_udfs()` in `worker.py`. The serializer class is kept for now since `TransformWithStateInPandasInitStateSerializer` (init-state variant, tracked separately) still subclasses it.
   
   ### Why are the changes needed?
   
   Part of [SPARK-55388](https://issues.apache.org/jira/browse/SPARK-55388). Keeping serializers as pure Arrow stream I/O and concentrating eval-type-specific logic in `worker.py` makes the per-eval-type data flow explicit.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Existing tests (`pyspark.sql.tests.pandas.streaming.test_pandas_transform_with_state`). No behavior change: replaying identical worker input through the old and new code paths produces byte-identical worker output across 12 scenario/UDF combinations.
   
   ASV comparison (`bench_eval_type.TransformWithStatePandasUDFTimeBench` / `TransformWithStatePandasUDFPeakmemBench`, `-a repeat=3`): before = `upstream/master`, after = this PR. Values are from one representative run per side; the conclusion is consistent across the 2 runs performed per side.
   
   ```text
   time_worker
   
   scenario         udf            before        after         diff
   ---------------  ------------  ------------  ------------  ------
   few_groups_sm    identity_udf   754+-5ms      743+-3ms      -1.5%
   few_groups_sm    sort_udf       781+-20ms     760+-0.6ms    -2.7%
   few_groups_sm    count_udf      726+-3ms      722+-1ms      -0.6%
   few_groups_lg    identity_udf   7.01+-0.03s   6.99+-0.01s   -0.3%
   few_groups_lg    sort_udf       7.21+-0.03s   7.10+-0.02s   -1.5%
   few_groups_lg    count_udf      6.67+-0.04s   6.60+-0.01s   -1.0%
   many_groups_sm   identity_udf   6.24+-0.03s   6.13+-0.01s   -1.8%
   many_groups_sm   sort_udf       6.64+-0.01s   6.69+-0.02s   +0.8%
   many_groups_sm   count_udf      5.76+-0.07s   5.89+-0.1s    +2.3%
   many_groups_lg   identity_udf   3.54+-0.02s   3.53+-0.04s   -0.3%
   many_groups_lg   sort_udf       3.65+-0.03s   3.63+-0.01s   -0.5%
   many_groups_lg   count_udf      3.40+-0.04s   3.39+-0.03s   -0.3%
   wide_cols        identity_udf   7.45+-0.07s   7.35+-0.03s   -1.3%
   wide_cols        sort_udf       7.51+-0.03s   7.42+-0.03s   -1.2%
   wide_cols        count_udf      6.85+-0.03s   6.80+-0.01s   -0.7%
   mixed_cols       identity_udf   3.15+-0.01s   3.16+-0.01s   +0.3%
   mixed_cols       sort_udf       3.36+-0.06s   3.28+-0s      -2.4%
   mixed_cols       count_udf      2.85+-0.02s   2.91+-0.04s   +2.1%
   nested_struct    identity_udf   7.47+-0.03s   7.33+-0.01s   -1.9%
   nested_struct    sort_udf       8.63+-0.2s    8.36+-0.07s   -3.1%
   nested_struct    count_udf      5.27+-0.03s   5.16+-0.01s   -2.1%
   ```
   
   ```text
   peakmem_worker
   
   scenario         udf            before   after
   ---------------  ------------  -------  -------
   few_groups_sm    identity_udf   104M     106M
   few_groups_sm    sort_udf       108M     107M
   few_groups_sm    count_udf      97.9M    98.1M
   few_groups_lg    identity_udf   200M     200M
   few_groups_lg    sort_udf       201M     189M
   few_groups_lg    count_udf      161M     161M
   many_groups_sm   identity_udf   130M     130M
   many_groups_sm   sort_udf       131M     131M
   many_groups_sm   count_udf      113M     113M
   many_groups_lg   identity_udf   136M     133M
   many_groups_lg   sort_udf       132M     134M
   many_groups_lg   count_udf      113M     113M
   wide_cols        identity_udf   240M     242M
   wide_cols        sort_udf       241M     241M
   wide_cols        count_udf      205M     205M
   mixed_cols       identity_udf   182M     182M
   mixed_cols       sort_udf       182M     182M
   mixed_cols       count_udf      182M     182M
   nested_struct    identity_udf   210M     210M
   nested_struct    sort_udf       210M     210M
   nested_struct    count_udf      210M     210M
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #56464:
URL: https://github.com/apache/spark/pull/56464#issuecomment-4784058870

   @funrollloops Thanks for taking a look! Replied to both inline comments. Regarding `TransformWithStateInPandasSerializer` not being removed: right, it stays because `TransformWithStateInPandasInitStateSerializer` still subclasses it and to be removed later


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #56464:
URL: https://github.com/apache/spark/pull/56464#discussion_r3463340073


##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
         return func, None, ser, ser
 
     if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
-        # We assume there is only one UDF here because grouped map doesn't
-        # support combining multiple UDFs.
-        assert num_udfs == 1
+        import pyarrow as pa
+        import pandas as pd
+
+        assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+        udf, arg_offsets, return_type = udfs[0]
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes
-        arg_offsets, f = udfs[0]
         parsed_offsets = extract_key_value_indexes(arg_offsets)
-        ser.key_offsets = parsed_offsets[0][0]
+        assert len(parsed_offsets) == 1, (
+            "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+        )
+
+        key_offsets = parsed_offsets[0][0]
+        value_offsets = parsed_offsets[0][1]
+        output_schema = StructType([StructField("_0", return_type)])
+
         stateful_processor_api_client = StatefulProcessorApiClient(
             eval_conf.state_server_socket_port, eval_conf.grouping_key_schema
         )
 
-        def mapper(a):
-            mode = a[0]
+        arrow_max_records_per_batch = runner_conf.arrow_max_records_per_batch
+        arrow_max_records_per_batch = (
+            arrow_max_records_per_batch if arrow_max_records_per_batch > 0 else 2**31 - 1
+        )
+        arrow_max_bytes_per_batch = runner_conf.arrow_max_bytes_per_batch
 
-            if mode == TransformWithStateInPandasFuncMode.PROCESS_DATA:
-                key = a[1]
+        def transform_with_state_func(
+            split_index: int,
+            batches: Iterator[pa.RecordBatch],
+        ) -> Iterator[pa.RecordBatch]:
+            """Apply transformWithStateInPandas UDF.
+
+            Data chunks for the same grouping key appear sequentially in the
+            input batches but may span batch boundaries, so rows are regrouped
+            by key and re-chunked into pandas DataFrames bounded by
+            arrow_max_records_per_batch and arrow_max_bytes_per_batch. The UDF
+            is invoked once per grouping key with a lazy iterator of chunks,
+            then once for PROCESS_TIMER and once for COMPLETE.
+            """
+            total_bytes = 0
+            total_rows = 0
+            average_arrow_row_size = 0.0
+
+            def row_stream():
+                nonlocal total_bytes, total_rows, average_arrow_row_size
+                for batch in batches:
+                    # Short circuit batch size stats if the batch size is
+                    # unlimited as computing batch size is computationally
+                    # expensive.
+                    if arrow_max_bytes_per_batch != 2**31 - 1 and batch.num_rows > 0:
+                        total_bytes += sum(
+                            buf.size
+                            for col in batch.columns
+                            for buf in col.buffers()
+                            if buf is not None
+                        )
+                        total_rows += batch.num_rows
+                        average_arrow_row_size = total_bytes / total_rows
+                    data_pandas = ArrowBatchTransformer.to_pandas(
+                        batch,
+                        timezone=runner_conf.timezone,
+                        prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,

Review Comment:
   those arguments are passed in as general apis of serializers before. as we decouple the logic from serializer, we only need to supply parameters needed for this particular UDF eval type. The omitted args are exactly the defaults of `ArrowBatchTransformer.to_pandas(schema=None, struct_in_pandas="dict", ndarray_as_list=False, df_for_struct=False)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #56464:
URL: https://github.com/apache/spark/pull/56464#issuecomment-4784758383

   Thanks @HyukjinKwon @uros-b and @funrollloops, merged to master/4.x


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #56464: [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF
URL: https://github.com/apache/spark/pull/56464


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Posted by "funrollloops (via GitHub)" <gi...@apache.org>.
funrollloops commented on code in PR #56464:
URL: https://github.com/apache/spark/pull/56464#discussion_r3461827665


##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
         return func, None, ser, ser
 
     if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
-        # We assume there is only one UDF here because grouped map doesn't
-        # support combining multiple UDFs.
-        assert num_udfs == 1
+        import pyarrow as pa
+        import pandas as pd
+
+        assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+        udf, arg_offsets, return_type = udfs[0]
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes
-        arg_offsets, f = udfs[0]
         parsed_offsets = extract_key_value_indexes(arg_offsets)
-        ser.key_offsets = parsed_offsets[0][0]
+        assert len(parsed_offsets) == 1, (
+            "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+        )
+
+        key_offsets = parsed_offsets[0][0]
+        value_offsets = parsed_offsets[0][1]
+        output_schema = StructType([StructField("_0", return_type)])
+
         stateful_processor_api_client = StatefulProcessorApiClient(
             eval_conf.state_server_socket_port, eval_conf.grouping_key_schema
         )
 
-        def mapper(a):
-            mode = a[0]
+        arrow_max_records_per_batch = runner_conf.arrow_max_records_per_batch
+        arrow_max_records_per_batch = (
+            arrow_max_records_per_batch if arrow_max_records_per_batch > 0 else 2**31 - 1
+        )
+        arrow_max_bytes_per_batch = runner_conf.arrow_max_bytes_per_batch
 
-            if mode == TransformWithStateInPandasFuncMode.PROCESS_DATA:
-                key = a[1]
+        def transform_with_state_func(
+            split_index: int,
+            batches: Iterator[pa.RecordBatch],
+        ) -> Iterator[pa.RecordBatch]:
+            """Apply transformWithStateInPandas UDF.
+
+            Data chunks for the same grouping key appear sequentially in the
+            input batches but may span batch boundaries, so rows are regrouped
+            by key and re-chunked into pandas DataFrames bounded by
+            arrow_max_records_per_batch and arrow_max_bytes_per_batch. The UDF
+            is invoked once per grouping key with a lazy iterator of chunks,
+            then once for PROCESS_TIMER and once for COMPLETE.
+            """
+            total_bytes = 0
+            total_rows = 0
+            average_arrow_row_size = 0.0
+
+            def row_stream():
+                nonlocal total_bytes, total_rows, average_arrow_row_size
+                for batch in batches:
+                    # Short circuit batch size stats if the batch size is
+                    # unlimited as computing batch size is computationally
+                    # expensive.
+                    if arrow_max_bytes_per_batch != 2**31 - 1 and batch.num_rows > 0:
+                        total_bytes += sum(
+                            buf.size
+                            for col in batch.columns
+                            for buf in col.buffers()
+                            if buf is not None
+                        )
+                        total_rows += batch.num_rows
+                        average_arrow_row_size = total_bytes / total_rows
+                    data_pandas = ArrowBatchTransformer.to_pandas(
+                        batch,
+                        timezone=runner_conf.timezone,
+                        prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,

Review Comment:
   we're passing far fewer arguments than in python/pyspark/sql/pandas/serializers.py:1037; why?
   
   in particular we're missing schema, struct_in_pandas, ndarray_as_list, and df_for_struct



##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
         return func, None, ser, ser
 
     if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
-        # We assume there is only one UDF here because grouped map doesn't
-        # support combining multiple UDFs.
-        assert num_udfs == 1
+        import pyarrow as pa
+        import pandas as pd
+
+        assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+        udf, arg_offsets, return_type = udfs[0]
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes
-        arg_offsets, f = udfs[0]
         parsed_offsets = extract_key_value_indexes(arg_offsets)
-        ser.key_offsets = parsed_offsets[0][0]
+        assert len(parsed_offsets) == 1, (
+            "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+        )
+
+        key_offsets = parsed_offsets[0][0]
+        value_offsets = parsed_offsets[0][1]
+        output_schema = StructType([StructField("_0", return_type)])

Review Comment:
   Did we previously wrap the returned value with a struct of one field? I can't find that logic anywhere in the touched code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #56464:
URL: https://github.com/apache/spark/pull/56464#discussion_r3463111993


##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
         return func, None, ser, ser
 
     if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
-        # We assume there is only one UDF here because grouped map doesn't
-        # support combining multiple UDFs.
-        assert num_udfs == 1
+        import pyarrow as pa
+        import pandas as pd
+
+        assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+        udf, arg_offsets, return_type = udfs[0]
 
         # See TransformWithStateInPandasExec for how arg_offsets are used to
         # distinguish between grouping attributes and data attributes
-        arg_offsets, f = udfs[0]
         parsed_offsets = extract_key_value_indexes(arg_offsets)
-        ser.key_offsets = parsed_offsets[0][0]
+        assert len(parsed_offsets) == 1, (
+            "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+        )
+
+        key_offsets = parsed_offsets[0][0]
+        value_offsets = parsed_offsets[0][1]
+        output_schema = StructType([StructField("_0", return_type)])

Review Comment:
   Yes, this is the same wrapping as before, just inlined. Previously the output went through `ArrowStreamPandasUDFSerializer.dump_stream` -> `create_batch`, which builds `StructType([StructField(f"_{i}", t) for  i, t in enumerate(types)])`. The TWS path always emits a single `(pdf, return_type)`, so `types == [return_type]` and that produced exactly the one `field _0` struct. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org