You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/06/12 01:17:43 UTC
[PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Yicong-Huang opened a new pull request, #56464:
URL: https://github.com/apache/spark/pull/56464
### What changes were proposed in this pull request?
This PR refactors `SQL_TRANSFORM_WITH_STATE_PANDAS_UDF` so that the worker uses the plain `ArrowStreamSerializer` for pure Arrow stream I/O, moving the per-eval-type logic (regrouping rows by grouping key, re-chunking into pandas DataFrames bounded by `arrow_max_records_per_batch`/`arrow_max_bytes_per_batch`, and converting result DataFrames back to Arrow) from `TransformWithStateInPandasSerializer` into `read_udfs()` in `worker.py`. The serializer class is kept for now since `TransformWithStateInPandasInitStateSerializer` (init-state variant, tracked separately) still subclasses it.
### Why are the changes needed?
Part of [SPARK-55388](https://issues.apache.org/jira/browse/SPARK-55388). Keeping serializers as pure Arrow stream I/O and concentrating eval-type-specific logic in `worker.py` makes the per-eval-type data flow explicit.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests (`pyspark.sql.tests.pandas.streaming.test_pandas_transform_with_state`). No behavior change: replaying identical worker input through the old and new code paths produces byte-identical worker output across 12 scenario/UDF combinations.
ASV comparison (`bench_eval_type.TransformWithStatePandasUDFTimeBench` / `TransformWithStatePandasUDFPeakmemBench`, `-a repeat=3`): before = `upstream/master`, after = this PR. Values are from one representative run per side; the conclusion is consistent across the 2 runs performed per side.
```text
time_worker
scenario udf before after diff
--------------- ------------ ------------ ------------ ------
few_groups_sm identity_udf 754+-5ms 743+-3ms -1.5%
few_groups_sm sort_udf 781+-20ms 760+-0.6ms -2.7%
few_groups_sm count_udf 726+-3ms 722+-1ms -0.6%
few_groups_lg identity_udf 7.01+-0.03s 6.99+-0.01s -0.3%
few_groups_lg sort_udf 7.21+-0.03s 7.10+-0.02s -1.5%
few_groups_lg count_udf 6.67+-0.04s 6.60+-0.01s -1.0%
many_groups_sm identity_udf 6.24+-0.03s 6.13+-0.01s -1.8%
many_groups_sm sort_udf 6.64+-0.01s 6.69+-0.02s +0.8%
many_groups_sm count_udf 5.76+-0.07s 5.89+-0.1s +2.3%
many_groups_lg identity_udf 3.54+-0.02s 3.53+-0.04s -0.3%
many_groups_lg sort_udf 3.65+-0.03s 3.63+-0.01s -0.5%
many_groups_lg count_udf 3.40+-0.04s 3.39+-0.03s -0.3%
wide_cols identity_udf 7.45+-0.07s 7.35+-0.03s -1.3%
wide_cols sort_udf 7.51+-0.03s 7.42+-0.03s -1.2%
wide_cols count_udf 6.85+-0.03s 6.80+-0.01s -0.7%
mixed_cols identity_udf 3.15+-0.01s 3.16+-0.01s +0.3%
mixed_cols sort_udf 3.36+-0.06s 3.28+-0s -2.4%
mixed_cols count_udf 2.85+-0.02s 2.91+-0.04s +2.1%
nested_struct identity_udf 7.47+-0.03s 7.33+-0.01s -1.9%
nested_struct sort_udf 8.63+-0.2s 8.36+-0.07s -3.1%
nested_struct count_udf 5.27+-0.03s 5.16+-0.01s -2.1%
```
```text
peakmem_worker
scenario udf before after
--------------- ------------ ------- -------
few_groups_sm identity_udf 104M 106M
few_groups_sm sort_udf 108M 107M
few_groups_sm count_udf 97.9M 98.1M
few_groups_lg identity_udf 200M 200M
few_groups_lg sort_udf 201M 189M
few_groups_lg count_udf 161M 161M
many_groups_sm identity_udf 130M 130M
many_groups_sm sort_udf 131M 131M
many_groups_sm count_udf 113M 113M
many_groups_lg identity_udf 136M 133M
many_groups_lg sort_udf 132M 134M
many_groups_lg count_udf 113M 113M
wide_cols identity_udf 240M 242M
wide_cols sort_udf 241M 241M
wide_cols count_udf 205M 205M
mixed_cols identity_udf 182M 182M
mixed_cols sort_udf 182M 182M
mixed_cols count_udf 182M 182M
nested_struct identity_udf 210M 210M
nested_struct sort_udf 210M 210M
nested_struct count_udf 210M 210M
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #56464:
URL: https://github.com/apache/spark/pull/56464#issuecomment-4784058870
@funrollloops Thanks for taking a look! Replied to both inline comments. Regarding `TransformWithStateInPandasSerializer` not being removed: right, it stays because `TransformWithStateInPandasInitStateSerializer` still subclasses it and to be removed later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #56464:
URL: https://github.com/apache/spark/pull/56464#discussion_r3463340073
##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
return func, None, ser, ser
if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
- # We assume there is only one UDF here because grouped map doesn't
- # support combining multiple UDFs.
- assert num_udfs == 1
+ import pyarrow as pa
+ import pandas as pd
+
+ assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+ udf, arg_offsets, return_type = udfs[0]
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes
- arg_offsets, f = udfs[0]
parsed_offsets = extract_key_value_indexes(arg_offsets)
- ser.key_offsets = parsed_offsets[0][0]
+ assert len(parsed_offsets) == 1, (
+ "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+ )
+
+ key_offsets = parsed_offsets[0][0]
+ value_offsets = parsed_offsets[0][1]
+ output_schema = StructType([StructField("_0", return_type)])
+
stateful_processor_api_client = StatefulProcessorApiClient(
eval_conf.state_server_socket_port, eval_conf.grouping_key_schema
)
- def mapper(a):
- mode = a[0]
+ arrow_max_records_per_batch = runner_conf.arrow_max_records_per_batch
+ arrow_max_records_per_batch = (
+ arrow_max_records_per_batch if arrow_max_records_per_batch > 0 else 2**31 - 1
+ )
+ arrow_max_bytes_per_batch = runner_conf.arrow_max_bytes_per_batch
- if mode == TransformWithStateInPandasFuncMode.PROCESS_DATA:
- key = a[1]
+ def transform_with_state_func(
+ split_index: int,
+ batches: Iterator[pa.RecordBatch],
+ ) -> Iterator[pa.RecordBatch]:
+ """Apply transformWithStateInPandas UDF.
+
+ Data chunks for the same grouping key appear sequentially in the
+ input batches but may span batch boundaries, so rows are regrouped
+ by key and re-chunked into pandas DataFrames bounded by
+ arrow_max_records_per_batch and arrow_max_bytes_per_batch. The UDF
+ is invoked once per grouping key with a lazy iterator of chunks,
+ then once for PROCESS_TIMER and once for COMPLETE.
+ """
+ total_bytes = 0
+ total_rows = 0
+ average_arrow_row_size = 0.0
+
+ def row_stream():
+ nonlocal total_bytes, total_rows, average_arrow_row_size
+ for batch in batches:
+ # Short circuit batch size stats if the batch size is
+ # unlimited as computing batch size is computationally
+ # expensive.
+ if arrow_max_bytes_per_batch != 2**31 - 1 and batch.num_rows > 0:
+ total_bytes += sum(
+ buf.size
+ for col in batch.columns
+ for buf in col.buffers()
+ if buf is not None
+ )
+ total_rows += batch.num_rows
+ average_arrow_row_size = total_bytes / total_rows
+ data_pandas = ArrowBatchTransformer.to_pandas(
+ batch,
+ timezone=runner_conf.timezone,
+ prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,
Review Comment:
those arguments are passed in as general apis of serializers before. as we decouple the logic from serializer, we only need to supply parameters needed for this particular UDF eval type. The omitted args are exactly the defaults of `ArrowBatchTransformer.to_pandas(schema=None, struct_in_pandas="dict", ndarray_as_list=False, df_for_struct=False)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #56464:
URL: https://github.com/apache/spark/pull/56464#issuecomment-4784758383
Thanks @HyukjinKwon @uros-b and @funrollloops, merged to master/4.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #56464: [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF
URL: https://github.com/apache/spark/pull/56464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Posted by "funrollloops (via GitHub)" <gi...@apache.org>.
funrollloops commented on code in PR #56464:
URL: https://github.com/apache/spark/pull/56464#discussion_r3461827665
##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
return func, None, ser, ser
if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
- # We assume there is only one UDF here because grouped map doesn't
- # support combining multiple UDFs.
- assert num_udfs == 1
+ import pyarrow as pa
+ import pandas as pd
+
+ assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+ udf, arg_offsets, return_type = udfs[0]
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes
- arg_offsets, f = udfs[0]
parsed_offsets = extract_key_value_indexes(arg_offsets)
- ser.key_offsets = parsed_offsets[0][0]
+ assert len(parsed_offsets) == 1, (
+ "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+ )
+
+ key_offsets = parsed_offsets[0][0]
+ value_offsets = parsed_offsets[0][1]
+ output_schema = StructType([StructField("_0", return_type)])
+
stateful_processor_api_client = StatefulProcessorApiClient(
eval_conf.state_server_socket_port, eval_conf.grouping_key_schema
)
- def mapper(a):
- mode = a[0]
+ arrow_max_records_per_batch = runner_conf.arrow_max_records_per_batch
+ arrow_max_records_per_batch = (
+ arrow_max_records_per_batch if arrow_max_records_per_batch > 0 else 2**31 - 1
+ )
+ arrow_max_bytes_per_batch = runner_conf.arrow_max_bytes_per_batch
- if mode == TransformWithStateInPandasFuncMode.PROCESS_DATA:
- key = a[1]
+ def transform_with_state_func(
+ split_index: int,
+ batches: Iterator[pa.RecordBatch],
+ ) -> Iterator[pa.RecordBatch]:
+ """Apply transformWithStateInPandas UDF.
+
+ Data chunks for the same grouping key appear sequentially in the
+ input batches but may span batch boundaries, so rows are regrouped
+ by key and re-chunked into pandas DataFrames bounded by
+ arrow_max_records_per_batch and arrow_max_bytes_per_batch. The UDF
+ is invoked once per grouping key with a lazy iterator of chunks,
+ then once for PROCESS_TIMER and once for COMPLETE.
+ """
+ total_bytes = 0
+ total_rows = 0
+ average_arrow_row_size = 0.0
+
+ def row_stream():
+ nonlocal total_bytes, total_rows, average_arrow_row_size
+ for batch in batches:
+ # Short circuit batch size stats if the batch size is
+ # unlimited as computing batch size is computationally
+ # expensive.
+ if arrow_max_bytes_per_batch != 2**31 - 1 and batch.num_rows > 0:
+ total_bytes += sum(
+ buf.size
+ for col in batch.columns
+ for buf in col.buffers()
+ if buf is not None
+ )
+ total_rows += batch.num_rows
+ average_arrow_row_size = total_bytes / total_rows
+ data_pandas = ArrowBatchTransformer.to_pandas(
+ batch,
+ timezone=runner_conf.timezone,
+ prefer_int_ext_dtype=runner_conf.prefer_int_ext_dtype,
Review Comment:
we're passing far fewer arguments than in python/pyspark/sql/pandas/serializers.py:1037; why?
in particular we're missing schema, struct_in_pandas, ndarray_as_list, and df_for_struct
##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
return func, None, ser, ser
if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
- # We assume there is only one UDF here because grouped map doesn't
- # support combining multiple UDFs.
- assert num_udfs == 1
+ import pyarrow as pa
+ import pandas as pd
+
+ assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+ udf, arg_offsets, return_type = udfs[0]
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes
- arg_offsets, f = udfs[0]
parsed_offsets = extract_key_value_indexes(arg_offsets)
- ser.key_offsets = parsed_offsets[0][0]
+ assert len(parsed_offsets) == 1, (
+ "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+ )
+
+ key_offsets = parsed_offsets[0][0]
+ value_offsets = parsed_offsets[0][1]
+ output_schema = StructType([StructField("_0", return_type)])
Review Comment:
Did we previously wrap the returned value with a struct of one field? I can't find that logic anywhere in the touched code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57400][PYTHON] Refactor SQL_TRANSFORM_WITH_STATE_PANDAS_UDF [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #56464:
URL: https://github.com/apache/spark/pull/56464#discussion_r3463111993
##########
python/pyspark/worker.py:
##########
@@ -3373,37 +3349,147 @@ def map_batch(batch):
return func, None, ser, ser
if eval_type == PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF:
- # We assume there is only one UDF here because grouped map doesn't
- # support combining multiple UDFs.
- assert num_udfs == 1
+ import pyarrow as pa
+ import pandas as pd
+
+ assert num_udfs == 1, "One TRANSFORM_WITH_STATE_PANDAS UDF expected here."
+ udf, arg_offsets, return_type = udfs[0]
# See TransformWithStateInPandasExec for how arg_offsets are used to
# distinguish between grouping attributes and data attributes
- arg_offsets, f = udfs[0]
parsed_offsets = extract_key_value_indexes(arg_offsets)
- ser.key_offsets = parsed_offsets[0][0]
+ assert len(parsed_offsets) == 1, (
+ "Expected one pair of offsets for TRANSFORM_WITH_STATE_PANDAS UDF."
+ )
+
+ key_offsets = parsed_offsets[0][0]
+ value_offsets = parsed_offsets[0][1]
+ output_schema = StructType([StructField("_0", return_type)])
Review Comment:
Yes, this is the same wrapping as before, just inlined. Previously the output went through `ArrowStreamPandasUDFSerializer.dump_stream` -> `create_batch`, which builds `StructType([StructField(f"_{i}", t) for i, t in enumerate(types)])`. The TWS path always emits a single `(pdf, return_type)`, so `types == [return_type]` and that produced exactly the one `field _0` struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org