You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/05/15 19:37:00 UTC
[jira] [Updated] (SPARK-55388) Refactor PythonEvalType processing logic
[ https://issues.apache.org/jira/browse/SPARK-55388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-55388:
----------------------------------
Affects Version/s: 4.3.0
(was: 4.2.0)
> Refactor PythonEvalType processing logic
> ----------------------------------------
>
> Key: SPARK-55388
> URL: https://issues.apache.org/jira/browse/SPARK-55388
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Yicong Huang
> Priority: Major
>
> In the current Python worker UDF execution path, the logic for each {{PythonEvalType}} is spread across three layers:
> - *Wrapper functions* (e.g. {{wrap_arrow_batch_iter_udf}}, {{wrap_scalar_arrow_udf}}, {{wrap_grouped_map_arrow_udf}}, etc.) that pre-process UDF arguments and post-process results.
> - *Mapper blocks* in {{read_udfs}} that handle input column selection, UDF invocation, and output assembly — often shared across unrelated eval types.
> - *Serializers* (e.g. {{ArrowStreamUDFSerializer}}) that mix data transformation (struct flatten/wrap, pandas conversion) with I/O concerns.
> To understand the full data flow for a single eval type, one has to trace through all three layers and reconstruct the implicit contracts between them (e.g. the serializer yields {{[batch]}} lists, the mapper indexes into them, the wrapper attaches {{(result, type)}} tuples).
> The goal is to make each eval type self-contained: all input transformation, UDF invocation, result verification, and output transformation should be co-located in one place in {{read_udfs}}, with serializers reduced to pure I/O.
> This can be done incrementally, one eval type at a time, starting with the simpler Arrow-based ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org