You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2026/08/29 21:00:01 UTC
[jira] [Resolved] (SPARK-59113) Fix PySpark UDF parity in the Rust drop-in: string args, pandas_udf family, applyInPandas/mapInPandas/mapInArrow/cogroup, and udf.register
[ https://issues.apache.org/jira/browse/SPARK-59113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-59113.
----------------------------------
Fix Version/s: connect-rust-4.2.0
connect-rust-4.3.0
Resolution: Fixed
> Fix PySpark UDF parity in the Rust drop-in: string args, pandas_udf family, applyInPandas/mapInPandas/mapInArrow/cogroup, and udf.register
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59113
> URL: https://issues.apache.org/jira/browse/SPARK-59113
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.2.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.2.0, connect-rust-4.3.0
>
>
> A full behavioral UDF sweep of the Rust drop-in against real PySpark 4.2.0 (live Spark Connect server) found the UDF subsystem broadly broken; this fixes it end to end (battery 32/32, offline suite 236 passed).
> - Scalar/arrow-optimized python UDFs treated a string column arg as a literal (udf(f)("c") vs udf(f)(F.col("c"))); and @udf/@pandas_udf/@arrow_udf decorator forms raised TypeError. Fixed the UDF __call__ arg coercion (str->col) and the decorator/return-type handling.
> - pandas_udf family: the command pickled a DDL string instead of a concrete DataType (worker Arrow serializer failed), and eval-type wasn't inferred from type hints. Now pickles a real DataType and infers eval type (scalar 200 / iterator 204 / grouped-agg 202).
> - Grouped/map UDFs (applyInPandas, mapInPandas, mapInArrow, applyInArrow, cogroup): the GroupMap/MapPartitions/CoGroupMap relation didn't pass input column names, so the worker DataFrame was unnamed (KeyError). Now passes the child columns as the UDF arguments.
> - spark.udf.register(name, f, T) didn't notify the server (UNRESOLVED_ROUTINE in SQL). Now sends a RegisterFunction command with the cloudpickled PythonUDF.
> Verified live: pandas_udf scalar/iterator/grouped-agg, applyInPandas/mapInPandas/mapInArrow/applyInArrow/cogroup, register+SQL all return correct results.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org