You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2026/08/29 20:25:00 UTC

[jira] [Created] (SPARK-59113) Fix PySpark UDF parity in the Rust drop-in: string args, pandas_udf family, applyInPandas/mapInPandas/mapInArrow/cogroup, and udf.register

Hyukjin Kwon created SPARK-59113:
------------------------------------

             Summary: Fix PySpark UDF parity in the Rust drop-in: string args, pandas_udf family, applyInPandas/mapInPandas/mapInArrow/cogroup, and udf.register
                 Key: SPARK-59113
                 URL: https://issues.apache.org/jira/browse/SPARK-59113
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.2.0
            Reporter: Hyukjin Kwon


A full behavioral UDF sweep of the Rust drop-in against real PySpark 4.2.0 (live Spark Connect server) found the UDF subsystem broadly broken; this fixes it end to end (battery 32/32, offline suite 236 passed).

- Scalar/arrow-optimized python UDFs treated a string column arg as a literal (udf(f)("c") vs udf(f)(F.col("c"))); and @udf/@pandas_udf/@arrow_udf decorator forms raised TypeError. Fixed the UDF __call__ arg coercion (str->col) and the decorator/return-type handling.
- pandas_udf family: the command pickled a DDL string instead of a concrete DataType (worker Arrow serializer failed), and eval-type wasn't inferred from type hints. Now pickles a real DataType and infers eval type (scalar 200 / iterator 204 / grouped-agg 202).
- Grouped/map UDFs (applyInPandas, mapInPandas, mapInArrow, applyInArrow, cogroup): the GroupMap/MapPartitions/CoGroupMap relation didn't pass input column names, so the worker DataFrame was unnamed (KeyError). Now passes the child columns as the UDF arguments.
- spark.udf.register(name, f, T) didn't notify the server (UNRESOLVED_ROUTINE in SQL). Now sends a RegisterFunction command with the cloudpickled PythonUDF.

Verified live: pandas_udf scalar/iterator/grouped-agg, applyInPandas/mapInPandas/mapInArrow/applyInArrow/cogroup, register+SQL all return correct results.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org