[jira] [Created] (SPARK-55206) Support basic single parameter addition Python UDF transpilation

["Holden Karau (Jira)" <ji...@apache.org>]

Holden Karau created SPARK-55206:
------------------------------------

             Summary: Support basic single parameter addition Python UDF transpilation
                 Key: SPARK-55206
                 URL: https://issues.apache.org/jira/browse/SPARK-55206
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 4.3
            Reporter: Holden Karau


For the sake of having code land in testable/reviewable chunks I think the first chunk we should do should be the bare bones for simple numerical PyUDF transpilation. To accomplish that we'll need to include the following:
 * Parameter for enabing transpilation (clearly marked as experimental)
 * AST extraction
 * AST transpilation
 * UDF node extended to support both transpiled and "original" UDF code (possible with an explanation string for why no transpilation)
 * UDF node printout in the query plan should be clear if transpiled or not
 * Property-based tests for transpiled UDFS ensuring correctness
 * Type promotion equivalence (does not need to be complete given numeric only)

My theory here is that by putting all of this together in one first PR, we both establish a solid framework for future work and ensure the framework we land is actually usable.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org