[jira] [Updated] (SPARK-54559) Refactor the UDF protocol

["Tian Gao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-54559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tian Gao updated SPARK-54559:
-----------------------------
    Priority: Major  (was: Minor)

> Refactor the UDF protocol
> -------------------------
>
>                 Key: SPARK-54559
>                 URL: https://issues.apache.org/jira/browse/SPARK-54559
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.2.0
>            Reporter: Tian Gao
>            Priority: Major
>
> We have a very ad-hoc UDF protocol now. People just add random `read_int` to the UDF reading logic and it crashes the maintainability.
> We have requests regarding using grpc for UDF workers, but it's too much for a single change.
> Nonetheless, we should still try to make the current code more maintainable and move towards a "defined protocol" (it doesn't have to be grpc but we should have some validation for wrong messages, instead of hang at random places).
> The ultimate goal for now is to separate the whole socket communication from the UDF logic - read everything at once and validate it, then use it for UDF/UDTF creation.
> We will divide this effort into the following steps:
>  # Create a single data structure for runner_conf
>  # Unify the protocol for runner_conf so every UDF (specifically vanilla Python) sends a runner_conf (it can have a length of 0).
>  # Unify how the argument offsets are passed - they should use the same protocol so it's not determined based on eval type.
>  # Move all the socket reading from read_udf/read_udtf to a unified place. We read the whole thing (except for data) and build functions based on it.
>  # Add some sanity validation to our data (for example, for some integer value, we are expecting a sane number. A super large number suggests protocol issue).
>  # Potential cleanups for some other stuff that can be unified (it would be clear after we put all the logic together).
> We expect most of the changes done in worker.py - with some minor changes on JVM side to unify protocols.
> Once this is done, we should be easily convert it to any fancy RPC protocols if we need to.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org