[jira] [Created] (SPARK-54559) Refactor the UDF protocol

["Tian Gao (Jira)" <ji...@apache.org>]

Tian Gao created SPARK-54559:
--------------------------------

             Summary: Refactor the UDF protocol
                 Key: SPARK-54559
                 URL: https://issues.apache.org/jira/browse/SPARK-54559
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Tian Gao


We have a very ad-hoc UDF protocol now. People just add random `read_int` to the UDF reading logic and it crashes the maintainability.

We have requests regarding using grpc for UDF workers, but it's too much for a single change.

Nonetheless, we should still try to make the current code more maintainable and move towards a "defined protocol" (it doesn't have to be grpc but we should have some validation for wrong messages, instead of hang at random places).

The ultimate goal for now is to separate the whole socket communication from the UDF logic - read everything at once and validate it, then use it for UDF/UDTF creation.

We will divide this effort into the following steps (to make sure code sync is not a disaster).
 # Create a single data structure for runner_conf
 # Unify the protocol for runner_conf so every UDF (specifically vanilla Python) sends a runner_conf (it can have a length of 0).
 # Unify how the argument offsets are passed - they should use the same protocol so it's not determined based on eval type.
 # Move all the socket reading from read_udf/read_udtf to a unified place. We read the whole thing (except for data) and build functions based on it.
 # Add some sanity validation to our data (for example, for some integer value, we are expecting a sane number. A super large number suggests protocol issue).
 # Potential cleanups for some other stuff that can be unified (it would be clear after we put all the logic together).

We expect most of the changes done in worker.py - with some minor changes on JVM side to unify protocols.

Once this is done, we should be easily convert it to any fancy RPC protocols if we need to.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org