You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Haiyang Sun (Jira)" <ji...@apache.org> on 2026/09/25 14:20:00 UTC
[jira] [Updated] (SPARK-59049) Support udf chaining logic for external udfs
[ https://issues.apache.org/jira/browse/SPARK-59049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haiyang Sun updated SPARK-59049:
--------------------------------
Description:
PySpark can currently chain compatible UDFs and execute them in a single worker round trip. This requires both:
- Planner logic that groups the UDF expressions into one execution node.
- Worker support for executing multiple UDFs together.
To generalize this behavior for external UDFs, chaining should be modeled as an explicit engine-worker capability:
1. Not every worker supports UDF chaining. The worker specification should advertise this capability so the engine can skip the chaining optimization for unsupported workers.
2. Chaining should have a language-neutral protocol representation. A generic planning rule can combine UDFs into an expression tree with metadata describing how inputs are mapped. Workers implementing the protocol can then execute the combined UDF in one round trip. The representation should support simple linear chains initially while allowing more complex trees in the future.
3. Language-specific client and payload serialization should remain independent of chaining. The planner should package the component UDFs as one logical composite UDF, which downstream planning treats like any other UDF and maps to one UDF session.
4. External UDF planning should enforce a one-to-one mapping between a Spark execution node and a UDF session. Each node therefore contains one logical UDF, which may be either a single UDF or a composite chained UDF.
was:
PySpark can currently chain compatible UDFs and execute them in a single worker round trip. This requires both:
- Planner logic that groups the UDF expressions into one execution node.
- Worker support for executing multiple UDFs together.
To generalize this behavior for external UDFs, chaining should be modeled as an explicit engine-worker capability:
1. Not every worker supports UDF chaining. The worker specification should advertise this capability so the engine can skip the chaining optimization for unsupported workers.
2. Chaining should have a language-neutral protocol representation. A generic planning rule can combine UDFs into an expression tree with metadata describing how inputs are mapped. Workers implementing the protocol can then execute the combined UDF in one round trip. The representation should support simple linear chains initially while allowing more complex trees in the future.
3. Language-specific client and payload serialization should remain independent of chaining. The planner should package the component UDFs as one logical composite UDF, which downstream planning treats like any other UDF and maps to one UDF session.
4. External UDF planning should enforce a one-to-one mapping between a Spark execution node and a UDF session. Each node therefore contains one logical UDF, which may be either a single UDF or a composite chained UDF.
> Support udf chaining logic for external udfs
> --------------------------------------------
>
> Key: SPARK-59049
> URL: https://issues.apache.org/jira/browse/SPARK-59049
> Project: Spark
> Issue Type: Sub-task
> Components: UDF
> Affects Versions: 4.3.0
> Reporter: Haiyang Sun
> Priority: Major
>
> PySpark can currently chain compatible UDFs and execute them in a single worker round trip. This requires both:
> - Planner logic that groups the UDF expressions into one execution node.
> - Worker support for executing multiple UDFs together.
> To generalize this behavior for external UDFs, chaining should be modeled as an explicit engine-worker capability:
> 1. Not every worker supports UDF chaining. The worker specification should advertise this capability so the engine can skip the chaining optimization for unsupported workers.
> 2. Chaining should have a language-neutral protocol representation. A generic planning rule can combine UDFs into an expression tree with metadata describing how inputs are mapped. Workers implementing the protocol can then execute the combined UDF in one round trip. The representation should support simple linear chains initially while allowing more complex trees in the future.
> 3. Language-specific client and payload serialization should remain independent of chaining. The planner should package the component UDFs as one logical composite UDF, which downstream planning treats like any other UDF and maps to one UDF session.
> 4. External UDF planning should enforce a one-to-one mapping between a Spark execution node and a UDF session. Each node therefore contains one logical UDF, which may be either a single UDF or a composite chained UDF.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org