You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/09/14 09:39:32 UTC
(spark) branch branch-4.x updated: [SPARK-59435][UDF] Qualify UDF worker proto import paths
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new d7bd595fd6ab [SPARK-59435][UDF] Qualify UDF worker proto import paths
d7bd595fd6ab is described below
commit d7bd595fd6ab422c52caa35859e3df2f3dfe97ce
Author: Haiyang Sun <ha...@databricks.com>
AuthorDate: Mon Sep 14 18:38:59 2026 +0900
[SPARK-59435][UDF] Qualify UDF worker proto import paths
### What changes were proposed in this pull request?
This PR relocates the UDF worker protobuf definitions beneath
`udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/` and updates
their intra-proto imports to use the corresponding qualified paths.
This follows the namespaced source layouts used by other Spark protobuf
definitions, including:
- [Spark Connect common.proto](https://github.com/apache/spark/blob/master/sql/connect/common/src/main/protobuf/spark/connect/common.proto)
- [configuration schema](https://github.com/apache/spark/blob/master/common/config/src/main/protobuf/org/apache/spark/config/config_schema.proto)
- [status store types](https://github.com/apache/spark/blob/master/core/src/main/protobuf/org/apache/spark/status/protobuf/store_types.proto)
- [streaming state message](https://github.com/apache/spark/blob/master/sql/core/src/main/protobuf/org/apache/spark/sql/execution/streaming/StateMessage.proto)
### Why are the changes needed?
The UDF worker definitions previously used generic bare import names such as
`common.proto` and `udf_message.proto`. Relocating them establishes stable,
qualified import paths, avoiding ambiguity if Spark protobuf sources are later
collected, indexed, or compiled alongside definitions with similarly named
files.
The message definitions, generated Java package, and wire format are unchanged.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `protoc` compilation of the relocated UDF worker protobuf definitions.
- `build/mvn -pl udf/worker/proto,udf/worker/grpc -am -DskipTests compile`
- `build/mvn -pl udf/worker/grpc -am test`
- UDF worker core: 59 tests passed.
- UDF worker gRPC: 60 tests passed.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Closes #58738 from haiyangsun-db/SPARK-59435-qualify-udf-worker-proto-imports.
Authored-by: Haiyang Sun <ha...@databricks.com>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
(cherry picked from commit f3b1883379162accd389c4b7b9a776d10550d4bd)
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
udf/worker/README.md | 20 +++++++++++---------
.../{ => org/apache/spark/udf/worker}/common.proto | 0
.../apache/spark/udf/worker}/udf_message.proto | 2 +-
.../apache/spark/udf/worker}/udf_service.proto | 2 +-
.../apache/spark/udf/worker}/worker_spec.proto | 2 +-
5 files changed, 14 insertions(+), 12 deletions(-)
diff --git a/udf/worker/README.md b/udf/worker/README.md
index 9e364bdb3c4c..50ee7c6a60bc 100644
--- a/udf/worker/README.md
+++ b/udf/worker/README.md
@@ -34,10 +34,11 @@ provisioning service or daemon).
```
udf/worker/
├── proto/ -- protobuf message classes only (protobuf-java)
-│ worker_spec.proto -- UDFWorkerSpecification protobuf
-│ udf_message.proto -- UDF execution protocol messages (Init, UdfPayload, ...)
-│ udf_service.proto -- UdfWorker gRPC service (Execute, Manage)
-│ common.proto -- shared enums (UDFWorkerDataFormat, etc.)
+│ └── src/main/protobuf/org/apache/spark/udf/worker/
+│ worker_spec.proto -- UDFWorkerSpecification protobuf
+│ udf_message.proto -- UDF execution protocol messages (Init, UdfPayload, ...)
+│ udf_service.proto -- UdfWorker gRPC service (Execute, Manage)
+│ common.proto -- shared enums (UDFWorkerDataFormat, etc.)
│
├── core/ -- abstract interfaces
│ WorkerDispatcher.scala -- creates sessions, manages worker lifecycle
@@ -56,7 +57,7 @@ udf/worker/
GrpcWorkerChannel.scala -- owns a Netty gRPC channel and event loop
GrpcWorkerSession.scala -- one UDF execution over an Execute stream
UnixDomainSocketTransport.scala -- selects native epoll/kqueue transport
- (generated) -- UdfWorkerGrpc stubs from proto/udf_service.proto
+ (generated) -- UdfWorkerGrpc stubs from the UDF worker service proto
```
The `core/` package defines abstract interfaces that are independent of how
@@ -67,7 +68,7 @@ obtaining workers from a provisioning service or daemon.
The `grpc/` module owns the concrete direct dispatcher, channel and session
implementations, native Unix-domain-socket transport selection, gRPC service-stub
-generation (from `proto/`'s `udf_service.proto`), and the gRPC runtime dependencies.
+generation (from the UDF worker service proto above), and the gRPC runtime dependencies.
Keeping gRPC here means `proto/`, `core/`, and their consumers (`core`, `catalyst`,
`sql/core`) carry no gRPC dependency on their classpath.
@@ -81,9 +82,10 @@ Engine -> Worker: Init -> PayloadChunk* -> (DataRequest)* -> Finish (Cancel)?
Worker -> Engine: InitResponse -> (DataResponse)* -> (ErrorResponse)? -> (FinishResponse | CancelResponse)
```
-See `udf/worker/proto/src/main/protobuf/udf_message.proto` for the complete
-message definitions, ordering invariants, and error contract, and
-`udf_service.proto` for the gRPC service.
+See `udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_message.proto`
+for the complete message definitions, ordering invariants, and error contract, and
+`udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_service.proto`
+for the gRPC service.
### Direct worker creation
diff --git a/udf/worker/proto/src/main/protobuf/common.proto b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/common.proto
similarity index 100%
rename from udf/worker/proto/src/main/protobuf/common.proto
rename to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/common.proto
diff --git a/udf/worker/proto/src/main/protobuf/udf_message.proto b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_message.proto
similarity index 99%
rename from udf/worker/proto/src/main/protobuf/udf_message.proto
rename to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_message.proto
index dd7c9ceb963b..e1be17b15d9f 100644
--- a/udf/worker/proto/src/main/protobuf/udf_message.proto
+++ b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_message.proto
@@ -17,7 +17,7 @@
syntax = "proto3";
-import "common.proto";
+import "org/apache/spark/udf/worker/common.proto";
package org.apache.spark.udf.worker;
diff --git a/udf/worker/proto/src/main/protobuf/udf_service.proto b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_service.proto
similarity index 98%
rename from udf/worker/proto/src/main/protobuf/udf_service.proto
rename to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_service.proto
index 2140d4f48bca..ec68325b4fd7 100644
--- a/udf/worker/proto/src/main/protobuf/udf_service.proto
+++ b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/udf_service.proto
@@ -17,7 +17,7 @@
syntax = "proto3";
-import "udf_message.proto";
+import "org/apache/spark/udf/worker/udf_message.proto";
package org.apache.spark.udf.worker;
diff --git a/udf/worker/proto/src/main/protobuf/worker_spec.proto b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/worker_spec.proto
similarity index 99%
rename from udf/worker/proto/src/main/protobuf/worker_spec.proto
rename to udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/worker_spec.proto
index 83dac4f962e5..5e9ceaf0a933 100644
--- a/udf/worker/proto/src/main/protobuf/worker_spec.proto
+++ b/udf/worker/proto/src/main/protobuf/org/apache/spark/udf/worker/worker_spec.proto
@@ -17,7 +17,7 @@
syntax = "proto3";
-import "common.proto";
+import "org/apache/spark/udf/worker/common.proto";
package org.apache.spark.udf.worker;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org