You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by dt...@apache.org on 2026/04/29 18:07:50 UTC
(spark) branch master updated: [SPARK-56412] Implement DirectWorkerDispatcher that spawns UDF workers as local callable
This is an automated email from the ASF dual-hosted git repository.
dtenedor pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0ae136e491c3 [SPARK-56412] Implement DirectWorkerDispatcher that spawns UDF workers as local callable
0ae136e491c3 is described below
commit 0ae136e491c3f911e49b57ea544bda44188d069b
Author: Haiyang Sun <ha...@databricks.com>
AuthorDate: Wed Apr 29 11:07:36 2026 -0700
[SPARK-56412] Implement DirectWorkerDispatcher that spawns UDF workers as local callable
### What changes were proposed in this pull request?
Adds the engine-side foundation for the language-agnostic UDF framework introduced by [SPARK-55278](https://issues.apache.org/jira/browse/SPARK-55278). Lives under `udf/worker/core` with no dependency
beyond `spark-tags` and the existing `udf/worker/proto`.
Public API in `org.apache.spark.udf.worker.core`:
- `WorkerDispatcher` — owns workers for one `UDFWorkerSpecification`; hands callers `WorkerSession`s.
- `WorkerSession` — one UDF execution. `init` (exactly once) and `process` (at most once, after init) are `final` with AtomicBoolean guards; subclasses implement `doInit` / `doProcess`. `InitMessage` is
a placeholder until the UDF wire protocol lands.
- `WorkerConnection` — transport channel; one server-side endpoint shared by all sessions.
- `UnixSocketWorkerConnection` — UDS subclass; owns the socket path and removes the file on close.
- `WorkerSecurityScope`, `WorkerLogger` — pool-partitioning identity and minimal logging surface (no SLF4J dep).
Direct-creation impl in `.direct`:
- `DirectWorkerDispatcher` (abstract, transport-agnostic) — process spawn, environment lifecycle (`installation` / `environment_verification` / `environment_cleanup`), worker registry, race-safe close.
Five protected hooks for transport plug-in: `newEndpointAddress`, `waitForReady`, `cleanupEndpointAddress`, `closeTransport`, `validateTransportSupport`.
- `DirectUnixSocketWorkerDispatcher` (abstract) — UDS implementation of the hooks (private 0700 socket dir, file-existence polling, file deletion).
- `DirectWorkerProcess` + `WorkerArtifacts` (process + connection + output log) — close sends SIGTERM, waits `gracefulTimeoutMs`, then artifacts handle connection close + SIGKILL+reap + file cleanup.
- `DirectWorkerException` / `DirectWorkerTimeoutException` — runtime-failure types; the timeout subclass lets callers discriminate retry policy.
Hardening:
- CAS-idempotent `close()`; acquire-before-publish + post-publish `closed` re-check makes `createSession` racing with `close` either throw or get a session whose worker is reaped.
- SIGKILL paths use a bounded-wait reap helper so the kernel actually reaps the child; stuck zombies WARN with a context tag.
- Proto-provided init/graceful timeouts are clamped to a 30s engine maximum (WARN on clamp).
- All install failures (timeout or non-zero exit) transition the environment to `Failed` permanently.
- UDS socket directory is created 0700; non-POSIX fallback WARNs if the platform refuses the perms.
Proto:
- `UDFWorkerProperties.connection` is singular (one server-side endpoint shared by sessions).
- `WorkerConnectionSpec` proto message renamed to disambiguate from the core Scala `WorkerConnection`.
Notably out of scope (TODOs in code): connection pooling, security-scope partitioning, indirect creation (daemon), TCP transport (a sibling subclass implementing the same five hooks),
retriable-vs-permanent failure classification, shared shutdown hook across dispatchers.
### Why are the changes needed?
This is the Spark-independent foundation the rest of the language-agnostic UDF work will build on. The dispatcher is already factored to support transports beyond UDS.
### Does this PR introduce _any_ user-facing change?
No. The module is not wired into any planner/executor path; classes are `Experimental`.
### How was this patch tested?
`DirectWorkerDispatcherSuite` (31 tests) covering: worker spawn / SIGTERM / SIGKILL escalation; concurrent createSession (distinct processes and socket paths); the close-vs-in-flight-createSession race;
createSession-after-close rejection; socket-dir 0700 + cleanup; init / callable / graceful timeout caps and behaviour; environment lifecycle (skip-install-on-verify, install-once-under-concurrency,
cached failure, install-timeout permanence, cleanup on close); spec validation (missing/non-UDS connection, verify-without-install, securityScope).
build/mvn test -pl udf/worker/core -am
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Closes #55406 from haiyangsun-db/SPARK-56412.
Authored-by: Haiyang Sun <ha...@databricks.com>
Signed-off-by: Daniel Tenedorio <da...@databricks.com>
---
udf/worker/README.md | 158 +++-
.../worker/core/UnixSocketWorkerConnection.scala} | 20 +-
.../spark/udf/worker/core/WorkerConnection.scala | 49 +
.../spark/udf/worker/core/WorkerDispatcher.scala | 6 +-
.../spark/udf/worker/core/WorkerLogger.scala | 51 ++
.../spark/udf/worker/core/WorkerSession.scala | 129 ++-
.../direct/DirectUnixSocketWorkerDispatcher.scala | 145 +++
.../core/direct/DirectWorkerDispatcher.scala | 532 +++++++++++
.../worker/core/direct/DirectWorkerException.scala | 46 +
.../worker/core/direct/DirectWorkerProcess.scala | 162 ++++
.../worker/core/direct/DirectWorkerSession.scala | 56 ++
.../worker/core/DirectWorkerDispatcherSuite.scala | 981 +++++++++++++++++++++
.../udf/worker/core/WorkerAbstractionSuite.scala | 25 -
udf/worker/proto/src/main/protobuf/common.proto | 7 +
.../proto/src/main/protobuf/worker_spec.proto | 34 +-
15 files changed, 2329 insertions(+), 72 deletions(-)
diff --git a/udf/worker/README.md b/udf/worker/README.md
index fa27430b62b6..b843c430d0e0 100644
--- a/udf/worker/README.md
+++ b/udf/worker/README.md
@@ -5,44 +5,162 @@ Package structure for the UDF worker framework described in
## Overview
-Spark processes a UDF by first obtaining a **WorkerDispatcher** from the worker
-specification (plus context such as security scope). The dispatcher manages the
-actual worker processes behind the scenes -- pooling, reuse, and termination are
-all invisible to Spark.
+Spark processes a UDF by obtaining a **WorkerDispatcher** from a worker
+specification. The dispatcher manages workers behind the scenes. From
+the dispatcher, Spark gets a **WorkerSession** -- one per UDF invocation --
+with an Iterator-to-Iterator `process` API that streams input batches
+through the worker and returns result batches.
-From the dispatcher, Spark gets a **WorkerSession**, which represents one single
-UDF execution and can carry per-execution state. A WorkerSession is not 1-to-1
-mapped to an actual worker -- multiple sessions may share the same underlying
-worker when it is reused. Worker reuse is managed by each dispatcher
-implementation based on the worker specification.
+```
+UDFWorkerSpecification -- how to create and configure workers
+ |
+ v
+WorkerDispatcher -- manages workers, creates sessions
+ |
+ v
+WorkerSession -- one UDF execution
+ | 1. session.init(InitMessage(payload, inputSchema, outputSchema))
+ | 2. val results = session.process(inputBatches)
+ | 3. session.close()
+```
+
+How workers are created depends on the dispatcher implementation. The
+framework currently provides **direct worker creation** (local OS
+processes) and is designed for future **indirect creation** (via a
+provisioning service or daemon).
## Sub-packages
```
udf/worker/
-├── proto/ Protobuf definition of the worker specification
-│ (UDFWorkerSpecification).
-│ WorkerSpecification -- typed Scala wrapper around the protobuf spec.
-└── core/ Engine-side APIs (all @Experimental):
- WorkerDispatcher -- manages workers for one spec; creates sessions.
- WorkerSession -- represents one single UDF execution.
- WorkerSecurityScope -- security boundary for connection pooling.
+├── proto/
+│ worker_spec.proto -- UDFWorkerSpecification protobuf (+ generated Java classes)
+│ common.proto -- shared enums (UDFWorkerDataFormat, etc.)
+│
+└── core/ -- abstract interfaces
+ WorkerDispatcher.scala -- creates sessions, manages worker lifecycle
+ WorkerSession.scala -- per-UDF init/process/cancel/close + InitMessage
+ WorkerConnection.scala -- transport channel abstraction
+ WorkerSecurityScope.scala -- security boundary for worker pooling
+ │
+ └── direct/ -- "direct" creation: local OS processes
+ DirectWorkerDispatcher.scala -- spawns processes, env lifecycle
+ DirectWorkerProcess.scala -- OS process + connection + UDS socket
+ DirectWorkerSession.scala -- session backed by a direct process
+```
+
+The `core/` package defines abstract interfaces that are independent of how
+workers are created. The `core/direct/` sub-package implements "direct"
+worker creation where Spark spawns local OS processes. Future packages
+(e.g., `core/indirect/`) can implement alternative creation modes such as
+obtaining workers from a provisioning service or daemon.
+
+### Direct worker creation
+
+`DirectWorkerDispatcher` spawns worker processes locally. On the first
+session, it runs the optional environment lifecycle callables from the
+`UDFWorkerSpecification`:
+
+- **`environmentVerification`** -- checks if the environment is ready
+ (exit 0 = ready). When it succeeds, installation is skipped.
+- **`installation`** -- prepares the environment (installs runtime,
+ dependencies, worker binaries). Only runs when verification is absent
+ or fails.
+- **`environmentCleanup`** -- runs after the dispatcher is closed or on
+ JVM shutdown to clean up temporary resources.
+
+Environment setup runs **once per dispatcher** (not per session).
+Workers are terminated via SIGTERM/SIGKILL when the dispatcher is closed.
+
+## Basic usage (Scala)
+
+```scala
+import org.apache.spark.udf.worker.{
+ DirectWorker, ProcessCallable, UDFProtoCommunicationPattern,
+ UDFWorkerDataFormat, UDFWorkerProperties, UDFWorkerSpecification,
+ UnixDomainSocket, WorkerCapabilities, WorkerConnectionSpec, WorkerEnvironment}
+import org.apache.spark.udf.worker.core._
+
+// 1. Define a worker spec (direct creation mode).
+val spec = UDFWorkerSpecification.newBuilder()
+ .setEnvironment(WorkerEnvironment.newBuilder()
+ .setEnvironmentVerification(ProcessCallable.newBuilder()
+ .addCommand("python").addCommand("-c").addCommand("import my_udf_worker").build())
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("pip").addCommand("install").addCommand("my_udf_worker").build())
+ .build())
+ .setCapabilities(WorkerCapabilities.newBuilder()
+ .addSupportedDataFormats(UDFWorkerDataFormat.ARROW)
+ .addSupportedCommunicationPatterns(
+ UDFProtoCommunicationPattern.BIDIRECTIONAL_STREAMING)
+ .build())
+ .setDirect(DirectWorker.newBuilder()
+ .setRunner(ProcessCallable.newBuilder()
+ .addCommand("python").addCommand("-m").addCommand("my_udf_worker").build())
+ .setProperties(UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setUnixDomainSocket(UnixDomainSocket.getDefaultInstance).build())
+ .build())
+ .build())
+ .build()
+
+// 2. Create a dispatcher. Use a protocol-specific subclass of
+// DirectWorkerDispatcher (e.g., gRPC over UDS).
+val dispatcher: WorkerDispatcher = ...
+
+// 3. Create a session for one UDF execution.
+val session = dispatcher.createSession(securityScope = None)
+try {
+ // 4. Initialize with the serialized function and schemas.
+ session.init(InitMessage(
+ functionPayload = serializedFunction,
+ inputSchema = arrowInputSchema,
+ outputSchema = arrowOutputSchema))
+
+ // 5. Process data -- Iterator in, Iterator out.
+ val results: Iterator[Array[Byte]] =
+ session.process(inputBatches)
+
+ // Consume results lazily.
+ results.foreach(processResultBatch)
+} finally {
+ session.close()
+}
+
+// 6. Shut down all workers.
+dispatcher.close()
```
## Build
SBT:
```
-build/sbt "udf-worker-core/compile"
-build/sbt "udf-worker-core/test"
+build/sbt "udf-worker-proto/compile" "udf-worker-core/compile"
```
Maven:
```
-./build/mvn -pl udf/worker/proto,udf/worker/core -am compile
-./build/mvn -pl udf/worker/proto,udf/worker/core -am test
+build/mvn compile -pl udf/worker/proto,udf/worker/core -am
```
+## Test
+
+SBT:
+```
+build/sbt "udf-worker-core/test"
+```
+
+## Current status
+
+This is the **first MVP** providing the core abstraction layer and the
+direct worker dispatcher.
+The following are left as TODOs:
+
+- **Connection pooling** -- reuse workers across sessions
+- **Security scope isolation** -- partition pools by `WorkerSecurityScope`
+- **Indirect worker creation** -- obtain workers from a service or daemon
+- **Protocol-specific implementations** -- e.g., gRPC over UDS
+
## Design references
* [SPIP Language-agnostic UDF Protocol for Spark](https://docs.google.com/document/d/19Whzq127QxVt2Luk0EClgaDtcpBsFUp67NcVdKKyPF8/edit?tab=t.0)
diff --git a/udf/worker/proto/src/main/scala/org/apache/spark/udf/worker/WorkerSpecification.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/UnixSocketWorkerConnection.scala
similarity index 57%
rename from udf/worker/proto/src/main/scala/org/apache/spark/udf/worker/WorkerSpecification.scala
rename to udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/UnixSocketWorkerConnection.scala
index e25b99b69990..b3b40d16e744 100644
--- a/udf/worker/proto/src/main/scala/org/apache/spark/udf/worker/WorkerSpecification.scala
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/UnixSocketWorkerConnection.scala
@@ -14,14 +14,28 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.spark.udf.worker
+package org.apache.spark.udf.worker.core
+
+import java.io.File
import org.apache.spark.annotation.Experimental
/**
* :: Experimental ::
- * Typed Scala wrapper around the protobuf [[UDFWorkerSpecification]].
+ * A [[WorkerConnection]] over a Unix domain socket. Owns the socket
+ * path and removes the socket file on [[close]]. Subclasses provide the
+ * protocol-specific channel (e.g. gRPC over UDS) and may override
+ * [[close]] to add transport-level shutdown -- they should call
+ * `super.close()` to ensure the socket file is removed.
+ *
+ * [[close]] is idempotent: deleting an already-removed file is a no-op.
*/
@Experimental
-class WorkerSpecification(val proto: UDFWorkerSpecification) {
+abstract class UnixSocketWorkerConnection(val socketPath: String)
+ extends WorkerConnection {
+
+ override def close(): Unit = {
+ val f = new File(socketPath)
+ if (f.exists()) f.delete()
+ }
}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerConnection.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerConnection.scala
new file mode 100644
index 000000000000..82b2fff8df58
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerConnection.scala
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * A transport-level connection to a running UDF worker process.
+ *
+ * A [[WorkerConnection]] represents the communication channel between the
+ * Spark engine and a single worker process (e.g., a gRPC channel over a
+ * Unix domain socket, or a raw TCP socket). It is owned by a worker
+ * process wrapper (e.g., [[direct.DirectWorkerProcess]]) and shared
+ * across all [[WorkerSession]]s that use that process.
+ *
+ * One connection, many sessions: the worker exposes a single server-side
+ * endpoint that all sessions share. For gRPC, per-session work lives on
+ * multiplexed streams over this channel.
+ *
+ * Implementations expose only lifecycle. Data transmission happens at
+ * the [[WorkerSession]] level -- this class is solely about whether the
+ * channel is open.
+ *
+ * '''Relationship to other classes (direct creation mode):'''
+ * {{{
+ * DirectWorkerProcess 1 --- 1 WorkerConnection (transport over UDS)
+ * DirectWorkerProcess 1 --- * WorkerSession (UDF executions)
+ * }}}
+ */
+@Experimental
+abstract class WorkerConnection extends AutoCloseable {
+ /** Returns true if the underlying transport channel is still usable. */
+ def isActive: Boolean
+}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerDispatcher.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerDispatcher.scala
index 58fabbaea00d..008cfc2993a0 100644
--- a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerDispatcher.scala
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerDispatcher.scala
@@ -17,11 +17,11 @@
package org.apache.spark.udf.worker.core
import org.apache.spark.annotation.Experimental
-import org.apache.spark.udf.worker.WorkerSpecification
+import org.apache.spark.udf.worker.UDFWorkerSpecification
/**
* :: Experimental ::
- * Manages workers for a single [[WorkerSpecification]] and hides worker details from Spark.
+ * Manages workers for a single [[UDFWorkerSpecification]] and hides worker details from Spark.
*
* A [[WorkerDispatcher]] is created from a worker specification (plus context such
* as security scope). It owns the underlying worker processes and connections,
@@ -31,7 +31,7 @@ import org.apache.spark.udf.worker.WorkerSpecification
@Experimental
trait WorkerDispatcher extends AutoCloseable {
- def workerSpec: WorkerSpecification
+ def workerSpec: UDFWorkerSpecification
/**
* Creates a [[WorkerSession]] that maps to one single UDF execution.
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerLogger.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerLogger.scala
new file mode 100644
index 000000000000..a8f135f68890
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerLogger.scala
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Minimal logging surface used by the udf/worker framework.
+ *
+ * The framework deliberately does not depend on SLF4J (or any other
+ * concrete logging backend) so callers can embed it without dragging a
+ * specific logger onto the classpath. Embedders should supply an
+ * adapter that forwards to their preferred backend (Spark's `Logging`
+ * trait, SLF4J, java.util.logging, etc.).
+ *
+ * Only the methods actually used by the framework are exposed.
+ * Messages are passed by-name so the formatting cost is avoided when
+ * the backend decides to drop the event.
+ */
+@Experimental
+trait WorkerLogger {
+ def warn(msg: => String): Unit
+ def warn(msg: => String, t: Throwable): Unit
+ def debug(msg: => String): Unit
+ def debug(msg: => String, t: Throwable): Unit
+}
+
+object WorkerLogger {
+ /** Discards all messages. Default for callers that don't wire up logging. */
+ val NoOp: WorkerLogger = new WorkerLogger {
+ override def warn(msg: => String): Unit = ()
+ override def warn(msg: => String, t: Throwable): Unit = ()
+ override def debug(msg: => String): Unit = ()
+ override def debug(msg: => String, t: Throwable): Unit = ()
+ }
+}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerSession.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerSession.scala
index 83c392a895b6..f4c4091688c9 100644
--- a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerSession.scala
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/WorkerSession.scala
@@ -16,22 +16,133 @@
*/
package org.apache.spark.udf.worker.core
+import java.util.concurrent.atomic.AtomicBoolean
+
import org.apache.spark.annotation.Experimental
/**
* :: Experimental ::
- * Represents one single UDF execution.
+ * Carries all information needed to initialize a UDF execution on a worker.
+ *
+ * This message is passed to [[WorkerSession#init]] and contains the function
+ * definition, schemas, and any additional configuration.
*
- * A [[WorkerSession]] is obtained from [[WorkerDispatcher#createSession]] and
- * can carry per-execution state for that UDF invocation. Implementations may
- * add concrete data-processing methods and lifecycle hooks as needed.
+ * Placeholder: will be replaced by a generated proto message once the
+ * UDF wire protocol lands. Do not rely on case-class equality --
+ * `Array[Byte]` fields compare by reference.
*
- * A WorkerSession is not 1-to-1 mapped to an actual worker process. Multiple
- * WorkerSessions may be backed by the same worker when the worker is reused.
- * Worker reuse and pooling are managed by each [[WorkerDispatcher]]
- * implementation based on the [[WorkerSpecification]].
+ * @param functionPayload serialized function (e.g., pickled Python, JVM bytes)
+ * @param inputSchema serialized input schema (e.g., Arrow schema bytes)
+ * @param outputSchema serialized output schema (e.g., Arrow schema bytes)
+ * @param properties additional key-value configuration. Can carry
+ * protocol-specific or engine-specific metadata that
+ * does not yet have a dedicated field.
+ */
+@Experimental
+case class InitMessage(
+ functionPayload: Array[Byte],
+ inputSchema: Array[Byte],
+ outputSchema: Array[Byte],
+ properties: Map[String, String] = Map.empty)
+
+/**
+ * :: Experimental ::
+ * One UDF execution on a worker -- the main interface Spark uses to run UDFs.
+ *
+ * A [[WorkerSession]] is the '''per-UDF-invocation''' handle that Spark
+ * obtains from [[WorkerDispatcher#createSession]]. It carries the full
+ * init / data-stream / finish lifecycle for a single UDF evaluation.
+ *
+ * A [[WorkerSession]] does ''not'' own the underlying worker or its
+ * transport channel -- those are managed by the [[WorkerDispatcher]].
+ * Multiple sessions may share the same worker when the worker supports
+ * concurrency.
+ *
+ * '''Usage:'''
+ * {{{
+ * val session = dispatcher.createSession(securityScope = None)
+ * try {
+ * session.init(InitMessage(functionPayload, inputSchema, outputSchema))
+ * val results = session.process(inputBatches)
+ * results.foreach(handleBatch)
+ * } finally {
+ * session.close()
+ * }
+ * }}}
+ *
+ * '''Lifecycle:'''
+ * - [[init]] must be called exactly once before [[process]].
+ * - [[process]] must be called at most once per session.
+ * - [[close]] must always be called (use try-finally).
+ * - [[cancel]] may be called at any time to abort execution.
+ *
+ * The lifecycle is enforced here: [[init]] and [[process]] are `final`
+ * and delegate to [[doInit]] / [[doProcess]] after AtomicBoolean guards.
+ * Subclasses implement the protocol-specific work and do not re-check
+ * the contract.
*/
@Experimental
abstract class WorkerSession extends AutoCloseable {
- override def close(): Unit = {}
+
+ private val initialized = new AtomicBoolean(false)
+ private val processed = new AtomicBoolean(false)
+
+ /**
+ * Initializes the UDF execution. Must be called exactly once before
+ * [[process]].
+ *
+ * Throws `IllegalStateException` if called more than once.
+ *
+ * @param message the initialization parameters including the serialized
+ * function, input/output schemas, and configuration.
+ */
+ final def init(message: InitMessage): Unit = {
+ if (!initialized.compareAndSet(false, true)) {
+ throw new IllegalStateException("init has already been called on this session")
+ }
+ doInit(message)
+ }
+
+ /**
+ * Processes input data through the worker and returns results.
+ *
+ * Follows Spark's Iterator-to-Iterator pattern: input batches are streamed
+ * to the worker, and result batches are lazily pulled from the returned
+ * iterator. The session sends a Finish signal to the worker when the input
+ * iterator is exhausted.
+ *
+ * Must be called after [[init]] and at most once per session.
+ * Throws `IllegalStateException` if called before [[init]] or more than once.
+ *
+ * @param input iterator of raw input data batches (e.g., Arrow IPC)
+ * @return iterator of raw result data batches
+ */
+ final def process(input: Iterator[Array[Byte]]): Iterator[Array[Byte]] = {
+ if (!initialized.get()) {
+ throw new IllegalStateException("process called before init")
+ }
+ if (!processed.compareAndSet(false, true)) {
+ throw new IllegalStateException("process has already been called on this session")
+ }
+ doProcess(input)
+ }
+
+ /** Subclass hook for [[init]]. Called once, after the guard. */
+ protected def doInit(message: InitMessage): Unit
+
+ /** Subclass hook for [[process]]. Called at most once, after the guard. */
+ protected def doProcess(input: Iterator[Array[Byte]]): Iterator[Array[Byte]]
+
+ /**
+ * Requests cancellation of the current UDF execution.
+ *
+ * '''Thread-safety:''' implementations must allow [[cancel]] to be called
+ * from a thread different from the one driving [[process]] (typically a
+ * task interruption thread). It may be invoked at any point after
+ * [[init]] and should be a no-op if execution has already finished.
+ */
+ def cancel(): Unit
+
+ /** Closes this session and releases resources. */
+ override def close(): Unit
}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectUnixSocketWorkerDispatcher.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectUnixSocketWorkerDispatcher.scala
new file mode 100644
index 000000000000..8da0354187e4
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectUnixSocketWorkerDispatcher.scala
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core.direct
+
+import java.io.File
+import java.nio.file.{Files, Path}
+import java.nio.file.attribute.PosixFilePermissions
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.udf.worker.UDFWorkerSpecification
+import org.apache.spark.udf.worker.core.{UnixSocketWorkerConnection, WorkerLogger}
+import org.apache.spark.udf.worker.core.direct.DirectWorkerDispatcher.SOCKET_POLL_INTERVAL_MS
+
+/**
+ * :: Experimental ::
+ * A [[DirectWorkerDispatcher]] using Unix domain sockets as the worker
+ * transport. Allocates a private 0700 socket directory at construction;
+ * each worker is given a UDS path inside it.
+ *
+ * Concrete subclasses implement [[createConnection]] (with a UDS protocol
+ * of choice) and [[createSessionForWorker]].
+ */
+@Experimental
+abstract class DirectUnixSocketWorkerDispatcher(
+ workerSpec: UDFWorkerSpecification,
+ logger: WorkerLogger = WorkerLogger.NoOp)
+ extends DirectWorkerDispatcher(workerSpec, logger) {
+
+ // Removed explicitly in closeTransport(). deleteOnExit is avoided because
+ // the JDK retains the path for the JVM lifetime, which leaks in
+ // long-lived drivers.
+ private val socketDir: Path = createPrivateTempDirectory()
+
+ override protected def newEndpointAddress(workerId: String): String =
+ socketDir.resolve(s"worker-$workerId.sock").toString
+
+ override protected def waitForReady(
+ address: String,
+ process: Process,
+ outputFile: File): Unit = {
+ val file = new File(address)
+ // At least one poll so very small initTimeouts don't trip a premature
+ // timeout before the worker has any chance to create the socket.
+ val maxAttempts = math.max(1, (initTimeoutMs / SOCKET_POLL_INTERVAL_MS).toInt)
+ var attempts = 0
+ while (!file.exists() && attempts < maxAttempts) {
+ if (!process.isAlive) throwWorkerExitedBeforeSocket(process, address, outputFile)
+ Thread.sleep(SOCKET_POLL_INTERVAL_MS)
+ attempts += 1
+ }
+ if (!file.exists()) {
+ if (process.isAlive) {
+ DirectWorkerDispatcher.destroyForciblyAndReap(
+ process, logger, s"init timeout $address")
+ val tail = readOutputTail(outputFile)
+ throw new DirectWorkerTimeoutException(
+ s"Worker did not create socket at $address within ${initTimeoutMs}ms\n$tail")
+ } else {
+ // Worker exited after the last poll without creating the socket;
+ // prefer the exit-code message over the ambiguous "did not create".
+ throwWorkerExitedBeforeSocket(process, address, outputFile)
+ }
+ }
+ }
+
+ override protected def cleanupEndpointAddress(address: String): Unit = {
+ Files.deleteIfExists(new File(address).toPath)
+ }
+
+ override protected def closeTransport(): Unit = {
+ val dir = socketDir.toFile
+ if (dir.exists()) {
+ val remaining = dir.listFiles()
+ if (remaining != null) remaining.foreach(_.delete())
+ dir.delete()
+ }
+ }
+
+ override protected def validateTransportSupport(): Unit = {
+ val props = workerSpec.getDirect.getProperties
+ require(props.hasConnection,
+ "DirectWorker.properties.connection must be set")
+ val conn = props.getConnection
+ require(conn.hasUnixDomainSocket,
+ "DirectUnixSocketWorkerDispatcher requires UNIX domain socket transport, " +
+ s"got ${conn.getTransportCase}")
+ }
+
+ override protected def createConnection(address: String): UnixSocketWorkerConnection
+
+ private def throwWorkerExitedBeforeSocket(
+ process: Process,
+ address: String,
+ outputFile: File): Nothing = {
+ val tail = readOutputTail(outputFile)
+ throw new DirectWorkerException(
+ s"Worker exited with code ${process.exitValue()} " +
+ s"before creating socket at $address\n$tail")
+ }
+
+ /**
+ * Creates a temp directory with owner-only permissions (0700 on POSIX).
+ * On non-POSIX filesystems falls back to best-effort `File.setXxx`,
+ * which is TOCTOU-racy and weaker; a WARN surfaces if the platform
+ * refuses the setters.
+ */
+ private def createPrivateTempDirectory(): Path = {
+ val attr = PosixFilePermissions.asFileAttribute(
+ PosixFilePermissions.fromString("rwx------"))
+ try {
+ Files.createTempDirectory("spark-udf-worker", attr)
+ } catch {
+ case _: UnsupportedOperationException =>
+ val dir = Files.createTempDirectory("spark-udf-worker")
+ val f = dir.toFile
+ // `&` (non-short-circuiting) so every setter is attempted even if
+ // an earlier one refused.
+ val applied =
+ f.setReadable(false, false) & f.setWritable(false, false) &
+ f.setExecutable(false, false) & f.setReadable(true, true) &
+ f.setWritable(true, true) & f.setExecutable(true, true)
+ if (!applied) {
+ logger.warn(
+ s"Could not fully restrict permissions on $dir; socket " +
+ s"directory may be accessible to other local users on this " +
+ s"filesystem")
+ }
+ dir
+ }
+ }
+}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala
new file mode 100644
index 000000000000..afaf23791d80
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala
@@ -0,0 +1,532 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core.direct
+
+import java.io.{BufferedReader, File, FileInputStream, InputStreamReader}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+import java.util.UUID
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.AtomicBoolean
+
+import scala.collection.mutable.{Queue => MQueue}
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.udf.worker.{ProcessCallable, UDFWorkerSpecification}
+import org.apache.spark.udf.worker.core.{WorkerConnection, WorkerDispatcher,
+ WorkerLogger, WorkerSecurityScope, WorkerSession}
+import org.apache.spark.udf.worker.core.direct.DirectWorkerDispatcher.{CallableResult,
+ DEFAULT_CALLABLE_TIMEOUT_MS, DEFAULT_GRACEFUL_TIMEOUT_MS, DEFAULT_INIT_TIMEOUT_MS,
+ ENGINE_MAX_TIMEOUT_MS, EnvironmentState, MAX_OUTPUT_SCAN_BYTES,
+ PROCESS_OUTPUT_TAIL_LINES}
+
+/**
+ * :: Experimental ::
+ * A [[WorkerDispatcher]] that creates workers by spawning local OS processes
+ * ("direct" creation mode from the worker specification).
+ *
+ * On the first [[createSession]], the dispatcher ensures the environment is
+ * ready (verify / install) and registers the cleanup hook. Each session
+ * currently gets a fresh worker that is terminated when the session closes
+ * (the single-reference case of the future pooling policy).
+ *
+ * Subclasses implement [[createConnection]] and [[createSessionForWorker]]
+ * to provide protocol-specific behavior (e.g., gRPC, raw sockets).
+ *
+ * For workers obtained through a provisioning service or daemon (indirect
+ * creation), see the `indirect` package (TODO).
+ *
+ * @param workerSpec worker specification (proto)
+ * @param logger [[WorkerLogger]] used for dispatcher-internal messages.
+ * The framework does not depend on any concrete logging
+ * backend; callers should pass an adapter that forwards
+ * to their preferred logger (Spark's `Logging` trait,
+ * SLF4J, etc.). Defaults to [[WorkerLogger.NoOp]].
+ */
+@Experimental
+abstract class DirectWorkerDispatcher(
+ override val workerSpec: UDFWorkerSpecification,
+ protected val logger: WorkerLogger = WorkerLogger.NoOp)
+ extends WorkerDispatcher {
+
+ // TODO: Connection pooling -- reuse idle workers across sessions.
+ // TODO: Security scope isolation -- partition pool by WorkerSecurityScope.
+
+ validateTransportSupport()
+ validateEnvironmentCallables()
+
+ /**
+ * Maximum time to wait for a setup/verify/cleanup callable to finish.
+ * Subclasses may override this to accommodate slow installation steps
+ * (e.g., a large dependency install). Defaults to 120 seconds.
+ */
+ protected def callableTimeoutMs: Long = DEFAULT_CALLABLE_TIMEOUT_MS
+
+ // Proto-provided timeouts are clamped to ENGINE_MAX_TIMEOUT_MS. The
+ // dispatcher-internal callableTimeoutMs above is subclass-controlled and
+ // not subject to the cap.
+ // Package-private for test access.
+ private[core] val initTimeoutMs: Long = {
+ val props = workerSpec.getDirect.getProperties
+ val raw = if (props.hasInitializationTimeoutMs && props.getInitializationTimeoutMs > 0) {
+ props.getInitializationTimeoutMs.toLong
+ } else {
+ DEFAULT_INIT_TIMEOUT_MS
+ }
+ clampTimeout("initialization_timeout_ms", raw)
+ }
+
+ private val gracefulTimeoutMs: Long = {
+ val props = workerSpec.getDirect.getProperties
+ val raw = if (props.hasGracefulTerminationTimeoutMs &&
+ props.getGracefulTerminationTimeoutMs > 0) {
+ props.getGracefulTerminationTimeoutMs.toLong
+ } else {
+ DEFAULT_GRACEFUL_TIMEOUT_MS
+ }
+ clampTimeout("graceful_termination_timeout_ms", raw)
+ }
+
+ private def clampTimeout(field: String, raw: Long): Long = {
+ if (raw > ENGINE_MAX_TIMEOUT_MS) {
+ logger.warn(
+ s"Worker-provided $field=${raw}ms exceeds engine maximum " +
+ s"${ENGINE_MAX_TIMEOUT_MS}ms; using ${ENGINE_MAX_TIMEOUT_MS}ms instead")
+ ENGINE_MAX_TIMEOUT_MS
+ } else {
+ raw
+ }
+ }
+
+ private[this] val workers = new ConcurrentHashMap[String, DirectWorkerProcess]()
+ private[this] val closed = new AtomicBoolean(false)
+
+ @volatile private var environmentState: EnvironmentState = EnvironmentState.Pending
+ private val environmentLock = new Object
+ private[this] var cleanupHook: Option[Thread] = None
+
+ /**
+ * Allocates a fresh endpoint address for a new worker. The string is
+ * passed to the worker binary as `--connection <address>`.
+ */
+ protected def newEndpointAddress(workerId: String): String
+
+ /**
+ * Waits for the worker process to be ready to accept connections at
+ * `address`. Throws [[DirectWorkerTimeoutException]] on timeout, or
+ * [[DirectWorkerException]] if the process exits early.
+ */
+ protected def waitForReady(
+ address: String,
+ process: Process,
+ outputFile: File): Unit
+
+ /**
+ * Best-effort per-endpoint cleanup, called from the spawn-failure path
+ * before any [[WorkerArtifacts]] / [[WorkerConnection]] exists.
+ */
+ protected def cleanupEndpointAddress(address: String): Unit
+
+ /**
+ * Cleans up dispatcher-level transport state (e.g., a UDS socket
+ * directory). Called from [[close]].
+ */
+ protected def closeTransport(): Unit
+
+ /**
+ * Validates the worker spec's transport choice. Subclasses declare
+ * which transports they support. Called from the base constructor;
+ * implementations must only read base-class state (`workerSpec`).
+ */
+ protected def validateTransportSupport(): Unit
+
+ /** Creates a protocol-specific connection to a worker at the given address. */
+ protected def createConnection(address: String): WorkerConnection
+
+ /** Creates a protocol-specific session for the given worker. */
+ protected def createSessionForWorker(worker: DirectWorkerProcess): WorkerSession
+
+ override def createSession(
+ securityScope: Option[WorkerSecurityScope]): WorkerSession = {
+ require(securityScope.isEmpty,
+ "securityScope is not supported yet; pass None until pooling lands")
+ if (closed.get()) throwClosed()
+ ensureEnvironmentReady()
+ val worker = spawnWorker()
+ // Acquire before publish: a concurrent close() iterating `workers` must
+ // not tear down this worker before we hand it to the caller.
+ worker.acquireSession()
+ workers.put(worker.id, worker)
+ // Re-check for close() that ran concurrently. Releasing fires the
+ // ref-count callback, which removes and tears down the worker.
+ if (closed.get()) {
+ worker.releaseSession()
+ throwClosed()
+ }
+ try {
+ createSessionForWorker(worker)
+ } catch {
+ case e: InterruptedException =>
+ Thread.currentThread().interrupt()
+ worker.releaseSession()
+ throw e
+ case NonFatal(e) =>
+ worker.releaseSession()
+ throw e
+ }
+ }
+
+ /**
+ * Invoked when a worker's last session closes. Terminates the worker
+ * today; future pooling can reuse it here instead. Safe to call after
+ * dispatcher close -- the worker's own CAS-idempotent close makes a
+ * second teardown a no-op.
+ */
+ private def releaseWorker(worker: DirectWorkerProcess): Unit = {
+ workers.remove(worker.id)
+ try {
+ worker.close()
+ } catch {
+ case NonFatal(e) =>
+ logger.warn(s"Error closing worker ${worker.id}", e)
+ }
+ }
+
+ private def throwClosed(): Nothing =
+ throw new IllegalStateException("Dispatcher is closed")
+
+ /**
+ * Terminates tracked workers, removes the socket directory, and runs
+ * environment cleanup. Idempotent via CAS. Does not drain in-flight
+ * createSession calls -- a worker spawned racing with close tears
+ * itself down through the ref-count callback, which may outlive this
+ * method.
+ */
+ override def close(): Unit = {
+ if (!closed.compareAndSet(false, true)) {
+ return
+ }
+ // TODO: close workers in parallel -- today shutdown is serialised at
+ // N * gracefulTimeoutMs worst case.
+ workers.values().iterator().asScala.foreach { w =>
+ try {
+ w.close()
+ } catch {
+ case NonFatal(e) =>
+ logger.warn(s"Error closing worker ${w.id}", e)
+ }
+ }
+ workers.clear()
+ try closeTransport() catch {
+ case NonFatal(e) =>
+ logger.warn("Error cleaning up transport state", e)
+ }
+ deregisterEnvironmentCleanupHook()
+ runEnvironmentCleanup()
+ }
+
+ // -- Environment lifecycle -------------------------------------------------
+
+ // TODO: distinguish retriable vs permanent environment failures.
+ private def ensureEnvironmentReady(): Unit = {
+ environmentLock.synchronized {
+ environmentState match {
+ case EnvironmentState.Ready | EnvironmentState.CleanedUp =>
+ case EnvironmentState.Failed(msg) =>
+ throw new DirectWorkerException(s"Environment setup previously failed: $msg")
+ case EnvironmentState.Pending =>
+ val env = workerSpec.getEnvironment
+ // Register up front so a partially-successful install still gets
+ // torn down at JVM shutdown if dispatcher.close is never called.
+ // No-op when environment_cleanup is not configured.
+ registerEnvironmentCleanupHook()
+ val verified = env.hasEnvironmentVerification &&
+ runCallable(env.getEnvironmentVerification).exitCode == 0
+ if (!verified && env.hasInstallation) {
+ // Treat any install failure (timeout or non-zero exit) as
+ // permanent. A partially-completed install can leave files on
+ // disk that a retry would race with; retry policy belongs in
+ // the future predicate (see TODO above).
+ val result = try {
+ runCallable(env.getInstallation)
+ } catch {
+ case e: DirectWorkerException =>
+ environmentState = EnvironmentState.Failed(
+ s"installation failed: ${e.getMessage}")
+ throw e
+ }
+ if (result.exitCode != 0) {
+ val detail = s"exit code ${result.exitCode}\n${result.outputTail}"
+ environmentState = EnvironmentState.Failed(detail)
+ throw new DirectWorkerException(
+ s"Environment installation failed with $detail")
+ }
+ }
+ environmentState = EnvironmentState.Ready
+ }
+ }
+ }
+
+ // TODO: share one JVM shutdown hook across all dispatchers in the
+ // process. Each live dispatcher is retained by the JVM until shutdown.
+
+ /** Registers the JVM shutdown hook that runs the cleanup callable. */
+ private def registerEnvironmentCleanupHook(): Unit = {
+ if (!Thread.holdsLock(environmentLock)) {
+ throw new IllegalStateException(
+ "registerEnvironmentCleanupHook must be called while holding environmentLock")
+ }
+ if (cleanupHook.isDefined) return
+ if (workerSpec.getEnvironment.hasEnvironmentCleanup) {
+ val hook = new Thread(() => runEnvironmentCleanup(), "udf-env-cleanup")
+ cleanupHook = Some(hook)
+ // scalastyle:off runtimeaddshutdownhook
+ Runtime.getRuntime.addShutdownHook(hook)
+ // scalastyle:on runtimeaddshutdownhook
+ }
+ }
+
+ private def deregisterEnvironmentCleanupHook(): Unit = {
+ environmentLock.synchronized {
+ cleanupHook.foreach { hook =>
+ try {
+ Runtime.getRuntime.removeShutdownHook(hook)
+ } catch {
+ case _: IllegalStateException => // JVM already shutting down
+ }
+ cleanupHook = None
+ }
+ }
+ }
+
+ private def runEnvironmentCleanup(): Unit = {
+ environmentLock.synchronized {
+ environmentState match {
+ case EnvironmentState.CleanedUp =>
+ case _ =>
+ if (workerSpec.getEnvironment.hasEnvironmentCleanup) {
+ try {
+ val result = runCallable(workerSpec.getEnvironment.getEnvironmentCleanup)
+ if (result.exitCode != 0) {
+ logger.warn(s"Environment cleanup exited with code ${result.exitCode}" +
+ s"\n${result.outputTail}")
+ }
+ } catch {
+ case NonFatal(e) => logger.warn("Environment cleanup failed", e)
+ }
+ }
+ environmentState = EnvironmentState.CleanedUp
+ }
+ }
+ }
+
+ // -- Process helpers -------------------------------------------------------
+
+ /**
+ * Runs a [[ProcessCallable]] synchronously and returns the result.
+ * Always throws on timeout; callers check `exitCode` for non-timeout failures.
+ */
+ private[core] def runCallable(callable: ProcessCallable): CallableResult = {
+ val cmd = (callable.getCommandList.asScala ++ callable.getArgumentsList.asScala).toSeq
+ require(cmd.nonEmpty,
+ "ProcessCallable must have at least one entry in command or arguments")
+ val outputFile = Files.createTempFile("udf-callable-", ".log")
+ try {
+ val process = launchProcess(
+ cmd, callable.getEnvironmentVariablesMap.asScala.toMap, outputFile.toFile)
+ val timeoutMs = callableTimeoutMs
+ if (!process.waitFor(timeoutMs, TimeUnit.MILLISECONDS)) {
+ DirectWorkerDispatcher.destroyForciblyAndReap(
+ process, logger, s"callable timeout: ${cmd.head}")
+ val tail = readOutputTail(outputFile.toFile)
+ throw new DirectWorkerTimeoutException(
+ s"Callable timed out after ${timeoutMs}ms: " +
+ s"${cmd.mkString(" ")}\n$tail")
+ }
+ val tail = readOutputTail(outputFile.toFile)
+ CallableResult(process.exitValue(), tail)
+ } finally {
+ Files.deleteIfExists(outputFile)
+ }
+ }
+
+ private def spawnWorker(): DirectWorkerProcess = {
+ val runner = workerSpec.getDirect.getRunner
+ val baseCmd = (runner.getCommandList.asScala ++ runner.getArgumentsList.asScala).toSeq
+ require(baseCmd.nonEmpty,
+ "DirectWorker.runner must have at least one entry in command or arguments")
+ val workerId = UUID.randomUUID().toString
+ val address = newEndpointAddress(workerId)
+ // Proto contract: the engine must pass --id and --connection.
+ val cmd = baseCmd ++ Seq("--id", workerId, "--connection", address)
+ val env = runner.getEnvironmentVariablesMap.asScala.toMap
+ val outputFile = Files.createTempFile("udf-worker-", ".log")
+ val process = launchProcess(cmd, env, outputFile.toFile)
+
+ try {
+ waitForReady(address, process, outputFile.toFile)
+ val connection = createConnection(address)
+ val artifacts = new WorkerArtifacts(process, connection, outputFile, logger)
+ new DirectWorkerProcess(
+ workerId, artifacts, gracefulTimeoutMs, logger,
+ onLastSessionReleased = releaseWorker)
+ } catch {
+ case e: InterruptedException =>
+ Thread.currentThread().interrupt()
+ cleanupRawSpawn(process, address, outputFile)
+ throw e
+ case NonFatal(e) =>
+ cleanupRawSpawn(process, address, outputFile)
+ throw e
+ }
+ }
+
+ // Pre-WorkerArtifacts cleanup: the connection has not been built yet,
+ // so we have no bundle to close(). Each step is independent.
+ private def cleanupRawSpawn(p: Process, address: String, outputFile: Path): Unit = {
+ DirectWorkerDispatcher.destroyForciblyAndReap(p, logger, "failed spawn")
+ try cleanupEndpointAddress(address) catch {
+ case NonFatal(e) =>
+ logger.debug(s"Failed to clean up endpoint address $address", e)
+ }
+ try Files.deleteIfExists(outputFile) catch {
+ case NonFatal(e) =>
+ logger.debug(s"Failed to clean up worker output file $outputFile", e)
+ }
+ }
+
+ /**
+ * Starts an OS process. stdout and stderr are merged and redirected to the
+ * given file so that output can be read back for error reporting.
+ */
+ private def launchProcess(
+ command: Seq[String],
+ env: Map[String, String],
+ outputFile: File): Process = {
+ val builder = new ProcessBuilder(command: _*)
+ env.foreach { case (k, v) => builder.environment().put(k, v) }
+ builder.redirectErrorStream(true)
+ builder.redirectOutput(outputFile)
+ builder.start()
+ }
+
+ // Bounded scan so a runaway worker that writes gigabytes of output does
+ // not OOM the caller during error reporting.
+ protected def readOutputTail(file: File): String = {
+ if (!file.exists() || file.length() == 0) return ""
+ val fileLen = file.length()
+ val startPos = math.max(0L, fileLen - MAX_OUTPUT_SCAN_BYTES)
+ val fis = new FileInputStream(file)
+ try {
+ if (startPos > 0) fis.getChannel.position(startPos)
+ val reader = new BufferedReader(
+ new InputStreamReader(fis, StandardCharsets.UTF_8))
+ // Discard the first (partial) line when we seeked into the middle.
+ if (startPos > 0) reader.readLine()
+ val buffer = new MQueue[String]()
+ var line = reader.readLine()
+ while (line != null) {
+ if (buffer.size >= PROCESS_OUTPUT_TAIL_LINES) buffer.dequeue()
+ buffer.enqueue(line)
+ line = reader.readLine()
+ }
+ if (buffer.isEmpty) ""
+ else "Process output (last lines):\n" + buffer.mkString("\n")
+ } catch {
+ case NonFatal(e) =>
+ logger.debug(s"Failed to read process output from $file", e)
+ ""
+ } finally {
+ fis.close()
+ }
+ }
+
+ // -- Spec validation -------------------------------------------------------
+
+ // Verification exists to short-circuit installation when the environment
+ // is already prepared, so requiring installation alongside verification
+ // catches user errors at spec-validation time.
+ private def validateEnvironmentCallables(): Unit = {
+ val env = workerSpec.getEnvironment
+ require(!env.hasEnvironmentVerification || env.hasInstallation,
+ "WorkerEnvironment.environment_verification requires installation to be set")
+ }
+}
+
+private[direct] object DirectWorkerDispatcher {
+ private[direct] val SOCKET_POLL_INTERVAL_MS = 100L
+ private[direct] val DEFAULT_INIT_TIMEOUT_MS = 10000L
+ private[direct] val DEFAULT_CALLABLE_TIMEOUT_MS = 120000L
+ private[direct] val DEFAULT_GRACEFUL_TIMEOUT_MS = 5000L
+ // Engine-side cap on proto-provided worker timeouts. The defaults below
+ // must stay at or under this cap so the clamp only fires on
+ // user-provided values.
+ private[direct] val ENGINE_MAX_TIMEOUT_MS = 30000L
+ require(DEFAULT_INIT_TIMEOUT_MS <= ENGINE_MAX_TIMEOUT_MS &&
+ DEFAULT_GRACEFUL_TIMEOUT_MS <= ENGINE_MAX_TIMEOUT_MS,
+ "default timeouts must not exceed ENGINE_MAX_TIMEOUT_MS")
+ private[direct] val PROCESS_OUTPUT_TAIL_LINES = 50
+ private[direct] val MAX_OUTPUT_SCAN_BYTES = 1024L * 1024L // 1 MiB
+ // 5s bounds the wait for the kernel to reap a SIGKILL'd child. SIGKILL
+ // is unblockable, so exceeding this usually means the process is stuck
+ // in uninterruptible I/O (D-state) and further waiting will not help.
+ private[direct] val SIGKILL_REAP_TIMEOUT_MS = 5000L
+
+ /**
+ * SIGKILL `process` and wait up to [[SIGKILL_REAP_TIMEOUT_MS]] for the
+ * kernel to reap it. `destroyForcibly()` alone returns before the child
+ * is reaped, which leaks a zombie until JVM exit. On reap-timeout logs
+ * a warning; on interrupt re-raises the interrupt and returns.
+ *
+ * @param context short tag included in the timeout warning so operators
+ * can correlate a stuck child with its source.
+ */
+ private[direct] def destroyForciblyAndReap(
+ process: Process,
+ logger: WorkerLogger,
+ context: String = ""): Unit = {
+ if (!process.isAlive) return
+ process.destroyForcibly()
+ val reaped = try {
+ process.waitFor(SIGKILL_REAP_TIMEOUT_MS, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: InterruptedException =>
+ Thread.currentThread().interrupt()
+ return
+ }
+ if (!reaped && process.isAlive) {
+ val suffix = if (context.nonEmpty) s" [$context]" else ""
+ logger.warn(
+ s"Process ${process.pid()}$suffix still alive ${SIGKILL_REAP_TIMEOUT_MS}ms " +
+ s"after SIGKILL; leaving behind as zombie " +
+ s"(likely stuck in uninterruptible kernel state)")
+ }
+ }
+
+ /** Result of running a [[ProcessCallable]]. */
+ private[core] case class CallableResult(exitCode: Int, outputTail: String)
+
+ private[direct] sealed trait EnvironmentState
+ private[direct] object EnvironmentState {
+ case object Pending extends EnvironmentState
+ case object Ready extends EnvironmentState
+ case class Failed(detail: String) extends EnvironmentState
+ case object CleanedUp extends EnvironmentState
+ }
+}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerException.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerException.scala
new file mode 100644
index 000000000000..b0ece15eae38
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerException.scala
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core.direct
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Thrown by [[DirectWorkerDispatcher]] for runtime failures: worker
+ * spawn problems, environment setup or cleanup failures, callable
+ * timeouts, and socket-establishment timeouts.
+ *
+ * Distinguished from `IllegalArgumentException` (bad spec) and
+ * `IllegalStateException` (using a closed dispatcher), which indicate
+ * programming errors. Catching this type lets callers handle runtime
+ * failures specifically without catching every `RuntimeException`.
+ */
+@Experimental
+class DirectWorkerException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+/**
+ * :: Experimental ::
+ * A [[DirectWorkerException]] caused specifically by a timeout: a worker
+ * that did not bind its socket within `initialization_timeout_ms`, or a
+ * setup callable (verify / install / cleanup) that exceeded
+ * `callableTimeoutMs`. Exposed as a distinct type so callers can choose
+ * different retry / escalation paths for timeouts vs other failures.
+ */
+@Experimental
+class DirectWorkerTimeoutException(message: String, cause: Throwable = null)
+ extends DirectWorkerException(message, cause)
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerProcess.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerProcess.scala
new file mode 100644
index 000000000000..f4b5c1df6319
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerProcess.scala
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core.direct
+
+import java.nio.file.{Files, Path}
+import java.util.concurrent.TimeUnit
+import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger}
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.udf.worker.core.{WorkerConnection, WorkerLogger}
+
+/**
+ * :: Experimental ::
+ * A locally-spawned OS process running a UDF worker, together with its
+ * transport connection. Wraps a [[WorkerArtifacts]] bundle (process +
+ * connection + output log) plus a session ref-count scaffolding for
+ * future pooling -- today one process per session.
+ *
+ * Closing sends SIGTERM, waits up to [[gracefulTimeoutMs]], then
+ * delegates connection close + forced kill + file cleanup to
+ * [[WorkerArtifacts.close]].
+ *
+ * @param id stable worker identifier (UUID passed to the binary as `--id`).
+ * @param artifacts process + connection + output-log, disposed together.
+ * @param gracefulTimeoutMs wait after SIGTERM before escalating to SIGKILL.
+ * @param logger [[WorkerLogger]] for process-level messages.
+ * @param onLastSessionReleased fires when the ref-count hits 0. Runs on
+ * the thread calling [[releaseSession]]. May fire more than once
+ * across a worker's lifetime; a concurrent `acquireSession` can
+ * re-increment the count before the callback returns, so pooling
+ * dispatchers must arbitrate reuse themselves.
+ */
+@Experimental
+class DirectWorkerProcess(
+ val id: String,
+ private[direct] val artifacts: WorkerArtifacts,
+ val gracefulTimeoutMs: Long,
+ protected val logger: WorkerLogger = WorkerLogger.NoOp,
+ private[direct] val onLastSessionReleased: DirectWorkerProcess => Unit = _ => ())
+ extends AutoCloseable {
+
+ // TODO: idle-timeout tracking and concurrent session capacity.
+
+ private val activeSessionCount = new AtomicInteger(0)
+ private val closed = new AtomicBoolean(false)
+
+ /** The OS process handle for this worker. */
+ def process: Process = artifacts.process
+
+ /** The transport connection for this worker. */
+ def connection: WorkerConnection = artifacts.connection
+
+ /** Path to the merged stdout/stderr log for this worker. */
+ def outputFile: Path = artifacts.outputFile
+
+ /** Number of sessions currently using this worker. */
+ def activeSessions: Int = activeSessionCount.get()
+
+ /** Increments the active session count. */
+ def acquireSession(): Unit = activeSessionCount.incrementAndGet()
+
+ /**
+ * Decrements the active session count. Fires [[onLastSessionReleased]]
+ * on the 0-transition. A negative count indicates an unbalanced
+ * acquire/release; we log and reset to 0 rather than silently mask it.
+ */
+ def releaseSession(): Unit = {
+ val c = activeSessionCount.decrementAndGet()
+ if (c < 0) {
+ logger.warn(
+ s"releaseSession called without a matching acquireSession (count=$c)")
+ activeSessionCount.set(0)
+ } else if (c == 0) {
+ // Swallow callback errors so session.close cannot throw.
+ try onLastSessionReleased(this) catch {
+ case NonFatal(e) =>
+ logger.warn(s"onLastSessionReleased callback failed for worker $id", e)
+ }
+ }
+ }
+
+ /** Returns true if the OS process is running and the connection is usable. */
+ def isAlive: Boolean = process.isAlive && connection.isActive
+
+ /**
+ * Sends SIGTERM, waits up to [[gracefulTimeoutMs]] for the worker to
+ * exit, then disposes artifacts (connection close + SIGKILL + file
+ * cleanup). Idempotent via CAS.
+ */
+ override def close(): Unit = {
+ if (!closed.compareAndSet(false, true)) return
+
+ if (process.isAlive) {
+ process.destroy() // SIGTERM
+ try {
+ // Ignore the return value: artifacts.close() SIGKILLs if still
+ // alive and no-ops if already dead.
+ process.waitFor(gracefulTimeoutMs, TimeUnit.MILLISECONDS)
+ } catch {
+ case _: InterruptedException =>
+ Thread.currentThread().interrupt()
+ }
+ }
+
+ artifacts.close()
+ }
+}
+
+/**
+ * Closeable bundle of per-worker OS resources: the child [[Process]], its
+ * transport [[WorkerConnection]], and its merged stdout/stderr log.
+ * [[close]] runs connection close (which for UDS removes the socket
+ * file), then SIGKILL-reaps the process, then deletes the output log.
+ * Graceful SIGTERM is the higher layer's responsibility (see
+ * [[DirectWorkerProcess#close]]).
+ */
+private[direct] final class WorkerArtifacts(
+ val process: Process,
+ val connection: WorkerConnection,
+ val outputFile: Path,
+ private[this] val logger: WorkerLogger) extends AutoCloseable {
+
+ private[this] val closed = new AtomicBoolean(false)
+
+ /**
+ * Idempotently closes the connection (transport teardown + any
+ * transport-specific cleanup such as deleting a UDS socket file),
+ * SIGKILL-reaps the process, and deletes the output log. Each step
+ * is guarded so a failure in one does not skip the next.
+ */
+ override def close(): Unit = {
+ if (!closed.compareAndSet(false, true)) return
+
+ try connection.close() catch {
+ case NonFatal(e) =>
+ logger.warn("Error closing worker connection", e)
+ }
+
+ DirectWorkerDispatcher.destroyForciblyAndReap(process, logger, "worker artifacts")
+
+ try Files.deleteIfExists(outputFile) catch {
+ case NonFatal(e) =>
+ logger.warn(s"Error cleaning up worker output file $outputFile", e)
+ }
+ }
+}
diff --git a/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerSession.scala b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerSession.scala
new file mode 100644
index 000000000000..7cdc5329350e
--- /dev/null
+++ b/udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerSession.scala
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core.direct
+
+import java.util.concurrent.atomic.AtomicBoolean
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.udf.worker.core.{WorkerConnection, WorkerSession}
+
+/**
+ * :: Experimental ::
+ * A [[WorkerSession]] backed by a locally-spawned [[DirectWorkerProcess]].
+ *
+ * This is the session type returned by [[DirectWorkerDispatcher]]. It ties
+ * the session lifecycle to the worker's ref-count: the dispatcher increments
+ * the count before construction, and [[close]] decrements it, so the
+ * dispatcher knows when a worker process is idle and can be terminated or
+ * reused.
+ *
+ * Subclasses implement the protocol-specific data transmission
+ * ([[init]], [[process]], [[cancel]]).
+ *
+ * @param workerProcess the direct worker process backing this session.
+ * Internal to the `core` package and test code -- the
+ * worker handle is a dispatcher implementation detail,
+ * not part of the public WorkerSession API.
+ */
+@Experimental
+abstract class DirectWorkerSession(
+ private[core] val workerProcess: DirectWorkerProcess) extends WorkerSession {
+
+ private val released = new AtomicBoolean(false)
+
+ /** The connection to the worker for this session. */
+ def connection: WorkerConnection = workerProcess.connection
+
+ override def close(): Unit = {
+ if (released.compareAndSet(false, true)) {
+ workerProcess.releaseSession()
+ }
+ }
+}
diff --git a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/DirectWorkerDispatcherSuite.scala b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/DirectWorkerDispatcherSuite.scala
new file mode 100644
index 000000000000..60f5e2211b70
--- /dev/null
+++ b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/DirectWorkerDispatcherSuite.scala
@@ -0,0 +1,981 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.udf.worker.core
+
+import java.io.File
+import java.nio.file.{Files, Path}
+import java.nio.file.attribute.PosixFileAttributeView
+
+import scala.jdk.CollectionConverters._
+
+// scalastyle:off funsuite
+import org.scalatest.BeforeAndAfterEach
+import org.scalatest.funsuite.AnyFunSuite
+
+import org.apache.spark.udf.worker.{
+ DirectWorker, LocalTcpConnection, ProcessCallable, UDFWorkerProperties,
+ UDFWorkerSpecification, UnixDomainSocket, WorkerConnectionSpec,
+ WorkerEnvironment}
+import org.apache.spark.udf.worker.core.direct.{DirectUnixSocketWorkerDispatcher,
+ DirectWorkerException, DirectWorkerProcess, DirectWorkerSession,
+ DirectWorkerTimeoutException}
+
+/**
+ * A [[WorkerConnection]] test implementation that considers the connection
+ * active as long as the socket file exists on disk. Inherits socket-file
+ * deletion from [[UnixSocketWorkerConnection.close]].
+ */
+class SocketFileConnection(socketPath: String)
+ extends UnixSocketWorkerConnection(socketPath) {
+ override def isActive: Boolean = new File(socketPath).exists()
+}
+
+/**
+ * A stub [[DirectWorkerSession]] for process-lifecycle tests that don't
+ * need actual data transmission.
+ *
+ * TODO: [[cancel]] is a no-op here. Once a concrete [[DirectWorkerSession]]
+ * with real data-plane wiring lands, add tests exercising cancel() in
+ * particular: cancel from a different thread than process(), cancel
+ * after process() has returned, and cancel before init (should be a
+ * no-op). Tracking the thread-safety contract in the docstring on
+ * [[org.apache.spark.udf.worker.core.WorkerSession.cancel]].
+ */
+class StubWorkerSession(
+ workerProcess: DirectWorkerProcess) extends DirectWorkerSession(workerProcess) {
+
+ override protected def doInit(message: InitMessage): Unit = {}
+
+ override protected def doProcess(
+ input: Iterator[Array[Byte]]): Iterator[Array[Byte]] =
+ Iterator.empty
+
+ override def cancel(): Unit = {}
+}
+
+/**
+ * A [[DirectUnixSocketWorkerDispatcher]] subclass for testing that uses
+ * a socket-file connection and stub sessions instead of a real protocol
+ * implementation.
+ */
+class TestDirectWorkerDispatcher(spec: UDFWorkerSpecification)
+ extends DirectUnixSocketWorkerDispatcher(spec) {
+
+ override protected def createConnection(
+ socketPath: String): UnixSocketWorkerConnection =
+ new SocketFileConnection(socketPath)
+
+ override protected def createSessionForWorker(
+ worker: DirectWorkerProcess): WorkerSession =
+ new StubWorkerSession(worker)
+}
+
+/**
+ * Tests for [[DirectWorkerDispatcher]] process lifecycle: spawning workers
+ * and terminating them on close.
+ */
+class DirectWorkerDispatcherSuite
+ extends AnyFunSuite with BeforeAndAfterEach {
+// scalastyle:on funsuite
+
+ private val echoWorkerScript =
+ """
+ |#!/bin/bash
+ |SOCKET_PATH=""
+ |while [[ $# -gt 0 ]]; do
+ | case "$1" in
+ | --connection) SOCKET_PATH="$2"; shift 2 ;;
+ | *) shift ;;
+ | esac
+ |done
+ |cleanup() { rm -f "$SOCKET_PATH"; exit 0; }
+ |trap cleanup SIGTERM
+ |touch "$SOCKET_PATH"
+ |while true; do sleep 1; done
+ """.stripMargin.trim
+
+ private def defaultRunner: ProcessCallable = ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c").addCommand(echoWorkerScript).addCommand("--")
+ .build()
+
+ private def udsProperties: UDFWorkerProperties = UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setUnixDomainSocket(UnixDomainSocket.getDefaultInstance)
+ .build())
+ .build()
+
+ private def directWorker(runner: ProcessCallable): DirectWorker =
+ DirectWorker.newBuilder().setRunner(runner).setProperties(udsProperties).build()
+
+ private def specWithRunner(runner: ProcessCallable): UDFWorkerSpecification =
+ UDFWorkerSpecification.newBuilder()
+ .setDirect(directWorker(runner))
+ .build()
+
+ private def specWithEnv(
+ runner: ProcessCallable = defaultRunner,
+ env: WorkerEnvironment): UDFWorkerSpecification =
+ UDFWorkerSpecification.newBuilder()
+ .setEnvironment(env)
+ .setDirect(directWorker(runner))
+ .build()
+
+ private var dispatcher: TestDirectWorkerDispatcher = _
+
+ override def afterEach(): Unit = {
+ if (dispatcher != null) {
+ dispatcher.close()
+ dispatcher = null
+ }
+ super.afterEach()
+ }
+
+ // Narrow the publicly-typed WorkerSession returned by `createSession` back
+ // down to StubWorkerSession in one place, with a descriptive failure if
+ // the cast is ever wrong, so individual tests don't scatter `asInstanceOf`
+ // (which would throw ClassCastException rather than a useful message).
+ private def createStubSession(): StubWorkerSession =
+ dispatcher.createSession(None) match {
+ case stub: StubWorkerSession => stub
+ case other => fail(
+ s"Expected StubWorkerSession, got ${other.getClass.getSimpleName}")
+ }
+
+ // The whole suite uses UDS as the only transport, so reaching past the
+ // generic WorkerConnection abstraction to read the socket path is fine.
+ private def udsPath(w: DirectWorkerProcess): String = w.connection match {
+ case uds: UnixSocketWorkerConnection => uds.socketPath
+ case other => fail(
+ s"Expected UnixSocketWorkerConnection, got ${other.getClass.getSimpleName}")
+ }
+
+ test("creates a worker and session") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+
+ val session = createStubSession()
+ val worker = session.workerProcess
+
+ assert(worker.isAlive, "worker should be alive after creation")
+ assert(worker.activeSessions == 1, "should have 1 active session")
+ assert(new File(udsPath(worker)).exists(), "socket file should exist")
+
+ session.close()
+ assert(worker.activeSessions == 0, "should have 0 sessions after close")
+ }
+
+ test("concurrent createSession calls produce distinct workers") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+
+ val threads = 8
+ val sessions = new java.util.concurrent.ConcurrentLinkedQueue[StubWorkerSession]()
+ val startGate = new java.util.concurrent.CountDownLatch(1)
+ val doneGate = new java.util.concurrent.CountDownLatch(threads)
+ val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]()
+
+ (1 to threads).foreach { _ =>
+ new Thread(() => {
+ try {
+ startGate.await()
+ sessions.add(createStubSession())
+ } catch {
+ case t: Throwable => errors.add(t)
+ } finally {
+ doneGate.countDown()
+ }
+ }).start()
+ }
+ startGate.countDown()
+ assert(doneGate.await(30, java.util.concurrent.TimeUnit.SECONDS),
+ "createSession threads did not finish in time")
+
+ assert(errors.isEmpty,
+ s"unexpected errors during concurrent createSession: ${errors.toArray.mkString(", ")}")
+ assert(sessions.size == threads, "expected one session per thread")
+
+ val sessionList = sessions.asScala.toList
+ val workerObjects = sessionList.map(_.workerProcess)
+ assert(workerObjects.distinct.length == threads,
+ "each session should have its own DirectWorkerProcess")
+ // Object-identity is not sufficient on its own: a future regression
+ // that accidentally shared underlying transport resources could still
+ // hand out distinct DirectWorkerProcess wrappers pointing at the same
+ // socket. Verify socket paths are unique too.
+ val socketPaths = workerObjects.map(udsPath)
+ assert(socketPaths.distinct.length == threads,
+ s"each worker should have its own socket path, got $socketPaths")
+
+ sessionList.foreach(_.close())
+ }
+
+ test("close shuts down all workers via SIGTERM") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+
+ val session1 = createStubSession()
+ val session2 = createStubSession()
+
+ val worker1 = session1.workerProcess
+ val worker2 = session2.workerProcess
+
+ session1.close()
+ session2.close()
+ dispatcher.close()
+ dispatcher = null
+
+ assert(!worker1.process.isAlive, "worker1 should be terminated")
+ assert(!worker2.process.isAlive, "worker2 should be terminated")
+ }
+
+ test("close escalates to SIGKILL when worker ignores SIGTERM") {
+ // The worker traps SIGTERM so the graceful stop is ineffective; the
+ // dispatcher must escalate to SIGKILL via destroyForciblyAndReap.
+ // Using a short gracefulTimeoutMs (500ms) keeps the test bounded:
+ // max close time is gracefulTimeoutMs + SIGKILL_REAP_TIMEOUT_MS.
+ val sigtermIgnoringScript =
+ """
+ |#!/bin/bash
+ |SOCKET_PATH=""
+ |while [[ $# -gt 0 ]]; do
+ | case "$1" in
+ | --connection) SOCKET_PATH="$2"; shift 2 ;;
+ | *) shift ;;
+ | esac
+ |done
+ |touch "$SOCKET_PATH"
+ |trap '' SIGTERM
+ |while true; do sleep 1; done
+ """.stripMargin.trim
+ val runner = ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c").addCommand(sigtermIgnoringScript).addCommand("--")
+ .build()
+ val shortGracefulProps = UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setUnixDomainSocket(UnixDomainSocket.getDefaultInstance).build())
+ .setGracefulTerminationTimeoutMs(500)
+ .build()
+ val spec = UDFWorkerSpecification.newBuilder()
+ .setDirect(DirectWorker.newBuilder()
+ .setRunner(runner).setProperties(shortGracefulProps).build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(spec)
+
+ val session = createStubSession()
+ val worker = session.workerProcess
+ assert(worker.process.isAlive, "worker should be alive before close")
+
+ val closeStart = System.nanoTime()
+ session.close()
+ val closeElapsedMs = (System.nanoTime() - closeStart) / 1000000L
+
+ assert(!worker.process.isAlive,
+ s"worker should have been SIGKILLed after ignoring SIGTERM (took ${closeElapsedMs}ms)")
+ assert(closeElapsedMs >= 500L,
+ s"close should have waited for gracefulTimeoutMs before escalating, " +
+ s"took ${closeElapsedMs}ms")
+ }
+
+ test("closing a session terminates its worker") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+
+ val session = createStubSession()
+ val worker = session.workerProcess
+ val socketFile = new File(udsPath(worker))
+
+ assert(worker.process.isAlive, "worker should be alive before session close")
+ assert(socketFile.exists(), "socket file should exist before session close")
+
+ session.close()
+
+ // The session-close path is synchronous: SIGTERM is sent and the process
+ // is reaped before `close` returns.
+ assert(!worker.process.isAlive,
+ "worker process should be terminated when the session closes")
+ assert(!socketFile.exists(),
+ "socket file should be cleaned up when the session closes")
+ }
+
+ test("concurrent session.close and dispatcher.close do not double-close the worker") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+
+ val sessions = (1 to 4).map(_ => createStubSession())
+ val workers = sessions.map(_.workerProcess)
+
+ val barrier = new java.util.concurrent.CyclicBarrier(sessions.size + 1)
+ val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]()
+
+ val sessionThreads = sessions.map { s =>
+ val t = new Thread(() => {
+ try {
+ barrier.await()
+ s.close()
+ } catch {
+ case t: Throwable => errors.add(t)
+ }
+ })
+ t.start()
+ t
+ }
+
+ val dispatcherThread = new Thread(() => {
+ try {
+ barrier.await()
+ dispatcher.close()
+ } catch {
+ case t: Throwable => errors.add(t)
+ }
+ })
+ dispatcherThread.start()
+
+ sessionThreads.foreach(_.join(30000))
+ dispatcherThread.join(30000)
+ dispatcher = null
+
+ assert(errors.isEmpty,
+ s"unexpected errors during concurrent close: ${errors.toArray.mkString(", ")}")
+ workers.foreach { w =>
+ assert(!w.process.isAlive,
+ s"worker at ${udsPath(w)} should be terminated after concurrent close")
+ }
+ }
+
+ test("close racing with in-flight createSession does not leak the worker") {
+ // The acquire-before-publish + post-publish closed re-check pattern in
+ // createSession is designed for this race: thread A is mid-spawn when
+ // thread B calls close(). Thread A must either throw IllegalStateException
+ // (post-publish check caught the close) or receive a session whose worker
+ // is reaped by close()'s iteration. No orphan process or socket file
+ // should remain in either case.
+ val readyLatch = new java.util.concurrent.CountDownLatch(1)
+ val releaseLatch = new java.util.concurrent.CountDownLatch(1)
+ val capturedWorkers =
+ new java.util.concurrent.ConcurrentLinkedQueue[DirectWorkerProcess]()
+ val racing = new DirectUnixSocketWorkerDispatcher(specWithRunner(defaultRunner)) {
+ override protected def createConnection(
+ socketPath: String): UnixSocketWorkerConnection =
+ new SocketFileConnection(socketPath)
+ override protected def createSessionForWorker(
+ worker: DirectWorkerProcess): WorkerSession = {
+ capturedWorkers.add(worker)
+ readyLatch.countDown()
+ // Block here so dispatcher.close() runs while createSession is in
+ // flight. Use a generous wait so a slow CI doesn't time out.
+ if (!releaseLatch.await(30, java.util.concurrent.TimeUnit.SECONDS)) {
+ fail("releaseLatch never fired -- test orchestration broken")
+ }
+ new StubWorkerSession(worker)
+ }
+ }
+ try {
+ val outcome =
+ new java.util.concurrent.atomic.AtomicReference[Either[Throwable, WorkerSession]]()
+ val createThread = new Thread(() => {
+ try {
+ val s = racing.createSession(None)
+ outcome.set(Right(s))
+ } catch {
+ case t: Throwable => outcome.set(Left(t))
+ }
+ }, "createSession-racer")
+ createThread.start()
+
+ // Wait for thread A to have published the worker and entered the
+ // blocking override.
+ assert(readyLatch.await(10, java.util.concurrent.TimeUnit.SECONDS),
+ "createSession thread never reached createSessionForWorker")
+
+ val closeThread = new Thread(() => racing.close(), "close-racer")
+ closeThread.start()
+ // Give close() time to flip `closed` and iterate workers.
+ Thread.sleep(200)
+
+ // Now release the in-flight createSession.
+ releaseLatch.countDown()
+
+ createThread.join(10000)
+ closeThread.join(10000)
+ assert(!createThread.isAlive, "createSession thread did not finish")
+ assert(!closeThread.isAlive, "close thread did not finish")
+
+ val captured = capturedWorkers.toArray(Array.empty[DirectWorkerProcess])
+ assert(captured.length == 1,
+ s"expected exactly one worker spawned, got ${captured.length}")
+ val worker = captured(0)
+
+ outcome.get() match {
+ case Left(e: IllegalStateException) =>
+ // Contractually allowed, but unreachable with this orchestration:
+ // readyLatch only fires after createSession has cleared both
+ // `closed` checks, so B's close cannot flip `closed` in time for
+ // A to observe it. Kept defensive so a future internal change
+ // that introduces a new window is still covered.
+ assert(e.getMessage.contains("closed"),
+ s"expected dispatcher-closed error, got: ${e.getMessage}")
+ case Left(other) =>
+ fail(s"unexpected exception from racing createSession: $other")
+ case Right(_) =>
+ // close() iterated the published worker and tore it down; the
+ // returned session points at a worker that should now be dead.
+ }
+
+ // Whichever path won, the worker must not still be running and the
+ // socket file must be gone.
+ val deadline = System.currentTimeMillis() + 5000
+ while (worker.process.isAlive && System.currentTimeMillis() < deadline) {
+ Thread.sleep(50)
+ }
+ val sockPath = udsPath(worker)
+ assert(!worker.process.isAlive,
+ s"worker process should be terminated after close, still alive at $sockPath")
+ assert(!new java.io.File(sockPath).exists(),
+ s"socket file $sockPath should have been removed")
+ } finally {
+ releaseLatch.countDown()
+ racing.close()
+ }
+ }
+
+ test("worker-provided graceful timeout is capped at the engine-side maximum") {
+ // The proto documents an engine-configurable maximum (fixed at 30s today).
+ // A 60s spec value should be clamped down.
+ val oversizedProps = UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setUnixDomainSocket(UnixDomainSocket.getDefaultInstance).build())
+ .setGracefulTerminationTimeoutMs(60000)
+ .build()
+ val spec = UDFWorkerSpecification.newBuilder()
+ .setDirect(DirectWorker.newBuilder()
+ .setRunner(defaultRunner).setProperties(oversizedProps).build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(spec)
+
+ val session = createStubSession()
+ assert(session.workerProcess.gracefulTimeoutMs == 30000L,
+ s"graceful timeout should be capped at 30000ms, " +
+ s"got ${session.workerProcess.gracefulTimeoutMs}")
+ session.close()
+ }
+
+ test("worker-provided init timeout is capped at the engine-side maximum") {
+ val oversizedProps = UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setUnixDomainSocket(UnixDomainSocket.getDefaultInstance).build())
+ .setInitializationTimeoutMs(60000)
+ .build()
+ val spec = UDFWorkerSpecification.newBuilder()
+ .setDirect(DirectWorker.newBuilder()
+ .setRunner(defaultRunner).setProperties(oversizedProps).build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(spec)
+
+ assert(dispatcher.initTimeoutMs == 30000L,
+ s"init timeout should be capped at 30000ms, got ${dispatcher.initTimeoutMs}")
+ }
+
+ test("createSession after close is rejected") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+ dispatcher.close()
+
+ val ex = intercept[IllegalStateException] {
+ dispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("closed"),
+ s"expected dispatcher-closed error, got: ${ex.getMessage}")
+ dispatcher = null
+ }
+
+ test("socket directory is owner-only (0700) on POSIX") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+ // Drive one createSession so a worker (and therefore the socket dir) is
+ // observable via the UDS connection's path.
+ val session = createStubSession()
+ val socketDir: Path = new File(udsPath(session.workerProcess)).toPath.getParent
+ session.close()
+
+ val view = Files.getFileAttributeView(socketDir, classOf[PosixFileAttributeView])
+ // Skip explicitly on non-POSIX filesystems rather than silently pass,
+ // so a CI environment without POSIX attributes is visible in the
+ // test report instead of giving false confidence.
+ assume(view != null, s"POSIX file attributes required to check $socketDir")
+ val perms = view.readAttributes().permissions().asScala.toSet
+ val expected = java.nio.file.attribute.PosixFilePermissions
+ .fromString("rwx------").asScala.toSet
+ assert(perms == expected,
+ s"socket directory $socketDir should be 0700, got ${perms.mkString(",")}")
+ }
+
+ test("socket directory is removed after dispatcher.close") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+ val session = createStubSession()
+ val socketDir = new File(udsPath(session.workerProcess)).toPath.getParent.toFile
+ assert(socketDir.exists(),
+ s"socket directory $socketDir should exist while a session is open")
+ session.close()
+
+ dispatcher.close()
+ dispatcher = null
+
+ assert(!socketDir.exists(),
+ s"socket directory $socketDir should be removed after dispatcher.close")
+ }
+
+ // -- Error-path tests -------------------------------------------------------
+
+ test("worker is cleaned up when createSessionForWorker throws") {
+ // A dispatcher whose createSessionForWorker always throws. The spawned
+ // worker must be terminated rather than leaked until dispatcher.close().
+ var capturedWorker: DirectWorkerProcess = null
+ val failingDispatcher =
+ new DirectUnixSocketWorkerDispatcher(specWithRunner(defaultRunner)) {
+ override protected def createConnection(
+ socketPath: String): UnixSocketWorkerConnection =
+ new SocketFileConnection(socketPath)
+ override protected def createSessionForWorker(
+ worker: DirectWorkerProcess): WorkerSession = {
+ capturedWorker = worker
+ throw new RuntimeException("session creation failed")
+ }
+ }
+
+ try {
+ val ex = intercept[RuntimeException] {
+ failingDispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("session creation failed"))
+ assert(capturedWorker != null, "worker should have been spawned before the failure")
+ assert(!capturedWorker.process.isAlive,
+ "worker process should have been terminated after session creation failed")
+ assert(capturedWorker.activeSessions == 0,
+ "worker session count should be released after failure")
+ } finally {
+ failingDispatcher.close()
+ }
+ }
+
+ test("DirectWorker without a connection is rejected") {
+ val badSpec = UDFWorkerSpecification.newBuilder()
+ .setDirect(DirectWorker.newBuilder().setRunner(defaultRunner).build())
+ .build()
+ val ex = intercept[IllegalArgumentException] {
+ new TestDirectWorkerDispatcher(badSpec)
+ }
+ assert(ex.getMessage.contains("connection must be set"),
+ s"expected missing-connection error, got: ${ex.getMessage}")
+ }
+
+ test("DirectWorker with non-UDS transport is rejected") {
+ val tcpProperties = UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setTcp(LocalTcpConnection.getDefaultInstance).build())
+ .build()
+ val badSpec = UDFWorkerSpecification.newBuilder()
+ .setDirect(DirectWorker.newBuilder()
+ .setRunner(defaultRunner).setProperties(tcpProperties).build())
+ .build()
+ val ex = intercept[IllegalArgumentException] {
+ new TestDirectWorkerDispatcher(badSpec)
+ }
+ assert(ex.getMessage.contains("UNIX domain socket"),
+ s"expected UDS-only error, got: ${ex.getMessage}")
+ }
+
+ test("socket file is cleaned up when createConnection throws") {
+ val capturedSocketPaths = new java.util.concurrent.ConcurrentLinkedQueue[String]()
+ val failingDispatcher =
+ new DirectUnixSocketWorkerDispatcher(specWithRunner(defaultRunner)) {
+ override protected def createConnection(
+ socketPath: String): UnixSocketWorkerConnection = {
+ capturedSocketPaths.add(socketPath)
+ throw new RuntimeException("connection creation failed")
+ }
+ override protected def createSessionForWorker(
+ worker: DirectWorkerProcess): WorkerSession =
+ new StubWorkerSession(worker)
+ }
+ try {
+ val ex = intercept[RuntimeException] {
+ failingDispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("connection creation failed"))
+ assert(capturedSocketPaths.size == 1, "createConnection should have been called once")
+ val socketPath = capturedSocketPaths.peek()
+ assert(!new File(socketPath).exists(),
+ s"socket file $socketPath should have been cleaned up")
+ } finally {
+ failingDispatcher.close()
+ }
+ }
+
+ test("empty ProcessCallable command is rejected with a clear error") {
+ val emptyRunner = ProcessCallable.newBuilder().build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(emptyRunner))
+ val ex = intercept[IllegalArgumentException] {
+ dispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("at least one entry"),
+ s"expected explicit empty-command error, got: ${ex.getMessage}")
+ }
+
+ test("spawnWorker fails when worker process exits immediately") {
+ val runner = ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand("echo 'fatal: bad config' >&2; exit 42").addCommand("--")
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(runner))
+
+ val ex = intercept[RuntimeException] {
+ dispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("exited with code 42"),
+ s"expected early-exit error, got: ${ex.getMessage}")
+ assert(ex.getMessage.contains("fatal: bad config"),
+ s"expected process output in error, got: ${ex.getMessage}")
+ }
+
+ test("spawnWorker times out when worker stays alive but never creates socket") {
+ // Distinct from the "process exits immediately" case: here the worker
+ // process is healthy but simply doesn't bind the socket, so the
+ // dispatcher must time out and SIGKILL-reap it rather than wait forever.
+ val hangingRunner = ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand("while true; do sleep 1; done").addCommand("--")
+ .build()
+ val shortInitProps = UDFWorkerProperties.newBuilder()
+ .setConnection(WorkerConnectionSpec.newBuilder()
+ .setUnixDomainSocket(UnixDomainSocket.getDefaultInstance).build())
+ .setInitializationTimeoutMs(500)
+ .build()
+ val spec = UDFWorkerSpecification.newBuilder()
+ .setDirect(DirectWorker.newBuilder()
+ .setRunner(hangingRunner).setProperties(shortInitProps).build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(spec)
+
+ val ex = intercept[DirectWorkerTimeoutException] {
+ dispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("did not create socket"),
+ s"expected init-timeout error, got: ${ex.getMessage}")
+ assert(ex.getMessage.contains("500ms"),
+ s"expected timeout value in error, got: ${ex.getMessage}")
+ }
+
+ // -- Environment lifecycle tests -------------------------------------------
+
+ test("skips installation when verification succeeds") {
+ val markerFile = Files.createTempFile("env-install-marker", ".txt").toFile
+ markerFile.delete()
+
+ val env = WorkerEnvironment.newBuilder()
+ .setEnvironmentVerification(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c").addCommand("exit 0").build())
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(s"touch ${markerFile.getAbsolutePath}").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val session = dispatcher.createSession(None)
+ session.close()
+
+ assert(!markerFile.exists(),
+ "installation should not run when verification succeeds")
+ }
+
+ test("runs installation when verification fails") {
+ val markerFile = Files.createTempFile("env-install-marker", ".txt").toFile
+ markerFile.delete()
+
+ val env = WorkerEnvironment.newBuilder()
+ .setEnvironmentVerification(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c").addCommand("exit 1").build())
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(s"touch ${markerFile.getAbsolutePath}").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val session = dispatcher.createSession(None)
+ session.close()
+
+ assert(markerFile.exists(),
+ "installation should run when verification fails")
+ markerFile.delete()
+ }
+
+ test("runs installation when no verification callable is provided") {
+ val markerFile = Files.createTempFile("env-install-marker", ".txt").toFile
+ markerFile.delete()
+
+ val env = WorkerEnvironment.newBuilder()
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(s"touch ${markerFile.getAbsolutePath}").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val session = dispatcher.createSession(None)
+ session.close()
+
+ assert(markerFile.exists(),
+ "installation should run when no verification is defined")
+ markerFile.delete()
+ }
+
+ test("installation failure throws with process output and prevents worker creation") {
+ val env = WorkerEnvironment.newBuilder()
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand("echo 'missing dependency: libfoo' >&2; exit 7").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val ex = intercept[RuntimeException] {
+ dispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("exit code 7"),
+ s"expected installation failure, got: ${ex.getMessage}")
+ assert(ex.getMessage.contains("missing dependency: libfoo"),
+ s"expected process output in error, got: ${ex.getMessage}")
+ }
+
+ test("installation that exceeds callableTimeoutMs is killed and reported") {
+ // Installation sleeps longer than callableTimeoutMs; the dispatcher
+ // must SIGKILL-reap it and surface a "Callable timed out" error
+ // rather than hang the caller.
+ val slowInstall = ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand("sleep 30").build()
+ val env = WorkerEnvironment.newBuilder().setInstallation(slowInstall).build()
+ val shortTimeoutDispatcher =
+ new DirectUnixSocketWorkerDispatcher(specWithEnv(env = env)) {
+ override protected def callableTimeoutMs: Long = 500L
+ override protected def createConnection(
+ socketPath: String): UnixSocketWorkerConnection =
+ new SocketFileConnection(socketPath)
+ override protected def createSessionForWorker(
+ worker: DirectWorkerProcess): WorkerSession =
+ new StubWorkerSession(worker)
+ }
+ try {
+ val ex = intercept[DirectWorkerTimeoutException] {
+ shortTimeoutDispatcher.createSession(None)
+ }
+ assert(ex.getMessage.contains("Callable timed out"),
+ s"expected callable-timeout error, got: ${ex.getMessage}")
+ assert(ex.getMessage.contains("500ms"),
+ s"expected timeout value in error, got: ${ex.getMessage}")
+ } finally {
+ shortTimeoutDispatcher.close()
+ }
+ }
+
+ test("environment setup runs only once across multiple sessions") {
+ val counterFile = Files.createTempFile("env-counter", ".txt").toFile
+ counterFile.delete()
+
+ val env = WorkerEnvironment.newBuilder()
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(s"echo invoked >> ${counterFile.getAbsolutePath}").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val s1 = dispatcher.createSession(None); s1.close()
+ val s2 = dispatcher.createSession(None); s2.close()
+
+ val src = scala.io.Source.fromFile(counterFile)
+ val lines = try src.getLines().toList finally src.close()
+ assert(lines.size == 1,
+ s"installation should run exactly once, but ran ${lines.size} time(s)")
+ counterFile.delete()
+ }
+
+ test("concurrent createSession still installs exactly once") {
+ // The sequential single-install test above cannot catch a missing
+ // lock around ensureEnvironmentReady. Race many createSession calls
+ // with an install script that takes long enough for the threads to
+ // queue on environmentLock, then verify it still ran exactly once.
+ val counterFile = Files.createTempFile("env-concurrent-install", ".txt").toFile
+ counterFile.delete()
+
+ val env = WorkerEnvironment.newBuilder()
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(
+ s"sleep 0.2; echo invoked >> ${counterFile.getAbsolutePath}").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val threads = 4
+ val startGate = new java.util.concurrent.CountDownLatch(1)
+ val doneGate = new java.util.concurrent.CountDownLatch(threads)
+ val sessions = new java.util.concurrent.ConcurrentLinkedQueue[WorkerSession]()
+ val errors = new java.util.concurrent.ConcurrentLinkedQueue[Throwable]()
+
+ (1 to threads).foreach { _ =>
+ new Thread(() => {
+ try {
+ startGate.await()
+ sessions.add(dispatcher.createSession(None))
+ } catch {
+ case t: Throwable => errors.add(t)
+ } finally {
+ doneGate.countDown()
+ }
+ }).start()
+ }
+ startGate.countDown()
+ assert(doneGate.await(30, java.util.concurrent.TimeUnit.SECONDS),
+ "createSession threads did not finish in time")
+ assert(errors.isEmpty,
+ s"unexpected errors during concurrent createSession: ${errors.toArray.mkString(", ")}")
+
+ val src = scala.io.Source.fromFile(counterFile)
+ val lines = try src.getLines().toList finally src.close()
+ assert(lines.size == 1,
+ s"installation should run exactly once under concurrent createSession, " +
+ s"but ran ${lines.size} time(s)")
+
+ sessions.asScala.foreach(_.close())
+ counterFile.delete()
+ }
+
+ test("failed environment setup is not retried on subsequent createSession") {
+ val counterFile = Files.createTempFile("env-failed-counter", ".txt").toFile
+ counterFile.delete()
+
+ // Installation script appends a line every time it runs, then always
+ // fails. The first createSession should run it; the second should be
+ // rejected immediately without re-running.
+ val env = WorkerEnvironment.newBuilder()
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(
+ s"echo invoked >> ${counterFile.getAbsolutePath}; exit 1").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val first = intercept[RuntimeException] { dispatcher.createSession(None) }
+ assert(first.getMessage.contains("installation failed"),
+ s"expected first-attempt installation failure, got: ${first.getMessage}")
+
+ val second = intercept[RuntimeException] { dispatcher.createSession(None) }
+ assert(second.getMessage.contains("previously failed"),
+ s"expected cached failure on retry, got: ${second.getMessage}")
+
+ val src = scala.io.Source.fromFile(counterFile)
+ val lines = try src.getLines().toList finally src.close()
+ assert(lines.size == 1,
+ s"installation should run only once across failed retries, got ${lines.size}")
+ counterFile.delete()
+ }
+
+ test("installation timeout transitions to Failed and is not retried") {
+ val counterFile = Files.createTempFile("env-timeout-counter", ".txt").toFile
+ counterFile.delete()
+
+ // Install appends to a counter file, then sleeps past callableTimeoutMs
+ // so runCallable times out. The dispatcher must mark the env Failed
+ // and reject the next createSession without re-running install.
+ val env = WorkerEnvironment.newBuilder()
+ .setInstallation(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(
+ s"echo invoked >> ${counterFile.getAbsolutePath}; sleep 30").build())
+ .build()
+ val timeoutDispatcher =
+ new DirectUnixSocketWorkerDispatcher(specWithEnv(env = env)) {
+ override protected def callableTimeoutMs: Long = 500L
+ override protected def createConnection(
+ socketPath: String): UnixSocketWorkerConnection =
+ new SocketFileConnection(socketPath)
+ override protected def createSessionForWorker(
+ worker: DirectWorkerProcess): WorkerSession =
+ new StubWorkerSession(worker)
+ }
+ try {
+ val first = intercept[DirectWorkerTimeoutException] {
+ timeoutDispatcher.createSession(None)
+ }
+ assert(first.getMessage.contains("Callable timed out"),
+ s"expected callable-timeout error, got: ${first.getMessage}")
+
+ val second = intercept[DirectWorkerException] {
+ timeoutDispatcher.createSession(None)
+ }
+ assert(second.getMessage.contains("previously failed"),
+ s"expected cached failure on retry, got: ${second.getMessage}")
+
+ val src = scala.io.Source.fromFile(counterFile)
+ val lines = try src.getLines().toList finally src.close()
+ assert(lines.size == 1,
+ s"installation should run only once across timed-out retries, got ${lines.size}")
+ } finally {
+ timeoutDispatcher.close()
+ counterFile.delete()
+ }
+ }
+
+ test("non-None securityScope is rejected until pooling lands") {
+ dispatcher = new TestDirectWorkerDispatcher(specWithRunner(defaultRunner))
+ val scope = new WorkerSecurityScope {
+ override def equals(obj: Any): Boolean = obj.isInstanceOf[this.type]
+ override def hashCode(): Int = 0
+ }
+ val ex = intercept[IllegalArgumentException] {
+ dispatcher.createSession(Some(scope))
+ }
+ assert(ex.getMessage.contains("not supported yet"),
+ s"expected unsupported-scope error, got: ${ex.getMessage}")
+ }
+
+ test("verification without installation is rejected") {
+ val env = WorkerEnvironment.newBuilder()
+ .setEnvironmentVerification(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c").addCommand("exit 0").build())
+ .build()
+ val ex = intercept[IllegalArgumentException] {
+ new TestDirectWorkerDispatcher(specWithEnv(env = env))
+ }
+ assert(ex.getMessage.contains("installation"),
+ s"expected installation-required error, got: ${ex.getMessage}")
+ }
+
+ test("cleanup runs on dispatcher close") {
+ val cleanupMarker = Files.createTempFile("env-cleanup-marker", ".txt").toFile
+ cleanupMarker.delete()
+
+ val env = WorkerEnvironment.newBuilder()
+ .setEnvironmentCleanup(ProcessCallable.newBuilder()
+ .addCommand("bash").addCommand("-c")
+ .addCommand(s"touch ${cleanupMarker.getAbsolutePath}").build())
+ .build()
+ dispatcher = new TestDirectWorkerDispatcher(specWithEnv(env = env))
+
+ val session = dispatcher.createSession(None)
+ session.close()
+
+ assert(!cleanupMarker.exists(),
+ "cleanup should not run until dispatcher is closed")
+
+ dispatcher.close()
+ dispatcher = null
+
+ assert(cleanupMarker.exists(),
+ "cleanup should run when dispatcher is closed")
+ cleanupMarker.delete()
+ }
+}
diff --git a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/WorkerAbstractionSuite.scala b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/WorkerAbstractionSuite.scala
deleted file mode 100644
index 42f53af07424..000000000000
--- a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/WorkerAbstractionSuite.scala
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.spark.udf.worker.core
-
-import org.scalatest.funsuite.AnyFunSuite // scalastyle:ignore funsuite
-
-class WorkerAbstractionSuite
- extends AnyFunSuite { // scalastyle:ignore funsuite
-
- test("dummy") {}
-}
diff --git a/udf/worker/proto/src/main/protobuf/common.proto b/udf/worker/proto/src/main/protobuf/common.proto
index 9c50cdd7a7e4..ee032def73ef 100644
--- a/udf/worker/proto/src/main/protobuf/common.proto
+++ b/udf/worker/proto/src/main/protobuf/common.proto
@@ -32,6 +32,13 @@ enum UDFWorkerDataFormat {
}
// The UDF execution type/shape.
+//
+// BIDIRECTIONAL_STREAMING is the only pattern supported by the engine for
+// now. It may be possible to express all UDF types (scalar, mapPartitions,
+// and eventually UDAF/UDTF/streaming) on top of this single pattern by
+// framing their phases as messages on the stream, but that is a design
+// question worth revisiting as additional UDF types are added -- for
+// example, aggregation may prefer a multi-round or specialized pattern.
enum UDFProtoCommunicationPattern {
UDF_PROTO_COMMUNICATION_PATTERN_UNSPECIFIED = 0;
diff --git a/udf/worker/proto/src/main/protobuf/worker_spec.proto b/udf/worker/proto/src/main/protobuf/worker_spec.proto
index f2eacf2b3ce3..83dac4f962e5 100644
--- a/udf/worker/proto/src/main/protobuf/worker_spec.proto
+++ b/udf/worker/proto/src/main/protobuf/worker_spec.proto
@@ -140,13 +140,17 @@ message WorkerCapabilities {
// Whether multiple, concurrent UDF
// connections are supported by this worker
// (for example via multi-threading).
- //
+ //
// In the first implementation of the engine-side
// worker specification, this property will not be used.
- //
+ //
// Usage of this property can be enabled in the future if the
// engine implements more advanced resource management (TBD).
//
+ // TODO: wire this into planning/scheduling -- SPIP worker-spec §2.4
+ // "Parallelism" describes the intended use (e.g., multiplex tasks onto
+ // a single worker vs. spawn multiple workers per executor).
+ //
// (Optional)
optional bool supports_concurrent_udfs = 3;
@@ -190,25 +194,31 @@ message UDFWorkerProperties {
// (Optional)
optional int32 graceful_termination_timeout_ms = 2;
- // The connection this [[DirectWorker]] supports. Note that a single
- // connection is sufficient to run multiple UDFs and (gRPC) services.
+ // A [[DirectWorker]] exposes one server-side connection endpoint (a
+ // UDS path or a TCP port) that all sessions on the worker share.
+ // Multi-connection workers (e.g., separate data and control channels)
+ // are not supported in this release.
+ //
+ // On [[DirectWorker]] creation, connection information
+ // is passed to the callable as a string parameter.
+ // The string format depends on the [[WorkerConnectionSpec]]:
//
- // On [[DirectWorker]] creation, connection information
- // is passed to the callable as a string parameter.
- // The string format depends on the [[WorkerConnection]]:
- //
// For example, when using TCP, the callable argument will be:
// --connection PORT
// Here is a concrete example
// --connection 8080
- //
+ //
// For the format of each specific transport type, see the comments below.
//
// (Required)
- WorkerConnection connection = 3;
+ WorkerConnectionSpec connection = 3;
}
-message WorkerConnection {
+// Describes one connection (transport endpoint) that a [[DirectWorker]]
+// exposes. This is a configuration message -- the live transport object
+// used by the engine at runtime is the Scala abstraction
+// `org.apache.spark.udf.worker.core.WorkerConnection`.
+message WorkerConnectionSpec {
// (Required)
oneof transport {
UnixDomainSocket unix_domain_socket = 1;
@@ -275,7 +285,7 @@ message ProcessCallable {
//
// --connection
// The value of the connection argument is a string with
- // engine-assinged connection parameters. See [[UDFWorkerProperties]]
+ // engine-assigned connection parameters. See [[UDFWorkerProperties]]
// for details.
//
// (Optional)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org