You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "kz930 (via GitHub)" <gi...@apache.org> on 2026/08/01 02:40:37 UTC
[PR] build: let WorkflowOperator route Python-executing tests to amber-integration [texera]
kz930 opened a new pull request, #7207:
URL: https://github.com/apache/texera/pull/7207
### What changes were proposed in this PR?
The `amber` job runs the `WorkflowOperator` module's test suite and installs no Python packages. The runner's interpreter is there — `PythonCodeRawInvalidTextSpec` uses it for a stdlib-only `py_compile` check — but pandas and plotly are not importable, so a test that *executes* an operator's generated template rather than only compiling it fails its dependency probe and cancels. A cancellation is neither a pass nor a failure, so the suite still reports "All tests passed" and the missing coverage leaves no trace in the build.
This gives `WorkflowOperator` the split `amber` already has:
1. An `IntegrationTest` tag under `common/workflow-operator/src/test`. `amber`'s tag cannot be reused: it lives in `amber/src/test/integration`, and `amber` depends on `WorkflowOperator`, not the other way around.
2. A `Test / testOptions` filter in `common/workflow-operator/build.sbt` reading the same `AMBER_TEST_FILTER`. The `amber` job already sets that variable on the step that invokes `WorkflowOperator/jacoco`, so that job needs no change for the exclusion to take effect.
3. `"WorkflowOperator/test"` added to the `amber-integration` invocation, which already runs `integration-only` with `amber/requirements.txt` and `amber/operator-requirements.txt` installed (pandas 2.2.3, plotly 5.24.1).
With the selection logic now wanted in two modules, it moves to `project/TestFilters.scala` next to the build's other shared sbt helpers (`AddMetaInfLicenseFiles.scala`, `JdkOptions.scala`), and each module passes its own env var, tag, and any extra ScalaTest arguments.
`PythonCodeRawInvalidTextSpec` gains a tagged case asserting that pandas and plotly import in the interpreter it already resolves — `py_compile` only parses the emitted code, while running it needs the packages it imports. Being the first user of the tag it also exercises the routing, and it turns a missing install in `amber-integration` into a failure rather than the silent cancellation described above. Elsewhere a bare interpreter is a local-setup fact, so it cancels instead.
### Any related issues, documentation, discussions?
Fixes #7186. The silent cancellation was found while reviewing #7149, whose runtime test is the case that cancels today; tagging it is a one-line follow-up once this is in.
### How was this PR tested?
Behavior of the extraction is unchanged: `show Test/testOptions` for both modules under all three env states produces byte-identical arguments to the previous inline code — `-l <tag>` on `skip-integration`, `-n <tag>` on `integration-only`, nothing when unset.
The new case was exercised in all four states it can be in, running `WorkflowOperator/testOnly *PythonCodeRawInvalidTextSpec`: with the packages present it passes under `integration-only` (1 test selected); under `skip-integration` it is excluded while the spec's two existing tests still run (2 tests); pointed at a bare interpreter it fails under `integration-only` and cancels with no filter set.
`sbt scalafmtCheckAll` is clean, and `WorkflowOperator`'s `Compile` and `Test` scalafix checks pass.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3764569668
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,19 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it and amber-integration, which installs operator-requirements.txt,
+// runs it. The amber job already sets this env var on the step that invokes
+// WorkflowOperator/jacoco, so no workflow change is needed for the exclusion.
+//
+// PythonCodeRawInvalidTextSpec reads the same env var and value directly, to tell
+// a missing package in amber-integration (a defect) from one on a developer's
+// machine (a local-setup fact). Renaming either without the other leaves that test
+// cancelling in the job meant to fail it.
Review Comment:
"either" and "the other" have no resolvable pair. The preceding sentence names an env var *and* a value, while the pair that must stay in step is the two sites — this file and the spec. Naming both ends is the point of the comment.
```suggestion
// machine (a local-setup fact). Changing the variable or the value here without
// changing it there leaves that test cancelling in the job meant to fail it.
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
Review Comment:
This `cancel` is first forced inside `Future(call)(ec)` (`interceptBounded`, `:130`), so it lands on a worker thread. `TestCanceledException` is a `RuntimeException`, so `intercept` converts it into a `TestFailedException` rather than letting it cancel — five cases here fail where `:70` promises a cancel.
The cap case at `:166` is the sharper half: `Try` (`:176`) records that cancellation as the `Failure` its only assertion looks for. On a machine with no interpreter it reports green without ever reaching the cap.
Forcing `python` once on the test thread fixes both halves:
```scala
override def withFixture(test: NoArgTest): org.scalatest.Outcome = {
val _ = python // resolve here, so a machine without one cancels rather than fails
super.withFixture(test)
}
```
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,19 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it and amber-integration, which installs operator-requirements.txt,
Review Comment:
Reads as a compound object ("excludes it and amber-integration") until the verb three words later forces a re-parse.
```suggestion
// excludes it, and amber-integration, which installs operator-requirements.txt,
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `responseMillis` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startupMillis` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] for every worker the pool could
+ * not give out or keep — one that would not start, would not report ready, or
+ * died mid-job — so one `catch` covers a caller's whole fallback.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot — though not
+ * PYTHONPATH or any other PYTHON* var, which the `-I` above makes CPython ignore.
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool. `timeouts` is not — it bounds
+ * this call, so a caller whose jobs are slower than most can raise it without
+ * splitting the pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty,
+ timeouts: Timeouts = Timeouts.Default
+ ): Outcome = {
+ val pool = pools.computeIfAbsent(
+ Key(resourcePath, pythonExe, interpreterArgs.toList, launchArgs.toList, env.toList.sorted),
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request, timeouts)
+ }
+
+ /** What makes two launches the same worker. Compared field by field rather than
+ * as one joined string, so a value carrying whatever the separator was — a
+ * python path with a space in it, a PYTHONPATH — cannot make two different
+ * launches share a pool.
+ */
+ private final case class Key(
+ resourcePath: String,
+ pythonExe: String,
+ interpreterArgs: List[String],
+ launchArgs: List[String],
+ env: List[(String, String)]
+ )
+
+ /** How long a caller at the worker cap waits before re-examining it. Not a
+ * deadline — see [[Pool.borrow]].
+ */
+ private val CapRecheckMillis: Long = 250
+
+ private val pools = new ConcurrentHashMap[Key, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode, timeouts: Timeouts): Outcome = {
+ val w = borrow(timeouts)
+ try {
+ val outcome = w.run(request, timeouts.responseMillis)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ // Every throw, not only a WorkerDiedException: an interrupt on this
+ // thread — what an executor's `shutdownNow` sends — leaves the blocking
+ // read or write as an InterruptedException, which `NonFatal` excludes and
+ // [[Worker.run]] therefore does not wrap. A worker left neither returned
+ // nor discarded costs this sub-pool a slot for the life of the JVM, and
+ // its answer may still be in flight, so it cannot be reused either way.
+ case e: Throwable =>
+ discard(w)
+ throw e
+ }
+ }
+
+ @tailrec
+ private def borrow(timeouts: Timeouts): Worker = {
+ val existing = idle.poll()
+ if (existing != null) existing
+ else if (liveCount.getAndIncrement() < maxWorkers) {
+ try create(timeouts)
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ // At the cap. Waiting outright for a returned worker would strand this
+ // caller when the ones ahead are discarded instead: a discard frees a
+ // slot without putting anything back. So wait only briefly, then look at
+ // the cap again — the next pass starts a replacement. A long queue still
+ // waits as long as it takes; that is the caller's own backlog, not a hang.
+ val returned = idle.poll(CapRecheckMillis, TimeUnit.MILLISECONDS)
+ if (returned != null) returned else borrow(timeouts)
+ }
+ }
+
+ private def create(timeouts: Timeouts): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ // A worker that cannot be started is a worker death like any other, so it
+ // leaves through the same exception: `start` throws a bare IOException — no
+ // interpreter at that path, or the OS out of processes under a fan-out — and
+ // a caller's fallback is written against [[WorkerDiedException]].
+ val process =
+ try pb.start()
+ catch {
+ case NonFatal(e) =>
+ throw new WorkerDiedException(
+ s"could not start python worker for $resourcePath: ${e.getMessage}",
+ e
+ )
+ }
+ val w = new Worker(process, s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ // Anything thrown before the worker joins `all` — a startup that timed out,
+ // an interrupt on this thread — leaves an interpreter nothing else would
+ // reap: not the caller, which never gets the handle, and not the shutdown
+ // hook, which walks `all`. `destroy` is idempotent, so a worker that already
+ // killed itself on the way out is no exception to that.
+ try w.awaitReady(timeouts.startupMillis)
+ catch {
+ case e: Throwable =>
+ w.destroy()
+ throw e
+ }
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin and background drains of
+ // its stdout (the protocol) and stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ // Protocol lines the worker has written, `None` marking end of stream. A
+ // dedicated thread owns the blocking read so a caller can wait with a
+ // timeout: `readLine` on a process pipe answers neither an interrupt nor a
+ // deadline, and only closing the pipe — killing the process — releases it.
+ private val lines = new LinkedBlockingQueue[Option[String]]()
+
+ // Owns the writing end for the same reason.
+ private val writer: ExecutorService = Executors.newSingleThreadExecutor { r =>
+ val t = new Thread(r, "python-worker-stdin")
+ t.setDaemon(true)
+ t
+ }
+
+ private val outThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ lines.put(Some(line))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ finally lines.put(None)
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stdout")
+ t.start()
+ t
+ }
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Wait for the worker's startup `{"ready": true}`; if it dies first (e.g. an
+ * import failed) or never gets there, surface its stderr. A worker that
+ * fails here is killed: it is not in the pool's set yet, so nothing else
+ * will reap it.
+ *
+ * A first line that is not the protocol at all counts as not ready, rather
+ * than throwing whatever the parser throws: the caller's contract here is
+ * [[WorkerDiedException]], and an escape past it would leave this
+ * interpreter with nothing to reap it. The line goes into the message —
+ * stderr is empty when a script writes its noise to stdout.
+ */
+ def awaitReady(timeoutMillis: Long): Unit = {
+ val line = nextLine(timeoutMillis, "signal ready")
+ val ready =
+ try objectMapper.readTree(line).path("ready").asBoolean(false)
+ catch { case NonFatal(_) => false }
+ if (!ready) {
+ destroy()
+ throw new WorkerDiedException(
+ s"python worker [$label] did not signal ready; it wrote: ${abbreviate(line)}." +
+ s" stderr:\n${drainErr()}"
+ )
+ }
+ }
+
+ def run(request: ObjectNode, timeoutMillis: Long): Outcome =
+ try {
+ send(request, timeoutMillis)
+ val node = objectMapper.readTree(nextLine(timeoutMillis, "answer"))
+ Outcome(
+ node.path("exit").asInt(1),
Review Comment:
A reply that parses but carries no `exit` defaults to 1 here. The caller then reports the *descriptor* as failing `py_compile`, with `(no output)` as the evidence — the wrong party blamed.
`awaitReady:389-391` already refuses to guess on the startup line: a parse failure becomes "did not signal ready", quoting what arrived. The response side still coerces. `py_compile_worker.py` always writes all three fields, but the class doc sells this seam as generic and #7149 brings the second script — would a missing `exit` be better leaving as a `WorkerDiedException`, so the caller's existing fallback covers it?
##########
project/TestFilters.scala:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import sbt._
+
+/**
+ * Selects a module's tagged tests for the fast-unit job or the integration job:
+ * skip-integration excludes them, integration-only runs only them, unset runs
+ * everything, and any other value fails the build rather than quietly running
+ * everything in a job that selected a subset. Shared because the mapping is
+ * identical in every module, while the env var and the tag are not — the tag
+ * annotation has to live somewhere the module's own Test config can see.
+ */
+object TestFilters {
+
+ /** @param envVar the variable the two CI jobs set to opposite values; it has to
+ * be the one the workflow already sets on the step that invokes
+ * this module's tests, or neither subset is selected.
+ * @param tag fully-qualified name of the tag annotation, as
+ * `classOf[...].getName` gives it at the test site — ScalaTest
+ * matches these by string, so a rename that misses one side
+ * silently stops filtering.
+ */
+ def integrationSplit(envVar: String, tag: String): Seq[TestOption] =
+ sys.env.get(envVar) match {
+ case Some("skip-integration") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-l", tag))
+ case Some("integration-only") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-n", tag))
+ case Some(other) => sys.error(s"unrecognized $envVar value: $other")
Review Comment:
This fails at project load, so it is the first and only thing the developer sees — worth naming the accepted values there rather than sending them to find this file.
```suggestion
case Some(other) => sys.error(s"$envVar=$other: use skip-integration or integration-only")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3742251585
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+ private val CompileWorker = "/python/py_compile_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: the fixture never answers, so it cannot race.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private def python(): String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: com.fasterxml.jackson.databind.node.ObjectNode = objectMapper.createObjectNode()
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python(),
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = ShortTimeouts
+ )
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto a worker that goes quiet:
Review Comment:
`borrow` creates a worker per caller up to the cap, so this starts `maxWorkers` hanging workers, not one. The singular hides the mechanism the case exists to test: the discards that free slots happen on different workers from the one the capped caller is waiting behind.
```suggestion
// One caller more than there are workers, all onto workers that go quiet:
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool. `timeouts` is not — it bounds
+ * this call, so a caller whose jobs are slower than most can raise it without
+ * splitting the pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty,
+ timeouts: Timeouts = Timeouts.Default
+ ): Outcome = {
+ val pool = pools.computeIfAbsent(
+ Key(resourcePath, pythonExe, interpreterArgs.toList, launchArgs.toList, env.toList.sorted),
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request, timeouts)
+ }
+
+ /** What makes two launches the same worker. Compared field by field rather than
+ * as one joined string, so a value carrying whatever the separator was — a
+ * python path with a space in it, a PYTHONPATH — cannot make two different
+ * launches share a pool.
+ */
+ private final case class Key(
+ resourcePath: String,
+ pythonExe: String,
+ interpreterArgs: List[String],
+ launchArgs: List[String],
+ env: List[(String, String)]
+ )
+
+ /** How long a caller at the worker cap waits before re-examining it. Not a
+ * deadline — see [[Pool.borrow]].
+ */
+ private val CapRecheckMillis: Long = 250
+
+ private val pools = new ConcurrentHashMap[Key, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode, timeouts: Timeouts): Outcome = {
+ val w = borrow(timeouts)
+ try {
+ val outcome = w.run(request, timeouts.responseMillis)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ case e: WorkerDiedException =>
+ discard(w)
+ throw e
+ }
+ }
+
+ @tailrec
+ private def borrow(timeouts: Timeouts): Worker = {
+ val existing = idle.poll()
+ if (existing != null) existing
+ else if (liveCount.getAndIncrement() < maxWorkers) {
+ try create(timeouts)
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ // At the cap. Waiting outright for a returned worker would strand this
+ // caller when the ones ahead are discarded instead: a discard frees a
+ // slot without putting anything back. So wait only briefly, then look at
+ // the cap again — the next pass starts a replacement. A long queue still
+ // waits as long as it takes; that is the caller's own backlog, not a hang.
+ val returned = idle.poll(CapRecheckMillis, TimeUnit.MILLISECONDS)
+ if (returned != null) returned else borrow(timeouts)
+ }
+ }
+
+ private def create(timeouts: Timeouts): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ val w = new Worker(pb.start(), s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ w.awaitReady(timeouts.startupMillis)
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin and background drains of
+ // its stdout (the protocol) and stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ // Protocol lines the worker has written, `None` marking end of stream. A
+ // dedicated thread owns the blocking read so a caller can wait with a
+ // timeout: `readLine` on a process pipe answers neither an interrupt nor a
+ // deadline, and only closing the pipe — killing the process — releases it.
+ private val lines = new LinkedBlockingQueue[Option[String]]()
+
+ // Owns the writing end for the same reason.
+ private val writer: ExecutorService = Executors.newSingleThreadExecutor { r =>
+ val t = new Thread(r, "python-worker-stdin")
+ t.setDaemon(true)
+ t
+ }
+
+ private val outThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ lines.put(Some(line))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ finally lines.put(None)
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stdout")
+ t.start()
+ t
+ }
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Wait for the worker's startup `{"ready": true}`; if it dies first (e.g. an
+ * import failed) or never gets there, surface its stderr. A worker that
+ * fails here is killed: it is not in the pool's set yet, so nothing else
+ * will reap it.
+ */
+ def awaitReady(timeoutMillis: Long): Unit = {
+ val line = nextLine(timeoutMillis, "signal ready")
+ if (!objectMapper.readTree(line).path("ready").asBoolean(false)) {
Review Comment:
`Worker.run` converts a failed protocol parse into `WorkerDiedException` (`:370`); `awaitReady` parses the same protocol here and does not. A non-JSON first line throws a raw `JsonParseException` past the caller's fallback. It also orphans the interpreter: `destroy()` here runs only on the `ready == false` branch, and the worker is not in `all` yet (`:253`), so the shutdown hook cannot reap it.
Not reachable with the two scripts in this PR, since both write their JSON first under `-I`. But the class doc sells this seam as generic, and #7149 brings the second worker script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5273362761
Resolved the problems..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #7207:
URL: https://github.com/apache/texera/pull/7207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build: let WorkflowOperator route Python-executing tests to amber-integration [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5149385103
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 2 better · 🔴 4 worse · ⚪ 9 noise (<±5%) · 0 without baseline
> Compared against main `0107183` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/30680478258)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 372 | 0.227 | 25,925/32,986/32,986 us | 🟢 -13.3% / 🔴 +104.1% |
| 🔴 | bs=100 sw=10 sl=64 | 790 | 0.482 | 128,331/150,002/150,002 us | 🔴 +6.9% / 🔴 +35.4% |
| ⚪ | bs=1000 sw=10 sl=64 | 909 | 0.555 | 1,100,680/1,143,724/1,143,724 us | ⚪ within ±5% / 🔴 -8.4% |
<details><summary>Baseline details</summary>
Latest main `0107183` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 372 tuples/sec | 400 tuples/sec | 765.25 tuples/sec | -7.0% | -51.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.227 MB/s | 0.244 MB/s | 0.467 MB/s | -7.0% | -51.4% |
| bs=10 sw=10 sl=64 | p50 | 25,925 us | 24,173 us | 12,703 us | +7.3% | +104.1% |
| bs=10 sw=10 sl=64 | p95 | 32,986 us | 38,046 us | 16,199 us | -13.3% | +103.6% |
| bs=10 sw=10 sl=64 | p99 | 32,986 us | 38,046 us | 19,557 us | -13.3% | +68.7% |
| bs=100 sw=10 sl=64 | throughput | 790 tuples/sec | 819 tuples/sec | 965.22 tuples/sec | -3.5% | -18.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.482 MB/s | 0.5 MB/s | 0.589 MB/s | -3.6% | -18.2% |
| bs=100 sw=10 sl=64 | p50 | 128,331 us | 120,042 us | 103,830 us | +6.9% | +23.6% |
| bs=100 sw=10 sl=64 | p95 | 150,002 us | 143,661 us | 110,778 us | +4.4% | +35.4% |
| bs=100 sw=10 sl=64 | p99 | 150,002 us | 143,661 us | 118,713 us | +4.4% | +26.4% |
| bs=1000 sw=10 sl=64 | throughput | 909 tuples/sec | 927 tuples/sec | 992.33 tuples/sec | -1.9% | -8.4% |
| bs=1000 sw=10 sl=64 | MB/s | 0.555 MB/s | 0.566 MB/s | 0.606 MB/s | -1.9% | -8.4% |
| bs=1000 sw=10 sl=64 | p50 | 1,100,680 us | 1,077,483 us | 1,016,959 us | +2.2% | +8.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,143,724 us | 1,128,204 us | 1,062,501 us | +1.4% | +7.6% |
| bs=1000 sw=10 sl=64 | p99 | 1,143,724 us | 1,128,204 us | 1,082,611 us | +1.4% | +5.6% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,537.09,200,128000,372,0.227,25925.34,32985.57,32985.57
1,100,10,64,20,2530.59,2000,1280000,790,0.482,128330.97,150001.96,150001.96
2,1000,10,64,20,21993.53,20000,12800000,909,0.555,1100680.21,1143723.61,1143723.61
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3716364601
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{ConcurrentHashMap, LinkedBlockingQueue}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). Only a hard interpreter crash ends a
+ * worker; the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty
+ ): Outcome = {
+ val envKey = env.toSeq.sorted.map { case (k, v) => s"$k=$v" }
+ val key =
+ (resourcePath +: pythonExe +: (interpreterArgs ++ launchArgs ++ envKey)).mkString(" ")
+ val pool = pools.computeIfAbsent(
+ key,
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request)
+ }
+
+ private val pools = new ConcurrentHashMap[String, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode): Outcome = {
+ val w = borrow()
+ try {
+ val outcome = w.run(request)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ case e: WorkerDiedException =>
+ discard(w)
+ throw e
+ }
+ }
+
+ private def borrow(): Worker = {
+ val existing = idle.poll()
+ if (existing != null) return existing
+ if (liveCount.getAndIncrement() < maxWorkers) {
+ try create()
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ idle.take() // at the cap — block until a worker is returned
+ }
+ }
+
+ private def create(): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ val w = new Worker(pb.start(), s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ w.awaitReady()
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin/stdout and a background
+ // drain of its own stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val stdout: BufferedReader =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Block until the worker's startup `{"ready": true}` arrives; if it dies
+ * first (e.g. an import failed), surface its stderr.
+ */
+ def awaitReady(): Unit = {
+ val line = stdout.readLine()
+ if (line == null || !objectMapper.readTree(line).path("ready").asBoolean(false)) {
+ throw new WorkerDiedException(
+ s"python worker [$label] did not signal ready. stderr:\n${drainErr()}"
+ )
+ }
+ }
+
+ def run(request: ObjectNode): Outcome =
+ try {
+ stdin.write(objectMapper.writeValueAsString(request))
+ stdin.write("\n")
+ stdin.flush()
+ val line = stdout.readLine()
Review Comment:
Fixed in `138ff92d1`: a dedicated thread now owns the blocking read and hands lines over a queue, so a caller waits with a deadline and a worker that misses it is killed and discarded — falling back to the one-shot subprocess, as a crash already did. Response keeps the 30 seconds the one-shot path allowed, startup gets its own budget, and `PythonWorkerPoolSpec` covers both against a fixture worker that goes quiet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3769805150
##########
project/TestFilters.scala:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import sbt._
+
+/**
+ * Selects a module's tagged tests for the fast-unit job or the integration job:
+ * skip-integration excludes them, integration-only runs only them, unset runs
+ * everything, and any other value fails the build rather than quietly running
+ * everything in a job that selected a subset. Shared because the mapping is
+ * identical in every module, while the env var and the tag are not — the tag
+ * annotation has to live somewhere the module's own Test config can see.
+ */
+object TestFilters {
+
+ /** @param envVar the variable the two CI jobs set to opposite values; it has to
+ * be the one the workflow already sets on the step that invokes
+ * this module's tests, or neither subset is selected.
+ * @param tag fully-qualified name of the tag annotation, as
+ * `classOf[...].getName` gives it at the test site — ScalaTest
+ * matches these by string, so a rename that misses one side
+ * silently stops filtering.
+ */
+ def integrationSplit(envVar: String, tag: String): Seq[TestOption] =
+ sys.env.get(envVar) match {
+ case Some("skip-integration") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-l", tag))
+ case Some("integration-only") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-n", tag))
+ case Some(other) => sys.error(s"unrecognized $envVar value: $other")
Review Comment:
Applied in `ed51a857`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5247283394
You're right. Applied your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3716364601
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{ConcurrentHashMap, LinkedBlockingQueue}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). Only a hard interpreter crash ends a
+ * worker; the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty
+ ): Outcome = {
+ val envKey = env.toSeq.sorted.map { case (k, v) => s"$k=$v" }
+ val key =
+ (resourcePath +: pythonExe +: (interpreterArgs ++ launchArgs ++ envKey)).mkString(" ")
+ val pool = pools.computeIfAbsent(
+ key,
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request)
+ }
+
+ private val pools = new ConcurrentHashMap[String, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode): Outcome = {
+ val w = borrow()
+ try {
+ val outcome = w.run(request)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ case e: WorkerDiedException =>
+ discard(w)
+ throw e
+ }
+ }
+
+ private def borrow(): Worker = {
+ val existing = idle.poll()
+ if (existing != null) return existing
+ if (liveCount.getAndIncrement() < maxWorkers) {
+ try create()
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ idle.take() // at the cap — block until a worker is returned
+ }
+ }
+
+ private def create(): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ val w = new Worker(pb.start(), s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ w.awaitReady()
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin/stdout and a background
+ // drain of its own stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val stdout: BufferedReader =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Block until the worker's startup `{"ready": true}` arrives; if it dies
+ * first (e.g. an import failed), surface its stderr.
+ */
+ def awaitReady(): Unit = {
+ val line = stdout.readLine()
+ if (line == null || !objectMapper.readTree(line).path("ready").asBoolean(false)) {
+ throw new WorkerDiedException(
+ s"python worker [$label] did not signal ready. stderr:\n${drainErr()}"
+ )
+ }
+ }
+
+ def run(request: ObjectNode): Outcome =
+ try {
+ stdin.write(objectMapper.writeValueAsString(request))
+ stdin.write("\n")
+ stdin.flush()
+ val line = stdout.readLine()
Review Comment:
Fixed. Dedicated threads now own the blocking read and the blocking write, so a caller waits with a deadline and a worker that misses it is killed and discarded, falling back to the one-shot subprocess as a crash already did. Response keeps the 30 seconds the old path allowed, startup gets its own budget, and `PythonWorkerPoolSpec` covers both against a fixture worker that goes quiet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5246608518
All four are closed in `dd9fd725e`, and I retitled the PR to your wording.
The 96% is the imports' share of the spawn, not a share of the job. The sentence now says so. `python` is a `lazy val` now; there was only one call site.
Pool spec passes 8/8. scalafmt and scalafix are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3754182899
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
Review Comment:
Reopening: naming the referent was the letter of the ask, but the sentence still isn't self-checking. Against the same paragraph's ~25 ms bare boot, a 260-310 ms spawn leaves the imports at 88-93% of it — never ~96% — so a reader who divides the two figures here still lands somewhere else.
If the 96% comes from #6975's decomposition of something slightly different, the cheapest fix is to stop asserting a percentage these two numbers contradict:
```suggestion
* are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build: let WorkflowOperator route Python-executing tests to amber-integration [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5149333966
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@carloea2`
You can notify them by mentioning `@aglinxinyuan`, `@carloea2` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5197952105
@aglinxinyuan May you take a look at this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3715709136
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{ConcurrentHashMap, LinkedBlockingQueue}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). Only a hard interpreter crash ends a
+ * worker; the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty
+ ): Outcome = {
+ val envKey = env.toSeq.sorted.map { case (k, v) => s"$k=$v" }
+ val key =
+ (resourcePath +: pythonExe +: (interpreterArgs ++ launchArgs ++ envKey)).mkString(" ")
+ val pool = pools.computeIfAbsent(
+ key,
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request)
+ }
+
+ private val pools = new ConcurrentHashMap[String, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode): Outcome = {
+ val w = borrow()
+ try {
+ val outcome = w.run(request)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ case e: WorkerDiedException =>
+ discard(w)
+ throw e
+ }
+ }
+
+ private def borrow(): Worker = {
+ val existing = idle.poll()
+ if (existing != null) return existing
+ if (liveCount.getAndIncrement() < maxWorkers) {
+ try create()
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ idle.take() // at the cap — block until a worker is returned
+ }
+ }
+
+ private def create(): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ val w = new Worker(pb.start(), s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ w.awaitReady()
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin/stdout and a background
+ // drain of its own stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val stdout: BufferedReader =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Block until the worker's startup `{"ready": true}` arrives; if it dies
+ * first (e.g. an import failed), surface its stderr.
+ */
+ def awaitReady(): Unit = {
+ val line = stdout.readLine()
+ if (line == null || !objectMapper.readTree(line).path("ready").asBoolean(false)) {
+ throw new WorkerDiedException(
+ s"python worker [$label] did not signal ready. stderr:\n${drainErr()}"
+ )
+ }
+ }
+
+ def run(request: ObjectNode): Outcome =
+ try {
+ stdin.write(objectMapper.writeValueAsString(request))
+ stdin.write("\n")
+ stdin.flush()
+ val line = stdout.readLine()
Review Comment:
This read can block forever if the worker stays alive but stops responding. The suite timeout does not fix that because interrupting the executor cannot unblock a thread waiting on the process pipe. The old path killed the process after 30 seconds. Please add a real response timeout that kills and discards the worker. Startup needs the same protection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3770521649
##########
common/workflow-operator/src/test/resources/python/py_compile_worker.py:
##########
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Persistent worker that syntax-checks generated operator code, replacing one
+`python -I -S -B -m py_compile <file>` spawn per operator descriptor.
+
+`compile(source, path, "exec")` is what `py_compile` does before writing a
+`.pyc` and raises the same SyntaxError; skipping that write is why neither `-B`
+nor a temp file is needed here.
+
+Protocol (line-delimited JSON, both directions):
+
+ startup worker -> parent: {"ready": true}
+ request parent -> worker: {"source": "<code>", "name": "<label>"}\n
+ response worker -> parent: {"exit": 0, "stdout": "...", "stderr": "..."}\n
+
+`exit` is 1 when the source does not compile, with the SyntaxError in `stderr`,
+mirroring the spawn it replaces so the parent's reporting is unchanged. That
+does not end the worker; only a hard interpreter crash does.
+"""
+from __future__ import annotations
+
+import json
+import sys
+import traceback
+
+
+def _compile_one(source: str, name: str) -> "dict[str, object]":
+ """Compile one generated module. `name` is the filename the traceback shows,
+ so a report names the descriptor rather than a temp path.
+ """
+ try:
+ compile(source, name, "exec")
Review Comment:
More permissive than the `py_compile` spawn it replaces. The module's `from __future__ import annotations` is inherited here, because `compile` defaults to `dont_inherit=False`, while `py_compile` uses `dont_inherit=True`.
I ran both on `def f(x: (yield 1)): pass`: the spawn exits 1 with `SyntaxError: 'yield' outside function`, this call accepts it, and adding `dont_inherit=True` restores the rejection. A source the old check rejected is now waved through silently.
```suggestion
compile(source, name, "exec", dont_inherit=True)
```
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,19 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it, and amber-integration, which installs operator-requirements.txt,
Review Comment:
`operator-requirements.txt` carries plotly (`amber/operator-requirements.txt:19`), but pandas comes from `amber/requirements.txt:19` — `build.yml:419-420` installs both. Naming only one leaves the other looking trimmable from that job.
```suggestion
// excludes it, and amber-integration, which installs both requirements files,
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ /** Resolves [[python]] here so that the cancel lands on the test thread. Forced
+ * inside a case, it would be forced inside a `Future`, and by the time it
+ * reached `intercept` a cancellation is a RuntimeException like any other: the
+ * cases would fail on a machine without an interpreter instead of cancelling,
+ * and the one at the cap would record it as the failure it asserts on and pass
+ * without ever reaching the cap.
+ */
+ override def withFixture(test: NoArgTest): org.scalatest.Outcome = {
+ val _ = python
+ super.withFixture(test)
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def call(
+ launchArgs: Seq[String],
+ request: ObjectNode,
+ timeouts: PythonWorkerPool.Timeouts
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python,
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = timeouts
+ )
+
+ /** A job the fixture takes and never answers. `hang` travels in the request
+ * rather than in `launchArgs` so that [[healthyCall]] lands in the same
+ * sub-pool: `Key` covers the script, its arguments and the environment, and a
+ * job routed elsewhere would be served by a pool whose queue this suite never
+ * touched.
+ */
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: ObjectNode = objectMapper.createObjectNode(),
+ timeouts: PythonWorkerPool.Timeouts = ShortTimeouts
+ ): PythonWorkerPool.Outcome =
+ call(launchArgs, request.deepCopy().put("hang", true), timeouts)
+
+ /** A job in that same sub-pool which the fixture does answer. */
+ private def healthyCall(): PythonWorkerPool.Outcome =
+ call(Seq.empty, objectMapper.createObjectNode(), ShortTimeouts)
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto workers that go quiet:
+ // the callers holding one time out and their workers are discarded, which
+ // frees a slot without handing anything back, and the caller waiting at the
+ // cap has to notice that rather than wait for a hand-back that never comes.
+ // The discards it is waiting on happen on workers other than the one ahead
+ // of it.
+ val callers = PythonWorkerPool.maxWorkers + 1
+
+ val outcomes = onDaemonThreads(callers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(callers)(Future(Try(hangingCall(Seq.empty))))), Bound)
+ }
+
+ assert(outcomes.length == callers)
+ assert(outcomes.forall(_.isFailure))
+ }
+
+ test("the pool still serves jobs after it has discarded a timed-out worker") {
+ interceptBounded(hangingCall(Seq.empty))
+
+ // Deliberately the sub-pool the discard happened in — see [[hangingCall]] —
+ // so what is asserted is that a pool short one worker starts a replacement,
+ // not that an untouched pool works.
+ assert(healthyCall().exit == 0)
+ }
+
+ test("an interpreter that cannot be started reaches the caller as a worker death") {
+ // Not the IOException ProcessBuilder raises: a caller's fallback is written
+ // against WorkerDiedException, and a pool that cannot hand out a worker at all
+ // is the case that fallback exists for.
+ val thrown = intercept[PythonWorkerPool.WorkerDiedException] {
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = Seq.empty,
+ pythonExe = "no-such-python-on-this-machine",
+ request = objectMapper.createObjectNode(),
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = ShortTimeouts
+ )
+ }
+
+ assert(thrown.getMessage.contains("could not start python worker"))
+ }
+
+ test("a caller interrupted mid-job does not cost the pool a worker") {
+ val workers = PythonWorkerPool.maxWorkers
+
+ // Every slot taken by a job nobody will answer, and then the callers are
+ // interrupted — `shutdownNow` is what an executor does to a fan-out whose test
+ // has already failed. An interrupt is not a WorkerDiedException, so a worker
+ // left neither returned nor discarded would cost this sub-pool that slot for
+ // the rest of the JVM.
+ onDaemonThreads(workers) { implicit ec =>
+ Seq.fill(workers)(Future(Try(hangingCall(Seq.empty, timeouts = PatientTimeouts))))
+ // Enough for the callers to be waiting on a worker rather than starting one;
+ // the slot has to come back wherever the interrupt lands, so this only
+ // decides which of the two paths the case exercises.
+ Thread.sleep(500)
+ }
+
+ // Serving `workers` jobs again is the whole assertion: a leaked slot leaves
+ // these at the cap, rechecking it until [[Bound]] runs out.
+ val outcomes = onDaemonThreads(workers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(workers)(Future(healthyCall()))), Bound)
+ }
+
+ assert(outcomes.forall(_.exit == 0))
+ }
+
+ test("a worker that answers without an exit code is reported as the worker") {
+ // Not as a job that failed: a default would name the caller's own request as
+ // what went wrong, with an empty stdout as the evidence.
+ val request = objectMapper.createObjectNode().put("drop-exit", true)
+
+ val thrown = intercept[PythonWorkerPool.WorkerDiedException] {
+ call(Seq.empty, request, ShortTimeouts)
+ }
Review Comment:
The one wait in this suite with no `Bound`, and it shares the `Seq.empty` sub-pool with the cap and interrupt cases above.
`borrow`'s at-cap path (`PythonWorkerPool.scala:251-252`) rechecks on a `@tailrec` loop with no deadline. So a leaked slot — what the interrupt case exists to catch — leaves this case spinning on the test thread, wedging the build instead of failing it. `:209` gets away with a bare `intercept` because its unique `pythonExe` gives it a fresh sub-pool that never reaches the cap.
```suggestion
val thrown = interceptBounded(call(Seq.empty, request, ShortTimeouts))
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `responseMillis` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startupMillis` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] for every worker the pool could
+ * not give out or keep — one that would not start, would not report ready, or
+ * died mid-job — so one `catch` covers a caller's whole fallback.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot — though not
Review Comment:
No worker script in the tree accepts `--serve`; the real launch flags are `hanging_worker.py`'s `--deaf` / `--hang-before-ready` / `--babble`, so a reader greps for an option that does not exist. Same shape as the `PYTHONPATH` example corrected in the next clause.
```suggestion
* (e.g. `--deaf`), and `env` carries what a flag cannot — though not
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3770685252
##########
common/workflow-operator/src/test/resources/python/py_compile_worker.py:
##########
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Persistent worker that syntax-checks generated operator code, replacing one
+`python -I -S -B -m py_compile <file>` spawn per operator descriptor.
+
+`compile(source, path, "exec")` is what `py_compile` does before writing a
+`.pyc` and raises the same SyntaxError; skipping that write is why neither `-B`
+nor a temp file is needed here.
+
+Protocol (line-delimited JSON, both directions):
+
+ startup worker -> parent: {"ready": true}
+ request parent -> worker: {"source": "<code>", "name": "<label>"}\n
+ response worker -> parent: {"exit": 0, "stdout": "...", "stderr": "..."}\n
+
+`exit` is 1 when the source does not compile, with the SyntaxError in `stderr`,
+mirroring the spawn it replaces so the parent's reporting is unchanged. That
+does not end the worker; only a hard interpreter crash does.
+"""
+from __future__ import annotations
+
+import json
+import sys
+import traceback
+
+
+def _compile_one(source: str, name: str) -> "dict[str, object]":
+ """Compile one generated module. `name` is the filename the traceback shows,
+ so a report names the descriptor rather than a temp path.
+ """
+ try:
+ compile(source, name, "exec")
Review Comment:
Fixed in 8dd2ce3f2. The flag is right, but I could not reproduce the direction: on 3.12 and 3.14 the worker rejects `def f(x: (yield 1)): pass` too, only the message differs. The divergence I did find runs the other way. `def f(x: (y := 1)): pass` compiles under the spawn on 3.12 and raised here, so the old check was stricter, not more permissive. That is the case the docstring now names.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ /** Resolves [[python]] here so that the cancel lands on the test thread. Forced
+ * inside a case, it would be forced inside a `Future`, and by the time it
+ * reached `intercept` a cancellation is a RuntimeException like any other: the
+ * cases would fail on a machine without an interpreter instead of cancelling,
+ * and the one at the cap would record it as the failure it asserts on and pass
+ * without ever reaching the cap.
+ */
+ override def withFixture(test: NoArgTest): org.scalatest.Outcome = {
+ val _ = python
+ super.withFixture(test)
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def call(
+ launchArgs: Seq[String],
+ request: ObjectNode,
+ timeouts: PythonWorkerPool.Timeouts
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python,
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = timeouts
+ )
+
+ /** A job the fixture takes and never answers. `hang` travels in the request
+ * rather than in `launchArgs` so that [[healthyCall]] lands in the same
+ * sub-pool: `Key` covers the script, its arguments and the environment, and a
+ * job routed elsewhere would be served by a pool whose queue this suite never
+ * touched.
+ */
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: ObjectNode = objectMapper.createObjectNode(),
+ timeouts: PythonWorkerPool.Timeouts = ShortTimeouts
+ ): PythonWorkerPool.Outcome =
+ call(launchArgs, request.deepCopy().put("hang", true), timeouts)
+
+ /** A job in that same sub-pool which the fixture does answer. */
+ private def healthyCall(): PythonWorkerPool.Outcome =
+ call(Seq.empty, objectMapper.createObjectNode(), ShortTimeouts)
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto workers that go quiet:
+ // the callers holding one time out and their workers are discarded, which
+ // frees a slot without handing anything back, and the caller waiting at the
+ // cap has to notice that rather than wait for a hand-back that never comes.
+ // The discards it is waiting on happen on workers other than the one ahead
+ // of it.
+ val callers = PythonWorkerPool.maxWorkers + 1
+
+ val outcomes = onDaemonThreads(callers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(callers)(Future(Try(hangingCall(Seq.empty))))), Bound)
+ }
+
+ assert(outcomes.length == callers)
+ assert(outcomes.forall(_.isFailure))
+ }
+
+ test("the pool still serves jobs after it has discarded a timed-out worker") {
+ interceptBounded(hangingCall(Seq.empty))
+
+ // Deliberately the sub-pool the discard happened in — see [[hangingCall]] —
+ // so what is asserted is that a pool short one worker starts a replacement,
+ // not that an untouched pool works.
+ assert(healthyCall().exit == 0)
+ }
+
+ test("an interpreter that cannot be started reaches the caller as a worker death") {
+ // Not the IOException ProcessBuilder raises: a caller's fallback is written
+ // against WorkerDiedException, and a pool that cannot hand out a worker at all
+ // is the case that fallback exists for.
+ val thrown = intercept[PythonWorkerPool.WorkerDiedException] {
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = Seq.empty,
+ pythonExe = "no-such-python-on-this-machine",
+ request = objectMapper.createObjectNode(),
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = ShortTimeouts
+ )
+ }
+
+ assert(thrown.getMessage.contains("could not start python worker"))
+ }
+
+ test("a caller interrupted mid-job does not cost the pool a worker") {
+ val workers = PythonWorkerPool.maxWorkers
+
+ // Every slot taken by a job nobody will answer, and then the callers are
+ // interrupted — `shutdownNow` is what an executor does to a fan-out whose test
+ // has already failed. An interrupt is not a WorkerDiedException, so a worker
+ // left neither returned nor discarded would cost this sub-pool that slot for
+ // the rest of the JVM.
+ onDaemonThreads(workers) { implicit ec =>
+ Seq.fill(workers)(Future(Try(hangingCall(Seq.empty, timeouts = PatientTimeouts))))
+ // Enough for the callers to be waiting on a worker rather than starting one;
+ // the slot has to come back wherever the interrupt lands, so this only
+ // decides which of the two paths the case exercises.
+ Thread.sleep(500)
+ }
+
+ // Serving `workers` jobs again is the whole assertion: a leaked slot leaves
+ // these at the cap, rechecking it until [[Bound]] runs out.
+ val outcomes = onDaemonThreads(workers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(workers)(Future(healthyCall()))), Bound)
+ }
+
+ assert(outcomes.forall(_.exit == 0))
+ }
+
+ test("a worker that answers without an exit code is reported as the worker") {
+ // Not as a job that failed: a default would name the caller's own request as
+ // what went wrong, with an empty stdout as the evidence.
+ val request = objectMapper.createObjectNode().put("drop-exit", true)
+
+ val thrown = intercept[PythonWorkerPool.WorkerDiedException] {
+ call(Seq.empty, request, ShortTimeouts)
+ }
Review Comment:
Fixed in 3e69a7867, taken as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3732477055
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -49,8 +56,70 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
private val MaxDepth: Int = 3
private val AcceptPackages: Seq[String] = Seq("org.apache.texera.amber.operator")
+ /** Budget for one whole fanned-out pass over every descriptor. Deliberately far
+ * above a real run (well under a second), so it only ever fires on a hang.
+ */
+ private val PassTimeout: FiniteDuration = 10.minutes
+
+ /** Runs the given work concurrently and returns the results in submission
+ * order, rethrowing the first failure so it fails the test. Sized to the pool
+ * so the threads match the workers available to serve them.
+ */
+ private def awaitAll[T](work: Seq[() => T]): Seq[T] = {
+ val threads = Executors.newFixedThreadPool(PythonWorkerPool.maxWorkers)
+ try {
+ implicit val ec: ExecutionContext = ExecutionContext.fromExecutorService(threads)
+ Await.result(Future.sequence(work.map(w => Future(w()))), PassTimeout)
+ } finally threads.shutdownNow()
+ }
Review Comment:
awaitAll uses a non-daemon fixed thread pool; if any task gets stuck past PassTimeout (e.g., blocked on subprocess I/O), shutdownNow() may not terminate the threads and the JVM can stay alive after the test has failed/timed out. Use daemon threads (and optionally awaitTermination) so a wedged task can’t hang the build process.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -249,18 +320,45 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
val findings = checkResult.findings ++ pyCompileFindings
if (findings.isEmpty && checkResult.code.nonEmpty) {
- ok += 1
- println(s"[py-compile OK $ok/$total | checked $checked/$total] ${descriptorClass.getName}")
+ println(
+ s"[py-compile OK ${ok.incrementAndGet()}/$total | " +
+ s"checked ${checked.get()}/$total] ${descriptorClass.getName}"
+ )
}
findings
- }
+ }).flatten
- println(s"[py-compile SUMMARY] ok=$ok/$total")
+ println(s"[py-compile SUMMARY] ok=${ok.get()}/$total")
if (allFindings.nonEmpty) {
fail(PythonReflectionUtils.renderReport(allFindings, total = total))
}
}
+ /** py_compile above only parses the emitted code; running it needs the packages
+ * it imports. Tagged, so only amber-integration — the job that installs them —
+ * runs this. There a missing package is a defect; elsewhere it is a local-setup
+ * fact, so cancel rather than fail.
+ */
+ test(
+ "the Python interpreter operator templates run in should import pandas and plotly",
+ Tag(classOf[IntegrationTest].getName)
+ ) {
+ val provisioned = sys.env.get("AMBER_TEST_FILTER").contains("integration-only")
+ def unavailable(message: String): Nothing =
+ if (provisioned) fail(message) else cancel(message)
+
+ val python = loadPythonExeFromUdfConf().getOrElse(unavailable("no runnable python"))
+ val imported = Try {
+ val process = new ProcessBuilder(python, "-c", "import pandas, plotly")
+ .redirectErrorStream(true)
+ .start()
+ process.waitFor(60, TimeUnit.SECONDS) && process.exitValue() == 0
+ }
Review Comment:
If the import probe times out, the spawned Python process is left running (waitFor returns false) and can leak into the rest of the test run. Destroy the process on timeout before returning false.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,435 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool. `timeouts` is not — it bounds
+ * this call, so a caller whose jobs are slower than most can raise it without
+ * splitting the pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty,
+ timeouts: Timeouts = Timeouts.Default
+ ): Outcome = {
+ val envKey = env.toSeq.sorted.map { case (k, v) => s"$k=$v" }
+ val key =
+ (resourcePath +: pythonExe +: (interpreterArgs ++ launchArgs ++ envKey)).mkString(" ")
Review Comment:
Pool identity keys are built by joining parts with a single space. If any part contains spaces (pythonExe path, env values like PYTHONPATH, etc.), distinct configurations can collide and incorrectly share a worker pool. Use an unambiguous delimiter (or a structured key) to avoid collisions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5223843822
@Yicong-Huang, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3764522114
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] for every worker the pool could
+ * not give out or keep — one that would not start, would not report ready, or
+ * died mid-job — so one `catch` covers a caller's whole fallback.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
Review Comment:
Verified: `PythonWorkerPool.scala:152-153` now says `env` carries what a flag cannot "though not PYTHONPATH or any other PYTHON* var, which the `-I` above makes CPython ignore", so the example no longer points a caller at a variable `-I` discards.
##########
project/TestFilters.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import sbt._
+
+/**
+ * Selects a module's tagged tests for the fast-unit job or the integration job:
+ * skip-integration excludes them, integration-only runs only them, unset runs
+ * everything. Shared because the mapping is identical in every module, while the
+ * env var and the tag are not — the tag annotation has to live somewhere the
+ * module's own Test config can see.
+ */
+object TestFilters {
+
+ /** @param envVar the variable the two CI jobs set to opposite values; it has to
+ * be the one the workflow already sets on the step that invokes
+ * this module's tests, or neither subset is selected.
+ * @param tag fully-qualified name of the tag annotation, as
+ * `classOf[...].getName` gives it at the test site — ScalaTest
+ * matches these by string, so a rename that misses one side
+ * silently stops filtering.
+ */
+ def integrationSplit(envVar: String, tag: String): Seq[TestOption] =
+ sys.env.get(envVar) match {
+ case Some("skip-integration") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-l", tag))
+ case Some("integration-only") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-n", tag))
+ case _ => Nil
Review Comment:
Verified: `TestFilters.scala:45` now fails the build on an unrecognized value (`case Some(other) => sys.error(...)`) with `case None => Nil` split out, so a typo in either job's env var can no longer run the full suite silently. Both jobs still set only the two recognized values (`build.yml:295`, `:649`), so nothing breaks.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
Review Comment:
Verified: `PythonWorkerPool.scala:119-120` now names `responseMillis` and `startupMillis`, matching the case class below.
##########
common/workflow-operator/src/test/java/org/apache/texera/amber/operator/tags/IntegrationTest.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.tags;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+import org.scalatest.TagAnnotation;
+
+/**
+ * Marks a test in this module as needing more than a bare Python interpreter —
+ * pandas or plotly, which the {@code amber} job does not install. See the
+ * AMBER_TEST_FILTER block in {@code common/workflow-operator/build.sbt} for how
+ * it routes to {@code amber-integration}.
+ *
+ * <p>Apply it to a whole spec as an annotation, or to a single case — {@code
+ * test(name, Tag(classOf[IntegrationTest].getName))} in a FunSuite,
+ * {@code taggedAs} in a FlatSpec — so that a spec's cheaper assertions stay in
+ * the unit job and its coverage report.
Review Comment:
Verified: `IntegrationTest.java:35-38` now attaches the coverage rationale to the per-case form only.
##########
.github/workflows/build.yml:
##########
@@ -284,9 +284,13 @@ jobs:
# Test config (sbt's `test` task does not transit dependsOn),
# so common modules' tests are listed explicitly here.
#
- # AMBER_TEST_FILTER=skip-integration tells amber/build.sbt to
- # exclude @org.apache.texera.amber.tags.IntegrationTest specs;
- # those run in the amber-integration job below.
+ # AMBER_TEST_FILTER=skip-integration excludes the integration-tagged
+ # specs of two modules, each under its own tag: amber/build.sbt drops
Review Comment:
Verified: `build.yml:288` now reads "excludes the integration-tagged tests of two modules", so the per-case granularity is no longer hidden behind "specs".
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def call(
+ launchArgs: Seq[String],
+ request: ObjectNode,
+ timeouts: PythonWorkerPool.Timeouts
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python,
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = timeouts
+ )
+
+ /** A job the fixture takes and never answers. `hang` travels in the request
+ * rather than in `launchArgs` so that [[healthyCall]] lands in the same
+ * sub-pool: `Key` covers the script, its arguments and the environment, and a
+ * job routed elsewhere would be served by a pool whose queue this suite never
+ * touched.
+ */
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: ObjectNode = objectMapper.createObjectNode(),
+ timeouts: PythonWorkerPool.Timeouts = ShortTimeouts
+ ): PythonWorkerPool.Outcome =
+ call(launchArgs, request.deepCopy().put("hang", true), timeouts)
+
+ /** A job in that same sub-pool which the fixture does answer. */
+ private def healthyCall(): PythonWorkerPool.Outcome =
+ call(Seq.empty, objectMapper.createObjectNode(), ShortTimeouts)
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto workers that go quiet:
+ // those holding one time out and are discarded, which frees a slot without
+ // handing anything back, and the caller waiting at the cap has to notice that
Review Comment:
Verified: `PythonWorkerPoolSpec.scala:168-169` now reads "the callers holding one time out and their workers are discarded", so the workers are what gets discarded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3771144730
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ /** Resolves [[python]] here so that the cancel lands on the test thread. Forced
+ * inside a case, it would be forced inside a `Future`, and by the time it
+ * reached `intercept` a cancellation is a RuntimeException like any other: the
+ * cases would fail on a machine without an interpreter instead of cancelling,
+ * and the one at the cap would record it as the failure it asserts on and pass
+ * without ever reaching the cap.
+ */
+ override def withFixture(test: NoArgTest): org.scalatest.Outcome = {
+ val _ = python
+ super.withFixture(test)
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def call(
+ launchArgs: Seq[String],
+ request: ObjectNode,
+ timeouts: PythonWorkerPool.Timeouts
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python,
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = timeouts
+ )
+
+ /** A job the fixture takes and never answers. `hang` travels in the request
+ * rather than in `launchArgs` so that [[healthyCall]] lands in the same
+ * sub-pool: `Key` covers the script, its arguments and the environment, and a
+ * job routed elsewhere would be served by a pool whose queue this suite never
+ * touched.
+ */
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: ObjectNode = objectMapper.createObjectNode(),
+ timeouts: PythonWorkerPool.Timeouts = ShortTimeouts
+ ): PythonWorkerPool.Outcome =
+ call(launchArgs, request.deepCopy().put("hang", true), timeouts)
+
+ /** A job in that same sub-pool which the fixture does answer. */
+ private def healthyCall(): PythonWorkerPool.Outcome =
+ call(Seq.empty, objectMapper.createObjectNode(), ShortTimeouts)
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto workers that go quiet:
+ // the callers holding one time out and their workers are discarded, which
+ // frees a slot without handing anything back, and the caller waiting at the
+ // cap has to notice that rather than wait for a hand-back that never comes.
+ // The discards it is waiting on happen on workers other than the one ahead
+ // of it.
+ val callers = PythonWorkerPool.maxWorkers + 1
+
+ val outcomes = onDaemonThreads(callers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(callers)(Future(Try(hangingCall(Seq.empty))))), Bound)
+ }
+
+ assert(outcomes.length == callers)
+ assert(outcomes.forall(_.isFailure))
+ }
+
+ test("the pool still serves jobs after it has discarded a timed-out worker") {
+ interceptBounded(hangingCall(Seq.empty))
+
+ // Deliberately the sub-pool the discard happened in — see [[hangingCall]] —
+ // so what is asserted is that a pool short one worker starts a replacement,
+ // not that an untouched pool works.
+ assert(healthyCall().exit == 0)
+ }
+
+ test("an interpreter that cannot be started reaches the caller as a worker death") {
+ // Not the IOException ProcessBuilder raises: a caller's fallback is written
+ // against WorkerDiedException, and a pool that cannot hand out a worker at all
+ // is the case that fallback exists for.
+ val thrown = intercept[PythonWorkerPool.WorkerDiedException] {
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = Seq.empty,
+ pythonExe = "no-such-python-on-this-machine",
+ request = objectMapper.createObjectNode(),
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = ShortTimeouts
+ )
+ }
+
+ assert(thrown.getMessage.contains("could not start python worker"))
+ }
+
+ test("a caller interrupted mid-job does not cost the pool a worker") {
+ val workers = PythonWorkerPool.maxWorkers
+
+ // Every slot taken by a job nobody will answer, and then the callers are
+ // interrupted — `shutdownNow` is what an executor does to a fan-out whose test
+ // has already failed. An interrupt is not a WorkerDiedException, so a worker
+ // left neither returned nor discarded would cost this sub-pool that slot for
+ // the rest of the JVM.
+ onDaemonThreads(workers) { implicit ec =>
+ Seq.fill(workers)(Future(Try(hangingCall(Seq.empty, timeouts = PatientTimeouts))))
+ // Enough for the callers to be waiting on a worker rather than starting one;
+ // the slot has to come back wherever the interrupt lands, so this only
+ // decides which of the two paths the case exercises.
+ Thread.sleep(500)
+ }
+
+ // Serving `workers` jobs again is the whole assertion: a leaked slot leaves
+ // these at the cap, rechecking it until [[Bound]] runs out.
+ val outcomes = onDaemonThreads(workers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(workers)(Future(healthyCall()))), Bound)
+ }
+
+ assert(outcomes.forall(_.exit == 0))
+ }
+
+ test("a worker that answers without an exit code is reported as the worker") {
Review Comment:
The name stops one word short of the contrast it means — reported as the worker *dying*, not as the job failing. `WorkerDiedException` and the class doc ("leaves as a worker death") already name that outcome, and ScalaTest prints this string in the report.
```suggestion
test("a worker that answers without an exit code is reported as a worker death") {
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ /** Resolves [[python]] here so that the cancel lands on the test thread. Forced
+ * inside a case, it would be forced inside a `Future`, and by the time it
+ * reached `intercept` a cancellation is a RuntimeException like any other: the
+ * cases would fail on a machine without an interpreter instead of cancelling,
+ * and the one at the cap would record it as the failure it asserts on and pass
+ * without ever reaching the cap.
+ */
+ override def withFixture(test: NoArgTest): org.scalatest.Outcome = {
+ val _ = python
+ super.withFixture(test)
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def call(
+ launchArgs: Seq[String],
+ request: ObjectNode,
+ timeouts: PythonWorkerPool.Timeouts
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python,
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = timeouts
+ )
+
+ /** A job the fixture takes and never answers. `hang` travels in the request
+ * rather than in `launchArgs` so that [[healthyCall]] lands in the same
+ * sub-pool: `Key` covers the script, its arguments and the environment, and a
+ * job routed elsewhere would be served by a pool whose queue this suite never
+ * touched.
+ */
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: ObjectNode = objectMapper.createObjectNode(),
+ timeouts: PythonWorkerPool.Timeouts = ShortTimeouts
+ ): PythonWorkerPool.Outcome =
+ call(launchArgs, request.deepCopy().put("hang", true), timeouts)
+
+ /** A job in that same sub-pool which the fixture does answer. */
+ private def healthyCall(): PythonWorkerPool.Outcome =
+ call(Seq.empty, objectMapper.createObjectNode(), ShortTimeouts)
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto workers that go quiet:
+ // the callers holding one time out and their workers are discarded, which
Review Comment:
`the callers holding one time out` garden-paths: "holding one time out" reads as a noun phrase until "time out" turns out to be the verb. A relative pronoun and a comma settle it, and this comment is the only explanation of why the case exists.
```suggestion
// the callers that hold one time out, and their workers are discarded, which
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5247372489
All five addressed in e56e3bc9b.
The two build-side ones: the spec's `AMBER_TEST_FILTER` read now points at the build's, and vice versa, since nothing can link them at compile time. The workflow note now names both modules, both tag names, and why the second exclusion is what keeps the probe out of that job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3753548921
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
Review Comment:
Is the ~96% measuring something other than these two figures? A 260-310 ms spawn around ~4 ms of work makes the spawn ~99% of the job. I assume the 96% is #6975's decomposition and a share of something slightly different; naming what it is a share of would keep the sentence self-checking. As written, a reader who divides the two numbers here gets a different answer.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
Review Comment:
This doc did not move with the contract. `run`'s Scaladoc at `:144` now promises this exception for a worker that would not start or never signalled ready too, and `create` (`:266`) and `awaitReady` (`:393`) deliver on it. A caller reading only the exception's own doc would still write the narrower fallback the widening was meant to retire.
```suggestion
/** Thrown for a worker the pool could not give out or keep: one that would not
* start, one that never signalled ready, or one that died or fell silent
* mid-job. Callers catch this and fall back to a one-shot subprocess.
*/
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -49,8 +57,99 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
private val MaxDepth: Int = 3
private val AcceptPackages: Seq[String] = Seq("org.apache.texera.amber.operator")
+ /** Budget for one whole fanned-out pass over every descriptor. Deliberately far
+ * above a real run (well under a second), so it only ever fires on a hang.
+ */
+ private val PassTimeout: FiniteDuration = 10.minutes
+
+ /** Runs the given work concurrently and returns the results in submission
+ * order, rethrowing the first failure so it fails the test. Sized to the pool
+ * so the threads match the workers available to serve them.
+ *
+ * Daemon threads: a task parked on a subprocess pipe answers no interrupt, so
+ * `shutdownNow` need not end it, and a non-daemon one left there would hold the
+ * JVM — and the build — open after [[PassTimeout]] has already failed the test.
+ */
+ private def awaitAll[T](work: Seq[() => T]): Seq[T] = {
+ val threads = Executors.newFixedThreadPool(
+ PythonWorkerPool.maxWorkers,
+ (r: Runnable) => {
+ val t = new Thread(r, "py-compile-check")
+ t.setDaemon(true)
+ t
+ }
+ )
+ try {
+ implicit val ec: ExecutionContext = ExecutionContext.fromExecutorService(threads)
+ Await.result(Future.sequence(work.map(w => Future(w()))), PassTimeout)
+ } finally threads.shutdownNow()
+ }
+
+ /** Checks the pooled path could not serve. Not a failure — the spawn each one
+ * fell back to is the pre-pool behavior — but a number the summary has to
+ * carry, since a check passing says nothing about which path answered it.
+ */
+ private val spawnFallbacks = new AtomicInteger(0)
+
+ /** Syntax-checks one generated module, through a pooled worker when available.
+ *
+ * The worker is launched with the same `-I -S` isolation the one-shot path
+ * uses, so what the check accepts is unchanged; it just stops paying an
+ * interpreter boot — the whole cost of a check whose real work is under a
+ * millisecond — once per descriptor. A worker the pool cannot give out, or
+ * loses mid-job, falls back to the spawn, so behavior is never worse than
+ * before the pool.
+ */
+ private def syntaxCheck(
+ pythonExecutable: String,
+ pythonSource: String,
+ descriptorName: String
+ ): Either[String, Unit] = {
+ def viaPool: Either[String, Unit] = {
+ val request = objectMapper.createObjectNode()
+ request.put("source", pythonSource)
+ request.put("name", s"$descriptorName.py")
+ val outcome = PythonWorkerPool.run(
+ resourcePath = "/python/py_compile_worker.py",
+ launchArgs = Seq.empty,
+ pythonExe = pythonExecutable,
+ request = request,
+ interpreterArgs = Seq("-I", "-S")
+ )
+ if (outcome.exit == 0) Right(())
+ else {
+ val output = if (outcome.stderr.trim.nonEmpty) outcome.stderr.trim else "(no output)"
+ Left(
+ s"py_compile failed (exit=${outcome.exit})\nOutput:\n" +
+ truncateBlock(output, maxLines = 40, maxChars = 8000)
+ )
+ }
+ }
+
+ if (PythonWorkerPool.enabled) {
+ try viaPool
+ catch {
+ // Anything the pooled path throws leaves the spawn as the answer, which is
+ // what makes it never worse than before: not only a worker that died
+ // mid-job, but equally an interpreter that could not be started at all —
+ // `ProcessBuilder.start` throws a bare IOException, which the pool passes
+ // through, and 4-way concurrency is a live way to reach it. Counted, so a
+ // run the pool served none of does not read as a green pooled run.
Review Comment:
The rationale no longer matches the code. As of this round the pool does not pass that `IOException` through: `PythonWorkerPool.scala:266-270` converts it, and the new case at `PythonWorkerPoolSpec.scala:192` asserts so.
The wider catch is still right, for a different reason. `ensureScript()` (`PythonWorkerPool.scala:255`) runs before `create`'s wrap, so materializing the worker script is the one step that can still escape as something else.
```suggestion
// mid-job, but equally one the pool could not hand out at all. Those
// arrive as WorkerDiedException; NonFatal also covers the steps outside
// that contract, such as materializing the worker script. Counted, so a
// run the pool served none of does not read as a green pooled run.
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private def python(): String = {
Review Comment:
Being a `def`, this re-probes the interpreter on every call. `call` (`:100`) invokes it per pool call, so the suite spawns `python3 --version` about 19 times for an answer that cannot change mid-run — and in the cap case all five callers pay it concurrently before reaching the pool.
A `lazy val` fixes it, at the cost of dropping the parens at the four call sites. Minor, but this is the cost the PR exists to remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build: let WorkflowOperator route Python-executing tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5171003355
@Yicong-Huang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3769803777
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
Review Comment:
Fixed in `39e74b76` with the `withFixture` you suggested. Reproduced first by pointing the probe at names that do not exist: 5 failed and 2 succeeded before, 8 cancel after.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `responseMillis` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startupMillis` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] for every worker the pool could
+ * not give out or keep — one that would not start, would not report ready, or
+ * died mid-job — so one `catch` covers a caller's whole fallback.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot — though not
+ * PYTHONPATH or any other PYTHON* var, which the `-I` above makes CPython ignore.
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool. `timeouts` is not — it bounds
+ * this call, so a caller whose jobs are slower than most can raise it without
+ * splitting the pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty,
+ timeouts: Timeouts = Timeouts.Default
+ ): Outcome = {
+ val pool = pools.computeIfAbsent(
+ Key(resourcePath, pythonExe, interpreterArgs.toList, launchArgs.toList, env.toList.sorted),
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request, timeouts)
+ }
+
+ /** What makes two launches the same worker. Compared field by field rather than
+ * as one joined string, so a value carrying whatever the separator was — a
+ * python path with a space in it, a PYTHONPATH — cannot make two different
+ * launches share a pool.
+ */
+ private final case class Key(
+ resourcePath: String,
+ pythonExe: String,
+ interpreterArgs: List[String],
+ launchArgs: List[String],
+ env: List[(String, String)]
+ )
+
+ /** How long a caller at the worker cap waits before re-examining it. Not a
+ * deadline — see [[Pool.borrow]].
+ */
+ private val CapRecheckMillis: Long = 250
+
+ private val pools = new ConcurrentHashMap[Key, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode, timeouts: Timeouts): Outcome = {
+ val w = borrow(timeouts)
+ try {
+ val outcome = w.run(request, timeouts.responseMillis)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ // Every throw, not only a WorkerDiedException: an interrupt on this
+ // thread — what an executor's `shutdownNow` sends — leaves the blocking
+ // read or write as an InterruptedException, which `NonFatal` excludes and
+ // [[Worker.run]] therefore does not wrap. A worker left neither returned
+ // nor discarded costs this sub-pool a slot for the life of the JVM, and
+ // its answer may still be in flight, so it cannot be reused either way.
+ case e: Throwable =>
+ discard(w)
+ throw e
+ }
+ }
+
+ @tailrec
+ private def borrow(timeouts: Timeouts): Worker = {
+ val existing = idle.poll()
+ if (existing != null) existing
+ else if (liveCount.getAndIncrement() < maxWorkers) {
+ try create(timeouts)
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ // At the cap. Waiting outright for a returned worker would strand this
+ // caller when the ones ahead are discarded instead: a discard frees a
+ // slot without putting anything back. So wait only briefly, then look at
+ // the cap again — the next pass starts a replacement. A long queue still
+ // waits as long as it takes; that is the caller's own backlog, not a hang.
+ val returned = idle.poll(CapRecheckMillis, TimeUnit.MILLISECONDS)
+ if (returned != null) returned else borrow(timeouts)
+ }
+ }
+
+ private def create(timeouts: Timeouts): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ // A worker that cannot be started is a worker death like any other, so it
+ // leaves through the same exception: `start` throws a bare IOException — no
+ // interpreter at that path, or the OS out of processes under a fan-out — and
+ // a caller's fallback is written against [[WorkerDiedException]].
+ val process =
+ try pb.start()
+ catch {
+ case NonFatal(e) =>
+ throw new WorkerDiedException(
+ s"could not start python worker for $resourcePath: ${e.getMessage}",
+ e
+ )
+ }
+ val w = new Worker(process, s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ // Anything thrown before the worker joins `all` — a startup that timed out,
+ // an interrupt on this thread — leaves an interpreter nothing else would
+ // reap: not the caller, which never gets the handle, and not the shutdown
+ // hook, which walks `all`. `destroy` is idempotent, so a worker that already
+ // killed itself on the way out is no exception to that.
+ try w.awaitReady(timeouts.startupMillis)
+ catch {
+ case e: Throwable =>
+ w.destroy()
+ throw e
+ }
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin and background drains of
+ // its stdout (the protocol) and stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ // Protocol lines the worker has written, `None` marking end of stream. A
+ // dedicated thread owns the blocking read so a caller can wait with a
+ // timeout: `readLine` on a process pipe answers neither an interrupt nor a
+ // deadline, and only closing the pipe — killing the process — releases it.
+ private val lines = new LinkedBlockingQueue[Option[String]]()
+
+ // Owns the writing end for the same reason.
+ private val writer: ExecutorService = Executors.newSingleThreadExecutor { r =>
+ val t = new Thread(r, "python-worker-stdin")
+ t.setDaemon(true)
+ t
+ }
+
+ private val outThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ lines.put(Some(line))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ finally lines.put(None)
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stdout")
+ t.start()
+ t
+ }
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Wait for the worker's startup `{"ready": true}`; if it dies first (e.g. an
+ * import failed) or never gets there, surface its stderr. A worker that
+ * fails here is killed: it is not in the pool's set yet, so nothing else
+ * will reap it.
+ *
+ * A first line that is not the protocol at all counts as not ready, rather
+ * than throwing whatever the parser throws: the caller's contract here is
+ * [[WorkerDiedException]], and an escape past it would leave this
+ * interpreter with nothing to reap it. The line goes into the message —
+ * stderr is empty when a script writes its noise to stdout.
+ */
+ def awaitReady(timeoutMillis: Long): Unit = {
+ val line = nextLine(timeoutMillis, "signal ready")
+ val ready =
+ try objectMapper.readTree(line).path("ready").asBoolean(false)
+ catch { case NonFatal(_) => false }
+ if (!ready) {
+ destroy()
+ throw new WorkerDiedException(
+ s"python worker [$label] did not signal ready; it wrote: ${abbreviate(line)}." +
+ s" stderr:\n${drainErr()}"
+ )
+ }
+ }
+
+ def run(request: ObjectNode, timeoutMillis: Long): Outcome =
+ try {
+ send(request, timeoutMillis)
+ val node = objectMapper.readTree(nextLine(timeoutMillis, "answer"))
+ Outcome(
+ node.path("exit").asInt(1),
Review Comment:
Yes, a `WorkerDiedException` is the right answer, fixed in `12046730`. The worker is discarded and the caller's spawn fallback covers it; the fixture and spec gain a case that answers without the field.
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,19 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it and amber-integration, which installs operator-requirements.txt,
Review Comment:
Applied in `ed51a857`.
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,19 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it and amber-integration, which installs operator-requirements.txt,
+// runs it. The amber job already sets this env var on the step that invokes
+// WorkflowOperator/jacoco, so no workflow change is needed for the exclusion.
+//
+// PythonCodeRawInvalidTextSpec reads the same env var and value directly, to tell
+// a missing package in amber-integration (a defect) from one on a developer's
+// machine (a local-setup fact). Renaming either without the other leaves that test
+// cancelling in the job meant to fail it.
Review Comment:
Applied in `ed51a857`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5259257305
@Yicong-Huang May you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build: let WorkflowOperator route Python-executing tests to amber-integration [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5149344042
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/7207?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.58%. Comparing base ([`348fa88`](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e82a346`](https://app.codecov.io/gh/apache/texera/commit/e82a3469f5dd7b5459009cd0ccf087c6e2707e45?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 6 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #7207 +/- ##
=========================================
Coverage 80.58% 80.58%
Complexity 3890 3890
=========================================
Files 1162 1162
Lines 46264 46264
Branches 5153 5153
=========================================
Hits 37280 37280
Misses 7319 7319
Partials 1665 1665
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.42% <ø> (ø)` | | |
| [amber](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.59% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `20.49% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.97% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `66.80% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.22% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.89% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `97.36% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/7207/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.31% <ø> (ø)` | | Carriedforward from [348fa88](https://app.codecov.io/gh/apache/texera/commit/348fa884442990307b842fd7c0dd6ca005dd320c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/7207?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3741403525
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -49,8 +56,81 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
private val MaxDepth: Int = 3
private val AcceptPackages: Seq[String] = Seq("org.apache.texera.amber.operator")
+ /** Budget for one whole fanned-out pass over every descriptor. Deliberately far
+ * above a real run (well under a second), so it only ever fires on a hang.
+ */
+ private val PassTimeout: FiniteDuration = 10.minutes
+
+ /** Runs the given work concurrently and returns the results in submission
+ * order, rethrowing the first failure so it fails the test. Sized to the pool
+ * so the threads match the workers available to serve them.
+ *
+ * Daemon threads: a task parked on a subprocess pipe answers no interrupt, so
+ * `shutdownNow` need not end it, and a non-daemon one left there would hold the
+ * JVM — and the build — open after [[PassTimeout]] has already failed the test.
+ */
+ private def awaitAll[T](work: Seq[() => T]): Seq[T] = {
+ val threads = Executors.newFixedThreadPool(
+ PythonWorkerPool.maxWorkers,
+ (r: Runnable) => {
+ val t = new Thread(r, "py-compile-check")
+ t.setDaemon(true)
+ t
+ }
+ )
+ try {
+ implicit val ec: ExecutionContext = ExecutionContext.fromExecutorService(threads)
+ Await.result(Future.sequence(work.map(w => Future(w()))), PassTimeout)
+ } finally threads.shutdownNow()
+ }
+
+ /** Syntax-checks one generated module, through a pooled worker when available.
+ *
+ * The worker is launched with the same `-I -S` isolation the one-shot path
+ * uses, so what the check accepts is unchanged; it just stops paying an
+ * interpreter boot — the whole cost of a check whose real work is under a
+ * millisecond — once per descriptor. A hard worker crash falls back to the
+ * spawn, so behavior is never worse than before the pool.
+ */
+ private def syntaxCheck(
+ pythonExecutable: String,
+ pythonSource: String,
+ descriptorName: String
+ ): Either[String, Unit] = {
+ def viaPool: Either[String, Unit] = {
+ val request = objectMapper.createObjectNode()
+ request.put("source", pythonSource)
+ request.put("name", s"$descriptorName.py")
+ val outcome = PythonWorkerPool.run(
+ resourcePath = "/python/py_compile_worker.py",
+ launchArgs = Seq.empty,
+ pythonExe = pythonExecutable,
+ request = request,
+ interpreterArgs = Seq("-I", "-S")
+ )
+ if (outcome.exit == 0) Right(())
+ else {
+ val output = if (outcome.stderr.trim.nonEmpty) outcome.stderr.trim else "(no output)"
+ Left(
+ s"py_compile failed (exit=${outcome.exit})\nOutput:\n" +
+ truncateBlock(output, maxLines = 40, maxChars = 8000)
+ )
+ }
+ }
+
+ if (PythonWorkerPool.enabled) {
+ try viaPool
+ catch {
+ case _: PythonWorkerPool.WorkerDiedException =>
+ pyCompile(pythonExecutable, pythonSource)
+ }
Review Comment:
Pool startup can fail without a `WorkerDiedException`: `pb.start()` throws a bare `IOException` (`PythonWorkerPool.scala:250`), and `borrow` rethrows it verbatim. That escapes this fallback and aborts the suite. The pre-pool code wrapped the same call in `Try` and returned a named `Left` (:160-163), so the "never worse than before" claim at :93 does not hold here. `error=11` under this PR's 4-way concurrency is a live way to reach it.
Also: the fallback is silent, so a pool whose workers all die still reports green.
```suggestion
catch {
case _: PythonWorkerPool.WorkerDiedException | _: java.io.IOException =>
pyCompile(pythonExecutable, pythonSource)
}
```
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,18 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it and amber-integration, which installs operator-requirements.txt,
+// runs it. The amber job already sets this env var on the step that invokes
+// WorkflowOperator/jacoco, so no workflow change is needed for the exclusion.
+// -P4 bounds ScalaTest's ParallelTestExecution pool on the integration side, whose
+// tagged tests drive Python subprocesses. It matches PythonWorkerPool's default
+// worker cap so the two bounds agree rather than multiply.
+Test / testOptions ++= TestFilters.integrationSplit(
+ envVar = "AMBER_TEST_FILTER",
+ tag = "org.apache.texera.amber.operator.tags.IntegrationTest",
+ integrationOnlyExtra = Seq("-P4")
Review Comment:
`-P4` is inert here. No suite in the module mixes in `ParallelTestExecution` — grep finds the string only in the comment above — and the single tagged test is the pandas/plotly probe, which spawns its own subprocess and never touches the pool.
The rationale is also wrong on its own terms. `maxWorkers` caps each *sub-pool*, keyed by script/args/env, so parallel suites on distinct keys multiply rather than agree. It is env-overridable too, while `-P4` is a literal. I would drop the parameter and add it back with the first pooled tagged test.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool. `timeouts` is not — it bounds
+ * this call, so a caller whose jobs are slower than most can raise it without
+ * splitting the pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty,
+ timeouts: Timeouts = Timeouts.Default
+ ): Outcome = {
+ val pool = pools.computeIfAbsent(
+ Key(resourcePath, pythonExe, interpreterArgs.toList, launchArgs.toList, env.toList.sorted),
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request, timeouts)
+ }
+
+ /** What makes two launches the same worker. Compared field by field rather than
+ * as one joined string, so a value carrying whatever the separator was — a
+ * python path with a space in it, a PYTHONPATH — cannot make two different
+ * launches share a pool.
+ */
+ private final case class Key(
+ resourcePath: String,
+ pythonExe: String,
+ interpreterArgs: List[String],
+ launchArgs: List[String],
+ env: List[(String, String)]
+ )
+
+ /** How long a caller at the worker cap waits before re-examining it. Not a
+ * deadline — see [[Pool.borrow]].
+ */
+ private val CapRecheckMillis: Long = 250
+
+ private val pools = new ConcurrentHashMap[Key, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode, timeouts: Timeouts): Outcome = {
+ val w = borrow(timeouts)
+ try {
+ val outcome = w.run(request, timeouts.responseMillis)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ case e: WorkerDiedException =>
+ discard(w)
+ throw e
+ }
Review Comment:
`Worker.run` converts `NonFatal` into `WorkerDiedException`. But `scala.util.control.NonFatal` excludes `InterruptedException`, and both blocking calls on that path throw exactly that (`lines.poll` :415, `write.get` :392).
Such a worker stays in `all`, never returns to `idle`, and `liveCount` is never decremented — the slot is gone for the life of the JVM. After `maxWorkers` losses `borrow` spins its 250 ms cap loop with no log until the 10-minute `PassTimeout`. The interrupt comes from `awaitAll`'s own `finally threads.shutdownNow()`.
```suggestion
} catch {
case e: Throwable =>
discard(w)
throw e
}
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+ private val CompileWorker = "/python/py_compile_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: the fixture never answers, so it cannot race.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private def python(): String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: com.fasterxml.jackson.databind.node.ObjectNode = objectMapper.createObjectNode()
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python(),
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = ShortTimeouts
+ )
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto a worker that goes quiet:
+ // those holding a worker time out and are discarded, which frees a slot
+ // without handing anything back, and the caller waiting at the cap has to
+ // notice that rather than wait for a hand-back that never comes.
+ val callers = PythonWorkerPool.maxWorkers + 1
+
+ val outcomes = onDaemonThreads(callers) { implicit ec =>
+ Await.result(Future.sequence(Seq.fill(callers)(Future(Try(hangingCall(Seq.empty))))), Bound)
+ }
+
+ assert(outcomes.length == callers)
+ assert(outcomes.forall(_.isFailure))
+ }
+
+ test("the pool still serves jobs after it has discarded a timed-out worker") {
Review Comment:
`interceptBounded(hangingCall(...))` discards a worker in the `HangingWorker` sub-pool, but the follow-up job passes `resourcePath = CompileWorker` (:163). `Key` includes `resourcePath` (`PythonWorkerPool.scala:176-182`), so `computeIfAbsent` builds a fresh `Pool` with an empty `idle`.
The assertion passes whether or not the discard left the pool it acted on usable. Pointing the follow-up at the same sub-pool would make it test its name. The cap case at :142-155 does cover replacement-after-discard, so the mechanism is not wholly untested.
##########
project/TestFilters.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import sbt._
+
+/**
+ * Selects a module's tagged tests for the fast-unit job or the integration job:
+ * skip-integration excludes them, integration-only runs only them, unset runs
+ * everything. Shared because the mapping is identical in every module, while the
+ * env var and the tag are not — the tag annotation has to live somewhere the
+ * module's own Test config can see.
+ */
+object TestFilters {
+
+ /** @param integrationOnlyExtra further ScalaTest args for the integration side,
Review Comment:
This uses the `@param` form but documents only `integrationOnlyExtra`. `envVar` and `tag` are the two a caller has to get right: the env var must match what the workflow sets, and the tag must match the annotation. Worth a line each here.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -225,21 +305,23 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
}
val total = descriptorCandidates.size
- var ok = 0
- var checked = 0
-
- val allFindings = descriptorCandidates.flatMap { descriptorClass =>
- checked += 1
+ val ok = new AtomicInteger(0)
+ val checked = new AtomicInteger(0)
+ // Checked concurrently: the fan-out is what turns the pool's workers into
+ // parallel interpreters rather than a queue in front of one. The pool bounds
+ // it — a submission past the cap blocks until a worker is returned.
Review Comment:
Both halves of the second sentence are off. The fan-out is bounded by `awaitAll`'s executor, which is sized to `PythonWorkerPool.maxWorkers` (:73-74) — nothing is submitted past the cap. And `borrow` at the cap deliberately does *not* block for a returned worker; its own comment (`PythonWorkerPool.scala:235-239`) says waiting would strand the caller.
```suggestion
// Checked concurrently: the fan-out is what turns the pool's workers into
// parallel interpreters rather than a queue in front of one. The executor is
// sized to maxWorkers, so nothing is submitted past the cap.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3770686511
##########
common/workflow-operator/build.sbt:
##########
@@ -35,6 +35,19 @@ ThisBuild / conflictManager := ConflictManager.latestRevision
// Restrict parallel execution of tests to avoid conflicts
Global / concurrentRestrictions += Tags.limit(Tags.Test, 1)
+// A test needing more than a bare Python interpreter is tagged, so the amber job
+// excludes it, and amber-integration, which installs operator-requirements.txt,
Review Comment:
Fixed in 53c69bbf9. Wrote "amber's requirements files" rather than "both", since the job installs dev-requirements.txt alongside the two.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,503 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `responseMillis` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startupMillis` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] for every worker the pool could
+ * not give out or keep — one that would not start, would not report ready, or
+ * died mid-job — so one `catch` covers a caller's whole fallback.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot — though not
Review Comment:
Fixed in 53c69bbf9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5286027691
Resolved the comments. @aglinxinyuan May you take a look at it? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5249146081
All seven are addressed in `baded3f5a`, and I reproduced both design findings before changing anything.
`-I` does hide PYTHONPATH: `PYTHONPATH=<dir> python3 -I -S -c "import <mod>"` raises ModuleNotFoundError while dropping `-I` imports it, and a non-`PYTHON*` variable survives either way. The parameter stays; the example is now the thing that says so.
`TestFilters` errors on an unrecognized value. Only the two recognized values appear anywhere in the repo, so nothing existing changes, and a typo now fails the build instead of running the full suite in a job that asked for a subset.
The description's 96 % is gone the same way its code twin went. I also fixed one the round missed: the description still said the executor stops anything being submitted past the cap, which you corrected in the code last round.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5245502913
Thanks — all of it is addressed in the two commits above, and each of the three real defects was reproduced before being fixed.
An interrupted caller now discards its worker rather than losing the slot: a new case fills every slot with an unanswered job, interrupts the callers, then asks the pool to serve that many jobs again, and without the fix it spins at the cap until the bound runs out. Starting a worker is now part of the pool's `WorkerDiedException` contract rather than something one call site has to name, so a bare `IOException` no longer walks past a caller's fallback and the runtime driver gets the same protection when it converts. And a first line that is not the protocol counts as not ready, with the line quoted in the message — stderr is empty in exactly that case — and the interpreter killed, since nothing else would reap a worker rejected before it joined the set; the fixture grew a mode that writes such a line, which raised `JsonParseException` before. On the silent half of that concern: the `py_compile` check now counts the descriptors the pool could not serve into its summary. Pointed
at an interpreter that does not exist, all 117 fall back, all 117 still pass, and the summary reports 117 fallbacks instead of reading as a green pooled run.
`-P4` is dropped, together with the shared argument that existed to carry it and the `maxWorkers` rationale it rested on — you are right that the cap is per sub-pool and env-overridable — and it comes back with the first tagged test that pools workers. Whether the fixture answers now travels in the request rather than in a launch argument, so the case for serving jobs after a discard acts on the sub-pool the discard happened in rather than a fresh one. `TestFilters` documents `envVar` and `tag`, and the two comment corrections are in as written.
`WorkflowOperator/test` is green at 2163 tests, the pool spec at 8, `testOptions` in both modules is unchanged apart from the dropped parameter, and scalafmt and scalafix are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3754197809
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -249,18 +350,50 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
val findings = checkResult.findings ++ pyCompileFindings
if (findings.isEmpty && checkResult.code.nonEmpty) {
- ok += 1
- println(s"[py-compile OK $ok/$total | checked $checked/$total] ${descriptorClass.getName}")
+ println(
+ s"[py-compile OK ${ok.incrementAndGet()}/$total | " +
+ s"checked ${checked.get()}/$total] ${descriptorClass.getName}"
+ )
}
findings
- }
+ }).flatten
- println(s"[py-compile SUMMARY] ok=$ok/$total")
+ println(
+ s"[py-compile SUMMARY] ok=${ok.get()}/$total, spawn fallbacks=${spawnFallbacks.get()}"
+ )
if (allFindings.nonEmpty) {
fail(PythonReflectionUtils.renderReport(allFindings, total = total))
}
}
+ /** py_compile above only parses the emitted code; running it needs the packages
+ * it imports. Tagged, so only amber-integration — the job that installs them —
+ * runs this. There a missing package is a defect; elsewhere it is a local-setup
+ * fact, so cancel rather than fail.
+ */
+ test(
+ "the Python interpreter operator templates run in should import pandas and plotly",
+ Tag(classOf[IntegrationTest].getName)
+ ) {
+ val provisioned = sys.env.get("AMBER_TEST_FILTER").contains("integration-only")
Review Comment:
Keying fail-vs-cancel on `AMBER_TEST_FILTER` ties this test's severity to *how* the integration subset gets selected. It is right today. But say that selector is later replaced or renamed — which extracting `TestFilters.integrationSplit` makes easy, and which its own `@param tag` doc warns about. The test still runs in `amber-integration`, silently back to cancelling: the non-result this PR removes, with nothing going red.
`TestFilters` is build-scope, so the string cannot be imported here. Worth a line at each site pointing at the other?
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -225,21 +324,23 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
}
val total = descriptorCandidates.size
- var ok = 0
- var checked = 0
-
- val allFindings = descriptorCandidates.flatMap { descriptorClass =>
- checked += 1
+ val ok = new AtomicInteger(0)
+ val checked = new AtomicInteger(0)
+ // Checked concurrently: the fan-out is what turns the pool's workers into
+ // parallel interpreters rather than a queue in front of one. The executor is
+ // sized to maxWorkers, so nothing is submitted past the cap.
Review Comment:
My wording from round 3 was wrong here: `awaitAll` builds all 117 futures up front, so every task *is* submitted — the fixed-size executor caps how many run at once.
```suggestion
// sized to maxWorkers, so nothing runs past the cap.
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — the imports ~96% of that — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool — per sub-pool, so callers on distinct worker
Review Comment:
`per sub-pool` twice reads as a stutter; the following line already says what the bound is not shared with.
```suggestion
/** Max live workers per sub-pool, so callers on distinct worker
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/PythonCodeRawInvalidTextSpec.scala:
##########
@@ -49,8 +57,99 @@ final class PythonCodeRawInvalidTextSpec extends AnyFunSuite {
private val MaxDepth: Int = 3
private val AcceptPackages: Seq[String] = Seq("org.apache.texera.amber.operator")
+ /** Budget for one whole fanned-out pass over every descriptor. Deliberately far
+ * above a real run (well under a second), so it only ever fires on a hang.
+ */
+ private val PassTimeout: FiniteDuration = 10.minutes
+
+ /** Runs the given work concurrently and returns the results in submission
+ * order, rethrowing the first failure so it fails the test. Sized to the pool
+ * so the threads match the workers available to serve them.
+ *
+ * Daemon threads: a task parked on a subprocess pipe answers no interrupt, so
+ * `shutdownNow` need not end it, and a non-daemon one left there would hold the
+ * JVM — and the build — open after [[PassTimeout]] has already failed the test.
+ */
+ private def awaitAll[T](work: Seq[() => T]): Seq[T] = {
+ val threads = Executors.newFixedThreadPool(
+ PythonWorkerPool.maxWorkers,
+ (r: Runnable) => {
+ val t = new Thread(r, "py-compile-check")
+ t.setDaemon(true)
+ t
+ }
+ )
+ try {
+ implicit val ec: ExecutionContext = ExecutionContext.fromExecutorService(threads)
+ Await.result(Future.sequence(work.map(w => Future(w()))), PassTimeout)
+ } finally threads.shutdownNow()
+ }
+
+ /** Checks the pooled path could not serve. Not a failure — the spawn each one
Review Comment:
`Checks` is the noun here, but every other Scaladoc in this file opens with a verb, so it first reads as "Checks that ..." — the next clause is what disambiguates it.
```suggestion
/** Count of checks the pooled path could not serve. Not a failure — the spawn each one
```
##########
.github/workflows/build.yml:
##########
@@ -631,6 +631,8 @@ jobs:
# specs. The Java @TagAnnotation makes the marker visible to
# ScalaTest's reflection, so `-n TAG` correctly narrows the
# run.
+ # WorkflowOperator/test is here for the same reason, with its own tag:
Review Comment:
This note landed here, but the `amber` job's counterpart at `:287-289` still reads as though the env value only tells `amber/build.sbt` to exclude amber's tag. It now also drives `common/workflow-operator/build.sbt:42-44` and a differently-named tag — and that exclusion is what keeps the pandas probe out of a job installing neither.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python tests to amber-integration [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3754680059
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
+ * a worker imports its libraries before it reports ready, and a loaded CI
+ * machine makes that slow. Override in seconds via
+ * `TEXERA_TEST_PYTHON_WORKER_TIMEOUT` / `TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT`.
+ */
+ final case class Timeouts(responseMillis: Long, startupMillis: Long)
+
+ object Timeouts {
+ private def envSeconds(name: String, default: Long): Long =
+ sys.env
+ .get(name)
+ .flatMap(s => scala.util.Try(s.trim.toLong).toOption)
+ .filter(_ > 0)
+ .getOrElse(default) * 1000
+
+ val Default: Timeouts = Timeouts(
+ responseMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_TIMEOUT", 30),
+ startupMillis = envSeconds("TEXERA_TEST_PYTHON_WORKER_STARTUP_TIMEOUT", 60)
+ )
+ }
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] for every worker the pool could
+ * not give out or keep — one that would not start, would not report ready, or
+ * died mid-job — so one `catch` covers a caller's whole fallback.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
Review Comment:
`-I` implies `-E`, so a worker launched with the `-I -S` this paragraph recommends three lines up ignores PYTHONPATH and every other `PYTHON*` variable. Checked it: `PYTHONPATH=<dir> python3 -I -S -c "import <mod>"` raises `ModuleNotFoundError` and `sys.path` shows no such entry; drop `-I` and it imports.
The parameter itself is fine — non-`PYTHON*` variables do survive `-I`. It is the example that misleads, and #7149's template runner is the caller that will reach for PYTHONPATH.
```suggestion
* (e.g. `--serve`), and `env` carries what a flag cannot — though not
* PYTHONPATH or any other PYTHON* var, which the `-I` above makes CPython ignore.
```
##########
project/TestFilters.scala:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import sbt._
+
+/**
+ * Selects a module's tagged tests for the fast-unit job or the integration job:
+ * skip-integration excludes them, integration-only runs only them, unset runs
+ * everything. Shared because the mapping is identical in every module, while the
+ * env var and the tag are not — the tag annotation has to live somewhere the
+ * module's own Test config can see.
+ */
+object TestFilters {
+
+ /** @param envVar the variable the two CI jobs set to opposite values; it has to
+ * be the one the workflow already sets on the step that invokes
+ * this module's tests, or neither subset is selected.
+ * @param tag fully-qualified name of the tag annotation, as
+ * `classOf[...].getName` gives it at the test site — ScalaTest
+ * matches these by string, so a rename that misses one side
+ * silently stops filtering.
+ */
+ def integrationSplit(envVar: String, tag: String): Seq[TestOption] =
+ sys.env.get(envVar) match {
+ case Some("skip-integration") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-l", tag))
+ case Some("integration-only") =>
+ Seq(Tests.Argument(TestFrameworks.ScalaTest, "-n", tag))
+ case _ => Nil
Review Comment:
A value neither branch recognizes falls through to "run everything". So an `AMBER_TEST_FILTER` typo in either job runs the full suite in `amber-integration`, and the pandas probe then reads `provisioned = false` and cancels. That is the non-result this PR exists to remove, arriving with nothing red.
The inline code did the same, but it was one module's. This is now the single helper both jobs route through, which makes it the place to fail loudly instead.
```suggestion
case Some(other) => sys.error(s"unrecognized $envVar value: $other")
case None => Nil
```
##########
common/workflow-operator/src/test/java/org/apache/texera/amber/operator/tags/IntegrationTest.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.tags;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+import org.scalatest.TagAnnotation;
+
+/**
+ * Marks a test in this module as needing more than a bare Python interpreter —
+ * pandas or plotly, which the {@code amber} job does not install. See the
+ * AMBER_TEST_FILTER block in {@code common/workflow-operator/build.sbt} for how
+ * it routes to {@code amber-integration}.
+ *
+ * <p>Apply it to a whole spec as an annotation, or to a single case — {@code
+ * test(name, Tag(classOf[IntegrationTest].getName))} in a FunSuite,
+ * {@code taggedAs} in a FlatSpec — so that a spec's cheaper assertions stay in
+ * the unit job and its coverage report.
Review Comment:
The trailing clause reads as the purpose of both options, but annotating a whole spec takes its cheap assertions out of the unit job too. That gain belongs only to the per-case form.
```suggestion
* <p>Apply it to a whole spec as an annotation, or — so that a spec's cheaper
* assertions stay in the unit job and its coverage report — to a single case:
* {@code test(name, Tag(classOf[IntegrationTest].getName))} in a FunSuite,
* {@code taggedAs} in a FlatSpec.
```
##########
.github/workflows/build.yml:
##########
@@ -284,9 +284,13 @@ jobs:
# Test config (sbt's `test` task does not transit dependsOn),
# so common modules' tests are listed explicitly here.
#
- # AMBER_TEST_FILTER=skip-integration tells amber/build.sbt to
- # exclude @org.apache.texera.amber.tags.IntegrationTest specs;
- # those run in the amber-integration job below.
+ # AMBER_TEST_FILTER=skip-integration excludes the integration-tagged
+ # specs of two modules, each under its own tag: amber/build.sbt drops
Review Comment:
No spec is excluded on the workflow-operator side — the tag sits on one case inside `PythonCodeRawInvalidTextSpec`, whose other tests still run in this job. That per-case granularity is the departure from amber worth not hiding here.
```suggestion
# AMBER_TEST_FILTER=skip-integration excludes the integration-tagged
# tests of two modules, each under its own tag: amber/build.sbt drops
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPoolSpec.scala:
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.funsuite.AnyFunSuite
+
+import java.util.concurrent.{Executors, TimeUnit}
+import scala.concurrent.duration._
+import scala.concurrent.{Await, ExecutionContext, ExecutionContextExecutorService, Future}
+import scala.util.Try
+
+/**
+ * What the pool owes a caller when a worker misbehaves. An ordinary job failure
+ * is the other suites' business; this one is about a worker that stays alive and
+ * stops taking part, which is the case that does not end by itself: a crash
+ * closes the pipe and the pending read returns, while silence would hold the
+ * caller forever — neither a read nor a write on a process pipe answers an
+ * interrupt or a deadline, so no suite-level timeout can release one.
+ *
+ * Every wait here is bounded and runs on daemon threads, so a regression fails
+ * these tests instead of wedging the run: a lost non-daemon thread parked on a
+ * pipe would keep the JVM, and the build, alive.
+ *
+ * The fixture worker is stdlib-only and runs under `-I -S`, so this needs an
+ * interpreter but none of the operator packages.
+ */
+final class PythonWorkerPoolSpec extends AnyFunSuite {
+
+ private val HangingWorker = "/python/hanging_worker.py"
+
+ /** Short enough to keep the suite quick, far enough above process startup to
+ * not be mistaken for one: a fixture asked to hang never answers at all, so
+ * these deadlines cannot race it.
+ */
+ private val ShortTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 1500, startupMillis = 1500)
+
+ /** For the case that interrupts its own callers: far enough out that they are
+ * certainly still waiting on the worker, not already past their deadline.
+ */
+ private val PatientTimeouts: PythonWorkerPool.Timeouts =
+ PythonWorkerPool.Timeouts(responseMillis = 60000, startupMillis = 60000)
+
+ /** Ceiling on a whole case, well above the deadlines under test. Reaching it
+ * means something never gave up.
+ */
+ private val Bound: FiniteDuration = 25.seconds
+
+ /** Any interpreter serves — the fixture imports only `json` and `time` — so
+ * this deliberately skips the configured `python.path` the suites that need
+ * pandas resolve. A machine without one cancels rather than fails.
+ */
+ private lazy val python: String = {
+ def isRunnable(exe: String): Boolean =
+ Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()).toOption
+ .exists { p =>
+ if (p.waitFor(5, TimeUnit.SECONDS)) p.exitValue() == 0 else { p.destroyForcibly(); false }
+ }
+
+ List("python3", "python", "py").find(isRunnable).getOrElse(cancel("no runnable python"))
+ }
+
+ private def onDaemonThreads[T](threads: Int)(body: ExecutionContext => T): T = {
+ val pool = Executors.newFixedThreadPool(
+ threads,
+ (r: Runnable) => {
+ val t = new Thread(r, "pool-spec-caller")
+ t.setDaemon(true)
+ t
+ }
+ )
+ val ec: ExecutionContextExecutorService = ExecutionContext.fromExecutorService(pool)
+ try body(ec)
+ finally pool.shutdownNow()
+ }
+
+ private def call(
+ launchArgs: Seq[String],
+ request: ObjectNode,
+ timeouts: PythonWorkerPool.Timeouts
+ ): PythonWorkerPool.Outcome =
+ PythonWorkerPool.run(
+ resourcePath = HangingWorker,
+ launchArgs = launchArgs,
+ pythonExe = python,
+ request = request,
+ interpreterArgs = Seq("-I", "-S"),
+ timeouts = timeouts
+ )
+
+ /** A job the fixture takes and never answers. `hang` travels in the request
+ * rather than in `launchArgs` so that [[healthyCall]] lands in the same
+ * sub-pool: `Key` covers the script, its arguments and the environment, and a
+ * job routed elsewhere would be served by a pool whose queue this suite never
+ * touched.
+ */
+ private def hangingCall(
+ launchArgs: Seq[String],
+ request: ObjectNode = objectMapper.createObjectNode(),
+ timeouts: PythonWorkerPool.Timeouts = ShortTimeouts
+ ): PythonWorkerPool.Outcome =
+ call(launchArgs, request.deepCopy().put("hang", true), timeouts)
+
+ /** A job in that same sub-pool which the fixture does answer. */
+ private def healthyCall(): PythonWorkerPool.Outcome =
+ call(Seq.empty, objectMapper.createObjectNode(), ShortTimeouts)
+
+ /** The call, on a daemon thread and under [[Bound]], expected to give up. */
+ private def interceptBounded(call: => Any): PythonWorkerPool.WorkerDiedException =
+ intercept[PythonWorkerPool.WorkerDiedException] {
+ onDaemonThreads(1)(ec => Await.result(Future(call)(ec), Bound))
+ }
+
+ test("a worker that takes the job and stops answering is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq.empty))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not answer"))
+ assert(thrown.getMessage.contains("killed it"))
+ // Well under the default response budget: what fired is the timeout passed in,
+ // not a wait that happened to end.
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.responseMillis / 2)
+ }
+
+ test("a worker that never signals ready is killed and reported") {
+ val startedAt = System.nanoTime()
+ val thrown = interceptBounded(hangingCall(Seq("--hang-before-ready")))
+ val elapsedMillis = (System.nanoTime() - startedAt) / 1000000
+
+ assert(thrown.getMessage.contains("did not signal ready"))
+ assert(elapsedMillis < PythonWorkerPool.Timeouts.Default.startupMillis / 2)
+ }
+
+ test("a worker that never reads its request is killed and reported") {
+ val request = objectMapper.createObjectNode()
+ // Past any pipe buffer, so the write cannot simply be handed to the kernel and
+ // left there: it is the blocked write itself that has to be given up on.
+ request.put("source", "x" * (4 * 1024 * 1024))
+
+ val thrown = interceptBounded(hangingCall(Seq("--deaf"), request))
+
+ assert(thrown.getMessage.contains("did not read its request"))
+ assert(thrown.getMessage.contains("killed it"))
+ }
+
+ test("a caller waiting at the worker cap is not stranded by a discarded worker") {
+ // One caller more than there are workers, all onto workers that go quiet:
+ // those holding one time out and are discarded, which frees a slot without
+ // handing anything back, and the caller waiting at the cap has to notice that
Review Comment:
"those holding one time out and are discarded" makes the callers the thing discarded; it is their workers. This sentence carries the mechanism the case exists to test, so the mismatch costs a re-read at exactly the wrong place.
```suggestion
// the callers holding one time out and their workers are discarded, which
// frees a slot without handing anything back, and the caller waiting at the
// cap has to notice that
```
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{
+ Callable,
+ ConcurrentHashMap,
+ ExecutionException,
+ ExecutorService,
+ Executors,
+ LinkedBlockingQueue,
+ TimeUnit,
+ TimeoutException
+}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.annotation.tailrec
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — mostly the imports — dwarfing
+ * the ~4 ms of real work a job does. A worker pays that once at startup, then
+ * serves many jobs over its lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — one test fanning
+ * its cases out, or suites running in parallel — so each sub-pool holds up to
+ * [[maxWorkers]] workers, each serving one job at a time (borrow -> run ->
+ * return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). A hard interpreter crash ends a worker;
+ * the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path. A worker that stays
+ * alive but stops answering ends the same way, on the [[Timeouts]] below.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown for a worker the pool could not give out or keep: one that would not
+ * start, one that never signalled ready, or one that died or fell silent
+ * mid-job. Callers catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool, so callers on distinct worker
+ * scripts add up rather than share this bound. Defaults to 4, override via
+ * `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out jobs within one
+ * test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /** How long a caller waits on a worker before the pool kills and discards it.
+ * A read on a process pipe cannot be interrupted — a suite or executor timeout
+ * leaves the reading thread stuck on it — so a worker that stays alive without
+ * answering has to be bounded here. `response` keeps the 30 seconds the
+ * one-shot spawn this pool replaced allowed a job; `startup` is longer because
Review Comment:
`response` and `startup` are not the field names — the case class below declares `responseMillis` and `startupMillis`, which is also what a caller constructing a `Timeouts` has to type.
```suggestion
* answering has to be bounded here. `responseMillis` keeps the 30 seconds the
* one-shot spawn allowed a job; `startupMillis` is longer because
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #7207:
URL: https://github.com/apache/texera/pull/7207#issuecomment-5222836571
@aglinxinyuan Resolved copilot's comment. May you take a look at it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): route Python-executing tests to amber-integration and pool the interpreters they spawn [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on code in PR #7207:
URL: https://github.com/apache/texera/pull/7207#discussion_r3716364601
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/util/python/PythonWorkerPool.scala:
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.util.python
+
+import com.fasterxml.jackson.databind.node.ObjectNode
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+
+import java.io.{BufferedReader, BufferedWriter, InputStreamReader, OutputStreamWriter}
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path, StandardCopyOption}
+import java.util.concurrent.{ConcurrentHashMap, LinkedBlockingQueue}
+import java.util.concurrent.atomic.AtomicInteger
+import scala.collection.mutable
+import scala.jdk.CollectionConverters._
+import scala.util.control.NonFatal
+
+/**
+ * Pools of persistent Python "worker" processes that eliminate the per-call
+ * interpreter-boot + import cost a test otherwise pays on every subprocess
+ * spawn. Testing operators one at a time does not scale when each one costs a
+ * spawn: a bare `-I -S` interpreter boots in ~25 ms, and once pandas and plotly
+ * are imported a spawn costs ~260-310 ms — ~96% of a job whose real work is
+ * ~4 ms. A worker pays that once at startup, then serves many jobs over its
+ * lifetime, so N spawns become one.
+ *
+ * Lives in test scope here, rather than beside a single caller, because tests in
+ * several modules run generated operator code and would otherwise each
+ * hand-roll a driver, a stdout protocol and a timeout. Other modules reach it
+ * through a `test->test` dependency on this one.
+ *
+ * Generic over the worker script — the pool never interprets the payload — so
+ * one implementation serves a syntax check, template execution and DataFrame
+ * comparison alike. Each distinct (resource, interpreterArgs, launchArgs,
+ * python, env) combination gets its own sub-pool.
+ *
+ * Protocol (line-delimited JSON, shared by all worker scripts):
+ * startup worker -> pool: {"ready": true}
+ * request pool -> worker: <caller-supplied JSON object>\n
+ * response worker -> pool: {"exit": <int>, "stdout": "...", "stderr": "..."}\n
+ *
+ * Concurrency: callers submit from several threads at once — a spec run with
+ * ScalaTest's `-P4`, or one test fanning its cases out — so each sub-pool holds
+ * up to [[maxWorkers]] workers, each serving one job at a time (borrow -> run
+ * -> return). A worker script may chdir per job, so a worker must never run two
+ * jobs at once — the borrow/return discipline guarantees that.
+ *
+ * Robustness: an ordinary *job* failure comes back as an [[Outcome]] with
+ * `exit != 0` (worker keeps running). Only a hard interpreter crash ends a
+ * worker; the pool detects the EOF / broken pipe, discards it, and throws
+ * [[WorkerDiedException]] so the caller can fall back to a one-shot subprocess
+ * — behavior is then never worse than the pre-pool path.
+ */
+object PythonWorkerPool extends LazyLogging {
+
+ /** Worker response: process-like exit code plus captured streams. */
+ final case class Outcome(exit: Int, stdout: String, stderr: String)
+
+ /** Thrown when a worker dies mid-job (hard crash / broken pipe). Callers
+ * catch this and fall back to a one-shot subprocess.
+ */
+ final class WorkerDiedException(message: String, cause: Throwable = null)
+ extends RuntimeException(message, cause)
+
+ /** Feature toggle. `TEXERA_TEST_PYTHON_WORKER=0` (or `false`/`off`) forces the
+ * one-subprocess-per-call paths everywhere — an escape hatch for debugging a
+ * suspected isolation leak. Default on.
+ */
+ val enabled: Boolean =
+ !sys.env
+ .get("TEXERA_TEST_PYTHON_WORKER")
+ .map(_.trim.toLowerCase)
+ .exists(Set("0", "false", "off"))
+
+ /** Max live workers per sub-pool. Defaults to 4 to match ScalaTest's `-P4`, so
+ * the two concurrency bounds agree on how many interpreters may be live;
+ * override via `TEXERA_TEST_PYTHON_WORKERS`. Public so a caller fanning out
+ * jobs within one test can size that fan-out to the workers it will get.
+ */
+ val maxWorkers: Int =
+ sys.env
+ .get("TEXERA_TEST_PYTHON_WORKERS")
+ .flatMap(s => scala.util.Try(s.trim.toInt).toOption)
+ .filter(_ > 0)
+ .getOrElse(4)
+
+ /**
+ * Run one job through a pooled worker for `resourcePath`, launched as
+ * `pythonExe <interpreterArgs> <script> <launchArgs>` with extra environment
+ * `env`. `request` is the worker-specific JSON payload (the pool does not
+ * interpret it). Throws [[WorkerDiedException]] on a hard worker crash.
+ *
+ * `interpreterArgs` are the flags that must precede the script — a syntax
+ * checker wants `-I -S` so it validates under the same isolation a one-shot
+ * `python -I -S -m py_compile` gave it. `launchArgs` are the script's own
+ * (e.g. `--serve`), and `env` carries what a flag cannot (e.g. PYTHONPATH).
+ * All three are part of a worker's identity: one started differently is not
+ * interchangeable, so it gets its own sub-pool.
+ */
+ def run(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ request: ObjectNode,
+ env: Map[String, String] = Map.empty,
+ interpreterArgs: Seq[String] = Seq.empty
+ ): Outcome = {
+ val envKey = env.toSeq.sorted.map { case (k, v) => s"$k=$v" }
+ val key =
+ (resourcePath +: pythonExe +: (interpreterArgs ++ launchArgs ++ envKey)).mkString(" ")
+ val pool = pools.computeIfAbsent(
+ key,
+ _ => new Pool(resourcePath, launchArgs, pythonExe, env, interpreterArgs)
+ )
+ pool.run(request)
+ }
+
+ private val pools = new ConcurrentHashMap[String, Pool]()
+
+ Runtime.getRuntime.addShutdownHook(new Thread(() => shutdownAll()))
+
+ private def shutdownAll(): Unit =
+ pools.values().forEach(_.shutdown())
+
+ // A single sub-pool: up to `maxWorkers` live workers for one worker script.
+ private final class Pool(
+ resourcePath: String,
+ launchArgs: Seq[String],
+ pythonExe: String,
+ env: Map[String, String],
+ interpreterArgs: Seq[String]
+ ) {
+ private val idle = new LinkedBlockingQueue[Worker]()
+ private val liveCount = new AtomicInteger(0)
+ private val all = mutable.Set.empty[Worker] // guarded by `all`
+ @volatile private var script: Path = _
+
+ def run(request: ObjectNode): Outcome = {
+ val w = borrow()
+ try {
+ val outcome = w.run(request)
+ idle.offer(w) // healthy — return to pool
+ outcome
+ } catch {
+ case e: WorkerDiedException =>
+ discard(w)
+ throw e
+ }
+ }
+
+ private def borrow(): Worker = {
+ val existing = idle.poll()
+ if (existing != null) return existing
+ if (liveCount.getAndIncrement() < maxWorkers) {
+ try create()
+ catch {
+ case e: Throwable =>
+ liveCount.decrementAndGet()
+ throw e
+ }
+ } else {
+ liveCount.decrementAndGet()
+ idle.take() // at the cap — block until a worker is returned
+ }
+ }
+
+ private def create(): Worker = {
+ val cmd =
+ (((pythonExe +: interpreterArgs) :+ ensureScript().toString) ++ launchArgs).asJava
+ val pb = new ProcessBuilder(cmd).redirectErrorStream(false)
+ env.foreach { case (k, v) => pb.environment().put(k, v) }
+ val w = new Worker(pb.start(), s"$resourcePath ${launchArgs.mkString(" ")}".trim)
+ w.awaitReady()
+ all.synchronized(all.add(w))
+ logger.debug(s"Started python worker for $resourcePath (live=${liveCount.get}/$maxWorkers)")
+ w
+ }
+
+ private def discard(w: Worker): Unit = {
+ all.synchronized(all.remove(w))
+ liveCount.decrementAndGet()
+ w.destroy()
+ }
+
+ private def ensureScript(): Path = {
+ if (script == null) synchronized {
+ if (script == null) {
+ val stream = getClass.getResourceAsStream(resourcePath)
+ require(stream != null, s"worker script not found on classpath at $resourcePath")
+ try {
+ val tmp = Files.createTempFile("py-worker-", ".py")
+ Files.copy(stream, tmp, StandardCopyOption.REPLACE_EXISTING)
+ tmp.toFile.deleteOnExit()
+ script = tmp
+ } finally stream.close()
+ }
+ }
+ script
+ }
+
+ def shutdown(): Unit =
+ all.synchronized {
+ all.foreach(_.destroy())
+ all.clear()
+ }
+ }
+
+ // One live worker process plus its framed-JSON stdin/stdout and a background
+ // drain of its own stderr (only non-empty on a hard crash).
+ private final class Worker(process: Process, label: String) {
+ private val stdin: BufferedWriter =
+ new BufferedWriter(new OutputStreamWriter(process.getOutputStream, StandardCharsets.UTF_8))
+ private val stdout: BufferedReader =
+ new BufferedReader(new InputStreamReader(process.getInputStream, StandardCharsets.UTF_8))
+ private val errBuf = new StringBuilder
+
+ private val errThread: Thread = {
+ val t = new Thread(() => {
+ val r =
+ new BufferedReader(new InputStreamReader(process.getErrorStream, StandardCharsets.UTF_8))
+ try {
+ var line = r.readLine()
+ while (line != null) {
+ errBuf.synchronized(errBuf.append(line).append('\n'))
+ line = r.readLine()
+ }
+ } catch { case NonFatal(_) => () }
+ })
+ t.setDaemon(true)
+ t.setName("python-worker-stderr")
+ t.start()
+ t
+ }
+
+ /** Block until the worker's startup `{"ready": true}` arrives; if it dies
+ * first (e.g. an import failed), surface its stderr.
+ */
+ def awaitReady(): Unit = {
+ val line = stdout.readLine()
+ if (line == null || !objectMapper.readTree(line).path("ready").asBoolean(false)) {
+ throw new WorkerDiedException(
+ s"python worker [$label] did not signal ready. stderr:\n${drainErr()}"
+ )
+ }
+ }
+
+ def run(request: ObjectNode): Outcome =
+ try {
+ stdin.write(objectMapper.writeValueAsString(request))
+ stdin.write("\n")
+ stdin.flush()
+ val line = stdout.readLine()
Review Comment:
Fixed. Dedicated threads now own the blocking read and the blocking write, so a caller waits with a deadline and a worker that misses it is killed and discarded, falling back to the one-shot subprocess as a crash already did. Response keeps the 30 seconds the old path allowed, startup gets its own budget, and PythonWorkerPoolSpec covers both against a fixture worker that goes quiet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org