You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/08/24 07:15:26 UTC
(texera) branch main updated: test(workflow-operator): cover the SQL, R-UDF, scorer and ECDF descriptors (#7899)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new 10663c9aea test(workflow-operator): cover the SQL, R-UDF, scorer and ECDF descriptors (#7899)
10663c9aea is described below
commit 10663c9aea922d2a088cc121775a6381b1d0e699
Author: Xinyuan Lin <xi...@uci.edu>
AuthorDate: Mon Aug 24 06:45:11 2026 +0000
test(workflow-operator): cover the SQL, R-UDF, scorer and ECDF descriptors (#7899)
### What changes were proposed in this PR?
Four operator descriptors. 36 tests → 47.
Measured with a full-module `WorkflowOperator/jacoco`, one fresh sbt JVM
per run, and the same suite-name exclusion on both sides
(`FileScanSourceOpExecSpec` aborts inside a git worktree — a worktree's
`.git` is a file, so it throws `RepositoryNotFoundException` in
`beforeAll` — and since sbt-jacoco runs unforked, a failing test task
yields an all-zero report rather than a partial one).
| File | Codecov | Missed arms |
|---|---|---|
| `SQLSourceOpDesc.scala` | 88.9% → **100%** | 11 → **0** |
| `RUDFSourceOpDesc.scala` | 80.6% → **100%** | 3 → **0** |
| `MachineLearningScorerOpDesc.scala` | 84.8% → **100%** | 4 → **0** |
| `ECDFPlotOpDesc.scala` | 85.4% → **87.5%** (its ceiling) | 15 → **12**
|
| **bundle** | 173/203 = 85.2% → **192/203 = 94.6%** | 48 → **27** |
JaCoCo line-hit goes 199/203 → 203/203.
**`TimeSeriesOpDesc` was in the original scope and is deliberately
absent.** It is worth exactly zero: all five of its partials are Scala's
null-safe comparison against a String *literal*, where one arm requires
the constant itself to be null. A probe setting every field to null left
it unchanged at 23/28. And the null path is a crash rather than a
behaviour — `dropnaCols.map(c => pyb"$c")` NPEs inside the `pyb` macro
on Base64 of a null string. Pinning it would cement junk.
### Verification
28 mutations, **27 killed, 1 equivalent survivor.**
The first draft reported no survivors. Two adversarial reviewers
demonstrated **8 survivors** between them, and every one that could be
reproduced was real. All are now dead.
Representative repairs, each proven by the mutation it now kills:
- The JDBC type map pinned 9 of 19 keys; the remaining ten are now
covered (`TINYINT`→INTEGER, `FLOAT`/`REAL`→DOUBLE, `BIT`→BOOLEAN, and
the rest).
- Three error-message assertions used `endWith`, leaving the half of the
message that names the failing operator unpinned. They are now exact
equalities against the concrete subclass name the spec itself controls.
- ECDF argument assertions were presence-only; they now check payloads
through the spec's existing Base64 helper, and three index-arithmetic
assertions pin the emitted body's order (`pd.to_numeric(` before
`px.ecdf(` before `plotly.io.to_html(`).
- The scorer's actual and predicted columns were interchangeable; the
loose whole-code `include` checks are now line-anchored.
**The survivor:** deleting `.withParallelizable(false)` from the else
arm of RUDF's worker-count guard. `PhysicalOp.sourcePhysicalOp` already
defaults it, so nothing observable changes — an equivalent mutant, not a
coverage gap.
### Two reviewer suggestions were refused
- One proposed deleting the RUDF single-worker test as vacuous. The
premise was half right — the `withParallelizable(false)` call really is
unobservable — but the suggested fix would have made the suite
**weaker**, since the test pins more than that one call.
- Its optional follow-up asked for a production deletion, which this
pass forbids. Recorded as a limitation rather than actioned.
A collateral claim about the builder's own harness was also left
unasserted rather than repeated, on the grounds that it could not be
verified from the state the worktree was handed over in.
### A nondeterminism source found along the way
The module-wide branch total does not move by exactly the amount these
four files account for, and the residual is **not** measurement noise in
this bundle: it is `IntervalJoinOpExec.scala`, whose covering spec uses
an **unseeded RNG**, so its branch count drifts run to run. That is
worth knowing because it makes module-level branch totals unquotable as
evidence for any PR touching this module. Seeding that spec is already
part of #7800.
Same-input reproducibility is otherwise good: the final `jacoco.xml` is
byte-identical to a reviewer's independently produced report.
### Deliberately not included, with evidence
- **`ECDFPlotOpDesc` cannot exceed 42/48.** `javap` on
`createPlotlyFigure` shows the three-jump null-safe form, and the
`ifnull`-taken arm needs the literal `"sum"` to be null. A probe moved
those lines from `cb=3` to `cb=5` with `mb` unchanged — 8 arms, zero
lines. Two further lines would need a `hashCode` collision with
`"count"` or `"sum"` that is not equal to it: constructible in
principle, padding in practice.
- **The scorer's last two lines are reachable only through an erasure
violation.** The LUB of the two metric list types erases to
`java.lang.Enum`, so the lambda bridge's checkcast admits a foreign enum
— `List(java.time.DayOfWeek.MONDAY).asInstanceOf[…]` does reach the
`case _` throw and takes the file to 33/33. No production route exists,
because Jackson rejects an unknown enum name first, so it is excluded.
- **The six `require` message strings are lifted to
`$anonfun$querySchema$1..6`** and dropped by `SyntheticFilter`; their
source lines already read as covered because the closure allocation is
attributed to the enclosing method.
- One file's headline 100% is worth qualifying: the `foldLeft` body
making its only schema-typing decision is a `$anonfun`, so it is not in
the tracked denominator either way.
Every spec in the bundle carries a JSON round-trip test, per house
convention. No descriptor subtype is defined in any spec, because
`PythonCodeRawInvalidTextSpec` instantiates every subclass by reflection
and a test-only subtype would break it.
No production file is touched, and no stray `test_large_binary.txt` was
left behind.
### Any related issues, documentation, discussions?
Closes #7898
### How was this PR tested?
```
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.source.sql.SQLSourceOpDescSpec org.apache.texera.amber.operator.udf.r.RUDFSourceOpDescSpec org.apache.texera.amber.operator.machineLearning.Scorer.MachineLearningScorerOpDescSpec org.apache.texera.amber.operator.visualization.ecdfPlot.ECDFPlotOpDescSpec"
```
```
[info] Total number of tests run: 47
[info] Tests: succeeded 47, failed 0, canceled 0, ignored 0, pending 0
```
`Test/scalafmtCheck` and `Test/scalafix --check` both pass.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
---------
Signed-off-by: Xinyuan Lin <xi...@uci.edu>
Co-authored-by: Copilot Autofix powered by AI <17...@users.noreply.github.com>
---
.../Scorer/MachineLearningScorerOpDescSpec.scala | 85 +++++++++++++++++++++-
.../operator/source/sql/SQLSourceOpDescSpec.scala | 55 +++++++++++++-
.../operator/udf/r/RUDFSourceOpDescSpec.scala | 63 ++++++++++++++++
.../ecdfPlot/ECDFPlotOpDescSpec.scala | 46 ++++++++++--
4 files changed, 236 insertions(+), 13 deletions(-)
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/machineLearning/Scorer/MachineLearningScorerOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/machineLearning/Scorer/MachineLearningScorerOpDescSpec.scala
index 07e728596b..09f916a51d 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/machineLearning/Scorer/MachineLearningScorerOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/machineLearning/Scorer/MachineLearningScorerOpDescSpec.scala
@@ -19,6 +19,7 @@
package org.apache.texera.amber.operator.machineLearning.Scorer
+import com.fasterxml.jackson.databind.node.ObjectNode
import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
import org.apache.texera.amber.operator.LogicalOp
import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
@@ -31,6 +32,18 @@ import java.util.Base64
class MachineLearningScorerOpDescSpec extends AnyFlatSpec with Matchers {
+ /** An EncodableString field renders as a runtime decode site in the emitted code. */
+ private val decodeSite = "self.decode_python_template"
+
+ private def b64(s: String): String =
+ Base64.getEncoder.encodeToString(s.getBytes(StandardCharsets.UTF_8))
+
+ /** The one emitted line that assigns metric_list, isolated from the template body. */
+ private def metricListLine(code: String): String =
+ code.linesIterator
+ .find(_.contains("metric_list = ["))
+ .getOrElse(fail(s"no metric_list assignment in:\n$code"))
+
"MachineLearningScorerOpDesc.operatorInfo" should
"advertise the name and Machine Learning General group" in {
val info = (new MachineLearningScorerOpDesc).operatorInfo
@@ -60,6 +73,23 @@ class MachineLearningScorerOpDescSpec extends AnyFlatSpec with Matchers {
)
}
+ it should "append one DOUBLE column per selected classification metric, after Class" in {
+ // The metric columns carry numeric scores, so their type is the one schema
+ // decision this method makes. With empty metric lists the foldLeft body never
+ // runs and that type stays unpinned -- a scorer could advertise Accuracy as
+ // STRING and the suite would not notice.
+ val d = new MachineLearningScorerOpDesc
+ d.classificationMetrics =
+ List(classificationMetricsFnc.accuracy, classificationMetricsFnc.f1Score)
+ val port = d.operatorInfo.outputPorts.head.id
+ d.getOutputSchemas(Map.empty)(port).getAttributes.map(a => (a.getName, a.getType)) shouldBe
+ List(
+ ("Class", AttributeType.STRING),
+ ("Accuracy", AttributeType.DOUBLE),
+ ("F1 Score", AttributeType.DOUBLE)
+ )
+ }
+
it should "produce an empty schema for regression with no metrics" in {
val d = new MachineLearningScorerOpDesc
d.isRegression = true
@@ -68,6 +98,17 @@ class MachineLearningScorerOpDescSpec extends AnyFlatSpec with Matchers {
out(d.operatorInfo.outputPorts.head.id).getAttributes shouldBe empty
}
+ it should "emit one DOUBLE column per selected regression metric and no Class column" in {
+ val d = new MachineLearningScorerOpDesc
+ d.isRegression = true
+ d.regressionMetrics = List(regressionMetricsFnc.mse, regressionMetricsFnc.r2)
+ // the classification list must be ignored entirely once the task is regression
+ d.classificationMetrics = List(classificationMetricsFnc.accuracy)
+ val port = d.operatorInfo.outputPorts.head.id
+ d.getOutputSchemas(Map.empty)(port).getAttributes.map(a => (a.getName, a.getType)) shouldBe
+ List(("MSE", AttributeType.DOUBLE), ("R2", AttributeType.DOUBLE))
+ }
+
"MachineLearningScorerOpDesc.generatePythonCode" should "emit the scorer table operator" in {
val d = new MachineLearningScorerOpDesc
d.actualValueColumn = "y"
@@ -75,9 +116,15 @@ class MachineLearningScorerOpDescSpec extends AnyFlatSpec with Matchers {
val code = d.generatePythonCode()
code should include("class ProcessTableOperator(UDFTableOperator)")
code should include("from sklearn.metrics import")
- // actualValueColumn/predictValueColumn are EncodableString: base64-encoded into the emitted code.
- code should include(Base64.getEncoder.encodeToString("y".getBytes(StandardCharsets.UTF_8)))
- code should include(Base64.getEncoder.encodeToString("yhat".getBytes(StandardCharsets.UTF_8)))
+ // actualValueColumn/predictValueColumn are EncodableString: base64-encoded into
+ // the emitted code. Assert WHICH variable each payload is bound to, not just that
+ // both payloads appear somewhere: precision_score/recall_score are asymmetric in
+ // (y_true, y_pred), so a swap would silently report recall as precision.
+ code should include(s"y_true = table[$decodeSite('${b64("y")}')]")
+ code should include(s"y_pred = table[$decodeSite('${b64("yhat")}')]")
+ // isRegression defaults to false, so the emitted branch guard must read False;
+ // without this the `else "False"` arm of the flag is unpinned in both tests.
+ code should include("if False:")
}
it should "splice the selected metrics verbatim into a proper metric_list" in {
@@ -96,6 +143,38 @@ class MachineLearningScorerOpDescSpec extends AnyFlatSpec with Matchers {
code should not include encoded
}
+ it should "select the regression metrics and the regression branch when isRegression is set" in {
+ val d = new MachineLearningScorerOpDesc
+ d.isRegression = true
+ d.actualValueColumn = "y"
+ d.predictValueColumn = "yhat"
+ d.regressionMetrics = List(regressionMetricsFnc.mse, regressionMetricsFnc.r2)
+ // the classification list must be ignored entirely once the task is regression
+ d.classificationMetrics = List(classificationMetricsFnc.accuracy)
+ val code = d.generatePythonCode()
+
+ metricListLine(code) should include("['MSE','R2']")
+ // isRegression also picks the branch process_table takes at runtime
+ code should include("if True:")
+ code should not include "if False:"
+ }
+
+ it should "reject an unrecognized entry in the metric list loudly" in {
+ // A saved workflow can carry a null element in the metric array. Emitting
+ // `metric_list = ['']` for it would generate silently broken Python, so the
+ // descriptor must fail while the workflow is still being compiled.
+ val node = objectMapper
+ .readTree(objectMapper.writeValueAsString(new MachineLearningScorerOpDesc))
+ .asInstanceOf[ObjectNode]
+ node.putArray("classificationFlag").addNull()
+ val d =
+ objectMapper.treeToValue(node, classOf[LogicalOp]).asInstanceOf[MachineLearningScorerOpDesc]
+ d.classificationMetrics shouldBe List(null)
+
+ val ex = intercept[IllegalArgumentException](d.generatePythonCode())
+ ex.getMessage shouldBe "Unknown metric type"
+ }
+
"MachineLearningScorerOpDesc" should "round-trip its config fields through the polymorphic base" in {
val d = new MachineLearningScorerOpDesc
d.isRegression = true
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/sql/SQLSourceOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/sql/SQLSourceOpDescSpec.scala
index 077f138078..5d2b3bd99f 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/sql/SQLSourceOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/source/sql/SQLSourceOpDescSpec.scala
@@ -89,6 +89,16 @@ class SQLSourceOpDescSpec extends AnyFlatSpec with Matchers with MockFactory {
configure(new TestSQLSourceOpDesc(conn)).sourceSchema().getAttribute("c").getType
}
+ /** Every mandatory connection field, with the phrase its prompt must carry. */
+ private val connectionFields: Seq[(String, (SQLSourceOpDesc, String) => Unit, String)] = Seq(
+ ("host", (d, v) => d.host = v, "valid host name"),
+ ("port", (d, v) => d.port = v, "valid port"),
+ ("database", (d, v) => d.database = v, "valid database name"),
+ ("table", (d, v) => d.table = v, "valid table name"),
+ ("username", (d, v) => d.username = v, "valid username"),
+ ("password", (d, v) => d.password = v, "valid password")
+ )
+
"SQLSourceOpDesc.sourceSchema" should "prompt for the missing connection field" in {
val desc = configure(new TestSQLSourceOpDesc(mock[Connection]))
desc.table = null
@@ -96,14 +106,41 @@ class SQLSourceOpDescSpec extends AnyFlatSpec with Matchers with MockFactory {
ex.getMessage should include("table")
}
+ it should "prompt for each connection field left unset or blank, naming that field" in {
+ // A saved workflow can carry either an absent field (null) or one the user
+ // filled with whitespace; both must be rejected before any connection attempt.
+ for {
+ (fieldName, set, phrase) <- connectionFields
+ (label, value) <- Seq("unset" -> null.asInstanceOf[String], "blank" -> " ")
+ } withClue(s"$fieldName $label: ") {
+ val desc = configure(new TestSQLSourceOpDesc(mock[Connection]))
+ set(desc, value)
+ val ex = intercept[IllegalArgumentException](desc.sourceSchema())
+ ex.getMessage should include(phrase)
+ }
+ }
+
it should "map each JDBC data type to its Texera attribute type" in {
- attributeTypeFor(Types.INTEGER) shouldBe AttributeType.INTEGER
+ // All 19 keys of the match, not a sample of them. JaCoCo folds the lookupswitch's
+ // 19 cases into its 8 distinct jump targets, so this list moves coverage by
+ // exactly zero -- it is bought purely so that moving a key between arms fails.
+ attributeTypeFor(Types.TINYINT) shouldBe AttributeType.INTEGER
attributeTypeFor(Types.SMALLINT) shouldBe AttributeType.INTEGER
+ attributeTypeFor(Types.INTEGER) shouldBe AttributeType.INTEGER
+ attributeTypeFor(Types.FLOAT) shouldBe AttributeType.DOUBLE
+ attributeTypeFor(Types.REAL) shouldBe AttributeType.DOUBLE
attributeTypeFor(Types.DOUBLE) shouldBe AttributeType.DOUBLE
attributeTypeFor(Types.NUMERIC) shouldBe AttributeType.DOUBLE
+ attributeTypeFor(Types.BIT) shouldBe AttributeType.BOOLEAN
attributeTypeFor(Types.BOOLEAN) shouldBe AttributeType.BOOLEAN
attributeTypeFor(Types.BINARY) shouldBe AttributeType.BINARY
+ attributeTypeFor(Types.DATE) shouldBe AttributeType.STRING
+ attributeTypeFor(Types.TIME) shouldBe AttributeType.STRING
+ attributeTypeFor(Types.LONGVARCHAR) shouldBe AttributeType.STRING
+ attributeTypeFor(Types.CHAR) shouldBe AttributeType.STRING
attributeTypeFor(Types.VARCHAR) shouldBe AttributeType.STRING
+ attributeTypeFor(Types.NULL) shouldBe AttributeType.STRING
+ attributeTypeFor(Types.OTHER) shouldBe AttributeType.STRING
attributeTypeFor(Types.BIGINT) shouldBe AttributeType.LONG
attributeTypeFor(Types.TIMESTAMP) shouldBe AttributeType.TIMESTAMP
}
@@ -149,14 +186,26 @@ class SQLSourceOpDescSpec extends AnyFlatSpec with Matchers with MockFactory {
(rs.getString(_: String)).expects("COLUMN_NAME").returning("c")
(rs.getInt(_: String)).expects("DATA_TYPE").returning(Types.ARRAY)
val ex = intercept[RuntimeException](configure(new TestSQLSourceOpDesc(conn)).sourceSchema())
- ex.getMessage should endWith(": unknown data type: " + Types.ARRAY)
+ // Whole message, not endWith: the leading getSimpleName is the only thing that
+ // tells the user WHICH source operator in the workflow rejected the column.
+ ex.getMessage shouldBe s"TestSQLSourceOpDesc: unknown data type: ${Types.ARRAY}"
}
it should "wrap a SQLException raised while introspecting" in {
val conn = mock[Connection]
(conn.setReadOnly _).expects(true).throwing(new SQLException("boom"))
val ex = intercept[RuntimeException](configure(new TestSQLSourceOpDesc(conn)).sourceSchema())
- ex.getMessage should endWith(" failed to connect to the database. boom")
+ ex.getMessage shouldBe "TestSQLSourceOpDesc failed to connect to the database. boom"
+ }
+
+ it should "wrap a ClassCastException raised while introspecting" in {
+ // A driver handing back a mistyped metadata object surfaces as a CCE, which
+ // the descriptor declares as a connection failure rather than letting it escape.
+ val conn = mock[Connection]
+ (conn.setReadOnly _).expects(true).throwing(new ClassCastException("cce"))
+ val ex = intercept[RuntimeException](configure(new TestSQLSourceOpDesc(conn)).sourceSchema())
+ ex.getClass shouldBe classOf[RuntimeException]
+ ex.getMessage shouldBe "TestSQLSourceOpDesc failed to connect to the database. cce"
}
it should "fail when the base establishConn returns no connection" in {
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala
index 2cfca40e73..d03eba1280 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala
@@ -19,6 +19,7 @@
package org.apache.texera.amber.operator.udf.r
+import com.fasterxml.jackson.databind.node.ObjectNode
import org.apache.texera.amber.core.executor.OpExecWithCode
import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
@@ -49,6 +50,18 @@ class RUDFSourceOpDescSpec extends AnyFlatSpec with Matchers {
d.sourceSchema() shouldBe Schema().add(new Attribute("a", AttributeType.STRING))
}
+ it should "treat an absent column list as no columns" in {
+ // A saved workflow can carry "columns": null, which Jackson leaves as a null
+ // reference; the schema must come back empty instead of blowing up.
+ val node = objectMapper
+ .readTree(objectMapper.writeValueAsString(new RUDFSourceOpDesc))
+ .asInstanceOf[ObjectNode]
+ node.putNull("columns")
+ val d = objectMapper.treeToValue(node, classOf[LogicalOp]).asInstanceOf[RUDFSourceOpDesc]
+ d.columns shouldBe null
+ d.sourceSchema() shouldBe Schema()
+ }
+
"RUDFSourceOpDesc.getPhysicalOp" should
"wire OpExecWithCode(code, \"r-table\") as a source op with one output port" in {
val d = new RUDFSourceOpDesc
@@ -61,6 +74,56 @@ class RUDFSourceOpDescSpec extends AnyFlatSpec with Matchers {
case other => fail(s"expected OpExecWithCode, got $other")
}
physical.outputPorts.keySet shouldBe d.operatorInfo.outputPorts.map(_.id).toSet
+ // A source that fans out is marked one-to-many for the scheduler; the flag
+ // defaults to false, so nothing else in the spec would notice it flipping.
+ physical.isOneToManyOp shouldBe true
+ }
+
+ it should "advertise its own source schema through the propagation function" in {
+ // The propagation function is a closure: asserting the wiring above never
+ // evaluates it, so the physical op could hand an empty schema downstream for a
+ // fully configured column list and every other assertion here would still hold.
+ val d = new RUDFSourceOpDesc
+ d.code = "data.frame(a=1)"
+ d.columns = List(new Attribute("a", AttributeType.STRING))
+ val physical = d.getPhysicalOp(workflowId, executionId)
+ physical.propagateSchema.func(Map.empty) shouldBe Map(
+ d.operatorInfo.outputPorts.head.id -> Schema().add(new Attribute("a", AttributeType.STRING))
+ )
+ }
+
+ it should "wire OpExecWithCode(code, \"r-tuple\") when the Tuple API is selected" in {
+ val d = new RUDFSourceOpDesc
+ d.code = "coro::generator(function() {})"
+ d.useTupleAPI = true
+ d.getPhysicalOp(workflowId, executionId).opExecInitInfo match {
+ case OpExecWithCode(code, language) =>
+ language shouldBe "r-tuple"
+ code shouldBe "coro::generator(function() {})"
+ case other => fail(s"expected OpExecWithCode, got $other")
+ }
+ }
+
+ it should "parallelize at the requested worker count only when more than one worker is asked for" in {
+ val many = new RUDFSourceOpDesc
+ many.code = "x"
+ many.workers = 3
+ val parallel = many.getPhysicalOp(workflowId, executionId)
+ parallel.parallelizable shouldBe true
+ parallel.suggestedWorkerNum shouldBe Some(3)
+
+ // The single-worker half pins the GUARD'S BOUNDARY, not the else-arm's body:
+ // PhysicalOp.sourcePhysicalOp already builds with parallelizable = false and
+ // suggestedWorkerNum = None, so the `withParallelizable(false)` call in the else
+ // arm is a no-op that cannot be observed from outside. What these two lines do
+ // catch is the boundary moving -- `workers > 1` widened to `workers >= 1` sends
+ // a one-worker op down the parallel arm and both assertions fail.
+ val one = new RUDFSourceOpDesc
+ one.code = "x"
+ one.workers = 1
+ val serial = one.getPhysicalOp(workflowId, executionId)
+ serial.parallelizable shouldBe false
+ serial.suggestedWorkerNum shouldBe None
}
it should "reject a non-positive worker count" in {
diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/ecdfPlot/ECDFPlotOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/ecdfPlot/ECDFPlotOpDescSpec.scala
index b1fe949935..496d6c3c3c 100644
--- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/ecdfPlot/ECDFPlotOpDescSpec.scala
+++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/ecdfPlot/ECDFPlotOpDescSpec.scala
@@ -71,13 +71,17 @@ class ECDFPlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers {
assert(plain.contains("fig = px.ecdf(table"))
assert(plain.contains("ecdfnorm=None"))
- assert(plain.contains("ecdfmode=self.decode_python_template"))
assert(plain.contains("orientation='h'"))
assert(plain.contains("markers=True"))
- assert(plain.contains("marginal=self.decode_python_template"))
- assert(plain.contains("x=self.decode_python_template"))
- assert(plain.contains("color=self.decode_python_template"))
- assert(plain.contains("facet_col=self.decode_python_template"))
+ // Assert WHICH value reaches each keyword, not merely that a decode site is
+ // present there. Every one of these arguments renders to the same marker, so a
+ // presence-only check cannot tell x from color, or ecdfmode from marginal, and
+ // the payloads stay freely substitutable underneath it.
+ assert(plain.contains(s"x=$decodeSite('${b64("score")}')"))
+ assert(plain.contains(s"color=$decodeSite('${b64("group")}')"))
+ assert(plain.contains(s"facet_col=$decodeSite('${b64("category")}')"))
+ assert(plain.contains(s"ecdfmode=$decodeSite('${b64("reversed")}')"))
+ assert(plain.contains(s"marginal=$decodeSite('${b64("histogram")}')"))
}
it should "throw AssertionError naming the Value Column when valueColumn is left empty (manipulateTable)" in {
@@ -161,7 +165,9 @@ class ECDFPlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers {
val call = lineContaining(opDesc.createPlotlyFigure().plain, "px.ecdf(")
decodeSiteCount(call) shouldBe 1
- call should include(s"px.ecdf(table, x=$decodeSite")
+ // The payload, not just the marker: a decode site carrying colorColumn (empty by
+ // default) would satisfy a marker-only check while plotting nothing.
+ call should include(s"px.ecdf(table, x=$decodeSite('${b64("score")}')")
call should not include "ecdfnorm"
call should not include ", y="
call should not include "color="
@@ -178,7 +184,7 @@ class ECDFPlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers {
val call = lineContaining(opDesc.createPlotlyFigure().plain, "px.ecdf(")
call should include("ecdfnorm=None")
- call should include(s", y=$decodeSite")
+ call should include(s", y=$decodeSite('${b64("score")}')")
// the value column is spliced twice: once as x, once as y
decodeSiteCount(call) shouldBe 2
}
@@ -207,6 +213,21 @@ class ECDFPlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers {
call should not include "markers="
}
+ it should "fall back to probability normalization when yAxisMode arrives null" in {
+ // A saved workflow can carry "yAxisMode": null. The match must fall through to
+ // its default arm and emit the probability form -- neither throwing nor turning
+ // normalization off the way the explicit "count"/"sum" modes do.
+ val json =
+ """{"operatorType":"ECDFPlot","operatorID":"ECDFPlot-1","valueColumn":"score","yAxisMode":null}"""
+ val d = objectMapper.readValue(json, classOf[LogicalOp]).asInstanceOf[ECDFPlotOpDesc]
+ d.yAxisMode shouldBe null
+
+ val call = lineContaining(d.createPlotlyFigure().plain, "px.ecdf(")
+ call should not include "ecdfnorm"
+ call should not include ", y="
+ decodeSiteCount(call) shouldBe 1
+ }
+
// --- manipulateTable: the dropna column list -------------------------------
"ECDFPlotOpDesc.manipulateTable" should
@@ -252,6 +273,17 @@ class ECDFPlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers {
code should include("no valid rows left after removing missing or non-numeric values.")
code should include("plotly.io.to_html(fig, include_plotlyjs='cdn', auto_play=False)")
code should include("yield {'html-content': html}")
+
+ // Order matters, and a bag of unordered `include`s cannot see it. The cleaning
+ // step has to be spliced BEFORE the figure so the plot is built on cleaned rows,
+ // with the "no valid rows left" guard sitting between the two; swapping the two
+ // splices would plot uncleaned data and leave that second guard dead.
+ code should include("pd.to_numeric(")
+ code should include("px.ecdf(")
+ code.indexOf("pd.to_numeric(") should be < code.indexOf("px.ecdf(")
+ code.indexOf("px.ecdf(") should be < code.indexOf("plotly.io.to_html(")
+ code.indexOf("input table is empty.") should be <
+ code.indexOf("no valid rows left after removing missing or non-numeric values.")
}
// --- JSON round-trip --------------------------------------------------------