You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/06/22 00:59:40 UTC

[PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

aglinxinyuan opened a new pull request, #5877:
URL: https://github.com/apache/texera/pull/5877

   ### What changes were proposed in this PR?
   
   Pin behavior of three previously-untested Java/R UDF descriptors in `common/workflow-operator/udf/`. No production-code changes.
   
   | Spec | Source class | Tests |
   | --- | --- | --- |
   | `JavaUDFOpDescSpec` | `JavaUDFOpDesc` | 6 |
   | `RUDFOpDescSpec` | `RUDFOpDesc` | 6 |
   | `RUDFSourceOpDescSpec` | `RUDFSourceOpDesc` | 6 |
   
   **Behavior pinned**
   
   | Surface | Contract |
   | --- | --- |
   | `operatorInfo` | exact name + group (`Java`/`R`); `JavaUDF`/`RUDF` dynamic 1-in/1-out; `RUDFSource` zero inputs / one output |
   | `getPhysicalOp` | wires `OpExecWithCode` (Java tagged `"java"`); port identities carried; rejects `workers <= 0` |
   | Schema propagation | `JavaUDF`/`RUDF`: `retainInputColumns=false` → only output columns, `true` → input + output columns; `RUDFSource`: `sourceSchema()` from the `columns` field |
   | Round-trip | config fields preserved through the polymorphic base (incl. R `useTupleAPI`) |
   
   ### Any related issues, documentation, discussions?
   
   Closes #5874.
   
   ### How was this PR tested?
   
   - `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.udf.java.JavaUDFOpDescSpec org.apache.texera.amber.operator.udf.r.RUDFOpDescSpec org.apache.texera.amber.operator.udf.r.RUDFSourceOpDescSpec"` — 18 tests, all green
   - `sbt "WorkflowOperator/Test/scalafmtCheck"` and `sbt "WorkflowOperator/Test/scalafix --check"` — clean
   - CI to confirm
   
   ### Was this PR authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Opus 4.8 [1M context])


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on code in PR #5877:
URL: https://github.com/apache/texera/pull/5877#discussion_r3449389225


##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFOpDescSpec.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFOpDesc.operatorInfo" should
+    "advertise the name, R group, and a default 1-in/1-out shape" in {
+    val info = (new RUDFOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts should have length 1
+    info.outputPorts should have length 1
+  }
+
+  "RUDFOpDesc" should "default code/workers/useTupleAPI/retainInputColumns" in {
+    val d = new RUDFOpDesc
+    d.code shouldBe ""
+    d.workers shouldBe 1
+    d.useTupleAPI shouldBe false
+    d.retainInputColumns shouldBe false
+  }
+
+  "RUDFOpDesc.getPhysicalOp" should "wire OpExecWithCode and carry port identities" in {
+    val d = new RUDFOpDesc
+    d.code = "function(t) t"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "function(t) t"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }
+    physical.inputPorts.keySet shouldBe d.operatorInfo.inputPorts.map(_.id).toSet
+    physical.outputPorts.keySet shouldBe d.operatorInfo.outputPorts.map(_.id).toSet
+  }
+
+  it should "reject a non-positive worker count" in {
+    val d = new RUDFOpDesc
+    d.workers = 0
+    intercept[IllegalArgumentException] { d.getPhysicalOp(workflowId, executionId) }
+  }
+
+  "RUDFOpDesc schema propagation" should
+    "emit only the output columns when input columns are not retained (default)" in {
+    val d = new RUDFOpDesc
+    d.outputColumns = List(new Attribute("res", AttributeType.INTEGER))
+    val input = Schema().add(new Attribute("in", AttributeType.STRING))
+    d.getExternalOutputSchemas(Map(d.operatorInfo.inputPorts.head.id -> input)) shouldBe Map(
+      d.operatorInfo.outputPorts.head.id -> Schema().add(
+        new Attribute("res", AttributeType.INTEGER)
+      )
+    )
+  }

Review Comment:
   Added a retainInputColumns = true propagation test asserting the retained input column plus the output column.



##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFOpDescSpec.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFOpDesc.operatorInfo" should
+    "advertise the name, R group, and a default 1-in/1-out shape" in {
+    val info = (new RUDFOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts should have length 1
+    info.outputPorts should have length 1
+  }
+
+  "RUDFOpDesc" should "default code/workers/useTupleAPI/retainInputColumns" in {
+    val d = new RUDFOpDesc
+    d.code shouldBe ""
+    d.workers shouldBe 1
+    d.useTupleAPI shouldBe false
+    d.retainInputColumns shouldBe false
+  }
+
+  "RUDFOpDesc.getPhysicalOp" should "wire OpExecWithCode and carry port identities" in {
+    val d = new RUDFOpDesc
+    d.code = "function(t) t"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "function(t) t"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }

Review Comment:
   Done — the test now pins language to "r-table".



##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFSourceOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFSourceOpDesc.operatorInfo" should
+    "advertise the 1-out R UDF source (no inputs, one output)" in {
+    val info = (new RUDFSourceOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "1-out R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts shouldBe empty
+    info.outputPorts should have length 1
+  }
+
+  "RUDFSourceOpDesc.sourceSchema" should "be empty by default and reflect the configured columns" in {
+    (new RUDFSourceOpDesc).sourceSchema().getAttributes shouldBe empty
+    val d = new RUDFSourceOpDesc
+    d.columns = List(new Attribute("a", AttributeType.STRING))
+    d.sourceSchema() shouldBe Schema().add(new Attribute("a", AttributeType.STRING))
+  }
+
+  "RUDFSourceOpDesc.getPhysicalOp" should
+    "wire OpExecWithCode as a source op with one output port" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "data.frame(a=1)"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "data.frame(a=1)"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }

Review Comment:
   Done — pinned language to "r-table".



##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFSourceOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFSourceOpDesc.operatorInfo" should
+    "advertise the 1-out R UDF source (no inputs, one output)" in {
+    val info = (new RUDFSourceOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "1-out R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts shouldBe empty
+    info.outputPorts should have length 1
+  }
+
+  "RUDFSourceOpDesc.sourceSchema" should "be empty by default and reflect the configured columns" in {
+    (new RUDFSourceOpDesc).sourceSchema().getAttributes shouldBe empty
+    val d = new RUDFSourceOpDesc
+    d.columns = List(new Attribute("a", AttributeType.STRING))
+    d.sourceSchema() shouldBe Schema().add(new Attribute("a", AttributeType.STRING))
+  }
+
+  "RUDFSourceOpDesc.getPhysicalOp" should
+    "wire OpExecWithCode as a source op with one output port" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "data.frame(a=1)"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "data.frame(a=1)"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }
+    physical.outputPorts.keySet shouldBe d.operatorInfo.outputPorts.map(_.id).toSet
+  }
+
+  it should "reject a non-positive worker count" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "x"
+    d.workers = 0
+    intercept[IllegalArgumentException] { d.getPhysicalOp(workflowId, executionId) }
+  }
+
+  "RUDFSourceOpDesc" should "round-trip its config fields through the polymorphic base" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "f"
+    d.workers = 2
+    d.useTupleAPI = true
+    val restored = objectMapper.readValue(objectMapper.writeValueAsString(d), classOf[LogicalOp])
+    restored shouldBe a[RUDFSourceOpDesc]
+    val r = restored.asInstanceOf[RUDFSourceOpDesc]
+    r.code shouldBe "f"
+    r.workers shouldBe 2
+    r.useTupleAPI shouldBe true

Review Comment:
   Done — the round-trip now covers the columns field.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
xuang7 merged PR #5877:
URL: https://github.com/apache/texera/pull/5877


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5877:
URL: https://github.com/apache/texera/pull/5877#issuecomment-4763904217

   ## [Codecov](https://app.codecov.io/gh/apache/texera/pull/5877?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 53.91%. Comparing base ([`ed22158`](https://app.codecov.io/gh/apache/texera/commit/ed22158534c1ae503fea9b4338a03862b5231c97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c51fbc5`](https://app.codecov.io/gh/apache/texera/commit/c51fbc5f58f21cd097c0e03ef442063874f021f1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##               main    #5877   +/-   ##
   =========================================
     Coverage     53.91%   53.91%           
     Complexity     2780     2780           
   =========================================
     Files          1099     1099           
     Lines         42565    42565           
     Branches       4577     4577           
   =========================================
     Hits          22947    22947           
     Misses        18282    18282           
     Partials       1336     1336           
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
   |---|---|---|---|
   | [access-control-service](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.44% <ø> (ø)` | | |
   | [agent-service](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.36% <ø> (ø)` | | Carriedforward from [ed22158](https://app.codecov.io/gh/apache/texera/commit/ed22158534c1ae503fea9b4338a03862b5231c97?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [amber](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.26% <ø> (ø)` | | Carriedforward from [ed22158](https://app.codecov.io/gh/apache/texera/commit/ed22158534c1ae503fea9b4338a03862b5231c97?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `1.65% <ø> (ø)` | | |
   | [config-service](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.35% <ø> (ø)` | | |
   | [file-service](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.06% <ø> (ø)` | | |
   | [frontend](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.03% <ø> (ø)` | | Carriedforward from [ed22158](https://app.codecov.io/gh/apache/texera/commit/ed22158534c1ae503fea9b4338a03862b5231c97?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pyamber](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.19% <ø> (ø)` | | Carriedforward from [ed22158](https://app.codecov.io/gh/apache/texera/commit/ed22158534c1ae503fea9b4338a03862b5231c97?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [python](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.80% <ø> (ø)` | | Carriedforward from [ed22158](https://app.codecov.io/gh/apache/texera/commit/ed22158534c1ae503fea9b4338a03862b5231c97?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/5877/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.69% <ø> (ø)` | | |
   
   *This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/5877?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5877:
URL: https://github.com/apache/texera/pull/5877#issuecomment-4763878526

   <!-- texera-reviewer-suggestion -->
   ### Automated Reviewer Suggestions
   
   Based on the `git blame` history of the changed files, we recommend the following reviewers:
   
   - No candidates found from `git blame` history.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #5877:
URL: https://github.com/apache/texera/pull/5877#discussion_r3449345284


##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFOpDescSpec.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFOpDesc.operatorInfo" should
+    "advertise the name, R group, and a default 1-in/1-out shape" in {
+    val info = (new RUDFOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts should have length 1
+    info.outputPorts should have length 1
+  }
+
+  "RUDFOpDesc" should "default code/workers/useTupleAPI/retainInputColumns" in {
+    val d = new RUDFOpDesc
+    d.code shouldBe ""
+    d.workers shouldBe 1
+    d.useTupleAPI shouldBe false
+    d.retainInputColumns shouldBe false
+  }
+
+  "RUDFOpDesc.getPhysicalOp" should "wire OpExecWithCode and carry port identities" in {
+    val d = new RUDFOpDesc
+    d.code = "function(t) t"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "function(t) t"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }
+    physical.inputPorts.keySet shouldBe d.operatorInfo.inputPorts.map(_.id).toSet
+    physical.outputPorts.keySet shouldBe d.operatorInfo.outputPorts.map(_.id).toSet
+  }
+
+  it should "reject a non-positive worker count" in {
+    val d = new RUDFOpDesc
+    d.workers = 0
+    intercept[IllegalArgumentException] { d.getPhysicalOp(workflowId, executionId) }
+  }
+
+  "RUDFOpDesc schema propagation" should
+    "emit only the output columns when input columns are not retained (default)" in {
+    val d = new RUDFOpDesc
+    d.outputColumns = List(new Attribute("res", AttributeType.INTEGER))
+    val input = Schema().add(new Attribute("in", AttributeType.STRING))
+    d.getExternalOutputSchemas(Map(d.operatorInfo.inputPorts.head.id -> input)) shouldBe Map(
+      d.operatorInfo.outputPorts.head.id -> Schema().add(
+        new Attribute("res", AttributeType.INTEGER)
+      )
+    )
+  }

Review Comment:
   `RUDFOpDesc` schema propagation is only asserted for the default `retainInputColumns=false`. The operator’s implementation also supports `retainInputColumns=true` (input schema + outputColumns), and the PR description claims this behavior is pinned for RUDF as well, but there’s no test covering it.



##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFSourceOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFSourceOpDesc.operatorInfo" should
+    "advertise the 1-out R UDF source (no inputs, one output)" in {
+    val info = (new RUDFSourceOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "1-out R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts shouldBe empty
+    info.outputPorts should have length 1
+  }
+
+  "RUDFSourceOpDesc.sourceSchema" should "be empty by default and reflect the configured columns" in {
+    (new RUDFSourceOpDesc).sourceSchema().getAttributes shouldBe empty
+    val d = new RUDFSourceOpDesc
+    d.columns = List(new Attribute("a", AttributeType.STRING))
+    d.sourceSchema() shouldBe Schema().add(new Attribute("a", AttributeType.STRING))
+  }
+
+  "RUDFSourceOpDesc.getPhysicalOp" should
+    "wire OpExecWithCode as a source op with one output port" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "data.frame(a=1)"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "data.frame(a=1)"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }
+    physical.outputPorts.keySet shouldBe d.operatorInfo.outputPorts.map(_.id).toSet
+  }
+
+  it should "reject a non-positive worker count" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "x"
+    d.workers = 0
+    intercept[IllegalArgumentException] { d.getPhysicalOp(workflowId, executionId) }
+  }
+
+  "RUDFSourceOpDesc" should "round-trip its config fields through the polymorphic base" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "f"
+    d.workers = 2
+    d.useTupleAPI = true
+    val restored = objectMapper.readValue(objectMapper.writeValueAsString(d), classOf[LogicalOp])
+    restored shouldBe a[RUDFSourceOpDesc]
+    val r = restored.asInstanceOf[RUDFSourceOpDesc]
+    r.code shouldBe "f"
+    r.workers shouldBe 2
+    r.useTupleAPI shouldBe true

Review Comment:
   The round-trip test doesn’t include the `columns` field, even though `RUDFSourceOpDesc.sourceSchema()` is derived from it. Adding `columns` to the round-trip assertion better pins the config contract for this source descriptor.



##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFOpDescSpec.scala:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFOpDesc.operatorInfo" should
+    "advertise the name, R group, and a default 1-in/1-out shape" in {
+    val info = (new RUDFOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts should have length 1
+    info.outputPorts should have length 1
+  }
+
+  "RUDFOpDesc" should "default code/workers/useTupleAPI/retainInputColumns" in {
+    val d = new RUDFOpDesc
+    d.code shouldBe ""
+    d.workers shouldBe 1
+    d.useTupleAPI shouldBe false
+    d.retainInputColumns shouldBe false
+  }
+
+  "RUDFOpDesc.getPhysicalOp" should "wire OpExecWithCode and carry port identities" in {
+    val d = new RUDFOpDesc
+    d.code = "function(t) t"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "function(t) t"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }

Review Comment:
   This test matches `OpExecWithCode(code, _)`, so it won’t fail if the operator type accidentally changes (e.g., from the default `"r-table"`). Since `RUDFOpDesc` selects the OpExec type based on `useTupleAPI`, it’s worth pinning the default to avoid regressions.



##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/udf/r/RUDFSourceOpDescSpec.scala:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.udf.r
+
+import org.apache.texera.amber.core.executor.OpExecWithCode
+import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
+import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
+import org.apache.texera.amber.operator.LogicalOp
+import org.apache.texera.amber.operator.metadata.OperatorGroupConstants
+import org.apache.texera.amber.util.JSONUtils.objectMapper
+import org.scalatest.flatspec.AnyFlatSpec
+import org.scalatest.matchers.should.Matchers
+
+class RUDFSourceOpDescSpec extends AnyFlatSpec with Matchers {
+
+  private val workflowId = WorkflowIdentity(1L)
+  private val executionId = ExecutionIdentity(1L)
+
+  "RUDFSourceOpDesc.operatorInfo" should
+    "advertise the 1-out R UDF source (no inputs, one output)" in {
+    val info = (new RUDFSourceOpDesc).operatorInfo
+    info.userFriendlyName shouldBe "1-out R UDF"
+    info.operatorGroupName shouldBe OperatorGroupConstants.R_GROUP
+    info.inputPorts shouldBe empty
+    info.outputPorts should have length 1
+  }
+
+  "RUDFSourceOpDesc.sourceSchema" should "be empty by default and reflect the configured columns" in {
+    (new RUDFSourceOpDesc).sourceSchema().getAttributes shouldBe empty
+    val d = new RUDFSourceOpDesc
+    d.columns = List(new Attribute("a", AttributeType.STRING))
+    d.sourceSchema() shouldBe Schema().add(new Attribute("a", AttributeType.STRING))
+  }
+
+  "RUDFSourceOpDesc.getPhysicalOp" should
+    "wire OpExecWithCode as a source op with one output port" in {
+    val d = new RUDFSourceOpDesc
+    d.code = "data.frame(a=1)"
+    val physical = d.getPhysicalOp(workflowId, executionId)
+    physical.opExecInitInfo match {
+      case OpExecWithCode(code, _) => code shouldBe "data.frame(a=1)"
+      case other                   => fail(s"expected OpExecWithCode, got $other")
+    }

Review Comment:
   This test matches `OpExecWithCode(code, _)`, so it won’t catch regressions where the R operator type changes. Since `RUDFSourceOpDesc` chooses `"r-table"` vs `"r-tuple"` based on `useTupleAPI`, consider asserting the default operator type here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(workflow-operator): add unit test coverage for Java and R UDF operator descriptors [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5877:
URL: https://github.com/apache/texera/pull/5877#issuecomment-4763894163

   <!-- texera-benchmarks-comment -->
   ## ⚠️ Benchmark changes need a look
   
   🟢 0 better · 🔴 8 worse · ⚪ 7 noise (<±5%) · 0 without baseline
   
   > Compared against main `ed22158` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
   
   [Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/27923253118)
   
   | | config | throughput | MB/s | latency | max Δ latest / 7d |
   |---|---|---:|---:|---:|---:|
   | 🔴 | bs=10 sw=10 sl=64 | 382 | 0.233 | 25,421/37,199/37,199 us | 🔴 -5.7% / 🔴 -7.1% |
   | 🔴 | bs=100 sw=10 sl=64 | 794 | 0.485 | 123,223/160,478/160,478 us | 🔴 +14.0% / 🔴 +14.8% |
   | ⚪ | bs=1000 sw=10 sl=64 | 924 | 0.564 | 1,087,932/1,138,448/1,138,448 us | ⚪ within ±5% / 🔴 +11.8% |
   
   <details><summary>Baseline details</summary>
   
   Latest main `ed22158` from same runner
   
   | config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
   |---|---|---:|---:|---:|---:|---:|
   | bs=10 sw=10 sl=64 | throughput | 382 tuples/sec | 404 tuples/sec | 410.82 tuples/sec | -5.4% | -7.0% |
   | bs=10 sw=10 sl=64 | MB/s | 0.233 MB/s | 0.247 MB/s | 0.251 MB/s | -5.7% | -7.1% |
   | bs=10 sw=10 sl=64 | p50 | 25,421 us | 25,175 us | 23,785 us | +1.0% | +6.9% |
   | bs=10 sw=10 sl=64 | p95 | 37,199 us | 35,421 us | 34,980 us | +5.0% | +6.3% |
   | bs=10 sw=10 sl=64 | p99 | 37,199 us | 35,421 us | 34,980 us | +5.0% | +6.3% |
   | bs=100 sw=10 sl=64 | throughput | 794 tuples/sec | 841 tuples/sec | 891.94 tuples/sec | -5.6% | -11.0% |
   | bs=100 sw=10 sl=64 | MB/s | 0.485 MB/s | 0.514 MB/s | 0.544 MB/s | -5.6% | -10.9% |
   | bs=100 sw=10 sl=64 | p50 | 123,223 us | 117,680 us | 112,277 us | +4.7% | +9.7% |
   | bs=100 sw=10 sl=64 | p95 | 160,478 us | 140,741 us | 139,802 us | +14.0% | +14.8% |
   | bs=100 sw=10 sl=64 | p99 | 160,478 us | 140,741 us | 139,802 us | +14.0% | +14.8% |
   | bs=1000 sw=10 sl=64 | throughput | 924 tuples/sec | 925 tuples/sec | 1,041 tuples/sec | -0.1% | -11.2% |
   | bs=1000 sw=10 sl=64 | MB/s | 0.564 MB/s | 0.565 MB/s | 0.635 MB/s | -0.2% | -11.2% |
   | bs=1000 sw=10 sl=64 | p50 | 1,087,932 us | 1,084,837 us | 972,714 us | +0.3% | +11.8% |
   | bs=1000 sw=10 sl=64 | p95 | 1,138,448 us | 1,125,679 us | 1,023,057 us | +1.1% | +11.3% |
   | bs=1000 sw=10 sl=64 | p99 | 1,138,448 us | 1,125,679 us | 1,023,057 us | +1.1% | +11.3% |
   
   </details>
   
   <details><summary>Raw CSV</summary>
   
   ```csv
   config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
   0,10,10,64,20,523.80,200,128000,382,0.233,25421.21,37198.70,37198.70
   1,100,10,64,20,2518.90,2000,1280000,794,0.485,123223.43,160478.29,160478.29
   2,1000,10,64,20,21642.36,20000,12800000,924,0.564,1087932.27,1138447.68,1138447.68
   ```
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org