You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/09/30 21:11:00 UTC
[PR] feat(workflow-compiler): preserve validation details [texera]
carloea2 opened a new pull request, #8794:
URL: https://github.com/apache/texera/pull/8794
### What changes were proposed in this PR?
Collect internal stage errors before hiding internal schemas. Preserve the originating validation detail instead of reporting only a later missing-schema error. Valid output schemas and invalid-workflow rejection stay unchanged.
Classified as feat because this improves the user-facing diagnostic rather than restoring established functionality.
### Any related issues, documentation, discussions?
Closes #8793
### How was this PR tested?
Local candidate checks: 26 focused and 66 compiler-module cases passed. Three negative cases failed before implementation. Module scalafix and repository Scala formatting passed.
```text
sbt "WorkflowCompiler/testOnly org.apache.texera.common.compiler.WorkflowCompilerSpec"
sbt "WorkflowCompiler/test"
sbt "WorkflowCompiler/scalafix --check" "WorkflowCompiler/Test/scalafix --check" scalafmtCheckAll
```
On localhost, compile the same legacy SUM graph with a missing column: the response retains the column error after this change. Its valid counterpart keeps the same schema. Chrome at 1707 by 979 also verified the combined example below.
These local checks preceded today's rebase onto upstream main 98d3d4b1ec. Candidate changes are unchanged. Remote CI and the full operating-system matrix are not claimed.
Combined example uses the separate conditional-Aggregate candidate. That feature is not part of this compiler PR; independent reproduction used legacy SUM.
| Generic diagnostic | Originating diagnostic |
| --- | --- |
|  |  |
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (exact authoring version not recorded).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiler): preserve validation details [Social Sciences Domain Request] [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8794:
URL: https://github.com/apache/texera/pull/8794#issuecomment-5920164547
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8794?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 92.58%. Comparing base ([`98d3d4b`](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cdf2d73`](https://app.codecov.io/gh/apache/texera/commit/cdf2d7334ab250a4ab3ab9f1af153c183c550a63?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:white_check_mark: All tests successful. No failed tests found.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8794 +/- ##
=========================================
Coverage 92.58% 92.58%
Complexity 4963 4963
=========================================
Files 1242 1242
Lines 52885 52885
Branches 6546 6546
=========================================
Hits 48965 48965
Misses 2291 2291
Partials 1629 1629
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.38% <ø> (ø)` | | |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.16% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.05% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.41% <ø> (ø)` | | |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.37% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.55% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.73% <ø> (ø)` | | |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.56% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8794?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiler): preserve validation details [Social Sciences Domain Request] [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8794:
URL: https://github.com/apache/texera/pull/8794#issuecomment-5920157888
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 4 better · 🔴 5 worse · ⚪ 6 noise (<±5%) · 0 without baseline
> Compared against main `98d3d4b` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/36777747274)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 510 | 0.311 | 19,206/23,368/23,368 us | 🟢 -19.4% / 🔴 +54.8% |
| 🟢 | bs=100 sw=10 sl=64 | 1,192 | 0.727 | 83,976/102,781/102,781 us | 🟢 -20.7% / 🟢 -15.3% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,406 | 0.858 | 712,477/777,434/777,434 us | 🔴 +8.6% / 🟢 +29.3% |
<details><summary>Baseline details</summary>
Latest main `98d3d4b` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 510 tuples/sec | 547 tuples/sec | 818.62 tuples/sec | -6.8% | -37.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.311 MB/s | 0.334 MB/s | 0.5 MB/s | -6.9% | -37.8% |
| bs=10 sw=10 sl=64 | p50 | 19,206 us | 17,754 us | 12,404 us | +8.2% | +54.8% |
| bs=10 sw=10 sl=64 | p95 | 23,368 us | 29,004 us | 15,787 us | -19.4% | +48.0% |
| bs=10 sw=10 sl=64 | p99 | 23,368 us | 29,004 us | 18,928 us | -19.4% | +23.5% |
| bs=100 sw=10 sl=64 | throughput | 1,192 tuples/sec | 1,145 tuples/sec | 1,058 tuples/sec | +4.1% | +12.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.727 MB/s | 0.699 MB/s | 0.646 MB/s | +4.0% | +12.6% |
| bs=100 sw=10 sl=64 | p50 | 83,976 us | 81,707 us | 99,149 us | +2.8% | -15.3% |
| bs=100 sw=10 sl=64 | p95 | 102,781 us | 129,653 us | 106,165 us | -20.7% | -3.2% |
| bs=100 sw=10 sl=64 | p99 | 102,781 us | 129,653 us | 114,997 us | -20.7% | -10.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,406 tuples/sec | 1,437 tuples/sec | 1,088 tuples/sec | -2.2% | +29.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.858 MB/s | 0.877 MB/s | 0.664 MB/s | -2.2% | +29.2% |
| bs=1000 sw=10 sl=64 | p50 | 712,477 us | 700,818 us | 971,066 us | +1.7% | -26.6% |
| bs=1000 sw=10 sl=64 | p95 | 777,434 us | 715,767 us | 1,011,680 us | +8.6% | -23.2% |
| bs=1000 sw=10 sl=64 | p99 | 777,434 us | 715,767 us | 1,037,719 us | +8.6% | -25.1% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,392.29,200,128000,510,0.311,19206.16,23367.83,23367.83
1,100,10,64,20,1678.09,2000,1280000,1192,0.727,83976.41,102780.80,102780.80
2,1000,10,64,20,14227.13,20000,12800000,1406,0.858,712476.60,777434.35,777434.35
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiler): preserve validation details [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8794:
URL: https://github.com/apache/texera/pull/8794#issuecomment-5919977184
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@Yicong-Huang`, `@eugenegujing`
You can notify them by mentioning `@Yicong-Huang`, `@eugenegujing` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org