You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/09/30 21:11:00 UTC

[PR] feat(workflow-compiler): preserve validation details [texera]

carloea2 opened a new pull request, #8794:
URL: https://github.com/apache/texera/pull/8794

   ### What changes were proposed in this PR?
   
   Collect internal stage errors before hiding internal schemas. Preserve the originating validation detail instead of reporting only a later missing-schema error. Valid output schemas and invalid-workflow rejection stay unchanged.
   
   Classified as feat because this improves the user-facing diagnostic rather than restoring established functionality.
   
   ### Any related issues, documentation, discussions?
   
   Closes #8793
   
   ### How was this PR tested?
   
   Local candidate checks: 26 focused and 66 compiler-module cases passed. Three negative cases failed before implementation. Module scalafix and repository Scala formatting passed.
   
   ```text
   sbt "WorkflowCompiler/testOnly org.apache.texera.common.compiler.WorkflowCompilerSpec"
   sbt "WorkflowCompiler/test"
   sbt "WorkflowCompiler/scalafix --check" "WorkflowCompiler/Test/scalafix --check" scalafmtCheckAll
   ```
   
   On localhost, compile the same legacy SUM graph with a missing column: the response retains the column error after this change. Its valid counterpart keeps the same schema. Chrome at 1707 by 979 also verified the combined example below.
   
   These local checks preceded today's rebase onto upstream main 98d3d4b1ec. Candidate changes are unchanged. Remote CI and the full operating-system matrix are not claimed.
   
   Combined example uses the separate conditional-Aggregate candidate. That feature is not part of this compiler PR; independent reproduction used legacy SUM.
   
   | Generic diagnostic | Originating diagnostic |
   | --- | --- |
   | ![Generic diagnostic](https://github.com/user-attachments/assets/01f6d4fc-1ec7-47b3-a817-3a38ad966321) | ![Originating diagnostic](https://github.com/user-attachments/assets/04fb1af9-9144-4afb-9205-079d1c9b7a37) |
   
   ### Was this PR authored or co-authored using generative AI tooling?
   
   Generated-by: OpenAI Codex (exact authoring version not recorded).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(workflow-compiler): preserve validation details [Social Sciences Domain Request] [texera]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8794:
URL: https://github.com/apache/texera/pull/8794#issuecomment-5920164547

   ## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8794?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 92.58%. Comparing base ([`98d3d4b`](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`cdf2d73`](https://app.codecov.io/gh/apache/texera/commit/cdf2d7334ab250a4ab3ab9f1af153c183c550a63?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :white_check_mark: All tests successful. No failed tests found.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##               main    #8794   +/-   ##
   =========================================
     Coverage     92.58%   92.58%           
     Complexity     4963     4963           
   =========================================
     Files          1242     1242           
     Lines         52885    52885           
     Branches       6546     6546           
   =========================================
     Hits          48965    48965           
     Misses         2291     2291           
     Partials       1629     1629           
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
   |---|---|---|---|
   | [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.38% <ø> (ø)` | | |
   | [agent-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.16% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [amber](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.05% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.41% <ø> (ø)` | | |
   | [config-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.37% <ø> (ø)` | | |
   | [file-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [frontend](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.55% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.73% <ø> (ø)` | | |
   | [pyamber](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.56% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8794/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (ø)` | | Carriedforward from [98d3d4b](https://app.codecov.io/gh/apache/texera/commit/98d3d4b1ec8b02f611801d035c190fa31f56fa47?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   *This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8794?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(workflow-compiler): preserve validation details [Social Sciences Domain Request] [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8794:
URL: https://github.com/apache/texera/pull/8794#issuecomment-5920157888

   <!-- texera-benchmarks-comment -->
   ## ⚠️ Benchmark changes need a look
   
   🟢 4 better · 🔴 5 worse · ⚪ 6 noise (<±5%) · 0 without baseline
   
   > Compared against main `98d3d4b` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
   
   [Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/36777747274)
   
   | | config | throughput | MB/s | latency | max Δ latest / 7d |
   |---|---|---:|---:|---:|---:|
   | 🔴 | bs=10 sw=10 sl=64 | 510 | 0.311 | 19,206/23,368/23,368 us | 🟢 -19.4% / 🔴 +54.8% |
   | 🟢 | bs=100 sw=10 sl=64 | 1,192 | 0.727 | 83,976/102,781/102,781 us | 🟢 -20.7% / 🟢 -15.3% |
   | 🔴 | bs=1000 sw=10 sl=64 | 1,406 | 0.858 | 712,477/777,434/777,434 us | 🔴 +8.6% / 🟢 +29.3% |
   
   <details><summary>Baseline details</summary>
   
   Latest main `98d3d4b` from same runner
   
   | config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
   |---|---|---:|---:|---:|---:|---:|
   | bs=10 sw=10 sl=64 | throughput | 510 tuples/sec | 547 tuples/sec | 818.62 tuples/sec | -6.8% | -37.7% |
   | bs=10 sw=10 sl=64 | MB/s | 0.311 MB/s | 0.334 MB/s | 0.5 MB/s | -6.9% | -37.8% |
   | bs=10 sw=10 sl=64 | p50 | 19,206 us | 17,754 us | 12,404 us | +8.2% | +54.8% |
   | bs=10 sw=10 sl=64 | p95 | 23,368 us | 29,004 us | 15,787 us | -19.4% | +48.0% |
   | bs=10 sw=10 sl=64 | p99 | 23,368 us | 29,004 us | 18,928 us | -19.4% | +23.5% |
   | bs=100 sw=10 sl=64 | throughput | 1,192 tuples/sec | 1,145 tuples/sec | 1,058 tuples/sec | +4.1% | +12.6% |
   | bs=100 sw=10 sl=64 | MB/s | 0.727 MB/s | 0.699 MB/s | 0.646 MB/s | +4.0% | +12.6% |
   | bs=100 sw=10 sl=64 | p50 | 83,976 us | 81,707 us | 99,149 us | +2.8% | -15.3% |
   | bs=100 sw=10 sl=64 | p95 | 102,781 us | 129,653 us | 106,165 us | -20.7% | -3.2% |
   | bs=100 sw=10 sl=64 | p99 | 102,781 us | 129,653 us | 114,997 us | -20.7% | -10.6% |
   | bs=1000 sw=10 sl=64 | throughput | 1,406 tuples/sec | 1,437 tuples/sec | 1,088 tuples/sec | -2.2% | +29.3% |
   | bs=1000 sw=10 sl=64 | MB/s | 0.858 MB/s | 0.877 MB/s | 0.664 MB/s | -2.2% | +29.2% |
   | bs=1000 sw=10 sl=64 | p50 | 712,477 us | 700,818 us | 971,066 us | +1.7% | -26.6% |
   | bs=1000 sw=10 sl=64 | p95 | 777,434 us | 715,767 us | 1,011,680 us | +8.6% | -23.2% |
   | bs=1000 sw=10 sl=64 | p99 | 777,434 us | 715,767 us | 1,037,719 us | +8.6% | -25.1% |
   
   </details>
   
   <details><summary>Raw CSV</summary>
   
   ```csv
   config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
   0,10,10,64,20,392.29,200,128000,510,0.311,19206.16,23367.83,23367.83
   1,100,10,64,20,1678.09,2000,1280000,1192,0.727,83976.41,102780.80,102780.80
   2,1000,10,64,20,14227.13,20000,12800000,1406,0.858,712476.60,777434.35,777434.35
   ```
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(workflow-compiler): preserve validation details [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8794:
URL: https://github.com/apache/texera/pull/8794#issuecomment-5919977184

   <!-- texera-reviewer-suggestion -->
   ### Automated Reviewer Suggestions
   
   Based on the `git blame` history of the changed files, we recommend the following reviewers:
   
   - **Contributors with relevant context:** `@Yicong-Huang`, `@eugenegujing`
     You can notify them by mentioning `@Yicong-Huang`, `@eugenegujing` in a comment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org