You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/08/31 17:39:43 UTC
[PR] fix(pyamber): report batch processing errors [texera]
carloea2 opened a new pull request, #8298:
URL: https://github.com/apache/texera/pull/8298
### What changes were proposed in this PR?
Report per-row MainLoop failures through the worker exception path, enter exception pause, and stop consuming the affected batch.
Before: the worker logged the failure, continued later rows, and could finish with missing output.
After: the coordinator receives the error and the batch stops at the failed row.
### Any related issues, documentation, discussions?
Closes #8297
### How was this PR tested?
Added a regression covering the failure report, exception pause, and stopped batch consumption.
```powershell
$env:PYTHONDONTWRITEBYTECODE='1'; C:\Users\carlo\texera\texera\.venv312\Scripts\python.exe -c "import sys,pytest; sys.path[:0]=[r'C:\Users\carlo\texera\texera-worktrees\investigate-bug78\amber\src\main\python',r'C:\Users\carlo\texera\texera\amber\src\main\python']; raise SystemExit(pytest.main([r'amber/src/test/python/core/runnables/test_main_loop.py::TestMainLoop::test_failing_element_reports_exception_and_stops_batch','-p','no:cacheprovider','-q']))"
C:\Users\carlo\texera\texera\.venv312\Scripts\ruff.exe check amber/src/main/python/core/runnables/main_loop.py amber/src/test/python/core/runnables/test_main_loop.py
C:\Users\carlo\texera\texera\.venv312\Scripts\ruff.exe format --check amber/src/main/python/core/runnables/main_loop.py amber/src/test/python/core/runnables/test_main_loop.py
```
The focused regression passed. The full MainLoop test file was also run: 36 tests passed and 8 pre-existing integration cases failed during executor initialization because the shared generated protocol files do not yet contain `loop_start_port_uris`.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): report batch processing errors [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8298:
URL: https://github.com/apache/texera/pull/8298#issuecomment-5482222558
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 2 worse · ⚪ 13 noise (<±5%) · 0 without baseline
> Compared against main `cb0b613` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33420891999)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 408 | 0.249 | 23,548/34,606/34,606 us | 🔴 -9.1% / 🔴 +122.1% |
| ⚪ | bs=100 sw=10 sl=64 | 879 | 0.537 | 107,383/165,961/165,961 us | ⚪ within ±5% / 🔴 +54.8% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,052 | 0.642 | 944,531/1,043,907/1,043,907 us | ⚪ within ±5% / ⚪ within ±5% |
<details><summary>Baseline details</summary>
Latest main `cb0b613` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 408 tuples/sec | 448 tuples/sec | 787.98 tuples/sec | -8.9% | -48.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.249 MB/s | 0.274 MB/s | 0.481 MB/s | -9.1% | -48.2% |
| bs=10 sw=10 sl=64 | p50 | 23,548 us | 23,327 us | 12,593 us | +0.9% | +87.0% |
| bs=10 sw=10 sl=64 | p95 | 34,606 us | 34,519 us | 15,579 us | +0.3% | +122.1% |
| bs=10 sw=10 sl=64 | p99 | 34,606 us | 34,519 us | 18,786 us | +0.3% | +84.2% |
| bs=100 sw=10 sl=64 | throughput | 879 tuples/sec | 888 tuples/sec | 1,008 tuples/sec | -1.0% | -12.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.537 MB/s | 0.542 MB/s | 0.615 MB/s | -0.9% | -12.7% |
| bs=100 sw=10 sl=64 | p50 | 107,383 us | 103,988 us | 100,701 us | +3.3% | +6.6% |
| bs=100 sw=10 sl=64 | p95 | 165,961 us | 161,780 us | 107,244 us | +2.6% | +54.8% |
| bs=100 sw=10 sl=64 | p99 | 165,961 us | 161,780 us | 116,122 us | +2.6% | +42.9% |
| bs=1000 sw=10 sl=64 | throughput | 1,052 tuples/sec | 1,045 tuples/sec | 1,042 tuples/sec | +0.7% | +1.0% |
| bs=1000 sw=10 sl=64 | MB/s | 0.642 MB/s | 0.638 MB/s | 0.636 MB/s | +0.6% | +0.9% |
| bs=1000 sw=10 sl=64 | p50 | 944,531 us | 951,070 us | 981,959 us | -0.7% | -3.8% |
| bs=1000 sw=10 sl=64 | p95 | 1,043,907 us | 1,004,189 us | 1,023,080 us | +4.0% | +2.0% |
| bs=1000 sw=10 sl=64 | p99 | 1,043,907 us | 1,004,189 us | 1,051,697 us | +4.0% | -0.7% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,490.64,200,128000,408,0.249,23548.32,34605.84,34605.84
1,100,10,64,20,2275.30,2000,1280000,879,0.537,107383.37,165961.06,165961.06
2,1000,10,64,20,19019.27,20000,12800000,1052,0.642,944531.46,1043906.79,1043906.79
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): report batch processing errors [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8298:
URL: https://github.com/apache/texera/pull/8298#issuecomment-5482156596
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`
You can notify them by mentioning `@aglinxinyuan` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): report batch processing errors [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8298:
URL: https://github.com/apache/texera/pull/8298#issuecomment-5482153789
<!-- backport-auto-label-report -->
### Backport auto-label report
This `fix:` PR was checked against each actively-supported release branch. `release/*` labels drive the post-merge backport, so add or remove one to change where this fix lands.
| Release branch | Analysis |
| --- | --- |
| ⚠️ `release/v1.3` | Not labeled automatically — none of the files this PR modifies exist on this branch (`amber/src/main/python/core/runnables/main_loop.py`, `amber/src/test/python/core/runnables/test_main_loop.py`). The fix may target code that isn't on this release, or the files were moved/renamed after the branch was cut. **Please check and add `release/v1.3` by hand if this fix should be backported here.** |
| ✅ `release/v1.2` | Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7. |
_[Auto-label run](https://github.com/apache/texera/actions/runs/33420781352)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(pyamber): report batch processing errors [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #8298:
URL: https://github.com/apache/texera/pull/8298#issuecomment-5593842794
/request-review @aglinxinyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): report batch processing errors [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8298:
URL: https://github.com/apache/texera/pull/8298#issuecomment-5482204655
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8298?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 93.93%. Comparing base ([`50321e4`](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2c24b33`](https://app.codecov.io/gh/apache/texera/commit/2c24b33c5db95a2058734703227e464601870003?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8298 +/- ##
=========================================
Coverage 93.93% 93.93%
Complexity 4762 4762
=========================================
Files 1191 1191
Lines 48308 48310 +2
Branches 5377 5377
=========================================
+ Hits 45380 45382 +2
Misses 1494 1494
Partials 1434 1434
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.03% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.86% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.91% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.41% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.31% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.87% <100.00%> (+<0.01%)` | :arrow_up: | |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.19% <ø> (ø)` | | Carriedforward from [50321e4](https://app.codecov.io/gh/apache/texera/commit/50321e403c82df299a13deb50a7f9849dd93bdba?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8298?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(pyamber): report batch processing errors [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #8298:
URL: https://github.com/apache/texera/pull/8298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org