You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/08/31 02:14:27 UTC
[PR] fix(pyamber): avoid duplicate storage writers [texera]
carloea2 opened a new pull request, #8200:
URL: https://github.com/apache/texera/pull/8200
### What changes were proposed in this PR?
Initialize output-port storage writers under the same first-registration guard that protects the port schema. Replaying an existing port assignment now leaves its original tracked writer pair intact.
Before: one logical port remained registered, but each repeated assignment started another result and state writer pair.
After: the first assignment initializes the port and its writers; repeated assignments do neither.
### Any related issues, documentation, discussions?
Closes #8199
### How was this PR tested?
Regression test first:
$env:PYTHONDONTWRITEBYTECODE='1'; C:\Users\carlo\texera\texera\.venv312\Scripts\python.exe -c "import sys,pytest; sys.path[:0]=[r'C:\Users\carlo\texera\texera-worktrees\fix-pyamber-duplicate-output-writers\amber\src\main\python',r'C:\Users\carlo\texera\texera\amber\src\main\python']; raise SystemExit(pytest.main([r'amber\src\test\python\core\architecture\packaging\test_output_manager.py','-q','-p','no:cacheprovider']))"
Before the source change: 38 passed and 1 failed. The failure showed two setup calls for one registered port.
After the fix, the OutputManager and port storage-writer lifecycle suites reported 44 passed:
$env:PYTHONDONTWRITEBYTECODE='1'; C:\Users\carlo\texera\texera\.venv312\Scripts\python.exe -c "import sys,pytest; sys.path[:0]=[r'C:\Users\carlo\texera\texera-worktrees\fix-pyamber-duplicate-output-writers\amber\src\main\python',r'C:\Users\carlo\texera\texera\amber\src\main\python']; raise SystemExit(pytest.main([r'amber\src\test\python\core\architecture\packaging\test_output_manager.py',r'amber\src\test\python\core\storage\runnables\test_port_storage_writer.py','-q','-p','no:cacheprovider']))"
C:\Users\carlo\texera\texera\.venv312\Scripts\ruff.exe check amber/src/main/python amber/src/test/python
C:\Users\carlo\texera\texera\.venv312\Scripts\ruff.exe format --check amber/src/main/python amber/src/test/python
Result: all checks passed and 213 files were already formatted.
The production add_output_port probe now reports one registered port and one storage setup call after the assignment is repeated.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex, GPT-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): avoid duplicate storage writers [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8200:
URL: https://github.com/apache/texera/pull/8200#issuecomment-5472897686
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 4 better · 🔴 5 worse · ⚪ 6 noise (<±5%) · 0 without baseline
> Compared against main `70c2114` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33350067544)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 510 | 0.311 | 17,410/30,653/30,653 us | 🔴 +20.1% / 🔴 +107.3% |
| 🟢 | bs=100 sw=10 sl=64 | 1,196 | 0.73 | 82,949/120,359/120,359 us | 🟢 -9.6% / 🔴 +19.2% |
| 🟢 | bs=1000 sw=10 sl=64 | 1,423 | 0.869 | 703,351/772,537/772,537 us | 🟢 -5.9% / 🟢 +27.2% |
<details><summary>Baseline details</summary>
Latest main `70c2114` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 510 tuples/sec | 573 tuples/sec | 844.57 tuples/sec | -11.0% | -39.6% |
| bs=10 sw=10 sl=64 | MB/s | 0.311 MB/s | 0.35 MB/s | 0.515 MB/s | -11.1% | -39.7% |
| bs=10 sw=10 sl=64 | p50 | 17,410 us | 16,526 us | 11,837 us | +5.4% | +47.1% |
| bs=10 sw=10 sl=64 | p95 | 30,653 us | 25,529 us | 14,788 us | +20.1% | +107.3% |
| bs=10 sw=10 sl=64 | p99 | 30,653 us | 25,529 us | 17,651 us | +20.1% | +73.7% |
| bs=100 sw=10 sl=64 | throughput | 1,196 tuples/sec | 1,151 tuples/sec | 1,084 tuples/sec | +3.9% | +10.3% |
| bs=100 sw=10 sl=64 | MB/s | 0.73 MB/s | 0.702 MB/s | 0.662 MB/s | +4.0% | +10.3% |
| bs=100 sw=10 sl=64 | p50 | 82,949 us | 85,645 us | 94,505 us | -3.1% | -12.2% |
| bs=100 sw=10 sl=64 | p95 | 120,359 us | 133,126 us | 100,941 us | -9.6% | +19.2% |
| bs=100 sw=10 sl=64 | p99 | 120,359 us | 133,126 us | 108,642 us | -9.6% | +10.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,423 tuples/sec | 1,417 tuples/sec | 1,119 tuples/sec | +0.4% | +27.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.869 MB/s | 0.865 MB/s | 0.683 MB/s | +0.5% | +27.2% |
| bs=1000 sw=10 sl=64 | p50 | 703,351 us | 703,226 us | 924,404 us | +0.0% | -23.9% |
| bs=1000 sw=10 sl=64 | p95 | 772,537 us | 820,846 us | 967,785 us | -5.9% | -20.2% |
| bs=1000 sw=10 sl=64 | p99 | 772,537 us | 820,846 us | 995,630 us | -5.9% | -22.4% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,392.37,200,128000,510,0.311,17410.22,30653.12,30653.12
1,100,10,64,20,1672.31,2000,1280000,1196,0.730,82949.34,120359.46,120359.46
2,1000,10,64,20,14053.16,20000,12800000,1423,0.869,703350.51,772537.21,772537.21
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): avoid duplicate storage writers [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8200:
URL: https://github.com/apache/texera/pull/8200#issuecomment-5472864446
<!-- backport-auto-label-report -->
### Backport auto-label report
This `fix:` PR was checked against each actively-supported release branch. `release/*` labels drive the post-merge backport, so add or remove one to change where this fix lands.
| Release branch | Analysis |
| --- | --- |
| ⚠️ `release/v1.3` | Not labeled automatically — none of the files this PR modifies exist on this branch (`amber/src/main/python/core/architecture/packaging/output_manager.py`, `amber/src/test/python/core/architecture/packaging/test_output_manager.py`). The fix may target code that isn't on this release, or the files were moved/renamed after the branch was cut. **Please check and add `release/v1.3` by hand if this fix should be backported here.** |
| ✅ `release/v1.2` | Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7. |
_[Auto-label run](https://github.com/apache/texera/actions/runs/33350055079)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(pyamber): avoid duplicate storage writers [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #8200:
URL: https://github.com/apache/texera/pull/8200#issuecomment-6045434095
Closing this duplicate in favor of #8278.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): avoid duplicate storage writers [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8200:
URL: https://github.com/apache/texera/pull/8200#issuecomment-5472890869
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8200?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 93.94%. Comparing base ([`70c2114`](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`55c3e9e`](https://app.codecov.io/gh/apache/texera/commit/55c3e9e0a4123a2dc4bcada42fee5eb0be606a0f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8200 +/- ##
=========================================
Coverage 93.94% 93.94%
Complexity 4759 4759
=========================================
Files 1185 1185
Lines 48069 48069
Branches 5359 5359
=========================================
Hits 45159 45159
Misses 1480 1480
Partials 1430 1430
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.99% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.86% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.91% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.49% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.31% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.87% <100.00%> (ø)` | | |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8200/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.19% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8200?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(pyamber): avoid duplicate storage writers [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8200:
URL: https://github.com/apache/texera/pull/8200#issuecomment-5472864404
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@eugenegujing`
You can notify them by mentioning `@aglinxinyuan`, `@eugenegujing` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(pyamber): avoid duplicate storage writers [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 closed pull request #8200: feat(pyamber): avoid duplicate storage writers
URL: https://github.com/apache/texera/pull/8200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org