You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/07/27 03:48:06 UTC
[PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
aglinxinyuan opened a new pull request, #6916:
URL: https://github.com/apache/texera/pull/6916
### What changes were proposed in this PR?
**Root cause.** On every normal region termination, the coordinator sends `EndWorker` from *inside* the `portCompleted` handler, so the `ReturnInvocation` replying to that same `portCompleted` is always emitted **after** `EndWorker` on the same FIFO control channel:
```
Coordinator — portCompleted(W) handler, one synchronous continuation:
1. advanceRegionExecutions() ──> sends EndWorker to every worker (seq N on W's control channel)
2. returns EmptyReturn() ──> sends ReturnInvocation to W (seq N+1, same channel)
Worker W — processes EndWorker while ReturnInvocation(N+1) may already sit in its arrival queue:
old check: ANY queued element => IllegalStateException("worker still has unprocessed messages")
=> attempt 1 of 150 fails => 200 ms retry succeeds
```
Whether the trailing reply has landed when `EndHandler` runs is a thread-scheduling race, so healthy teardowns routinely fail the first termination attempt and log an alarming storm. A queued `ReturnInvocation` is not work: processing it only fulfills a promise for a request the worker already issued, and every worker-to-coordinator call discards its future (`DataProcessor.scala:179,187,224,312`), so no continuation can be lost.
**Fix.** Both workers now exclude queued `ReturnInvocation`s from the unprocessed-messages check. Everything that *is* work — `ControlInvocation`s, data, ECMs, timer-based controls, actor commands — still blocks termination, and the 150 × 200 ms retry/give-up contract is untouched.
| | Before | After |
|---|---|---|
| Normal teardown logs | 2 ERROR + 2 WARN + 2 stack traces | none |
| First `EndWorker` attempt | fails on a benign race | succeeds |
| Teardown latency | +200 ms retry | no retry |
| Real queued work | blocks termination | still blocks termination |
| Python: queued reply | silently **dropped** (destructive `get()` inside the warning f-string), then a bare `assert` passes; with 2+ elements the coordinator gets a blank-message `ControlError` | kept (non-destructive `peek()`); failure raises `RuntimeError` with the same message string as the Scala side |
Changes:
- **`EndHandler.scala`**: scan the arrival queue for pending *work* instead of testing bare emptiness; the warning names the payload type and channel instead of dumping the element; scaladoc no longer claims `EndWorker` is the last message a worker receives.
- **`end_worker_handler.py` / `internal_queue.py`**: new non-destructive `InternalQueue.peek()` and a reply-aware `has_unprocessed_work()` (an `AtomicInteger` work counter maintained in `put`/`get`, since the multi-queue offers no iteration; it also sees work in paused/backpressured sub-queues that `size()`/`get()` hide).
- **`RegionExecutionManager.scala`**: fixed the false doc ("This will be the last message each worker receives") and demoted the detail-free duplicate `"Error when terminating region X."` WARN to debug — `terminateWorkersWithRetry` logs the same failure with attempt context (WARN) or gives up loudly (ERROR), unchanged.
One deliberate behavior change to be aware of: a Python worker that used to "succeed" `EndWorker` by silently swallowing a queued message now honestly reports unprocessed work and lets the coordinator retry.
### Any related issues, documentation, discussions?
Related: #6796 — this addresses the `EndWorker` retry-storm item behaviorally (the expected race no longer fails the first attempt) rather than by adjusting log levels. Follow-up to the CI log-verbosity work in #6797.
### How was this PR tested?
- **`EndHandlerSpec`** (the three existing tests are untouched — control invocations and actor commands still fail `endWorker`): new regression tests for the exact CI payload (`ReturnInvocation(2, EmptyReturn())` queued → success), multiple queued replies, a `ControlInvocation` queued *behind* a reply (the scan must not stop at the head), and queue-size preservation on both paths. Written test-first: the two new success tests fail on `main`.
- **`test_end_worker_handler.py`** (new; this handler previously had zero Python coverage): success keeps the queued reply (`queue.size() == 1` afterwards — fails on `main`, where the element was destructively consumed), `RuntimeError` with the shared message string for queued invocations/data, and idempotence across coordinator retries.
- **`test_internal_queue.py`**: `peek()` (None on empty, non-destructive) and `has_unprocessed_work()` (ignores replies; counts invocations, data, ECMs, SYSTEM items; sees data in a disabled sub-queue; not counted for rejected elements).
- Untouched and green as evidence the retry contract is unchanged: `RegionExecutionManagerSpec`, `WorkflowExecutionManagerSpec`, `DPThreadSpec`, `WorkerSpec`, `AsyncRPCClientSpec` (43/43).
- Lint: `scalafmtCheck`, `scalafixAll --check`, `ruff check` + `ruff format --check` all pass.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #6916:
URL: https://github.com/apache/texera/pull/6916#discussion_r3662053730
##########
amber/src/main/python/core/models/internal_queue.py:
##########
@@ -67,12 +72,23 @@ def __init__(self):
self._queue_ids: Set[ChannelIdentity] = set()
self._queue_state: Set[InternalQueue.DisableType] = set()
self._lock = RLock()
+ # Number of queued elements that represent work (see _is_work). Counted at
+ # put/get time because the underlying multi-queue offers no iteration, and
+ # it must see elements in disabled (paused/backpressured) sub-queues too.
+ self._work_count = AtomicInteger()
def is_empty(self, key=None) -> bool:
return self._queue.is_empty(key)
def get(self) -> T:
- return self._queue.get()
+ item = self._queue.get()
+ if self._is_work(item):
+ self._work_count.dec()
+ return item
+
+ def peek(self) -> Optional[T]:
+ """Non-destructively return the next available item, or None when empty."""
+ return self._queue.peek()
Review Comment:
I feel I reviewed similar code before. Let's not make this queue peekable?
##########
amber/src/main/python/core/models/internal_queue.py:
##########
@@ -67,12 +72,23 @@ def __init__(self):
self._queue_ids: Set[ChannelIdentity] = set()
self._queue_state: Set[InternalQueue.DisableType] = set()
self._lock = RLock()
+ # Number of queued elements that represent work (see _is_work). Counted at
+ # put/get time because the underlying multi-queue offers no iteration, and
+ # it must see elements in disabled (paused/backpressured) sub-queues too.
+ self._work_count = AtomicInteger()
Review Comment:
this is a queue. let's not give it specific semantic like "work" to be done. A queue is a data structure for FIFO, please let it stay general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #6916:
URL: https://github.com/apache/texera/pull/6916#issuecomment-5099722503
Superseded by #6960. Per the review discussion on #6892, the root cause is the coordinator's send order — EndWorker was emitted from inside the portCompleted handler's continuation, before that handler's own reply, so the reply always trailed EndWorker on the same FIFO control channel. #6960 fixes that scenario directly (reply first, continuation after), which lets the worker-side unprocessed-messages check stay strict instead of being relaxed to tolerate the trailing ack as this PR did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan closed pull request #6916: fix(amber): treat a queued coordinator reply as processed work in EndWorker
URL: https://github.com/apache/texera/pull/6916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6916:
URL: https://github.com/apache/texera/pull/6916#issuecomment-5087061693
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@Yicong-Huang`, `@eugenegujing`
You can notify them by mentioning `@Yicong-Huang`, `@eugenegujing` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6916:
URL: https://github.com/apache/texera/pull/6916#issuecomment-5088439719
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/6916?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 78.95%. Comparing base ([`49f9e2c`](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`490f820`](https://app.codecov.io/gh/apache/texera/commit/490f820a63e0373a15689bf19462aecfd3c86d07?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #6916 +/- ##
=========================================
Coverage 78.95% 78.95%
+ Complexity 3795 3793 -2
=========================================
Files 1161 1161
Lines 46084 46094 +10
Branches 5110 5110
=========================================
+ Hits 36384 36394 +10
Misses 8077 8077
Partials 1623 1623
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.76% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.20% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `20.49% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `66.66% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.21% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.91% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.94% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `95.40% <100.00%> (+0.01%)` | :arrow_up: | |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/6916/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.14% <ø> (ø)` | | Carriedforward from [49f9e2c](https://app.codecov.io/gh/apache/texera/commit/49f9e2c414b116bf328bdb9154cc97dc4d1e7e9e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/6916?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(amber): treat a queued coordinator reply as processed work in EndWorker [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6916:
URL: https://github.com/apache/texera/pull/6916#issuecomment-5088462054
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 7 better · 🔴 1 worse · ⚪ 7 noise (<±5%) · 0 without baseline
> Compared against main `49f9e2c` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/30245590096)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🟢 | bs=10 sw=10 sl=64 | 402 | 0.245 | 23,911/34,743/34,743 us | 🟢 -12.9% / 🔴 +112.2% |
| 🔴 | bs=100 sw=10 sl=64 | 792 | 0.483 | 125,006/149,286/149,286 us | 🔴 +5.2% / 🔴 +36.6% |
| 🟢 | bs=1000 sw=10 sl=64 | 913 | 0.557 | 1,092,965/1,135,757/1,135,757 us | 🟢 -5.1% / 🔴 -8.6% |
<details><summary>Baseline details</summary>
Latest main `49f9e2c` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 402 tuples/sec | 369 tuples/sec | 759.72 tuples/sec | +8.9% | -47.1% |
| bs=10 sw=10 sl=64 | MB/s | 0.245 MB/s | 0.225 MB/s | 0.464 MB/s | +8.9% | -47.2% |
| bs=10 sw=10 sl=64 | p50 | 23,911 us | 27,465 us | 12,653 us | -12.9% | +89.0% |
| bs=10 sw=10 sl=64 | p95 | 34,743 us | 39,396 us | 16,371 us | -11.8% | +112.2% |
| bs=10 sw=10 sl=64 | p99 | 34,743 us | 39,396 us | 19,941 us | -11.8% | +74.2% |
| bs=100 sw=10 sl=64 | throughput | 792 tuples/sec | 826 tuples/sec | 966.78 tuples/sec | -4.1% | -18.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.483 MB/s | 0.504 MB/s | 0.59 MB/s | -4.2% | -18.1% |
| bs=100 sw=10 sl=64 | p50 | 125,006 us | 118,861 us | 103,654 us | +5.2% | +20.6% |
| bs=100 sw=10 sl=64 | p95 | 149,286 us | 155,008 us | 109,308 us | -3.7% | +36.6% |
| bs=100 sw=10 sl=64 | p99 | 149,286 us | 155,008 us | 116,369 us | -3.7% | +28.3% |
| bs=1000 sw=10 sl=64 | throughput | 913 tuples/sec | 914 tuples/sec | 997.95 tuples/sec | -0.1% | -8.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.557 MB/s | 0.558 MB/s | 0.609 MB/s | -0.2% | -8.6% |
| bs=1000 sw=10 sl=64 | p50 | 1,092,965 us | 1,087,281 us | 1,007,348 us | +0.5% | +8.5% |
| bs=1000 sw=10 sl=64 | p95 | 1,135,757 us | 1,196,827 us | 1,050,149 us | -5.1% | +8.2% |
| bs=1000 sw=10 sl=64 | p99 | 1,135,757 us | 1,196,827 us | 1,079,497 us | -5.1% | +5.2% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,497.68,200,128000,402,0.245,23911.47,34743.20,34743.20
1,100,10,64,20,2526.29,2000,1280000,792,0.483,125006.10,149285.52,149285.52
2,1000,10,64,20,21912.34,20000,12800000,913,0.557,1092964.81,1135757.19,1135757.19
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org