You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bvolpato (via GitHub)" <gi...@apache.org> on 2026/09/03 17:43:50 UTC
[PR] [Python] Fail bundles on state write errors [beam]
bvolpato opened a new pull request, #39992:
URL: https://github.com/apache/beam/pull/39992
A runner-reported append or clear failure can currently complete a Python bundle successfully because runtime state commits ignore `StateResponse.error`. A clear followed by an append also loses the clear response.
Check every pending write response before completing bag, set, and ordered-list state commits, and preserve both futures for clear-then-append. Regression tests cover failed appends and clears, multiple append chunks, set compaction, and successful commits. The new tests fail against the upstream implementation.
## Testing
From `sdks/python`, with the SDK and test dependencies installed:
```sh
python -m pytest apache_beam/runners/worker/bundle_processor_test.py apache_beam/runners/worker/sdk_worker_test.py -q
```
Python 3.12: 45 passed, plus 15 subtests passed. The new regression cases produced 15 failures against the unfixed implementation. YAPF 0.43.0, Ruff 0.15.22, and `git diff --check` pass for the changed files.
## Downsides
Previously hidden backend failures now fail the bundle and invoke the runner's existing failure/retry handling. The successful path adds no state RPCs. Retry uses the existing runner contract that failed state handlers must not reuse their cache token.
------------------------
- [x] Describe the bug and include a reproducible regression test; no separate Python issue is linked.
- [x] Update `CHANGES.md` with the behavior change.
- [ ] Apache Individual Contributor License Agreement, if required for this contribution.
See the [Contributor Guide](https://beam.apache.org/contribute) and [CI documentation](https://github.com/apache/beam/blob/master/CI.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail bundles on state write errors [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #39992:
URL: https://github.com/apache/beam/pull/39992#issuecomment-6139183216
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/39992?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 79.74%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`70f308b`](https://app.codecov.io/gh/apache/beam/commit/70f308bc254f6313f690041df14187791770c577?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 143 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #39992 +/- ##
=============================================
+ Coverage 56.16% 79.74% +23.57%
=============================================
Files 1124 487 -637
Lines 178336 82280 -96056
Branches 1489 0 -1489
=============================================
- Hits 100170 65613 -34557
+ Misses 75645 16667 -58978
+ Partials 2521 0 -2521
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/39992/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/39992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java](https://app.codecov.io/gh/apache/beam/pull/39992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/beam/pull/39992/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.74% <100.00%> (+0.18%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/39992?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail bundles on state write errors [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping merged PR #39992:
URL: https://github.com/apache/beam/pull/39992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail bundles on state write errors [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #39992:
URL: https://github.com/apache/beam/pull/39992#discussion_r4317493661
##########
sdks/python/apache_beam/runners/worker/bundle_processor.py:
##########
@@ -620,6 +620,17 @@ def __iter__(self) -> Iterator[Any]:
coder_impl.FastPrimitivesCoderImpl.register_iterable_like_type(_ConcatIterable)
+def _await_state_commit(futures) -> None:
+ # Resolve every write before reporting errors, including clears followed by
+ # appends. State futures return error responses rather than raising them.
+ responses = [future.get() for future in futures if future]
+ errors = [
+ response.error for response in responses if response and response.error
+ ]
+ if errors:
+ raise RuntimeError('\n'.join(errors))
Review Comment:
**🟢 Nit**: In `GrpcStateHandler._blocking_request` (`sdk_worker.py`), state responses with `response.error_reason == beam_fn_api_pb2.StateResponse.ErrorReason.CANCELLED` raise `WorkCancelledException` rather than `RuntimeError` so that `SdkHarness._execute` logs the cancellation at `INFO` level instead of `ERROR`. Consider checking `error_reason` here (and propagating it in `GlobalCachingStateHandler.extend`'s `_DeferredCall`) for consistency with state read error handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail bundles on state write errors [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #39992:
URL: https://github.com/apache/beam/pull/39992#issuecomment-6274574017
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] Fail bundles on state write errors [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39992:
URL: https://github.com/apache/beam/pull/39992#issuecomment-6244205019
Assigning reviewers:
R: @shunping for label python.
Backup reviewer: claudevdm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org