You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "scwhittle (via GitHub)" <gi...@apache.org> on 2025/03/25 13:47:48 UTC
[I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
scwhittle opened a new issue, #34419:
URL: https://github.com/apache/beam/issues/34419
### What happened?
Consider a pipeline where two steps A and B are fused and A -> B.
Runners such as Dataflow or the Fnapi harness wire up `output(...)` calls from A directly to `processElement(...)` calls to B. If B throws an exception during processing this exception then is just propagated up the stack and inside of A. If A doesn't catch this exception, it is caught at the runner level and results in a failed bundle.
However by raising the exception to A possible issues arise:
- the exception may interrupt A processing unexpectedly leaving it in some invalid state if it had bad error handling. Example could be not decrementing some static semaphore held in a final block.
- be caught within A if A itself has a try catch block for error handling. However the user may not realize that they are catching and supressing errors from B with this handling and it could lead to data loss as the processing of the input to B didn't complete.
Similar issues can occur if B itself outputs to a sink (such as fnapi data output or dataflow commit) which encounters an encoding error.
It seems like it would be preferable to catch such errors between DoFn boundaries instead of propagating them directly. Once detected, the process of the bundle could be cleanly stopped and the exceptions could be surfaced as failures to process the bundle.
This issue has been observed within Java execution on Dataflow but appears to be a common problem for the Java SDK and likely other sdks in which exceptions are thrown.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-3245181436
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-3269430694
I think that perhaps we could introduce this before beam3 behind a pipeline option and then change the option default for beam 3.
Currently `output` method from a dofn is wired up to the next dofn process element method via PCollectionConsumerRegistry [here](https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L198). I think that we could just modify where it currently logs and rethrows [here](https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java#L249) to additionally save the exception, not delegate future outputs and prevent the bundle from finishing successfully. If it has a way to truncate the input so the bundle fails faster that would also be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-4288655834
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-2751330476
Adding as 3.0.0 release milestone as this is potentially breaking behavior because existing pipelines may work due to relying on inline exception propagation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-3245211821
Still a feature request for 3.0, pinging to remove staleness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-3249993561
This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks.
* If this issue is solved, close it. If it is obsolete, close as not planned. If it is no longer necessary for Beam 3, move it out of the Milestone.
* If the work is pending, please be reminded that Milestone 1 is targeted for completion by Beam 2.72 cut (Feb 18, 2026)
* If this issue involves multiple milestones (e.g. remove a public API should first mark it as deprecated, scoped in Milestone 1; then finally remove it in Beam 3 release candidate, scoped in Milestone 3) and Milestone 1 portion has completed, consider move the Issue to Milestone 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #34419:
URL: https://github.com/apache/beam/issues/34419#issuecomment-3564251839
This sounds a proposed breaking change ideally making the switch between Beam 2->3. Currently stay at milestone 1 to track progress of
> we could introduce this before beam3 behind a pipeline option and then change the option default for beam 3.
We can move to milestone 3 when milestone 1 closes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org