You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 21:45:43 UTC
[GitHub] [beam] damccorm opened a new issue, #21164: Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true
damccorm opened a new issue, #21164:
URL: https://github.com/apache/beam/issues/21164
BeamFnDataInboundObserver drops the bytes instead of passing it forward on the last message: https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java#L58
data_plane.py drops the timers payload but seems to do the right thing for data:
https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/python/apache_beam/runners/worker/data_plane.py#L343
data_mgr.go drops the data payload (timers don't seem to be supported):
https://github.com/apache/beam/blob/cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec/sdks/go/pkg/beam/core/runtime/harness/datamgr.go#L297
Imported from Jira [BEAM-13142](https://issues.apache.org/jira/browse/BEAM-13142). Original Jira may contain additional context.
Reported by: lcwik.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #21164: Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #21164:
URL: https://github.com/apache/beam/issues/21164#issuecomment-1496102983
@lostluck should this be closed or is it just fixed for Go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #21164: Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #21164:
URL: https://github.com/apache/beam/issues/21164#issuecomment-1496261747
Ah good catch.
Java is correct: Bytes are not dropped for either [data](https://github.com/apache/beam/blob/master/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java#L154) or [timers](https://github.com/apache/beam/blob/master/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java#L188).
Python remains how it was on first filing:
Data is always propagated, timers are only propagated if not last in the In Memory channel:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/data_plane.py#L363
The GRPC channel just drops both.
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/data_plane.py#L516
I'm still unfamiliar with timers in that respect though (ask me again after I add them to Prism). I'd expect the intent behind this issue is to allow runners to avoid extra messages, which does matter for streaming. That benefit is superseded if the SDK supports the "Elements in ProcessBundle" extensions though, which if the Python SDK supports, then this issue may be obselete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck closed issue #21164: Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck closed issue #21164: Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true
URL: https://github.com/apache/beam/issues/21164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Ensure that the payload is consumed on Elements.Data and Elements.Timers when is_last = true [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #21164:
URL: https://github.com/apache/beam/issues/21164#issuecomment-1839540668
It has been determined that Go SDK was breaching FnAPI contract in how it implemented timer writers. By not eagerly creating them, we never had a clear is_last signal for timers for the runner to key off of.
That specific issue is filed in #29605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org