You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/11/03 17:26:01 UTC
[jira] [Updated] (BEAM-3304) Go triggering support
[ https://issues.apache.org/jira/browse/BEAM-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-3304:
--------------------------------
Labels: stale-assigned (was: )
> Go triggering support
> ---------------------
>
> Key: BEAM-3304
> URL: https://issues.apache.org/jira/browse/BEAM-3304
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Henning Rohde
> Assignee: Ritesh Ghorse
> Priority: P3
> Labels: stale-assigned
> Time Spent: 11h 40m
> Remaining Estimate: 0h
>
> `Add support for triggers.
> [https://beam.apache.org/documentation/programming-guide/#triggers]
> Triggers are special runner side behavior indicating how to handle data WRT the watermark and window. Commonly configuring the processing for “late data” and similar.
> These are not currently implemented for user use in the Go SDK. Reshuffle configures triggers, but it’s not accessible. A correct trigger implementation can at least re-implement Reshuffle in a user pipeline, rather than handled specially within the framework.
> * Requires extending the window package to be able to configure the various triggers.
> * Specifically being able to compose triggers as also permitted by the proto.
> ** [https://github.com/apache/beam/blob/6e7b1c44bc7275ee047afc059fd610cd3f4e5bee/model/pipeline/src/main/proto/beam_runner_api.proto#L1111]
> * Requires updating the graphx package translate.go to marshal (and unmarshal?) the triggers to and from Beam PipelineProto Windowing strategies.
> * Requires supporting triggers with the beam.WindowInto transform for user pipeline use as well as complete documentation on its use from the user side.
> ** [https://github.com/apache/beam/blob/6e7b1c44bc7275ee047afc059fd610cd3f4e5bee/sdks/go/pkg/beam/windowing.go]
> * Panes need to be decoded, otherwise triggering will cause runtime errors: [https://lists.apache.org/thread.html/r94c42d2d116f6464cd6b689543e5e578edf8310bf7c6e48a0958a56c%40%3Cdev.beam.apache.org%3E]
> * Handle pane propagation and observation in the exec package, and in user dofns.
> ** Panes indicate whether data was on time or not, and similar facets which may be relevant for processing.
> ** Might simply extend the existing window interface.
>
> Similar to windowing, many of the same places as https://issues.apache.org/jira/browse/BEAM-11100 need to be modified.
> At simplest though, it's mostly a runner side construction, with less concern on the exec side, and generally much simpler.
> Appropriate integration tests against portable runners must be implemented:
> [https://github.com/apache/beam/tree/master/sdks/go/test/integration/primitives]
> And optionally add support for the configurable triggers to the the Go Direct Runner. However, the results must be compared and validated against a semantically correct runner like the python portable runner first. At minimum, the Go Direct Runner should be made aware of triggers and produce a coherent error whenever there's a trigger it can't deal with.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)