[jira] [Assigned] (BEAM-3304) Go triggering support

["Eduardo Barrera (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-3304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eduardo Barrera reassigned BEAM-3304:
-------------------------------------

    Assignee: Eduardo Barrera

> Go triggering support
> ---------------------
>
>                 Key: BEAM-3304
>                 URL: https://issues.apache.org/jira/browse/BEAM-3304
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-go
>            Reporter: Henning Rohde
>            Assignee: Eduardo Barrera
>            Priority: P3
>
> `Add support for triggers.
> [https://beam.apache.org/documentation/programming-guide/#triggers] 
> Triggers are special runner side behavior indicating how to handle data WRT the watermark and window. Commonly configuring the processing for “late data” and similar.
> These are not currently implemented for user use in the Go SDK. Reshuffle configures triggers, but it’s not accessible. A correct trigger implementation can at least re-implement Reshuffle in a user pipeline, rather than handled specially within the framework.
>  * Requires extending the window package to be able to configure the various triggers.
>  * Specifically being able to compose triggers as also permitted by the proto.
>  ** [https://github.com/apache/beam/blob/6e7b1c44bc7275ee047afc059fd610cd3f4e5bee/model/pipeline/src/main/proto/beam_runner_api.proto#L1111] 
>  * Requires updating the graphx package translate.go to marshal (and unmarshal?) the triggers to and from Beam PipelineProto Windowing strategies.
>  * Requires supporting triggers with the beam.WindowInto transform for user pipeline use as well as complete documentation on its use from the user side.
>  ** [https://github.com/apache/beam/blob/6e7b1c44bc7275ee047afc059fd610cd3f4e5bee/sdks/go/pkg/beam/windowing.go] 
>  * Extension: Handle pane propagation and observation in the exec package, and in user dofns. 
>  ** Panes indicate whether data was on time or not, and similar facets which may be relevant for processing.
>  ** Might simply extend the existing window interface.
>  
> Similar to windowing,  many of the same places as https://issues.apache.org/jira/browse/BEAM-11100 need to be modified.
> At simplest though, it's mostly a runner side construction, with less concern on the exec side, and generally much simpler. 
> Appropriate integration tests against portable runners must be implemented:
> [https://github.com/apache/beam/tree/master/sdks/go/test/integration/primitives] 
> And optionally add support for the configurable triggers to the the Go Direct Runner. However, the results must be compared and validated against a semantically correct runner like the python portable runner first.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)