You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Amit Sela (JIRA)" <ji...@apache.org> on 2017/03/30 20:10:41 UTC
[jira] [Closed] (BEAM-696) Document: Side-Inputs non-deterministic
with merging main-input windows
[ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amit Sela closed BEAM-696.
--------------------------
Resolution: Not A Problem
Fix Version/s: Not applicable
Resolved by providing better documentation. See comments for details.
> Document: Side-Inputs non-deterministic with merging main-input windows
> -----------------------------------------------------------------------
>
> Key: BEAM-696
> URL: https://issues.apache.org/jira/browse/BEAM-696
> Project: Beam
> Issue Type: Task
> Components: beam-model
> Reporter: Ben Chambers
> Assignee: Amit Sela
> Fix For: Not applicable
>
>
> Side-Inputs are non-deterministic for several reasons:
> 1. Because they depend on triggering of the side-input (this is acceptable because triggers are by their nature non-deterministic).
> 2. They depend on the current state of the main-input window in order to lookup the side-input. This means that with merging
> 3. Any runner optimizations that affect when the side-input is looked up may cause problems with either or both of these.
> This issue focuses on #2 -- the non-determinism of side-inputs that execute within a Merging WindowFn.
> Possible solution would be to defer running anything that looks up the side-input until we need to extract an output, and using the main-window at that point. Specifically, if the main-window is a MergingWindowFn, don't execute any kind of pre-combine, instead buffer all the inputs and combine later.
> This could still run into some non-determinism if there are triggers controlling when we extract output.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)