[jira] [Comment Edited] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

["Pei He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563569#comment-15563569 ] 

Pei He edited comment on BEAM-696 at 10/10/16 9:26 PM:
-------------------------------------------------------

re: to "Sessions is not allowed in side inputs"
There are two WindowFns involved, one from main input PCollection and the other from side input view.
For example, you can have Sessions in the main input window, and looks up values in a fix windowed side input.
Sessions.java#L84 is saying side inputs cannot be windowed by Sessions.

This jira [BEAM-696] is about when the main input is windowed by Sessions, how Combine with side input should be executed.


was (Author: peihe0@gmail.com):
re: to "Sessions is not allowed in side inputs"
There are two WindowFns involved, one from main input PCollection and the other from side input view.
Now, you can have Sessions in the main input window, and looks up values in a fix windowed side input.
Sessions.java#L84 is saying side inputs cannot be windowed by Sessions.

This jira [BEAM-696] is about when the main input is windowed by Sessions, how Combine with side input should be executed.

> Side-Inputs non-deterministic with merging main-input windows
> -------------------------------------------------------------
>
>                 Key: BEAM-696
>                 URL: https://issues.apache.org/jira/browse/BEAM-696
>             Project: Beam
>          Issue Type: Bug
>          Components: beam-model
>            Reporter: Ben Chambers
>            Assignee: Pei He
>
> Side-Inputs are non-deterministic for several reasons:
> 1. Because they depend on triggering of the side-input (this is acceptable because triggers are by their nature non-deterministic).
> 2. They depend on the current state of the main-input window in order to lookup the side-input. This means that with merging
> 3. Any runner optimizations that affect when the side-input is looked up may cause problems with either or both of these.
> This issue focuses on #2 -- the non-determinism of side-inputs that execute within a Merging WindowFn.
> Possible solution would be to defer running anything that looks up the side-input until we need to extract an output, and using the main-window at that point. Specifically, if the main-window is a MergingWindowFn, don't execute any kind of pre-combine, instead buffer all the inputs and combine later.
> This could still run into some non-determinism if there are triggers controlling when we extract output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)