You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Mark Shields (JIRA)" <ji...@apache.org> on 2016/04/29 23:04:12 UTC

[jira] [Created] (BEAM-241) Not easy for runners to get late-data dropping

Mark Shields created BEAM-241:
---------------------------------

             Summary: Not easy for runners to get late-data dropping
                 Key: BEAM-241
                 URL: https://issues.apache.org/jira/browse/BEAM-241
             Project: Beam
          Issue Type: Bug
          Components: runner-core
            Reporter: Mark Shields
            Assignee: Frances Perry


Quite by accident realized the Flink runner delegates to GroupAlsoByWindowViaWindowSetDoFn for GBK, which in turn delegates to ReduceFnRunner. The latter now assumes no messages will arrive beyond the 'garbage collection' time of their target window(s).

The Dataflow runner interposes a LateDataDroppingDoFnRunner into the path so as to drop those too-late messages. That's done (I think) using DoFnRunners.createDefault.

I don't think the Flink runner does that.

We need a nice runner-friendly way of dealing with the too-late data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)