[jira] [Updated] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window

["Sam Whittle (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sam Whittle updated BEAM-3776:
------------------------------
    Affects Version/s: 2.4.0

> StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-3776
>                 URL: https://issues.apache.org/jira/browse/BEAM-3776
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>    Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0
>            Reporter: Sam Whittle
>            Assignee: Sam Whittle
>            Priority: Critical
>          Time Spent: 6h
>  Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the window depends only on window, sets a hold for the end of the window regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
>  .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
>  .withTrigger(
>  Repeatedly.forever(
>  AfterWatermark.pastEndOfWindow()
>  .withLateFirings(AfterPane.elementCountAtLeast(10))))
>  .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark until the allowedLateness has passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)