You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Sam Whittle (JIRA)" <ji...@apache.org> on 2018/05/18 17:46:00 UTC
[jira] [Resolved] (BEAM-3776) StateMerging.mergeWatermarks sets a
late watermark hold for late merging windows that depend only on the window
[ https://issues.apache.org/jira/browse/BEAM-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Whittle resolved BEAM-3776.
-------------------------------
Resolution: Fixed
Fix Version/s: 2.5.0
> StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window
> ---------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-3776
> URL: https://issues.apache.org/jira/browse/BEAM-3776
> Project: Beam
> Issue Type: Bug
> Components: runner-core
> Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Priority: Critical
> Fix For: 2.5.0
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the window depends only on window, sets a hold for the end of the window regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
> .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
> .withTrigger(
> Repeatedly.forever(
> AfterWatermark.pastEndOfWindow()
> .withLateFirings(AfterPane.elementCountAtLeast(10))))
> .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark until the allowedLateness has passed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)