[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=102212&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102212 ]

ASF GitHub Bot logged work on BEAM-3776:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 15/May/18 17:21
            Start Date: 15/May/18 17:21
    Worklog Time Spent: 10m 
      Work Description: scwhittle commented on issue #4793: [BEAM-3776] Fix issue with merging late windows where a watermark hold could be added behind the input watermark.
URL: https://github.com/apache/beam/pull/4793#issuecomment-389246274
 
 
    I am back from leave and will rebase and make sure tests pass.
   
   
   On Tue, Apr 17, 2018 at 9:09 AM Thomas Groh <no...@github.com>
   wrote:
   
   > Additionally, running flink:testCompileJava, I get:
   >
   > > Task :runners:flink:compileTestJava FAILED
   > /usr/local/google/home/tgroh/git/beam/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkSplitStateInternalsTest.java:115: error: method does not override or implement a method from a supertype
   >   @Override
   >   ^
   > /usr/local/google/home/tgroh/git/beam/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkSplitStateInternalsTest.java:119: error: method does not override or implement a method from a supertype
   >   @Override
   >   ^
   > /usr/local/google/home/tgroh/git/beam/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java:87: error: method does not override or implement a method from a supertype
   >   @Override
   >   ^
   > /usr/local/google/home/tgroh/git/beam/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java:91: error: method does not override or implement a method from a supertype
   >   @Override
   >   ^
   > /usr/local/google/home/tgroh/git/beam/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkKeyGroupStateInternalsTest.java:133: error: method does not override or implement a method from a supertype
   >     @Override
   >     ^
   > /usr/local/google/home/tgroh/git/beam/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/FlinkKeyGroupStateInternalsTest.java:137: error: method does not override or implement a method from a supertype
   >     @Override
   >     ^
   > Note: Some input files use unchecked or unsafe operations.
   > Note: Recompile with -Xlint:unchecked for details.
   > 6 errors
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/4793#issuecomment-382050277>, or mute
   > the thread
   > <https://github.com/notifications/unsubscribe-auth/AEOczCv1JHAgbNJD9pkqlYWfycdNvp3lks5tphPUgaJpZM4Sat_i>
   > .
   >
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 102212)
    Time Spent: 5h 40m  (was: 5.5h)

> StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-3776
>                 URL: https://issues.apache.org/jira/browse/BEAM-3776
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>    Affects Versions: 2.1.0, 2.2.0, 2.3.0
>            Reporter: Sam Whittle
>            Assignee: Sam Whittle
>            Priority: Critical
>          Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the window depends only on window, sets a hold for the end of the window regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
>  .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
>  .withTrigger(
>  Repeatedly.forever(
>  AfterWatermark.pastEndOfWindow()
>  .withLateFirings(AfterPane.elementCountAtLeast(10))))
>  .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark until the allowedLateness has passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)