[jira] [Commented] (FLINK-18996) Avoid disorder for time interval join

["Benchao Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182623#comment-17182623 ] 

Benchao Li commented on FLINK-18996:
------------------------------------

After some more deep-diving into the code with [~nilerzhou] . I think we found the root cause for this.

We have already considered the watermark lateness, see: [https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamExecIntervalJoin.scala#L335] 

However, the timer for state cleanup is later than the actual watermark, see it's logic here: [https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/interval/TimeIntervalJoin.java#L368]

Then, for example, we use {{left outer join}}, and there is 1 record in left stream, and no record in right stream. Then the trigger time for this record is {{rowTime + leftRelativeSize + minCleanUpInterval + allowedLateness + 1}}, however the actual watermark is {{Math.max(leftRelativeSize, rightRelativeSize) + allowedLateness}}, then this record will be late for downstream.

> Avoid disorder for time interval join
> -------------------------------------
>
>                 Key: FLINK-18996
>                 URL: https://issues.apache.org/jira/browse/FLINK-18996
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / API, Table SQL / Runtime
>            Reporter: Benchao Li
>            Priority: Major
>
> Currently, the time interval join will produce data with rowtime later than watermark. If we use the rowtime again in downstream, e.t. window aggregation, we'll lose some data.
>  
> reported from user-zh: [http://apache-flink.147419.n8.nabble.com/Re-flink-interval-join-tc4458.html#none]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)