You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Steven Zhen Wu (Jira)" <ji...@apache.org> on 2020/11/03 02:03:00 UTC

[jira] [Updated] (FLINK-19934) [FLIP-27 source] add new API: SplitEnumeratorContext.execute(Runnable)

     [ https://issues.apache.org/jira/browse/FLINK-19934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Zhen Wu updated FLINK-19934:
-----------------------------------
    Description: 
Here is the motivation use case. We are implementing event-time alignment across sources in Iceberg source. Basically, each Iceberg source/enumerator tracks its watermark using min/max timestamps captures in the column stats of the data files.

When the watermark from another source advances, notified source/enumerator can try `assignSplits` as constraints may be satisfied now. This callback is initiated from the coordinator thread from the other source. If we have `SplitEnumeratorContext.execute(Runnable r)` API, we can ensure that all the actions by enumerator and assigner are serialized by the coordinator thread. That can avoid the need of locks.

[~becket_qin] [~sewen] what do you think? cc [~sundaram]


  was:
Here is the motivation use case. We are implementing event-time alignment across sources in Iceberg source. Basically, each Iceberg source/enumerator tracks its watermark using min/max timestamps captures in the column stats of the data files.

When the watermark from another source advances, notified source/enumerator can try `assignSplits` as constraints may be satisfied now. This callback is initiated from the coordinator thread from the other source. If we have `SplitEnumeratorContext.execute(Runnable r)` API, we can ensure that all the actions by enumerator and assigner are serialized by the coordinator thread. That can avoid the need of locks.

[~becket_qin] [~sewen] what do you think?



> [FLIP-27 source] add new API: SplitEnumeratorContext.execute(Runnable)
> ----------------------------------------------------------------------
>
>                 Key: FLINK-19934
>                 URL: https://issues.apache.org/jira/browse/FLINK-19934
>             Project: Flink
>          Issue Type: New Feature
>          Components: API / DataStream
>    Affects Versions: 1.11.2
>            Reporter: Steven Zhen Wu
>            Priority: Major
>
> Here is the motivation use case. We are implementing event-time alignment across sources in Iceberg source. Basically, each Iceberg source/enumerator tracks its watermark using min/max timestamps captures in the column stats of the data files.
> When the watermark from another source advances, notified source/enumerator can try `assignSplits` as constraints may be satisfied now. This callback is initiated from the coordinator thread from the other source. If we have `SplitEnumeratorContext.execute(Runnable r)` API, we can ensure that all the actions by enumerator and assigner are serialized by the coordinator thread. That can avoid the need of locks.
> [~becket_qin] [~sewen] what do you think? cc [~sundaram]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)