You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Jocean shi <jo...@gmail.com> on 2019/12/23 13:46:45 UTC
Watermark options for TwoInputStreamOperator
Hi all:
Currently, The "TwoInputStreamOperator" such as
"CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
(Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
support compute watermark by two stream.
but we just need one stream to compute watermark in some case.
For example: one stream is only rule data or control command, it
don't hava event time.
I think user has right to choose which stream to compute watermark.
Best
Ocean
Re: Watermark options for TwoInputStreamOperator
Posted by Victor Wong <ji...@gmail.com>.
Hi Jocean,
How about assigning the control data stream with Long.MAX_VALUE as
watermark?
Best,
Victor
Jocean shi <jo...@gmail.com> 于2019年12月25日周三 上午9:53写道：
> Hi Congxian,
>
> It is not the same with Idling Source
>
> Best
> Ocean
>
> Congxian Qiu <qc...@gmail.com> 于2019年12月24日周二 下午4:39写道：
> >
> > Hi
> >
> > Maybe the doc[1] about Idling Source can help there.
> > [1]
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#idling-sources
> >
> > Best,
> > Congxian
> >
> >
> > Jocean shi <jo...@gmail.com> 于2019年12月24日周二 下午3:05写道：
> >
> > > Hi Jark,
> > >
> > > I got you. We have discussed this question in Flink Forward 2019.
> > > I know that i can custom operator to resolve this problem.
> > > but also has some other problems:
> > >
> > > First,
> > > This is a very common scene that we often meet
> > > I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
> > > and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
> > > bucause it's "protected",
> > > It's unfriendly for new user.
> > >
> > > Second.
> > > One stream even don't call "assignTimestampsAndWatermarks" method,
> > > It's mean this stream don't have time and watermark,
> > > It don't have time and watermark why It have a hand in computing of
> > > watermak and time default.
> > >
> > > Best
> > > Jark
> > >
> > > Jark Wu <im...@gmail.com> 于2019年12月24日周二 下午12:08写道：
> > > >
> > > > Hi Ocean,
> > > >
> > > > You can implement your custom operator by the
> "TwoInputStreamOperator"
> > > > interface.
> > > > The TwoInputStreamOperator interface provides "processWatermark1" and
> > > > "processWatermark2" which handles
> > > > watermarks for left stream and right stream. You can then ignore the
> > > > watermarks from right stream and forward
> > > > watermark from left stream.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Mon, 23 Dec 2019 at 21:47, Jocean shi <jo...@gmail.com>
> wrote:
> > > >
> > > > > Hi all:
> > > > > Currently, The "TwoInputStreamOperator" such as
> > > > > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > > > > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream"
> only
> > > > > support compute watermark by two stream.
> > > > > but we just need one stream to compute watermark in some case.
> > > > > For example: one stream is only rule data or control command,
> it
> > > > > don't hava event time.
> > > > > I think user has right to choose which stream to compute watermark.
> > > > >
> > > > > Best
> > > > > Ocean
> > > > >
> > >
>
--
Best,
Victor
Re: Watermark options for TwoInputStreamOperator
Posted by Jocean shi <jo...@gmail.com>.
Hi Congxian,
It is not the same with Idling Source
Best
Ocean
Congxian Qiu <qc...@gmail.com> 于2019年12月24日周二 下午4:39写道：
>
> Hi
>
> Maybe the doc[1] about Idling Source can help there.
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#idling-sources
>
> Best,
> Congxian
>
>
> Jocean shi <jo...@gmail.com> 于2019年12月24日周二 下午3:05写道：
>
> > Hi Jark,
> >
> > I got you. We have discussed this question in Flink Forward 2019.
> > I know that i can custom operator to resolve this problem.
> > but also has some other problems:
> >
> > First,
> > This is a very common scene that we often meet
> > I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
> > and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
> > bucause it's "protected",
> > It's unfriendly for new user.
> >
> > Second.
> > One stream even don't call "assignTimestampsAndWatermarks" method,
> > It's mean this stream don't have time and watermark,
> > It don't have time and watermark why It have a hand in computing of
> > watermak and time default.
> >
> > Best
> > Jark
> >
> > Jark Wu <im...@gmail.com> 于2019年12月24日周二 下午12:08写道：
> > >
> > > Hi Ocean,
> > >
> > > You can implement your custom operator by the "TwoInputStreamOperator"
> > > interface.
> > > The TwoInputStreamOperator interface provides "processWatermark1" and
> > > "processWatermark2" which handles
> > > watermarks for left stream and right stream. You can then ignore the
> > > watermarks from right stream and forward
> > > watermark from left stream.
> > >
> > > Best,
> > > Jark
> > >
> > > On Mon, 23 Dec 2019 at 21:47, Jocean shi <jo...@gmail.com> wrote:
> > >
> > > > Hi all:
> > > > Currently, The "TwoInputStreamOperator" such as
> > > > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > > > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> > > > support compute watermark by two stream.
> > > > but we just need one stream to compute watermark in some case.
> > > > For example: one stream is only rule data or control command, it
> > > > don't hava event time.
> > > > I think user has right to choose which stream to compute watermark.
> > > >
> > > > Best
> > > > Ocean
> > > >
> >
Re: Watermark options for TwoInputStreamOperator
Posted by Congxian Qiu <qc...@gmail.com>.
Hi
Maybe the doc[1] about Idling Source can help there.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#idling-sources
Best,
Congxian
Jocean shi <jo...@gmail.com> 于2019年12月24日周二 下午3:05写道：
> Hi Jark,
>
> I got you. We have discussed this question in Flink Forward 2019.
> I know that i can custom operator to resolve this problem.
> but also has some other problems:
>
> First,
> This is a very common scene that we often meet
> I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
> and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
> bucause it's "protected",
> It's unfriendly for new user.
>
> Second.
> One stream even don't call "assignTimestampsAndWatermarks" method,
> It's mean this stream don't have time and watermark,
> It don't have time and watermark why It have a hand in computing of
> watermak and time default.
>
> Best
> Jark
>
> Jark Wu <im...@gmail.com> 于2019年12月24日周二 下午12:08写道：
> >
> > Hi Ocean,
> >
> > You can implement your custom operator by the "TwoInputStreamOperator"
> > interface.
> > The TwoInputStreamOperator interface provides "processWatermark1" and
> > "processWatermark2" which handles
> > watermarks for left stream and right stream. You can then ignore the
> > watermarks from right stream and forward
> > watermark from left stream.
> >
> > Best,
> > Jark
> >
> > On Mon, 23 Dec 2019 at 21:47, Jocean shi <jo...@gmail.com> wrote:
> >
> > > Hi all:
> > > Currently, The "TwoInputStreamOperator" such as
> > > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> > > support compute watermark by two stream.
> > > but we just need one stream to compute watermark in some case.
> > > For example: one stream is only rule data or control command, it
> > > don't hava event time.
> > > I think user has right to choose which stream to compute watermark.
> > >
> > > Best
> > > Ocean
> > >
>
Re: Watermark options for TwoInputStreamOperator
Posted by Jocean shi <jo...@gmail.com>.
Hi Jark,
I got you. We have discussed this question in Flink Forward 2019.
I know that i can custom operator to resolve this problem.
but also has some other problems:
First,
This is a very common scene that we often meet
I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
bucause it's "protected",
It's unfriendly for new user.
Second.
One stream even don't call "assignTimestampsAndWatermarks" method,
It's mean this stream don't have time and watermark,
It don't have time and watermark why It have a hand in computing of
watermak and time default.
Best
Jark
Jark Wu <im...@gmail.com> 于2019年12月24日周二 下午12:08写道：
>
> Hi Ocean,
>
> You can implement your custom operator by the "TwoInputStreamOperator"
> interface.
> The TwoInputStreamOperator interface provides "processWatermark1" and
> "processWatermark2" which handles
> watermarks for left stream and right stream. You can then ignore the
> watermarks from right stream and forward
> watermark from left stream.
>
> Best,
> Jark
>
> On Mon, 23 Dec 2019 at 21:47, Jocean shi <jo...@gmail.com> wrote:
>
> > Hi all:
> > Currently, The "TwoInputStreamOperator" such as
> > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> > support compute watermark by two stream.
> > but we just need one stream to compute watermark in some case.
> > For example: one stream is only rule data or control command, it
> > don't hava event time.
> > I think user has right to choose which stream to compute watermark.
> >
> > Best
> > Ocean
> >
Re: Watermark options for TwoInputStreamOperator
Posted by Jark Wu <im...@gmail.com>.
Hi Ocean,
You can implement your custom operator by the "TwoInputStreamOperator"
interface.
The TwoInputStreamOperator interface provides "processWatermark1" and
"processWatermark2" which handles
watermarks for left stream and right stream. You can then ignore the
watermarks from right stream and forward
watermark from left stream.
Best,
Jark
On Mon, 23 Dec 2019 at 21:47, Jocean shi <jo...@gmail.com> wrote:
> Hi all:
> Currently, The "TwoInputStreamOperator" such as
> "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> support compute watermark by two stream.
> but we just need one stream to compute watermark in some case.
> For example: one stream is only rule data or control command, it
> don't hava event time.
> I think user has right to choose which stream to compute watermark.
>
> Best
> Ocean
>