You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Sidney Feiner <si...@startapp.com> on 2020/03/22 14:02:50 UTC

Windows on SinkFunctions

Hey,
I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well.
For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink.
Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in the pipeline?
Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

Thanks :)


Sidney Feiner / Data Platform Developer
M: +972.528197720 / Skype: sidney.feiner.startapp

[emailsignature]

Re: Windows on SinkFunctions

Posted by Robert Metzger <rm...@apache.org>.

Hey,

In your original email, you wrote:

 Because if I have multiple sinks that that only for one of them I need a
> Window, the second solution might be problematic.


You can also send the data of an operator to multiple sinks

Source --> MyComputationProcessFunction --> DataBatcher --> BatchedSink
                                         \
                                          \--> StreamingSink
(also here if it renders weirdly in email:
https://gist.github.com/rmetzger/9ef311c2926a82fdba2b2b7af9ad65d6 )

In this example "MyComputationProcessFunction" is sending the data to two
downstream operators: DataBatcher and StreamingSink.

For the "DataBatcher", you can also consider building something yourself
with ProcessFunction. I would just collect the data in a List, and emit it
once the list reached 500 elements, or a custom 5 minute trigger has
triggered.


On Sun, Mar 29, 2020 at 12:07 PM Sidney Feiner <si...@startapp.com>
wrote:

> Thanks!
> What am I supposed to put in the apply/process function for the sink to be
> invoked on a List of items?
>
> *Sidney Feiner* */* Data Platform Developer
> M: +972.528197720 */* Skype: sidney.feiner.startapp
>
> [image: emailsignature]
>
> ------------------------------
> *From:* tison <wa...@gmail.com>
> *Sent:* Sunday, March 22, 2020 4:19 PM
> *To:* Sidney Feiner <si...@startapp.com>
> *Cc:* user@flink.apache.org <us...@flink.apache.org>
> *Subject:* Re: Windows on SinkFunctions
>
> Hi Sidney,
>
> For the case, you can exactly write
>
> stream.
>   ...
>   .window()
>   .apply()
>   .addSink()
>
> Operator chain will chain these operators into one so that you don't have
> to worry about the efficiency.
>
> Best,
> tison.
>
>
> Sidney Feiner <si...@startapp.com> 于2020年3月22日周日 下午10:03写道：
>
> Hey,
> I wanted to know if it's possible to define a SinkFunction as a
> WindowFunction as well.
> For example, I would like the sink to be invoked every 5 minute or once
> 500 events reached the sink.
> Is there a way to do this inside the sink implementation? Or do I have to
> create the windows prior in the pipeline?
> Because if I have multiple sinks that that only for one of them I need a
> Window, the second solution might be problematic.
>
> Thanks :)
>
> *Sidney Feiner* */* Data Platform Developer
> M: +972.528197720 */* Skype: sidney.feiner.startapp
>
> [image: emailsignature]
>
>

Re: Windows on SinkFunctions

Posted by Sidney Feiner <si...@startapp.com>.

Thanks!
What am I supposed to put in the apply/process function for the sink to be invoked on a List of items?


Sidney Feiner / Data Platform Developer
M: +972.528197720 / Skype: sidney.feiner.startapp

[emailsignature]


________________________________
From: tison <wa...@gmail.com>
Sent: Sunday, March 22, 2020 4:19 PM
To: Sidney Feiner <si...@startapp.com>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: Windows on SinkFunctions

Hi Sidney,

For the case, you can exactly write

stream.
  ...
  .window()
  .apply()
  .addSink()

Operator chain will chain these operators into one so that you don't have to worry about the efficiency.

Best,
tison.


Sidney Feiner <si...@startapp.com>> 于2020年3月22日周日 下午10:03写道：
Hey,
I wanted to know if it's possible to define a SinkFunction as a WindowFunction as well.
For example, I would like the sink to be invoked every 5 minute or once 500 events reached the sink.
Is there a way to do this inside the sink implementation? Or do I have to create the windows prior in the pipeline?
Because if I have multiple sinks that that only for one of them I need a Window, the second solution might be problematic.

Thanks :)


Sidney Feiner / Data Platform Developer
M: +972.528197720 / Skype: sidney.feiner.startapp

[emailsignature]

Re: Windows on SinkFunctions

Posted by tison <wa...@gmail.com>.

Hi Sidney,

For the case, you can exactly write

stream.
  ...
  .window()
  .apply()
  .addSink()

Operator chain will chain these operators into one so that you don't have
to worry about the efficiency.

Best,
tison.


Sidney Feiner <si...@startapp.com> 于2020年3月22日周日 下午10:03写道：

> Hey,
> I wanted to know if it's possible to define a SinkFunction as a
> WindowFunction as well.
> For example, I would like the sink to be invoked every 5 minute or once
> 500 events reached the sink.
> Is there a way to do this inside the sink implementation? Or do I have to
> create the windows prior in the pipeline?
> Because if I have multiple sinks that that only for one of them I need a
> Window, the second solution might be problematic.
>
> Thanks :)
>
> *Sidney Feiner* */* Data Platform Developer
> M: +972.528197720 */* Skype: sidney.feiner.startapp
>
> [image: emailsignature]
>
>