You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Antonio Manzano <a....@gmail.com> on 2020/09/15 09:00:29 UTC

Emit event to kafka when finish sink

Hello guys,

i would like to know if there is any possibility to emit an event when a
sink has finished.
To put it in context, I have a simple ETL (streaming bounded) that reads
data from a database, maps, and inserts into another database. Once I
finish inserting the data I want to issue an event to kafka to notify
another process.

I can think of some solutions, but I am not convinced which one is the best
option:

   - dividing the stream into two paths and with a session window, in
   onTimer,  check if the last record is already in the database
   - Implement custom sink
   - ...


Any suggestions?
Thanks!
-- 

Antonio Manzano Tejón

Tel: 699 333 556

Re: Emit event to kafka when finish sink

Posted by Dawid Wysakowicz <dw...@apache.org>.

Hi,

I am not sure if I understand your first solution, but it sounds rather
complicated.

I think implementing a custom operator could be a valid approach. You
would have to make sure it is run with parallelism of 1. You could
additionally implement a BoundedOneInput interface and notify the
external process in the endOfInput method. This is so also quite
involved solution.

Another idea is you could register a JobListener in the
StreamExecutionEnvironment. There you can execute some code based on the
status of the whole job.

Best,

Dawid

On 15/09/2020 11:00, Antonio Manzano wrote:
>
> Hello guys, 
>
> i would like to know if there is any possibility to emit an event when
> a sink has finished.
> To put it in context, I have a simple ETL (streaming bounded) that
> reads data from a database, maps, and inserts into another database.
> Once I finish inserting the data I want to issue an event to kafka to
> notify another process.
>
> I can think of some solutions, but I am not convinced which one is the
> best option:
>
>   * dividing the stream into two paths and with a session window, in
>     onTimer,  check if the last record is already in the database
>   * Implement custom sink
>   * ...
>
>
> Any suggestions?
> Thanks!
> -- 
>
> Antonio Manzano Tejón
>
> Tel: 699 333 556
>
>