You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Carlos Freitas <ca...@talkdesk.com.INVALID> on 2023/01/23 12:34:14 UTC

Flink connector sinks

Hello everyone,

I have a question more on the curiosity side, unsure if this has been discussed before (sorry if It has). Given the recent deprecation of the `FlinkKafkaProducer` in favour of the new `KafkaSink` where we get away from the `SinkFunction` interface in favour of the “new”  `Sink` interface. My question here is if there is any plan on making a similar move in any other sink (eg. JDBC connector sink) either by using the same Sink interface that Kafka uses or any other one that does not make direct use of SinkFunction anymore.

PS. I do not have the context on why was decided to deprecate FlinkKafkaProducer in the first place.

Cumps,
Carlos Freitas

Re: Flink connector sinks

Posted by Carlos Freitas <ca...@talkdesk.com.INVALID>.
Hello Martijn,

Thanks a lot for the answer. Other question I have if you could help me.

Is there a proper way to define sequential sinks? Meaning if we could have a sequence of sinks that operate in order instead of operating in parallel

Best regards,
Carlos Freitas

> On 23 Jan 2023, at 12:52, Martijn Visser <ma...@apache.org> wrote:
> 
> Hi Carlos,
> 
> Yes, the new Sink interface is ultimately the target for all Sink
> connectors. A discussion on that topic has started recently, which you can
> see at https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx.
> Currently Kafka, Elasticsearch, Opensearch, Kinesis, Firehose, DynamoDB and
> more are already using the new interface. JDBC doesn't yet, but there is a
> PR open for that https://github.com/apache/flink-connector-jdbc/pull/2.
> 
> For context on why a new Sink interface was created, I would recommend
> checking out FLIP-143 that talked about this. You can find that at
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> 
> Best regards,
> 
> Martijn
> 
> Op ma 23 jan. 2023 om 13:35 schreef Carlos Freitas
> <ca...@talkdesk.com.invalid>:
> 
>> Hello everyone,
>> 
>> I have a question more on the curiosity side, unsure if this has been
>> discussed before (sorry if It has). Given the recent deprecation of the
>> `FlinkKafkaProducer` in favour of the new `KafkaSink` where we get away
>> from the `SinkFunction` interface in favour of the “new”  `Sink` interface.
>> My question here is if there is any plan on making a similar move in any
>> other sink (eg. JDBC connector sink) either by using the same Sink
>> interface that Kafka uses or any other one that does not make direct use of
>> SinkFunction anymore.
>> 
>> PS. I do not have the context on why was decided to deprecate
>> FlinkKafkaProducer in the first place.
>> 
>> Cumps,
>> Carlos Freitas


Re: Flink connector sinks

Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi,

I would recommend consolidating all those discussions like using Sink over
SinkFunction, SinkFunktion deprecation, etc. in [1] to keep everyone on the
same page. Martijn has already shared a very good big picture with more
details in that thread. Let's continue our discussion there. Thanks!

Best regards,
Jing

[1]https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx

On Mon, Jan 23, 2023 at 1:52 PM Martijn Visser <ma...@apache.org>
wrote:

> Hi Carlos,
>
> Yes, the new Sink interface is ultimately the target for all Sink
> connectors. A discussion on that topic has started recently, which you can
> see at https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx.
> Currently Kafka, Elasticsearch, Opensearch, Kinesis, Firehose, DynamoDB and
> more are already using the new interface. JDBC doesn't yet, but there is a
> PR open for that https://github.com/apache/flink-connector-jdbc/pull/2.
>
> For context on why a new Sink interface was created, I would recommend
> checking out FLIP-143 that talked about this. You can find that at
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
>
> Best regards,
>
> Martijn
>
> Op ma 23 jan. 2023 om 13:35 schreef Carlos Freitas
> <ca...@talkdesk.com.invalid>:
>
> > Hello everyone,
> >
> > I have a question more on the curiosity side, unsure if this has been
> > discussed before (sorry if It has). Given the recent deprecation of the
> > `FlinkKafkaProducer` in favour of the new `KafkaSink` where we get away
> > from the `SinkFunction` interface in favour of the “new”  `Sink`
> interface.
> > My question here is if there is any plan on making a similar move in any
> > other sink (eg. JDBC connector sink) either by using the same Sink
> > interface that Kafka uses or any other one that does not make direct use
> of
> > SinkFunction anymore.
> >
> > PS. I do not have the context on why was decided to deprecate
> > FlinkKafkaProducer in the first place.
> >
> > Cumps,
> > Carlos Freitas
>

Re: Flink connector sinks

Posted by Martijn Visser <ma...@apache.org>.
Hi Carlos,

Yes, the new Sink interface is ultimately the target for all Sink
connectors. A discussion on that topic has started recently, which you can
see at https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx.
Currently Kafka, Elasticsearch, Opensearch, Kinesis, Firehose, DynamoDB and
more are already using the new interface. JDBC doesn't yet, but there is a
PR open for that https://github.com/apache/flink-connector-jdbc/pull/2.

For context on why a new Sink interface was created, I would recommend
checking out FLIP-143 that talked about this. You can find that at
https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API

Best regards,

Martijn

Op ma 23 jan. 2023 om 13:35 schreef Carlos Freitas
<ca...@talkdesk.com.invalid>:

> Hello everyone,
>
> I have a question more on the curiosity side, unsure if this has been
> discussed before (sorry if It has). Given the recent deprecation of the
> `FlinkKafkaProducer` in favour of the new `KafkaSink` where we get away
> from the `SinkFunction` interface in favour of the “new”  `Sink` interface.
> My question here is if there is any plan on making a similar move in any
> other sink (eg. JDBC connector sink) either by using the same Sink
> interface that Kafka uses or any other one that does not make direct use of
> SinkFunction anymore.
>
> PS. I do not have the context on why was decided to deprecate
> FlinkKafkaProducer in the first place.
>
> Cumps,
> Carlos Freitas