You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kevin Lam <ke...@shopify.com> on 2022/01/11 20:57:50 UTC

Is FlinkKafkaProducer state compatible with KafkaSink sink? How to migrate?

Hi all,

We're looking to migrating from FlinkKafkaProducer to the new KafkaSink for
the new unified Sink API.

Is the state compatible across the two Kafka sink APIs? If not, what's the
best way to migrate from one to the other?

Thanks in advance,
Kevin

Re: Is FlinkKafkaProducer state compatible with KafkaSink sink? How to migrate?

Posted by Kevin Lam <ke...@shopify.com>.
Perfect, thanks for your consistently helpful and prompt responses Fabian!

On Wed, Jan 12, 2022 at 10:25 AM Fabian Paul <fp...@apache.org> wrote:

> Hi Kevin,
>
> No, the state is not compatible but it is also not necessary because
> if the FlinkKafkaProducer is stopped with a savepoint all transactions
> are finalized and the new KafkaSink uses a different mechanism to
> track transaction ids. [1]
>
> It should be enough to recover from the savepoint with the KafkaSink
> and ignore the FlinkKafkaProducer state.
>
> Best,
> Fabian
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.14/release-notes/flink-1.14/#port-kafkasink-to-new-unified-sink-api-flip-143
>
> On Tue, Jan 11, 2022 at 9:58 PM Kevin Lam <ke...@shopify.com> wrote:
> >
> > Hi all,
> >
> > We're looking to migrating from FlinkKafkaProducer to the new KafkaSink
> for the new unified Sink API.
> >
> > Is the state compatible across the two Kafka sink APIs? If not, what's
> the best way to migrate from one to the other?
> >
> > Thanks in advance,
> > Kevin
>

Re: Is FlinkKafkaProducer state compatible with KafkaSink sink? How to migrate?

Posted by Fabian Paul <fp...@apache.org>.
Hi Kevin,

No, the state is not compatible but it is also not necessary because
if the FlinkKafkaProducer is stopped with a savepoint all transactions
are finalized and the new KafkaSink uses a different mechanism to
track transaction ids. [1]

It should be enough to recover from the savepoint with the KafkaSink
and ignore the FlinkKafkaProducer state.

Best,
Fabian

[1] https://nightlies.apache.org/flink/flink-docs-release-1.14/release-notes/flink-1.14/#port-kafkasink-to-new-unified-sink-api-flip-143

On Tue, Jan 11, 2022 at 9:58 PM Kevin Lam <ke...@shopify.com> wrote:
>
> Hi all,
>
> We're looking to migrating from FlinkKafkaProducer to the new KafkaSink for the new unified Sink API.
>
> Is the state compatible across the two Kafka sink APIs? If not, what's the best way to migrate from one to the other?
>
> Thanks in advance,
> Kevin