You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexey Trenikhun <ye...@msn.com> on 2022/01/14 02:08:37 UTC
FlinkKafkaConsumer and FlinkKafkaProducer and Kafka Cluster Migration
Hello,
Currently we are using FlinkKafkaConsumer and FlinkKafkaProducer and planning to migrate to different Kafka cluster. Are boostrap servers, username and passwords part of FlinkKafkaConsumer and FlinkKafkaProducer ? So if we take savepoint change boostrap server and credentials and start job from savepoint, will it use new connection properties and old one from savepoint?
Assuming that we connected to new Kafka cluster, I think that FlinkKafkaConsumer offsets will be reset, because new Kafka cluster will be empty and FlinkKafkaConsumer will not be able to seek to stored offsets, am I right?
Thanks,
Alexey
Re: FlinkKafkaConsumer and FlinkKafkaProducer and Kafka Cluster Migration
Posted by Martijn Visser <ma...@ververica.com>.
Hi Alexey,
Just so you know, this feature most likely won't make it to 1.15
unfortunately.
Best regards,
Martijn
On Mon, 17 Jan 2022 at 22:47, Alexey Trenikhun <ye...@msn.com> wrote:
> Thank you Fabian.
>
> We are waiting for FLINK-18450
> <https://issues.apache.org/jira/browse/FLINK-18450> (watermark alignment)
> before switching to KafkaSource, currently we use extra logic on top of
> FlinkKafkaConsumer to support watermark alignment.
>
> Thanks,
> Alexey
> [FLINK-18450] Add watermark alignment logic to SourceReaderBase. - ASF
> JIRA - issues.apache.org
> <https://issues.apache.org/jira/browse/FLINK-18450>
> With the per-split watermark support, SourceReaderBase should be able to
> perform watermark alignment so that all the connectors that inherit from it
> would benefit.
> issues.apache.org
>
>
> ------------------------------
> *From:* Fabian Paul <fp...@apache.org>
> *Sent:* Friday, January 14, 2022 4:02 AM
> *To:* Alexey Trenikhun <ye...@msn.com>
> *Cc:* Flink User Mail List <us...@flink.apache.org>
> *Subject:* Re: FlinkKafkaConsumer and FlinkKafkaProducer and Kafka
> Cluster Migration
>
> Hi Alexey,
>
> The bootstrap servers are not part of the state so you are good to go
> although please stop all your jobs with a savepoint and resume from it
> with the new properties.
> I guess to migrate the FlinkKafkaConsumer to an empty topic you can
> discard the state if you ensure that all messages beginning from the
> latest checkpointed offset are in the new topic.
>
> Please be aware that we deprecated the FlinkKafkaConsumer and
> FlinkKafkaProducer with Flink 1.14 in favor of the KafkaSource and
> KafkaSink. In the future, we plan to remove both and they will not
> receive further updates.
>
> Best,
> Fabian
>
> On Fri, Jan 14, 2022 at 3:08 AM Alexey Trenikhun <ye...@msn.com> wrote:
> >
> > Hello,
> >
> > Currently we are using FlinkKafkaConsumer and FlinkKafkaProducer and
> planning to migrate to different Kafka cluster. Are boostrap servers,
> username and passwords part of FlinkKafkaConsumer and FlinkKafkaProducer ?
> So if we take savepoint change boostrap server and credentials and start
> job from savepoint, will it use new connection properties and old one from
> savepoint?
> > Assuming that we connected to new Kafka cluster, I think that
> FlinkKafkaConsumer offsets will be reset, because new Kafka cluster will be
> empty and FlinkKafkaConsumer will not be able to seek to stored offsets, am
> I right?
> >
> > Thanks,
> > Alexey
>
Re: FlinkKafkaConsumer and FlinkKafkaProducer and Kafka Cluster Migration
Posted by Alexey Trenikhun <ye...@msn.com>.
Thank you Fabian.
We are waiting for FLINK-18450<https://issues.apache.org/jira/browse/FLINK-18450> (watermark alignment) before switching to KafkaSource, currently we use extra logic on top of FlinkKafkaConsumer to support watermark alignment.
Thanks,
Alexey
[FLINK-18450] Add watermark alignment logic to SourceReaderBase. - ASF JIRA - issues.apache.org<https://issues.apache.org/jira/browse/FLINK-18450>
With the per-split watermark support, SourceReaderBase should be able to perform watermark alignment so that all the connectors that inherit from it would benefit.
issues.apache.org
________________________________
From: Fabian Paul <fp...@apache.org>
Sent: Friday, January 14, 2022 4:02 AM
To: Alexey Trenikhun <ye...@msn.com>
Cc: Flink User Mail List <us...@flink.apache.org>
Subject: Re: FlinkKafkaConsumer and FlinkKafkaProducer and Kafka Cluster Migration
Hi Alexey,
The bootstrap servers are not part of the state so you are good to go
although please stop all your jobs with a savepoint and resume from it
with the new properties.
I guess to migrate the FlinkKafkaConsumer to an empty topic you can
discard the state if you ensure that all messages beginning from the
latest checkpointed offset are in the new topic.
Please be aware that we deprecated the FlinkKafkaConsumer and
FlinkKafkaProducer with Flink 1.14 in favor of the KafkaSource and
KafkaSink. In the future, we plan to remove both and they will not
receive further updates.
Best,
Fabian
On Fri, Jan 14, 2022 at 3:08 AM Alexey Trenikhun <ye...@msn.com> wrote:
>
> Hello,
>
> Currently we are using FlinkKafkaConsumer and FlinkKafkaProducer and planning to migrate to different Kafka cluster. Are boostrap servers, username and passwords part of FlinkKafkaConsumer and FlinkKafkaProducer ? So if we take savepoint change boostrap server and credentials and start job from savepoint, will it use new connection properties and old one from savepoint?
> Assuming that we connected to new Kafka cluster, I think that FlinkKafkaConsumer offsets will be reset, because new Kafka cluster will be empty and FlinkKafkaConsumer will not be able to seek to stored offsets, am I right?
>
> Thanks,
> Alexey
Re: FlinkKafkaConsumer and FlinkKafkaProducer and Kafka Cluster Migration
Posted by Fabian Paul <fp...@apache.org>.
Hi Alexey,
The bootstrap servers are not part of the state so you are good to go
although please stop all your jobs with a savepoint and resume from it
with the new properties.
I guess to migrate the FlinkKafkaConsumer to an empty topic you can
discard the state if you ensure that all messages beginning from the
latest checkpointed offset are in the new topic.
Please be aware that we deprecated the FlinkKafkaConsumer and
FlinkKafkaProducer with Flink 1.14 in favor of the KafkaSource and
KafkaSink. In the future, we plan to remove both and they will not
receive further updates.
Best,
Fabian
On Fri, Jan 14, 2022 at 3:08 AM Alexey Trenikhun <ye...@msn.com> wrote:
>
> Hello,
>
> Currently we are using FlinkKafkaConsumer and FlinkKafkaProducer and planning to migrate to different Kafka cluster. Are boostrap servers, username and passwords part of FlinkKafkaConsumer and FlinkKafkaProducer ? So if we take savepoint change boostrap server and credentials and start job from savepoint, will it use new connection properties and old one from savepoint?
> Assuming that we connected to new Kafka cluster, I think that FlinkKafkaConsumer offsets will be reset, because new Kafka cluster will be empty and FlinkKafkaConsumer will not be able to seek to stored offsets, am I right?
>
> Thanks,
> Alexey