production ready for zookeeper to kraft migration

[Matthieu Patou <mp...@rockset.com>]

I looked at the notes for 3.7.x and the migration from ZK to Kraft is still
not marked as production ready.

I'm wondering what are the issues that people could be facing during the
migration.

If 4.0 is still planned to be the full removal for ZK, is there a plan for
something after 3.7 to mark ZK migration as production ready ?

Best.

Matthieu.

Re: production ready for zookeeper to kraft migration

[David Arthur <da...@confluent.io.INVALID>]

Matthieu,

There are a few things to look out for during the migration.

1) Migration not starting. This is pretty much always going to be due to
mis-configuration. Controller KRaftMigrationDriver logs should reveal what
its waiting for
2) Migration of metadata fails. If some poison record in ZK causes the
metadata migration to fail, you'll see stacktraces and ERRORs in the
controller logs. The brokers will also not see an active controller and
won't be getting metadata updates.
3) Something in the dual-write path fails. If the write back to ZK fails
during dual-write mode (with ZK brokers or KRaft brokers), you'll see an
increasing lag (ZkWriteBehindLag metric)

So, in general: lack of an active controller, metadata not being propagated
(e.g., ISRs not getting updated), errors in the logs.

I'll try to get this written up in the docs soon.

Cheers,
David


On Thu, Apr 4, 2024 at 12:56 PM Matthieu Patou <mp...@rockset.com> wrote:

> Hey Luke,
>
> Thank you for the update.
> Out of curiosity if the migration is not working what are the symptoms ? is
> it just that the controller won't show that the migration is complete ? or
> could the controller claim (wrongfully) that the migration is complete when
> it's not ?
>
> Best.
> Matthieu
>
> On Wed, Apr 3, 2024 at 4:53 PM Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Matthieu,
> >
> > Yes, the ZK migrating to KRaft feature is already GA in v3.6.0.
> > Sorry, we forgot to update the document in the Kafka-site repo.
> > I've filed a PR for it: https://github.com/apache/kafka-site/pull/594
> >
> > Thanks.
> > Luke
> >
> > On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou <mp...@rockset.com>
> wrote:
> >
> > > I looked at the notes for 3.7.x and the migration from ZK to Kraft is
> > still
> > > not marked as production ready.
> > >
> > > I'm wondering what are the issues that people could be facing during
> the
> > > migration.
> > >
> > > If 4.0 is still planned to be the full removal for ZK, is there a plan
> > for
> > > something after 3.7 to mark ZK migration as production ready ?
> > >
> > > Best.
> > >
> > > Matthieu.
> > >
> >
>


-- 
-David

Re: production ready for zookeeper to kraft migration

[Matthieu Patou <mp...@rockset.com>]

Hey Luke,

Thank you for the update.
Out of curiosity if the migration is not working what are the symptoms ? is
it just that the controller won't show that the migration is complete ? or
could the controller claim (wrongfully) that the migration is complete when
it's not ?

Best.
Matthieu

On Wed, Apr 3, 2024 at 4:53 PM Luke Chen <sh...@gmail.com> wrote:

> Hi Matthieu,
>
> Yes, the ZK migrating to KRaft feature is already GA in v3.6.0.
> Sorry, we forgot to update the document in the Kafka-site repo.
> I've filed a PR for it: https://github.com/apache/kafka-site/pull/594
>
> Thanks.
> Luke
>
> On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou <mp...@rockset.com> wrote:
>
> > I looked at the notes for 3.7.x and the migration from ZK to Kraft is
> still
> > not marked as production ready.
> >
> > I'm wondering what are the issues that people could be facing during the
> > migration.
> >
> > If 4.0 is still planned to be the full removal for ZK, is there a plan
> for
> > something after 3.7 to mark ZK migration as production ready ?
> >
> > Best.
> >
> > Matthieu.
> >
>

Re: production ready for zookeeper to kraft migration

[Luke Chen <sh...@gmail.com>]

Hi Matthieu,

Yes, the ZK migrating to KRaft feature is already GA in v3.6.0.
Sorry, we forgot to update the document in the Kafka-site repo.
I've filed a PR for it: https://github.com/apache/kafka-site/pull/594

Thanks.
Luke

On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou <mp...@rockset.com> wrote:

> I looked at the notes for 3.7.x and the migration from ZK to Kraft is still
> not marked as production ready.
>
> I'm wondering what are the issues that people could be facing during the
> migration.
>
> If 4.0 is still planned to be the full removal for ZK, is there a plan for
> something after 3.7 to mark ZK migration as production ready ?
>
> Best.
>
> Matthieu.
>