You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zhenyu Wang <st...@gmail.com> on 2023/03/07 03:55:39 UTC
Question about KRaft
Hi team,
Here is a question about KRaft from normal user, who starts to use and
learn Kafka since 3.2
Last month Kafka 3.4, the first bridge release was available, and I am
considering to have a plan to use KRaft (get rid of ZK) since this version
Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which is
both controller & broker, even ZK is installed on this node too (sorry I
know it is not distributed and I will try to improve it with more knowledge
learned in future)
When I read KIP-866, ZK to KRaft migration, from section Migration
Overview, seems like the document is for multi-nodes with no or almost no
downtime, enable KRaft node by node; however my case accepts downtime (one
node -_-!!), just want to have Kafka upgrade to 3.4 then start service
under KRaft mode, make sure everything works well and no log lost
Should I simply
1. download 3.4 binary
2. stop ZK & Kafka service
3. upgrade Kafka to 3.4
4. start only Kafka service with KRaft server.properties
Or any other thing I need to pay attention to?
If there is a documentation as guide that would be quite helpful
Really appreciate
Re: Question about KRaft
Posted by David Arthur <mu...@gmail.com>.
Hi Zhenyu,
> Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which is
both controller & broker, even ZK is installed on this node too (sorry I
know it is not distributed and I will try to improve it with more knowledge
learned in future)
Controllers are always colocated with brokers in ZK mode. Only in
KRaft mode do we separate the two concepts by introducing the
"process.roles" configuration.
As Luke mentioned, you can try a regular migration by following the
docs and it should work. Essentially, you would be bringing up a new
KRaft controller (on the same or different server) and letting it do a
migration of your single-node Kafka cluster. Once you've gone through
all the steps, you should have a single KRaft broker and a single
KRaft controller. At that point you can decommission ZooKeeper.
If you run into any trouble, feel free to reach out here on the users
list or file a JIRA (if you think you found a bug 😉)
https://issues.apache.org/jira/browse/KAFKA
Cheers,
David
On Fri, Mar 10, 2023 at 12:17 AM Luke Chen <sh...@gmail.com> wrote:
>
> For questions related to confluent, I think you'd better ask in their
> channel.
>
> Luke
>
> On Fri, Mar 10, 2023 at 12:54 PM sunil chaudhari <
> sunilmchaudhari05@gmail.com> wrote:
>
> > Hi Luke,
> > This docu is good.
> > Does it apply for confluent as well?
> >
> >
> >
> > On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi Zhenyu,
> > >
> > > Answering your question:
> > >
> > > > Should I simply
> > > 1. download 3.4 binary
> > > 2. stop ZK & Kafka service
> > > 3. upgrade Kafka to 3.4
> > > 4. start only Kafka service with KRaft server.properties
> > >
> > > That is not migrating, actually. That is just creating another kafka
> > > cluster in KRaft mode.
> > > The point for migration is to move metadata in ZK into KRaft controllers.
> > > You can follow the guide here to do migration:
> > > https://kafka.apache.org/documentation/#kraft_zk_migration
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang <st...@gmail.com>
> > > wrote:
> > >
> > > > Hi Sunil,
> > > >
> > > > As mentioned earlier in my question, I have only one "combined" node as
> > > > both controller and broker, and I totally accept downtime (stop
> > service)
> > > >
> > > > So just want to ask for my case, single node, if I want to upgrade to
> > 3.4
> > > > then start service under KRaft (get rid of ZK), what would be the
> > steps?
> > > >
> > > > Thanks~
> > > >
> > > > On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> > > > sunilmchaudhari05@gmail.com>
> > > > wrote:
> > > >
> > > > > How will you achieve zero downtime of you stop zookeeper and kafka?
> > > > > There must be some standard steps so that stop zookeeper one by one
> > and
> > > > > start kraft same time so that it will be migrated gradually.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang <st...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Here is a question about KRaft from normal user, who starts to use
> > > and
> > > > > > learn Kafka since 3.2
> > > > > >
> > > > > > Last month Kafka 3.4, the first bridge release was available, and I
> > > am
> > > > > > considering to have a plan to use KRaft (get rid of ZK) since this
> > > > > version
> > > > > >
> > > > > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node,
> > > which
> > > > > is
> > > > > > both controller & broker, even ZK is installed on this node too
> > > (sorry
> > > > I
> > > > > > know it is not distributed and I will try to improve it with more
> > > > > knowledge
> > > > > > learned in future)
> > > > > >
> > > > > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > > > > Overview, seems like the document is for multi-nodes with no or
> > > almost
> > > > no
> > > > > > downtime, enable KRaft node by node; however my case accepts
> > downtime
> > > > > (one
> > > > > > node -_-!!), just want to have Kafka upgrade to 3.4 then start
> > > service
> > > > > > under KRaft mode, make sure everything works well and no log lost
> > > > > >
> > > > > > Should I simply
> > > > > > 1. download 3.4 binary
> > > > > > 2. stop ZK & Kafka service
> > > > > > 3. upgrade Kafka to 3.4
> > > > > > 4. start only Kafka service with KRaft server.properties
> > > > > >
> > > > > > Or any other thing I need to pay attention to?
> > > > > >
> > > > > > If there is a documentation as guide that would be quite helpful
> > > > > >
> > > > > > Really appreciate
> > > > > >
> > > > >
> > > >
> > >
> >
--
David Arthur
Re: Question about KRaft
Posted by Luke Chen <sh...@gmail.com>.
For questions related to confluent, I think you'd better ask in their
channel.
Luke
On Fri, Mar 10, 2023 at 12:54 PM sunil chaudhari <
sunilmchaudhari05@gmail.com> wrote:
> Hi Luke,
> This docu is good.
> Does it apply for confluent as well?
>
>
>
> On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen <sh...@gmail.com> wrote:
>
> > Hi Zhenyu,
> >
> > Answering your question:
> >
> > > Should I simply
> > 1. download 3.4 binary
> > 2. stop ZK & Kafka service
> > 3. upgrade Kafka to 3.4
> > 4. start only Kafka service with KRaft server.properties
> >
> > That is not migrating, actually. That is just creating another kafka
> > cluster in KRaft mode.
> > The point for migration is to move metadata in ZK into KRaft controllers.
> > You can follow the guide here to do migration:
> > https://kafka.apache.org/documentation/#kraft_zk_migration
> >
> > Thank you.
> > Luke
> >
> > On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang <st...@gmail.com>
> > wrote:
> >
> > > Hi Sunil,
> > >
> > > As mentioned earlier in my question, I have only one "combined" node as
> > > both controller and broker, and I totally accept downtime (stop
> service)
> > >
> > > So just want to ask for my case, single node, if I want to upgrade to
> 3.4
> > > then start service under KRaft (get rid of ZK), what would be the
> steps?
> > >
> > > Thanks~
> > >
> > > On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> > > sunilmchaudhari05@gmail.com>
> > > wrote:
> > >
> > > > How will you achieve zero downtime of you stop zookeeper and kafka?
> > > > There must be some standard steps so that stop zookeeper one by one
> and
> > > > start kraft same time so that it will be migrated gradually.
> > > >
> > > >
> > > >
> > > > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang <st...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Here is a question about KRaft from normal user, who starts to use
> > and
> > > > > learn Kafka since 3.2
> > > > >
> > > > > Last month Kafka 3.4, the first bridge release was available, and I
> > am
> > > > > considering to have a plan to use KRaft (get rid of ZK) since this
> > > > version
> > > > >
> > > > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node,
> > which
> > > > is
> > > > > both controller & broker, even ZK is installed on this node too
> > (sorry
> > > I
> > > > > know it is not distributed and I will try to improve it with more
> > > > knowledge
> > > > > learned in future)
> > > > >
> > > > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > > > Overview, seems like the document is for multi-nodes with no or
> > almost
> > > no
> > > > > downtime, enable KRaft node by node; however my case accepts
> downtime
> > > > (one
> > > > > node -_-!!), just want to have Kafka upgrade to 3.4 then start
> > service
> > > > > under KRaft mode, make sure everything works well and no log lost
> > > > >
> > > > > Should I simply
> > > > > 1. download 3.4 binary
> > > > > 2. stop ZK & Kafka service
> > > > > 3. upgrade Kafka to 3.4
> > > > > 4. start only Kafka service with KRaft server.properties
> > > > >
> > > > > Or any other thing I need to pay attention to?
> > > > >
> > > > > If there is a documentation as guide that would be quite helpful
> > > > >
> > > > > Really appreciate
> > > > >
> > > >
> > >
> >
>
Re: Question about KRaft
Posted by sunil chaudhari <su...@gmail.com>.
Hi Luke,
This docu is good.
Does it apply for confluent as well?
On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen <sh...@gmail.com> wrote:
> Hi Zhenyu,
>
> Answering your question:
>
> > Should I simply
> 1. download 3.4 binary
> 2. stop ZK & Kafka service
> 3. upgrade Kafka to 3.4
> 4. start only Kafka service with KRaft server.properties
>
> That is not migrating, actually. That is just creating another kafka
> cluster in KRaft mode.
> The point for migration is to move metadata in ZK into KRaft controllers.
> You can follow the guide here to do migration:
> https://kafka.apache.org/documentation/#kraft_zk_migration
>
> Thank you.
> Luke
>
> On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang <st...@gmail.com>
> wrote:
>
> > Hi Sunil,
> >
> > As mentioned earlier in my question, I have only one "combined" node as
> > both controller and broker, and I totally accept downtime (stop service)
> >
> > So just want to ask for my case, single node, if I want to upgrade to 3.4
> > then start service under KRaft (get rid of ZK), what would be the steps?
> >
> > Thanks~
> >
> > On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> > sunilmchaudhari05@gmail.com>
> > wrote:
> >
> > > How will you achieve zero downtime of you stop zookeeper and kafka?
> > > There must be some standard steps so that stop zookeeper one by one and
> > > start kraft same time so that it will be migrated gradually.
> > >
> > >
> > >
> > > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang <st...@gmail.com>
> > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Here is a question about KRaft from normal user, who starts to use
> and
> > > > learn Kafka since 3.2
> > > >
> > > > Last month Kafka 3.4, the first bridge release was available, and I
> am
> > > > considering to have a plan to use KRaft (get rid of ZK) since this
> > > version
> > > >
> > > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node,
> which
> > > is
> > > > both controller & broker, even ZK is installed on this node too
> (sorry
> > I
> > > > know it is not distributed and I will try to improve it with more
> > > knowledge
> > > > learned in future)
> > > >
> > > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > > Overview, seems like the document is for multi-nodes with no or
> almost
> > no
> > > > downtime, enable KRaft node by node; however my case accepts downtime
> > > (one
> > > > node -_-!!), just want to have Kafka upgrade to 3.4 then start
> service
> > > > under KRaft mode, make sure everything works well and no log lost
> > > >
> > > > Should I simply
> > > > 1. download 3.4 binary
> > > > 2. stop ZK & Kafka service
> > > > 3. upgrade Kafka to 3.4
> > > > 4. start only Kafka service with KRaft server.properties
> > > >
> > > > Or any other thing I need to pay attention to?
> > > >
> > > > If there is a documentation as guide that would be quite helpful
> > > >
> > > > Really appreciate
> > > >
> > >
> >
>
Re: Question about KRaft
Posted by Luke Chen <sh...@gmail.com>.
Hi Zhenyu,
Answering your question:
> Should I simply
1. download 3.4 binary
2. stop ZK & Kafka service
3. upgrade Kafka to 3.4
4. start only Kafka service with KRaft server.properties
That is not migrating, actually. That is just creating another kafka
cluster in KRaft mode.
The point for migration is to move metadata in ZK into KRaft controllers.
You can follow the guide here to do migration:
https://kafka.apache.org/documentation/#kraft_zk_migration
Thank you.
Luke
On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang <st...@gmail.com> wrote:
> Hi Sunil,
>
> As mentioned earlier in my question, I have only one "combined" node as
> both controller and broker, and I totally accept downtime (stop service)
>
> So just want to ask for my case, single node, if I want to upgrade to 3.4
> then start service under KRaft (get rid of ZK), what would be the steps?
>
> Thanks~
>
> On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> sunilmchaudhari05@gmail.com>
> wrote:
>
> > How will you achieve zero downtime of you stop zookeeper and kafka?
> > There must be some standard steps so that stop zookeeper one by one and
> > start kraft same time so that it will be migrated gradually.
> >
> >
> >
> > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang <st...@gmail.com>
> wrote:
> >
> > > Hi team,
> > >
> > > Here is a question about KRaft from normal user, who starts to use and
> > > learn Kafka since 3.2
> > >
> > > Last month Kafka 3.4, the first bridge release was available, and I am
> > > considering to have a plan to use KRaft (get rid of ZK) since this
> > version
> > >
> > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which
> > is
> > > both controller & broker, even ZK is installed on this node too (sorry
> I
> > > know it is not distributed and I will try to improve it with more
> > knowledge
> > > learned in future)
> > >
> > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > Overview, seems like the document is for multi-nodes with no or almost
> no
> > > downtime, enable KRaft node by node; however my case accepts downtime
> > (one
> > > node -_-!!), just want to have Kafka upgrade to 3.4 then start service
> > > under KRaft mode, make sure everything works well and no log lost
> > >
> > > Should I simply
> > > 1. download 3.4 binary
> > > 2. stop ZK & Kafka service
> > > 3. upgrade Kafka to 3.4
> > > 4. start only Kafka service with KRaft server.properties
> > >
> > > Or any other thing I need to pay attention to?
> > >
> > > If there is a documentation as guide that would be quite helpful
> > >
> > > Really appreciate
> > >
> >
>
Re: Question about KRaft
Posted by Zhenyu Wang <st...@gmail.com>.
Hi Sunil,
As mentioned earlier in my question, I have only one "combined" node as
both controller and broker, and I totally accept downtime (stop service)
So just want to ask for my case, single node, if I want to upgrade to 3.4
then start service under KRaft (get rid of ZK), what would be the steps?
Thanks~
On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <su...@gmail.com>
wrote:
> How will you achieve zero downtime of you stop zookeeper and kafka?
> There must be some standard steps so that stop zookeeper one by one and
> start kraft same time so that it will be migrated gradually.
>
>
>
> On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang <st...@gmail.com> wrote:
>
> > Hi team,
> >
> > Here is a question about KRaft from normal user, who starts to use and
> > learn Kafka since 3.2
> >
> > Last month Kafka 3.4, the first bridge release was available, and I am
> > considering to have a plan to use KRaft (get rid of ZK) since this
> version
> >
> > Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which
> is
> > both controller & broker, even ZK is installed on this node too (sorry I
> > know it is not distributed and I will try to improve it with more
> knowledge
> > learned in future)
> >
> > When I read KIP-866, ZK to KRaft migration, from section Migration
> > Overview, seems like the document is for multi-nodes with no or almost no
> > downtime, enable KRaft node by node; however my case accepts downtime
> (one
> > node -_-!!), just want to have Kafka upgrade to 3.4 then start service
> > under KRaft mode, make sure everything works well and no log lost
> >
> > Should I simply
> > 1. download 3.4 binary
> > 2. stop ZK & Kafka service
> > 3. upgrade Kafka to 3.4
> > 4. start only Kafka service with KRaft server.properties
> >
> > Or any other thing I need to pay attention to?
> >
> > If there is a documentation as guide that would be quite helpful
> >
> > Really appreciate
> >
>
Re: Question about KRaft
Posted by sunil chaudhari <su...@gmail.com>.
How will you achieve zero downtime of you stop zookeeper and kafka?
There must be some standard steps so that stop zookeeper one by one and
start kraft same time so that it will be migrated gradually.
On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang <st...@gmail.com> wrote:
> Hi team,
>
> Here is a question about KRaft from normal user, who starts to use and
> learn Kafka since 3.2
>
> Last month Kafka 3.4, the first bridge release was available, and I am
> considering to have a plan to use KRaft (get rid of ZK) since this version
>
> Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which is
> both controller & broker, even ZK is installed on this node too (sorry I
> know it is not distributed and I will try to improve it with more knowledge
> learned in future)
>
> When I read KIP-866, ZK to KRaft migration, from section Migration
> Overview, seems like the document is for multi-nodes with no or almost no
> downtime, enable KRaft node by node; however my case accepts downtime (one
> node -_-!!), just want to have Kafka upgrade to 3.4 then start service
> under KRaft mode, make sure everything works well and no log lost
>
> Should I simply
> 1. download 3.4 binary
> 2. stop ZK & Kafka service
> 3. upgrade Kafka to 3.4
> 4. start only Kafka service with KRaft server.properties
>
> Or any other thing I need to pay attention to?
>
> If there is a documentation as guide that would be quite helpful
>
> Really appreciate
>