You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tony Liu <ji...@zuora.com> on 2018/01/04 04:39:35 UTC
Any suggestion about Kafka/Zookeeper migration.
Hi All,
This post here is aimed to ask experience about what did you do migration
`Kafka/zookeeper` ? :)
All of Kafka/zookeeper are running on AWS, because of some reason, we have
to replace all the existed server (you can simply think we will terminate
the old server and create new server to replace it), which is running with
Kafka/zookeeper.
the Kafka infrastructure :
- 3 zookeeper with exhibitor running together.
- 6 brokers.
requirements:
- terminate all of the 3 zookeeper with exhibitor, and also 6 brokers.
- launch 5 new zookeepers without exhibitor.
- launch 6 new brokers.
- 0 or minimum downtime, try our best no impact for the Kafka client.
so far, I have two prepared solution, but any of them has pros & cons, the
solution A is too much manual operations needed, the solution B is some
development effort required and also need to migration all of the Kafka
client to use the self-developed Kafka client with `fail-over` feature.
*Solution A*: simply speaking, this solution is mainly working on manual
operation to add znode / broker to the existed cluster, when all of the
thing ready, I will step by step to terminate the old one.
*Solution B:* simply speaking, the solution will work like fail-over way,
a new cluster will be launched, and control Kafka client's traffic to the
new cluster, and terminate the old cluster.
Re: Any suggestion about Kafka/Zookeeper migration.
Posted by Ted Yu <yu...@gmail.com>.
Did you intend to attach pictures following the two solutions ?
It seems the pictures didn't come thru.
FYI
On Wed, Jan 3, 2018 at 8:39 PM, Tony Liu <ji...@zuora.com> wrote:
> Hi All,
>
> This post here is aimed to ask experience about what did you do migration
> `Kafka/zookeeper` ? :)
>
> All of Kafka/zookeeper are running on AWS, because of some reason, we have
> to replace all the existed server (you can simply think we will terminate
> the old server and create new server to replace it), which is running with
> Kafka/zookeeper.
>
> the Kafka infrastructure :
>
> - 3 zookeeper with exhibitor running together.
> - 6 brokers.
>
> requirements:
>
> - terminate all of the 3 zookeeper with exhibitor, and also 6 brokers.
> - launch 5 new zookeepers without exhibitor.
> - launch 6 new brokers.
> - 0 or minimum downtime, try our best no impact for the Kafka client.
>
>
> so far, I have two prepared solution, but any of them has pros & cons, the
> solution A is too much manual operations needed, the solution B is some
> development effort required and also need to migration all of the Kafka
> client to use the self-developed Kafka client with `fail-over` feature.
>
> *Solution A*: simply speaking, this solution is mainly working on manual
> operation to add znode / broker to the existed cluster, when all of the
> thing ready, I will step by step to terminate the old one.
>
>
>
>
> *Solution B:* simply speaking, the solution will work like fail-over
> way, a new cluster will be launched, and control Kafka client's traffic to
> the new cluster, and terminate the old cluster.
>
>
>
>
>
>
Re: Any suggestion about Kafka/Zookeeper migration.
Posted by Tony Liu <ji...@zuora.com>.
sorry guys.
I attached the picture again.
On Wed, Jan 3, 2018 at 8:49 PM, Tony Liu <ji...@zuora.com> wrote:
> for solution A , actually including two parts:
>
> - zookeeper : manual operation to migrate zookeeper, so far I think it
> needs at least 6 steps. this part will be tricky, we will have 3 zookeeper
> exhibitor , and 5 new zookeeper without e
> - Kafka : this should be easier , try launch new one to join the
> brokers cluster, and manual migration / reassign partitions.
>
>
> By the way, I also have a concern about `Solution A`, you know we have 5
> production cluster right now, the manual effort will be big and dangerous.
>
> :), that's why I ask your guys experience firstly here.
>
>
> On Wed, Jan 3, 2018 at 8:39 PM, Tony Liu <ji...@zuora.com> wrote:
>
>> Hi All,
>>
>> This post here is aimed to ask experience about what did you do migration
>> `Kafka/zookeeper` ? :)
>>
>> All of Kafka/zookeeper are running on AWS, because of some reason, we
>> have to replace all the existed server (you can simply think we will
>> terminate the old server and create new server to replace it), which is
>> running with Kafka/zookeeper.
>>
>> the Kafka infrastructure :
>>
>> - 3 zookeeper with exhibitor running together.
>> - 6 brokers.
>>
>> requirements:
>>
>> - terminate all of the 3 zookeeper with exhibitor, and also 6 brokers.
>> - launch 5 new zookeepers without exhibitor.
>> - launch 6 new brokers.
>> - 0 or minimum downtime, try our best no impact for the Kafka client.
>>
>>
>> so far, I have two prepared solution, but any of them has pros & cons,
>> the solution A is too much manual operations needed, the solution B is some
>> development effort required and also need to migration all of the Kafka
>> client to use the self-developed Kafka client with `fail-over` feature.
>>
>> *Solution A*: simply speaking, this solution is mainly working on manual
>> operation to add znode / broker to the existed cluster, when all of the
>> thing ready, I will step by step to terminate the old one.
>>
>>
>>
>>
>> *Solution B:* simply speaking, the solution will work like fail-over
>> way, a new cluster will be launched, and control Kafka client's traffic to
>> the new cluster, and terminate the old cluster.
>>
>>
>>
>>
>>
>>
>
Re: Any suggestion about Kafka/Zookeeper migration.
Posted by Tony Liu <ji...@zuora.com>.
for solution A , actually including two parts:
- zookeeper : manual operation to migrate zookeeper, so far I think it
needs at least 6 steps. this part will be tricky, we will have 3 zookeeper
exhibitor , and 5 new zookeeper without e
- Kafka : this should be easier , try launch new one to join the brokers
cluster, and manual migration / reassign partitions.
By the way, I also have a concern about `Solution A`, you know we have 5
production cluster right now, the manual effort will be big and dangerous.
:), that's why I ask your guys experience firstly here.
On Wed, Jan 3, 2018 at 8:39 PM, Tony Liu <ji...@zuora.com> wrote:
> Hi All,
>
> This post here is aimed to ask experience about what did you do migration
> `Kafka/zookeeper` ? :)
>
> All of Kafka/zookeeper are running on AWS, because of some reason, we have
> to replace all the existed server (you can simply think we will terminate
> the old server and create new server to replace it), which is running with
> Kafka/zookeeper.
>
> the Kafka infrastructure :
>
> - 3 zookeeper with exhibitor running together.
> - 6 brokers.
>
> requirements:
>
> - terminate all of the 3 zookeeper with exhibitor, and also 6 brokers.
> - launch 5 new zookeepers without exhibitor.
> - launch 6 new brokers.
> - 0 or minimum downtime, try our best no impact for the Kafka client.
>
>
> so far, I have two prepared solution, but any of them has pros & cons, the
> solution A is too much manual operations needed, the solution B is some
> development effort required and also need to migration all of the Kafka
> client to use the self-developed Kafka client with `fail-over` feature.
>
> *Solution A*: simply speaking, this solution is mainly working on manual
> operation to add znode / broker to the existed cluster, when all of the
> thing ready, I will step by step to terminate the old one.
>
>
>
>
> *Solution B:* simply speaking, the solution will work like fail-over
> way, a new cluster will be launched, and control Kafka client's traffic to
> the new cluster, and terminate the old cluster.
>
>
>
>
>
>
Re: Any suggestion about Kafka/Zookeeper migration.
Posted by Tim Visher <ti...@gmail.com>.
On Thu, Jan 4, 2018 at 11:55 AM, Tony Liu <ji...@zuora.com> wrote:
> Thanks Timmy.
>
> I appreciate your suggestion, but for us, it's hard to do that , the
> problem are :
>
> - 6+ Kafka cluster.
> - more than 200 micro services or applications are talking with Kafka
> now.
>
> Any other suggestion for our case ?
>
I've never done it but the standard migration guide should get you an
upgrade in place. It's just a terribly tedious sounding process.
https://kafka.apache.org/documentation/#upgrade
With that kind of topology I'd probably recommend getting something in
front of kafka that can do a cut over for you like HAProxy or DNS.
Obviously that's a future consideration though. For now, definitely go with
the in place migration.
>
>
> On Thu, Jan 4, 2018 at 6:06 AM, Tim Visher <ti...@gmail.com> wrote:
>
> > On Wed, Jan 3, 2018 at 11:39 PM, Tony Liu <ji...@zuora.com>
> wrote:
> >
> > > This post here is aimed to ask experience about what did you do
> migration
> > > `Kafka/zookeeper` ? :)
> > >
> > >
> > I think the easiest way to migrate kafka is to do a blue/green deploy.
> > Stand up an entire new cluster, cut over producers to it, wait for the
> old
> > cluster to drain, cut over consumers, then shut down the old cluster.
> >
> > Especially in AWS or any other virtualized environment this should be
> very
> > easy assuming that you don't need to deploy many things to do the
> cutover.
> > DNS or some other load balancing solution could help but you may be in
> too
> > deep to put that in place now.
> >
> > As far as migrating zookeeper, you could stand up a new zookeeper cluster
> > when spinning up your new kafka cluster and do the same thing.
> >
> > My team is currently doing exactly that right now.
> >
> > --
> >
> > In Christ,
> >
> > Timmy V.
> >
> > https://blog.twonegatives.com
> > https://five.sentenc.es
> >
>
Re: Any suggestion about Kafka/Zookeeper migration.
Posted by Tony Liu <ji...@zuora.com>.
Thanks Timmy.
I appreciate your suggestion, but for us, it's hard to do that , the
problem are :
- 6+ Kafka cluster.
- more than 200 micro services or applications are talking with Kafka
now.
Any other suggestion for our case ?
On Thu, Jan 4, 2018 at 6:06 AM, Tim Visher <ti...@gmail.com> wrote:
> On Wed, Jan 3, 2018 at 11:39 PM, Tony Liu <ji...@zuora.com> wrote:
>
> > This post here is aimed to ask experience about what did you do migration
> > `Kafka/zookeeper` ? :)
> >
> >
> I think the easiest way to migrate kafka is to do a blue/green deploy.
> Stand up an entire new cluster, cut over producers to it, wait for the old
> cluster to drain, cut over consumers, then shut down the old cluster.
>
> Especially in AWS or any other virtualized environment this should be very
> easy assuming that you don't need to deploy many things to do the cutover.
> DNS or some other load balancing solution could help but you may be in too
> deep to put that in place now.
>
> As far as migrating zookeeper, you could stand up a new zookeeper cluster
> when spinning up your new kafka cluster and do the same thing.
>
> My team is currently doing exactly that right now.
>
> --
>
> In Christ,
>
> Timmy V.
>
> https://blog.twonegatives.com
> https://five.sentenc.es
>
Re: Any suggestion about Kafka/Zookeeper migration.
Posted by Tim Visher <ti...@gmail.com>.
On Wed, Jan 3, 2018 at 11:39 PM, Tony Liu <ji...@zuora.com> wrote:
> This post here is aimed to ask experience about what did you do migration
> `Kafka/zookeeper` ? :)
>
>
I think the easiest way to migrate kafka is to do a blue/green deploy.
Stand up an entire new cluster, cut over producers to it, wait for the old
cluster to drain, cut over consumers, then shut down the old cluster.
Especially in AWS or any other virtualized environment this should be very
easy assuming that you don't need to deploy many things to do the cutover.
DNS or some other load balancing solution could help but you may be in too
deep to put that in place now.
As far as migrating zookeeper, you could stand up a new zookeeper cluster
when spinning up your new kafka cluster and do the same thing.
My team is currently doing exactly that right now.
--
In Christ,
Timmy V.
https://blog.twonegatives.com
https://five.sentenc.es