How to upgrade Kafka broker to a newer version without impacting live traffic?

[Connie Yang <cy...@gmail.com>]

Hi All,

If I want to upgrade from Kafka 0.8.1.1 to a newer version, what are the
steps that I need to do without impacting production traffic?  More
specifically, how do I have the new brokers to continue writing to the
existing logs assuming the file format has not changed?

Is there a migration tool for us to use?

Thanks,
Connie

Re: How to upgrade Kafka broker to a newer version without impacting live traffic?

[Yury Ruchin <yu...@gmail.com>]

A couple of points to keep in mind during the rolling update:

- "Controlled shutdown" should be used to bring brokers down (
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.ControlledShutdown),
so that brokers gracefully transfer leadership before actually going offline
- after the last broker is brought back online, preferred replica election
should be initiated to restore the brokers' leadership for the respective
partitions


2014-08-28 3:21 GMT+04:00 Alexis Midon <al...@airbedandbreakfast.com>
:

> Assuming the partitions are replicated, a migration plan could be:
> - install latest kafka on broker $i, /etc/kakfa-NEW
> - update server config in  /etc/kakfa-NEW/config/server.properties to match
> the old one (in particular zookeeper.connect and log.dirs)
> - stop broker $i
> - start broker $i from /etc/kakfa-NEW/bin/....
>
> I successfully did that for a 6-broker cluster going from 0.8.1 to 0.8.1.1.
>
> Alexis
>
>
>
>
>
> On Wed, Aug 27, 2014 at 3:56 PM, Connie Yang <cy...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > If I want to upgrade from Kafka 0.8.1.1 to a newer version, what are the
> > steps that I need to do without impacting production traffic?  More
> > specifically, how do I have the new brokers to continue writing to the
> > existing logs assuming the file format has not changed?
> >
> > Is there a migration tool for us to use?
> >
> > Thanks,
> > Connie
> >
>

Re: How to upgrade Kafka broker to a newer version without impacting live traffic?

[Alexis Midon <al...@airbedandbreakfast.com>]

Assuming the partitions are replicated, a migration plan could be:
- install latest kafka on broker $i, /etc/kakfa-NEW
- update server config in  /etc/kakfa-NEW/config/server.properties to match
the old one (in particular zookeeper.connect and log.dirs)
- stop broker $i
- start broker $i from /etc/kakfa-NEW/bin/....

I successfully did that for a 6-broker cluster going from 0.8.1 to 0.8.1.1.

Alexis





On Wed, Aug 27, 2014 at 3:56 PM, Connie Yang <cy...@gmail.com> wrote:

> Hi All,
>
> If I want to upgrade from Kafka 0.8.1.1 to a newer version, what are the
> steps that I need to do without impacting production traffic?  More
> specifically, how do I have the new brokers to continue writing to the
> existing logs assuming the file format has not changed?
>
> Is there a migration tool for us to use?
>
> Thanks,
> Connie
>