0.7 -> 0.8 Protocol Upgrade in production environments

[Clark Breyman <cl...@breyman.com>]

I'm curious what the recommended best practice is for migrating a
production environment with replication from 0.7 to 0.8 given the protocol
upgrade. Some specific questions I have are:

a) Is it possible to mix 0.7 and 0.8 servers for a given partition during
the migration?

b) If we can't mix server versions, how do we do a zero-downtime upgrade?

c) Will 0.7 clients continue to communicate correctly on-the-wire with the
0.8 server until they can be upgraded?


Any war stories of production upgrades would be helpful. Thanks in advance.
- Clark

Re: 0.7 -> 0.8 Protocol Upgrade in production environments

[Jason Rosenberg <jb...@squareup.com>]

In my case, we just rolled out a separate 0.8 cluster, and migrated
producers to it over time (took several weeks to get everything
updated to the new cluster).  In the transition, we had consumers
running for both clusters.  Once no traffic was flowing on the old
cluster, we then shut down the 0.7 cluster.

I think this was simpler than trying to forward data from one to the other.

You can't have the 2 existing in a mixed state in the same cluster, at
all.  The protocols and storage formats are entirely different.

Jason

On Tue, Jan 21, 2014 at 5:38 PM, François Langelier
<f....@gmail.com> wrote:
> Hi,
>
> I'm also in the process to upgrade from 0.7 to 0.8. WIth the informations I
> found, your best friend is here :
> https://cwiki.apache.org/confluence/display/KAFKA/Migrating+from+0.7+to+0.8
> It's a migration tool from 0.7 to 0.8.
>
> You will have to install 0.8 while you still have you 0.7 running. When
> those and your 0.8 consumers run, you will have to lunch the migration tool
> that will consume your 0.7 message to produce them to your 0.8 environment.
> When all your message will be transfer to your 0.8, you will them be able
> to transfer your 0.7 producers to your 0.8 environment.
>
> A picture is worth 1000 words :
> [image: Inline image 1]
>
> So for you b) , yes it's possible, but as far as I know, you will need to
> have a tool to verify the duplication of your message because some message
> will probably be consume in 0.7 and 0.8 while you are migrating because the
> offsets are incompatible between 0.7 and 0.8...
>
> c) yes if you use the migration tool
>
>
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie
> Supérieure<http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génies <http://www.jdgets.com/> 2011 à 2014
> Argentier Fraternité du Piranhas <http://fraternitedupiranha.com/> 2012-2014
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On Tue, Jan 21, 2014 at 12:48 PM, Clark Breyman <cl...@breyman.com> wrote:
>
>> I'm curious what the recommended best practice is for migrating a
>> production environment with replication from 0.7 to 0.8 given the protocol
>> upgrade. Some specific questions I have are:
>>
>> a) Is it possible to mix 0.7 and 0.8 servers for a given partition during
>> the migration?
>>
>> b) If we can't mix server versions, how do we do a zero-downtime upgrade?
>>
>> c) Will 0.7 clients continue to communicate correctly on-the-wire with the
>> 0.8 server until they can be upgraded?
>>
>>
>> Any war stories of production upgrades would be helpful. Thanks in advance.
>> - Clark
>>

Re: 0.7 -> 0.8 Protocol Upgrade in production environments

[François Langelier <f....@gmail.com>]

Hi,

I'm also in the process to upgrade from 0.7 to 0.8. WIth the informations I
found, your best friend is here :
https://cwiki.apache.org/confluence/display/KAFKA/Migrating+from+0.7+to+0.8
It's a migration tool from 0.7 to 0.8.

You will have to install 0.8 while you still have you 0.7 running. When
those and your 0.8 consumers run, you will have to lunch the migration tool
that will consume your 0.7 message to produce them to your 0.8 environment.
When all your message will be transfer to your 0.8, you will them be able
to transfer your 0.7 producers to your 0.8 environment.

A picture is worth 1000 words :
[image: Inline image 1]

So for you b) , yes it's possible, but as far as I know, you will need to
have a tool to verify the duplication of your message because some message
will probably be consume in 0.7 and 0.8 while you are migrating because the
offsets are incompatible between 0.7 and 0.8...

c) yes if you use the migration tool





François Langelier
Étudiant en génie Logiciel - École de Technologie
Supérieure<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génies <http://www.jdgets.com/> 2011 à 2014
Argentier Fraternité du Piranhas <http://fraternitedupiranha.com/> 2012-2014
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior


On Tue, Jan 21, 2014 at 12:48 PM, Clark Breyman <cl...@breyman.com> wrote:

> I'm curious what the recommended best practice is for migrating a
> production environment with replication from 0.7 to 0.8 given the protocol
> upgrade. Some specific questions I have are:
>
> a) Is it possible to mix 0.7 and 0.8 servers for a given partition during
> the migration?
>
> b) If we can't mix server versions, how do we do a zero-downtime upgrade?
>
> c) Will 0.7 clients continue to communicate correctly on-the-wire with the
> 0.8 server until they can be upgraded?
>
>
> Any war stories of production upgrades would be helpful. Thanks in advance.
> - Clark
>