You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Anish Mashankar <an...@systeminsights.com> on 2017/11/21 08:54:49 UTC
Questions about upgrading to Kafka 1.0 from 0.10.0
Hello Kafka users!
The first question that I have is related to the documentation. I see that
we no longer have to change the message format version when upgrading to
1.0. So, will all clients continue to work after performing the rolling
upgrade?
We are running Kafka v0.10.0.0. The Kafka ecosystem consists of several
nodejs producer-consumers (NoKafka library which using v0.9 and v0.10
protocols), few kafka connect projects (v0.10.2) and a java consumer
running on v0.8 library. Which of the applications can I expect to break
after the rolling upgrade due to mismatch in protocol versions?
Also, I see the quote in the documentation saying that if downtimes were
acceptable, we need not go through changing the protocol versions and only
update the code. Does this mean even older clients will have no problems
connecting to the brokers post upgrade?
--
Regards,
Anish Samir Mashankar
Site Reliability Engineer
VIMANA
Phone: +91-9789870733
Re: Questions about upgrading to Kafka 1.0 from 0.10.0
Posted by Anish Mashankar <an...@systeminsights.com>.
I understand it now. I must've done something wrong last time.
Thank you.
On Wed 22 Nov, 2017, 5:21 PM Ismael Juma, <is...@juma.me.uk> wrote:
> Anish,
>
> That's correct, the broker will down convert messages for older consumers
> after log.message.format.version is increased. As I said, however, this has
> an impact on efficiency so you should only do it if the traffic from old
> consumers is low (as mentioned in the release notes). Also note that you
> _cannot_ safely downgrade log.message.format.version. If you do that, you
> will get errors.
>
> Ismael
>
> On Wed, Nov 22, 2017 at 10:13 AM, Anish Mashankar <
> anish@systeminsights.com>
> wrote:
>
> > Thanks Ismael.
> > Just need a clarification on something because I observed getting errors
> > from the v0.9 and v0.10 consumer for invalid message format.
> > Is it true that after bumping the consumer version post rolling upgrade
> > will not cause message format mismatch errors in the Consumer?
> >
> > On Tue, Nov 21, 2017 at 7:37 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Anish,
> > >
> > > The documentation is a bit misleading, see the following JIRA:
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-6238
> > >
> > > All of your clients will still work after the upgrade, but there is an
> > > efficiency hit if the message format used for the topic is newer than
> the
> > > message format supported by the consumer.
> > >
> > > Also, we found a memory leak in 1.0.0, so I'd recommend you upgrade to
> > > 0.11.0.2 or wait until 1.0.1.
> > >
> > > Ismael
> > >
> > > On Tue, Nov 21, 2017 at 8:54 AM, Anish Mashankar <
> > anish@systeminsights.com
> > > >
> > > wrote:
> > >
> > > > Hello Kafka users!
> > > > The first question that I have is related to the documentation. I see
> > > that
> > > > we no longer have to change the message format version when upgrading
> > to
> > > > 1.0. So, will all clients continue to work after performing the
> rolling
> > > > upgrade?
> > > > We are running Kafka v0.10.0.0. The Kafka ecosystem consists of
> several
> > > > nodejs producer-consumers (NoKafka library which using v0.9 and v0.10
> > > > protocols), few kafka connect projects (v0.10.2) and a java consumer
> > > > running on v0.8 library. Which of the applications can I expect to
> > break
> > > > after the rolling upgrade due to mismatch in protocol versions?
> > > > Also, I see the quote in the documentation saying that if downtimes
> > were
> > > > acceptable, we need not go through changing the protocol versions and
> > > only
> > > > update the code. Does this mean even older clients will have no
> > problems
> > > > connecting to the brokers post upgrade?
> > > > --
> > > >
> > > > Regards,
> > > >
> > > > Anish Samir Mashankar
> > > >
> > > > Site Reliability Engineer
> > > >
> > > > VIMANA
> > > >
> > > > Phone: +91-9789870733 <+91%2097898%2070733>
> > > >
> > >
> > --
> >
> > Regards,
> >
> > Anish Samir Mashankar
> >
> > Site Reliability Engineer
> >
> > VIMANA
> >
> > Phone: +91-9789870733
> >
>
--
Regards,
Anish Samir Mashankar
Site Reliability Engineer
VIMANA
Phone: +91-9789870733
Re: Questions about upgrading to Kafka 1.0 from 0.10.0
Posted by Ismael Juma <is...@juma.me.uk>.
Anish,
That's correct, the broker will down convert messages for older consumers
after log.message.format.version is increased. As I said, however, this has
an impact on efficiency so you should only do it if the traffic from old
consumers is low (as mentioned in the release notes). Also note that you
_cannot_ safely downgrade log.message.format.version. If you do that, you
will get errors.
Ismael
On Wed, Nov 22, 2017 at 10:13 AM, Anish Mashankar <an...@systeminsights.com>
wrote:
> Thanks Ismael.
> Just need a clarification on something because I observed getting errors
> from the v0.9 and v0.10 consumer for invalid message format.
> Is it true that after bumping the consumer version post rolling upgrade
> will not cause message format mismatch errors in the Consumer?
>
> On Tue, Nov 21, 2017 at 7:37 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Anish,
> >
> > The documentation is a bit misleading, see the following JIRA:
> >
> > https://issues.apache.org/jira/browse/KAFKA-6238
> >
> > All of your clients will still work after the upgrade, but there is an
> > efficiency hit if the message format used for the topic is newer than the
> > message format supported by the consumer.
> >
> > Also, we found a memory leak in 1.0.0, so I'd recommend you upgrade to
> > 0.11.0.2 or wait until 1.0.1.
> >
> > Ismael
> >
> > On Tue, Nov 21, 2017 at 8:54 AM, Anish Mashankar <
> anish@systeminsights.com
> > >
> > wrote:
> >
> > > Hello Kafka users!
> > > The first question that I have is related to the documentation. I see
> > that
> > > we no longer have to change the message format version when upgrading
> to
> > > 1.0. So, will all clients continue to work after performing the rolling
> > > upgrade?
> > > We are running Kafka v0.10.0.0. The Kafka ecosystem consists of several
> > > nodejs producer-consumers (NoKafka library which using v0.9 and v0.10
> > > protocols), few kafka connect projects (v0.10.2) and a java consumer
> > > running on v0.8 library. Which of the applications can I expect to
> break
> > > after the rolling upgrade due to mismatch in protocol versions?
> > > Also, I see the quote in the documentation saying that if downtimes
> were
> > > acceptable, we need not go through changing the protocol versions and
> > only
> > > update the code. Does this mean even older clients will have no
> problems
> > > connecting to the brokers post upgrade?
> > > --
> > >
> > > Regards,
> > >
> > > Anish Samir Mashankar
> > >
> > > Site Reliability Engineer
> > >
> > > VIMANA
> > >
> > > Phone: +91-9789870733 <+91%2097898%2070733>
> > >
> >
> --
>
> Regards,
>
> Anish Samir Mashankar
>
> Site Reliability Engineer
>
> VIMANA
>
> Phone: +91-9789870733
>
Re: Questions about upgrading to Kafka 1.0 from 0.10.0
Posted by Anish Mashankar <an...@systeminsights.com>.
Thanks Ismael.
Just need a clarification on something because I observed getting errors
from the v0.9 and v0.10 consumer for invalid message format.
Is it true that after bumping the consumer version post rolling upgrade
will not cause message format mismatch errors in the Consumer?
On Tue, Nov 21, 2017 at 7:37 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Anish,
>
> The documentation is a bit misleading, see the following JIRA:
>
> https://issues.apache.org/jira/browse/KAFKA-6238
>
> All of your clients will still work after the upgrade, but there is an
> efficiency hit if the message format used for the topic is newer than the
> message format supported by the consumer.
>
> Also, we found a memory leak in 1.0.0, so I'd recommend you upgrade to
> 0.11.0.2 or wait until 1.0.1.
>
> Ismael
>
> On Tue, Nov 21, 2017 at 8:54 AM, Anish Mashankar <anish@systeminsights.com
> >
> wrote:
>
> > Hello Kafka users!
> > The first question that I have is related to the documentation. I see
> that
> > we no longer have to change the message format version when upgrading to
> > 1.0. So, will all clients continue to work after performing the rolling
> > upgrade?
> > We are running Kafka v0.10.0.0. The Kafka ecosystem consists of several
> > nodejs producer-consumers (NoKafka library which using v0.9 and v0.10
> > protocols), few kafka connect projects (v0.10.2) and a java consumer
> > running on v0.8 library. Which of the applications can I expect to break
> > after the rolling upgrade due to mismatch in protocol versions?
> > Also, I see the quote in the documentation saying that if downtimes were
> > acceptable, we need not go through changing the protocol versions and
> only
> > update the code. Does this mean even older clients will have no problems
> > connecting to the brokers post upgrade?
> > --
> >
> > Regards,
> >
> > Anish Samir Mashankar
> >
> > Site Reliability Engineer
> >
> > VIMANA
> >
> > Phone: +91-9789870733 <+91%2097898%2070733>
> >
>
--
Regards,
Anish Samir Mashankar
Site Reliability Engineer
VIMANA
Phone: +91-9789870733
Re: Questions about upgrading to Kafka 1.0 from 0.10.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Anish,
The documentation is a bit misleading, see the following JIRA:
https://issues.apache.org/jira/browse/KAFKA-6238
All of your clients will still work after the upgrade, but there is an
efficiency hit if the message format used for the topic is newer than the
message format supported by the consumer.
Also, we found a memory leak in 1.0.0, so I'd recommend you upgrade to
0.11.0.2 or wait until 1.0.1.
Ismael
On Tue, Nov 21, 2017 at 8:54 AM, Anish Mashankar <an...@systeminsights.com>
wrote:
> Hello Kafka users!
> The first question that I have is related to the documentation. I see that
> we no longer have to change the message format version when upgrading to
> 1.0. So, will all clients continue to work after performing the rolling
> upgrade?
> We are running Kafka v0.10.0.0. The Kafka ecosystem consists of several
> nodejs producer-consumers (NoKafka library which using v0.9 and v0.10
> protocols), few kafka connect projects (v0.10.2) and a java consumer
> running on v0.8 library. Which of the applications can I expect to break
> after the rolling upgrade due to mismatch in protocol versions?
> Also, I see the quote in the documentation saying that if downtimes were
> acceptable, we need not go through changing the protocol versions and only
> update the code. Does this mean even older clients will have no problems
> connecting to the brokers post upgrade?
> --
>
> Regards,
>
> Anish Samir Mashankar
>
> Site Reliability Engineer
>
> VIMANA
>
> Phone: +91-9789870733
>