You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rajiv Kurian <ra...@signalfx.com> on 2015/12/01 17:51:39 UTC
Any gotchas upgrading to 0.9?
I plan to upgrade both the server and clients to 0.9. Had a few questions
before I went ahead with the upgrade:
1. Do all brokers need to be on 0.9? Currently we are running 0.8.2. We'd
ideally like to convert only a few brokers to 0.9 and only if we don't see
problems convert the rest.
2. Is it possible to run Kafka 0.9 clients (specifically the consumer) with
Kafka 0.8.2 brokers?
Any link to the upgrade path would be really useful.
Thanks,
Rajiv
Re: Any gotchas upgrading to 0.9?
Posted by Rajiv Kurian <ra...@signalfx.com>.
Scratch that. On more careful observation I do see this in the logs:
inter.broker.protocol.version = 0.8.2.X
On Mon, Dec 14, 2015 at 10:25 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> I am in the process of updating to 0.9 and had another question.
>
> The docs at http://kafka.apache.org/documentation.html#upgrade say that
> to do a smooth upgrade from 0.8.2.X to 0.9 we can use the
> inter.broker.protocol.version config to control what protocol to use. After
> upgrading how can we tell that we are running the right inter broker
> protocol? Is there any jmx metric I could look at to confirm that I started
> the broker correctly? Or maybe something in the logs? I grepped for
> inter.broker.protocol.version in the logs and didn't find anything.
>
> How about when I change the protocol version back to 0.9.0.0? I have
> followed the steps outlined in the guide on a test environment and my
> broker and client metrics look fine. But it would be great to get some kind
> of confirmation that each step (upgrade to 0.9 with 0.8.2.X protocol and
> then restart with 0.9.0.0 protocol) worked.
>
> Thanks,
> Rajiv
>
> On Tue, Dec 1, 2015 at 11:39 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Old clients should work well with the new servers, while vice versa is not
>> generally true mainly because the new consumer client uses a few new
>> request types that only the new brokers can recognize. So the normal
>> upgrade path is server-first, clients later.
>>
>> Filed https://issues.apache.org/jira/browse/KAFKA-2923 to update the
>> upgrade doc to include it.
>>
>> Guozhang
>>
>> On Tue, Dec 1, 2015 at 11:14 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
>>
>> > Thanks folks. Anywhere I can read about client compatibility i.e. old
>> > clients working with new servers and new clients working with old
>> servers?
>> >
>> > Thanks,
>> > Rajiv
>> >
>> > On Tue, Dec 1, 2015 at 10:54 AM, Jay Kreps <ja...@confluent.io> wrote:
>> >
>> > > I think the point is that we should ideally try to cover all these in
>> the
>> > > "upgrade" notes.
>> > >
>> > > -Jay
>> > >
>> > > On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar <
>> aauradkar@linkedin.com
>> > >
>> > > wrote:
>> > >
>> > > > Rajiv,
>> > > >
>> > > > By default, the quota is unlimited until you decide to configure
>> them
>> > > > explicitly.
>> > > > And yes, we did get rid of "replica.lag.max.messages", so that
>> > > > configuration will no longer apply.
>> > > >
>> > > > Aditya
>> > > >
>> > > > On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <
>> tsnyder@blackberry.com>
>> > > > wrote:
>> > > >
>> > > > > The quota page is here:
>> > > > > http://kafka.apache.org/documentation.html#design_quotas
>> > > > >
>> > > > > "By default, each unique client-id receives a fixed quota in
>> > bytes/sec
>> > > as
>> > > > > configured by the cluster (quota.producer.default,
>> > > > quota.consumer.default)"
>> > > > >
>> > > > >
>> > > > > I also noticed there's been a change in the replication
>> configuration
>> > > > > while reading:
>> > > > >
>> > > >
>> > >
>> >
>> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
>> > > > >
>> > > > > It may not break anything, but it may impact how you decide to
>> > > configure
>> > > > > and monitor replication
>> > > > >
>> > > > > "Now there is only one value you need to configure on the server
>> > which
>> > > is
>> > > > > replica.lag.time.max.ms. The interpretation of this has changed
>> to
>> > be
>> > > > the
>> > > > > time for which a replica has been out-of-sync with the leader.
>> Stuck
>> > or
>> > > > > failed replicas are detected the same way as before - if a replica
>> > > fails
>> > > > to
>> > > > > send a fetch request for longer than replica.lag.time.max.ms, it
>> is
>> > > > > considered dead and is removed from the ISR. The mechanism of
>> > detecting
>> > > > > slow replicas has changed - if a replica starts lagging behind the
>> > > leader
>> > > > > for longer than replica.lag.time.max.ms, then it is considered
>> too
>> > > slow
>> > > > > and is removed from the ISR. So even if there is a spike in
>> traffic
>> > and
>> > > > > large batches of messages are written on the leader, unless the
>> > replica
>> > > > > consistently remains behind the leader for
>> replica.lag.time.max.ms,
>> > it
>> > > > > will not shuffle in and out of the ISR."
>> > > > >
>> > > > >
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: Rajiv Kurian [mailto:rajiv@signalfx.com]
>> > > > > Sent: Tuesday, December 01, 2015 11:57
>> > > > > To: users@kafka.apache.org
>> > > > > Subject: Re: Any gotchas upgrading to 0.9?
>> > > > >
>> > > > > Also I remember reading (can't find now) something about default
>> > > traffic
>> > > > > quotas. I'd hope the default quotas are very large (infinite?) and
>> > not
>> > > > > small so that compatibility is maintained. It would be very
>> > unfortunate
>> > > > if
>> > > > > some of our traffic was throttled because of the upgrade because
>> of
>> > > magic
>> > > > > defaults. For example we have a certain cluster dedicated to
>> serving
>> > a
>> > > > > single important topic and we'd hate for it to be throttled
>> because
>> > of
>> > > > > incorrect defaults.
>> > > > >
>> > > > > Thanks,
>> > > > > Rajiv
>> > > > >
>> > > > > On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com>
>> > > wrote:
>> > > > >
>> > > > > > I saw the upgrade path documentation at
>> > > > > > http://kafka.apache.org/documentation.html and that kind of
>> > answers
>> > > > (1).
>> > > > > > Not sure if there is anything about client compatibility though.
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <
>> rajiv@signalfx.com>
>> > > > wrote:
>> > > > > >
>> > > > > >> I plan to upgrade both the server and clients to 0.9. Had a few
>> > > > > questions
>> > > > > >> before I went ahead with the upgrade:
>> > > > > >>
>> > > > > >> 1. Do all brokers need to be on 0.9? Currently we are running
>> > 0.8.2.
>> > > > > We'd
>> > > > > >> ideally like to convert only a few brokers to 0.9 and only if
>> we
>> > > don't
>> > > > > see
>> > > > > >> problems convert the rest.
>> > > > > >>
>> > > > > >> 2. Is it possible to run Kafka 0.9 clients (specifically the
>> > > consumer)
>> > > > > >> with Kafka 0.8.2 brokers?
>> > > > > >>
>> > > > > >> Any link to the upgrade path would be really useful.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> Rajiv
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
Re: Any gotchas upgrading to 0.9?
Posted by Rajiv Kurian <ra...@signalfx.com>.
I am in the process of updating to 0.9 and had another question.
The docs at http://kafka.apache.org/documentation.html#upgrade say that to
do a smooth upgrade from 0.8.2.X to 0.9 we can use the
inter.broker.protocol.version config to control what protocol to use. After
upgrading how can we tell that we are running the right inter broker
protocol? Is there any jmx metric I could look at to confirm that I started
the broker correctly? Or maybe something in the logs? I grepped for
inter.broker.protocol.version in the logs and didn't find anything.
How about when I change the protocol version back to 0.9.0.0? I have
followed the steps outlined in the guide on a test environment and my
broker and client metrics look fine. But it would be great to get some kind
of confirmation that each step (upgrade to 0.9 with 0.8.2.X protocol and
then restart with 0.9.0.0 protocol) worked.
Thanks,
Rajiv
On Tue, Dec 1, 2015 at 11:39 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Old clients should work well with the new servers, while vice versa is not
> generally true mainly because the new consumer client uses a few new
> request types that only the new brokers can recognize. So the normal
> upgrade path is server-first, clients later.
>
> Filed https://issues.apache.org/jira/browse/KAFKA-2923 to update the
> upgrade doc to include it.
>
> Guozhang
>
> On Tue, Dec 1, 2015 at 11:14 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
>
> > Thanks folks. Anywhere I can read about client compatibility i.e. old
> > clients working with new servers and new clients working with old
> servers?
> >
> > Thanks,
> > Rajiv
> >
> > On Tue, Dec 1, 2015 at 10:54 AM, Jay Kreps <ja...@confluent.io> wrote:
> >
> > > I think the point is that we should ideally try to cover all these in
> the
> > > "upgrade" notes.
> > >
> > > -Jay
> > >
> > > On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar <
> aauradkar@linkedin.com
> > >
> > > wrote:
> > >
> > > > Rajiv,
> > > >
> > > > By default, the quota is unlimited until you decide to configure them
> > > > explicitly.
> > > > And yes, we did get rid of "replica.lag.max.messages", so that
> > > > configuration will no longer apply.
> > > >
> > > > Aditya
> > > >
> > > > On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <tsnyder@blackberry.com
> >
> > > > wrote:
> > > >
> > > > > The quota page is here:
> > > > > http://kafka.apache.org/documentation.html#design_quotas
> > > > >
> > > > > "By default, each unique client-id receives a fixed quota in
> > bytes/sec
> > > as
> > > > > configured by the cluster (quota.producer.default,
> > > > quota.consumer.default)"
> > > > >
> > > > >
> > > > > I also noticed there's been a change in the replication
> configuration
> > > > > while reading:
> > > > >
> > > >
> > >
> >
> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
> > > > >
> > > > > It may not break anything, but it may impact how you decide to
> > > configure
> > > > > and monitor replication
> > > > >
> > > > > "Now there is only one value you need to configure on the server
> > which
> > > is
> > > > > replica.lag.time.max.ms. The interpretation of this has changed to
> > be
> > > > the
> > > > > time for which a replica has been out-of-sync with the leader.
> Stuck
> > or
> > > > > failed replicas are detected the same way as before - if a replica
> > > fails
> > > > to
> > > > > send a fetch request for longer than replica.lag.time.max.ms, it
> is
> > > > > considered dead and is removed from the ISR. The mechanism of
> > detecting
> > > > > slow replicas has changed - if a replica starts lagging behind the
> > > leader
> > > > > for longer than replica.lag.time.max.ms, then it is considered too
> > > slow
> > > > > and is removed from the ISR. So even if there is a spike in traffic
> > and
> > > > > large batches of messages are written on the leader, unless the
> > replica
> > > > > consistently remains behind the leader for replica.lag.time.max.ms
> ,
> > it
> > > > > will not shuffle in and out of the ISR."
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Rajiv Kurian [mailto:rajiv@signalfx.com]
> > > > > Sent: Tuesday, December 01, 2015 11:57
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Any gotchas upgrading to 0.9?
> > > > >
> > > > > Also I remember reading (can't find now) something about default
> > > traffic
> > > > > quotas. I'd hope the default quotas are very large (infinite?) and
> > not
> > > > > small so that compatibility is maintained. It would be very
> > unfortunate
> > > > if
> > > > > some of our traffic was throttled because of the upgrade because of
> > > magic
> > > > > defaults. For example we have a certain cluster dedicated to
> serving
> > a
> > > > > single important topic and we'd hate for it to be throttled because
> > of
> > > > > incorrect defaults.
> > > > >
> > > > > Thanks,
> > > > > Rajiv
> > > > >
> > > > > On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com>
> > > wrote:
> > > > >
> > > > > > I saw the upgrade path documentation at
> > > > > > http://kafka.apache.org/documentation.html and that kind of
> > answers
> > > > (1).
> > > > > > Not sure if there is anything about client compatibility though.
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <rajiv@signalfx.com
> >
> > > > wrote:
> > > > > >
> > > > > >> I plan to upgrade both the server and clients to 0.9. Had a few
> > > > > questions
> > > > > >> before I went ahead with the upgrade:
> > > > > >>
> > > > > >> 1. Do all brokers need to be on 0.9? Currently we are running
> > 0.8.2.
> > > > > We'd
> > > > > >> ideally like to convert only a few brokers to 0.9 and only if we
> > > don't
> > > > > see
> > > > > >> problems convert the rest.
> > > > > >>
> > > > > >> 2. Is it possible to run Kafka 0.9 clients (specifically the
> > > consumer)
> > > > > >> with Kafka 0.8.2 brokers?
> > > > > >>
> > > > > >> Any link to the upgrade path would be really useful.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Rajiv
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Any gotchas upgrading to 0.9?
Posted by Guozhang Wang <wa...@gmail.com>.
Old clients should work well with the new servers, while vice versa is not
generally true mainly because the new consumer client uses a few new
request types that only the new brokers can recognize. So the normal
upgrade path is server-first, clients later.
Filed https://issues.apache.org/jira/browse/KAFKA-2923 to update the
upgrade doc to include it.
Guozhang
On Tue, Dec 1, 2015 at 11:14 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> Thanks folks. Anywhere I can read about client compatibility i.e. old
> clients working with new servers and new clients working with old servers?
>
> Thanks,
> Rajiv
>
> On Tue, Dec 1, 2015 at 10:54 AM, Jay Kreps <ja...@confluent.io> wrote:
>
> > I think the point is that we should ideally try to cover all these in the
> > "upgrade" notes.
> >
> > -Jay
> >
> > On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar <aauradkar@linkedin.com
> >
> > wrote:
> >
> > > Rajiv,
> > >
> > > By default, the quota is unlimited until you decide to configure them
> > > explicitly.
> > > And yes, we did get rid of "replica.lag.max.messages", so that
> > > configuration will no longer apply.
> > >
> > > Aditya
> > >
> > > On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <ts...@blackberry.com>
> > > wrote:
> > >
> > > > The quota page is here:
> > > > http://kafka.apache.org/documentation.html#design_quotas
> > > >
> > > > "By default, each unique client-id receives a fixed quota in
> bytes/sec
> > as
> > > > configured by the cluster (quota.producer.default,
> > > quota.consumer.default)"
> > > >
> > > >
> > > > I also noticed there's been a change in the replication configuration
> > > > while reading:
> > > >
> > >
> >
> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
> > > >
> > > > It may not break anything, but it may impact how you decide to
> > configure
> > > > and monitor replication
> > > >
> > > > "Now there is only one value you need to configure on the server
> which
> > is
> > > > replica.lag.time.max.ms. The interpretation of this has changed to
> be
> > > the
> > > > time for which a replica has been out-of-sync with the leader. Stuck
> or
> > > > failed replicas are detected the same way as before - if a replica
> > fails
> > > to
> > > > send a fetch request for longer than replica.lag.time.max.ms, it is
> > > > considered dead and is removed from the ISR. The mechanism of
> detecting
> > > > slow replicas has changed - if a replica starts lagging behind the
> > leader
> > > > for longer than replica.lag.time.max.ms, then it is considered too
> > slow
> > > > and is removed from the ISR. So even if there is a spike in traffic
> and
> > > > large batches of messages are written on the leader, unless the
> replica
> > > > consistently remains behind the leader for replica.lag.time.max.ms,
> it
> > > > will not shuffle in and out of the ISR."
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Rajiv Kurian [mailto:rajiv@signalfx.com]
> > > > Sent: Tuesday, December 01, 2015 11:57
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Any gotchas upgrading to 0.9?
> > > >
> > > > Also I remember reading (can't find now) something about default
> > traffic
> > > > quotas. I'd hope the default quotas are very large (infinite?) and
> not
> > > > small so that compatibility is maintained. It would be very
> unfortunate
> > > if
> > > > some of our traffic was throttled because of the upgrade because of
> > magic
> > > > defaults. For example we have a certain cluster dedicated to serving
> a
> > > > single important topic and we'd hate for it to be throttled because
> of
> > > > incorrect defaults.
> > > >
> > > > Thanks,
> > > > Rajiv
> > > >
> > > > On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com>
> > wrote:
> > > >
> > > > > I saw the upgrade path documentation at
> > > > > http://kafka.apache.org/documentation.html and that kind of
> answers
> > > (1).
> > > > > Not sure if there is anything about client compatibility though.
> > > > >
> > > > >
> > > > > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com>
> > > wrote:
> > > > >
> > > > >> I plan to upgrade both the server and clients to 0.9. Had a few
> > > > questions
> > > > >> before I went ahead with the upgrade:
> > > > >>
> > > > >> 1. Do all brokers need to be on 0.9? Currently we are running
> 0.8.2.
> > > > We'd
> > > > >> ideally like to convert only a few brokers to 0.9 and only if we
> > don't
> > > > see
> > > > >> problems convert the rest.
> > > > >>
> > > > >> 2. Is it possible to run Kafka 0.9 clients (specifically the
> > consumer)
> > > > >> with Kafka 0.8.2 brokers?
> > > > >>
> > > > >> Any link to the upgrade path would be really useful.
> > > > >>
> > > > >> Thanks,
> > > > >> Rajiv
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
--
-- Guozhang
Re: Any gotchas upgrading to 0.9?
Posted by Rajiv Kurian <ra...@signalfx.com>.
Thanks folks. Anywhere I can read about client compatibility i.e. old
clients working with new servers and new clients working with old servers?
Thanks,
Rajiv
On Tue, Dec 1, 2015 at 10:54 AM, Jay Kreps <ja...@confluent.io> wrote:
> I think the point is that we should ideally try to cover all these in the
> "upgrade" notes.
>
> -Jay
>
> On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar <aa...@linkedin.com>
> wrote:
>
> > Rajiv,
> >
> > By default, the quota is unlimited until you decide to configure them
> > explicitly.
> > And yes, we did get rid of "replica.lag.max.messages", so that
> > configuration will no longer apply.
> >
> > Aditya
> >
> > On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <ts...@blackberry.com>
> > wrote:
> >
> > > The quota page is here:
> > > http://kafka.apache.org/documentation.html#design_quotas
> > >
> > > "By default, each unique client-id receives a fixed quota in bytes/sec
> as
> > > configured by the cluster (quota.producer.default,
> > quota.consumer.default)"
> > >
> > >
> > > I also noticed there's been a change in the replication configuration
> > > while reading:
> > >
> >
> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
> > >
> > > It may not break anything, but it may impact how you decide to
> configure
> > > and monitor replication
> > >
> > > "Now there is only one value you need to configure on the server which
> is
> > > replica.lag.time.max.ms. The interpretation of this has changed to be
> > the
> > > time for which a replica has been out-of-sync with the leader. Stuck or
> > > failed replicas are detected the same way as before - if a replica
> fails
> > to
> > > send a fetch request for longer than replica.lag.time.max.ms, it is
> > > considered dead and is removed from the ISR. The mechanism of detecting
> > > slow replicas has changed - if a replica starts lagging behind the
> leader
> > > for longer than replica.lag.time.max.ms, then it is considered too
> slow
> > > and is removed from the ISR. So even if there is a spike in traffic and
> > > large batches of messages are written on the leader, unless the replica
> > > consistently remains behind the leader for replica.lag.time.max.ms, it
> > > will not shuffle in and out of the ISR."
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Rajiv Kurian [mailto:rajiv@signalfx.com]
> > > Sent: Tuesday, December 01, 2015 11:57
> > > To: users@kafka.apache.org
> > > Subject: Re: Any gotchas upgrading to 0.9?
> > >
> > > Also I remember reading (can't find now) something about default
> traffic
> > > quotas. I'd hope the default quotas are very large (infinite?) and not
> > > small so that compatibility is maintained. It would be very unfortunate
> > if
> > > some of our traffic was throttled because of the upgrade because of
> magic
> > > defaults. For example we have a certain cluster dedicated to serving a
> > > single important topic and we'd hate for it to be throttled because of
> > > incorrect defaults.
> > >
> > > Thanks,
> > > Rajiv
> > >
> > > On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com>
> wrote:
> > >
> > > > I saw the upgrade path documentation at
> > > > http://kafka.apache.org/documentation.html and that kind of answers
> > (1).
> > > > Not sure if there is anything about client compatibility though.
> > > >
> > > >
> > > > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com>
> > wrote:
> > > >
> > > >> I plan to upgrade both the server and clients to 0.9. Had a few
> > > questions
> > > >> before I went ahead with the upgrade:
> > > >>
> > > >> 1. Do all brokers need to be on 0.9? Currently we are running 0.8.2.
> > > We'd
> > > >> ideally like to convert only a few brokers to 0.9 and only if we
> don't
> > > see
> > > >> problems convert the rest.
> > > >>
> > > >> 2. Is it possible to run Kafka 0.9 clients (specifically the
> consumer)
> > > >> with Kafka 0.8.2 brokers?
> > > >>
> > > >> Any link to the upgrade path would be really useful.
> > > >>
> > > >> Thanks,
> > > >> Rajiv
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Any gotchas upgrading to 0.9?
Posted by Jay Kreps <ja...@confluent.io>.
I think the point is that we should ideally try to cover all these in the
"upgrade" notes.
-Jay
On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar <aa...@linkedin.com>
wrote:
> Rajiv,
>
> By default, the quota is unlimited until you decide to configure them
> explicitly.
> And yes, we did get rid of "replica.lag.max.messages", so that
> configuration will no longer apply.
>
> Aditya
>
> On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <ts...@blackberry.com>
> wrote:
>
> > The quota page is here:
> > http://kafka.apache.org/documentation.html#design_quotas
> >
> > "By default, each unique client-id receives a fixed quota in bytes/sec as
> > configured by the cluster (quota.producer.default,
> quota.consumer.default)"
> >
> >
> > I also noticed there's been a change in the replication configuration
> > while reading:
> >
> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
> >
> > It may not break anything, but it may impact how you decide to configure
> > and monitor replication
> >
> > "Now there is only one value you need to configure on the server which is
> > replica.lag.time.max.ms. The interpretation of this has changed to be
> the
> > time for which a replica has been out-of-sync with the leader. Stuck or
> > failed replicas are detected the same way as before - if a replica fails
> to
> > send a fetch request for longer than replica.lag.time.max.ms, it is
> > considered dead and is removed from the ISR. The mechanism of detecting
> > slow replicas has changed - if a replica starts lagging behind the leader
> > for longer than replica.lag.time.max.ms, then it is considered too slow
> > and is removed from the ISR. So even if there is a spike in traffic and
> > large batches of messages are written on the leader, unless the replica
> > consistently remains behind the leader for replica.lag.time.max.ms, it
> > will not shuffle in and out of the ISR."
> >
> >
> >
> > -----Original Message-----
> > From: Rajiv Kurian [mailto:rajiv@signalfx.com]
> > Sent: Tuesday, December 01, 2015 11:57
> > To: users@kafka.apache.org
> > Subject: Re: Any gotchas upgrading to 0.9?
> >
> > Also I remember reading (can't find now) something about default traffic
> > quotas. I'd hope the default quotas are very large (infinite?) and not
> > small so that compatibility is maintained. It would be very unfortunate
> if
> > some of our traffic was throttled because of the upgrade because of magic
> > defaults. For example we have a certain cluster dedicated to serving a
> > single important topic and we'd hate for it to be throttled because of
> > incorrect defaults.
> >
> > Thanks,
> > Rajiv
> >
> > On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> >
> > > I saw the upgrade path documentation at
> > > http://kafka.apache.org/documentation.html and that kind of answers
> (1).
> > > Not sure if there is anything about client compatibility though.
> > >
> > >
> > > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com>
> wrote:
> > >
> > >> I plan to upgrade both the server and clients to 0.9. Had a few
> > questions
> > >> before I went ahead with the upgrade:
> > >>
> > >> 1. Do all brokers need to be on 0.9? Currently we are running 0.8.2.
> > We'd
> > >> ideally like to convert only a few brokers to 0.9 and only if we don't
> > see
> > >> problems convert the rest.
> > >>
> > >> 2. Is it possible to run Kafka 0.9 clients (specifically the consumer)
> > >> with Kafka 0.8.2 brokers?
> > >>
> > >> Any link to the upgrade path would be really useful.
> > >>
> > >> Thanks,
> > >> Rajiv
> > >>
> > >
> > >
> >
>
Re: Any gotchas upgrading to 0.9?
Posted by Aditya Auradkar <aa...@linkedin.com>.
Rajiv,
By default, the quota is unlimited until you decide to configure them
explicitly.
And yes, we did get rid of "replica.lag.max.messages", so that
configuration will no longer apply.
Aditya
On Tue, Dec 1, 2015 at 10:24 AM, Todd Snyder <ts...@blackberry.com> wrote:
> The quota page is here:
> http://kafka.apache.org/documentation.html#design_quotas
>
> "By default, each unique client-id receives a fixed quota in bytes/sec as
> configured by the cluster (quota.producer.default, quota.consumer.default)"
>
>
> I also noticed there's been a change in the replication configuration
> while reading:
> http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
>
> It may not break anything, but it may impact how you decide to configure
> and monitor replication
>
> "Now there is only one value you need to configure on the server which is
> replica.lag.time.max.ms. The interpretation of this has changed to be the
> time for which a replica has been out-of-sync with the leader. Stuck or
> failed replicas are detected the same way as before - if a replica fails to
> send a fetch request for longer than replica.lag.time.max.ms, it is
> considered dead and is removed from the ISR. The mechanism of detecting
> slow replicas has changed - if a replica starts lagging behind the leader
> for longer than replica.lag.time.max.ms, then it is considered too slow
> and is removed from the ISR. So even if there is a spike in traffic and
> large batches of messages are written on the leader, unless the replica
> consistently remains behind the leader for replica.lag.time.max.ms, it
> will not shuffle in and out of the ISR."
>
>
>
> -----Original Message-----
> From: Rajiv Kurian [mailto:rajiv@signalfx.com]
> Sent: Tuesday, December 01, 2015 11:57
> To: users@kafka.apache.org
> Subject: Re: Any gotchas upgrading to 0.9?
>
> Also I remember reading (can't find now) something about default traffic
> quotas. I'd hope the default quotas are very large (infinite?) and not
> small so that compatibility is maintained. It would be very unfortunate if
> some of our traffic was throttled because of the upgrade because of magic
> defaults. For example we have a certain cluster dedicated to serving a
> single important topic and we'd hate for it to be throttled because of
> incorrect defaults.
>
> Thanks,
> Rajiv
>
> On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
>
> > I saw the upgrade path documentation at
> > http://kafka.apache.org/documentation.html and that kind of answers (1).
> > Not sure if there is anything about client compatibility though.
> >
> >
> > On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> >
> >> I plan to upgrade both the server and clients to 0.9. Had a few
> questions
> >> before I went ahead with the upgrade:
> >>
> >> 1. Do all brokers need to be on 0.9? Currently we are running 0.8.2.
> We'd
> >> ideally like to convert only a few brokers to 0.9 and only if we don't
> see
> >> problems convert the rest.
> >>
> >> 2. Is it possible to run Kafka 0.9 clients (specifically the consumer)
> >> with Kafka 0.8.2 brokers?
> >>
> >> Any link to the upgrade path would be really useful.
> >>
> >> Thanks,
> >> Rajiv
> >>
> >
> >
>
RE: Any gotchas upgrading to 0.9?
Posted by Todd Snyder <ts...@blackberry.com>.
The quota page is here: http://kafka.apache.org/documentation.html#design_quotas
"By default, each unique client-id receives a fixed quota in bytes/sec as configured by the cluster (quota.producer.default, quota.consumer.default)"
I also noticed there's been a change in the replication configuration while reading: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
It may not break anything, but it may impact how you decide to configure and monitor replication
"Now there is only one value you need to configure on the server which is replica.lag.time.max.ms. The interpretation of this has changed to be the time for which a replica has been out-of-sync with the leader. Stuck or failed replicas are detected the same way as before - if a replica fails to send a fetch request for longer than replica.lag.time.max.ms, it is considered dead and is removed from the ISR. The mechanism of detecting slow replicas has changed - if a replica starts lagging behind the leader for longer than replica.lag.time.max.ms, then it is considered too slow and is removed from the ISR. So even if there is a spike in traffic and large batches of messages are written on the leader, unless the replica consistently remains behind the leader for replica.lag.time.max.ms, it will not shuffle in and out of the ISR."
-----Original Message-----
From: Rajiv Kurian [mailto:rajiv@signalfx.com]
Sent: Tuesday, December 01, 2015 11:57
To: users@kafka.apache.org
Subject: Re: Any gotchas upgrading to 0.9?
Also I remember reading (can't find now) something about default traffic
quotas. I'd hope the default quotas are very large (infinite?) and not
small so that compatibility is maintained. It would be very unfortunate if
some of our traffic was throttled because of the upgrade because of magic
defaults. For example we have a certain cluster dedicated to serving a
single important topic and we'd hate for it to be throttled because of
incorrect defaults.
Thanks,
Rajiv
On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> I saw the upgrade path documentation at
> http://kafka.apache.org/documentation.html and that kind of answers (1).
> Not sure if there is anything about client compatibility though.
>
>
> On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
>
>> I plan to upgrade both the server and clients to 0.9. Had a few questions
>> before I went ahead with the upgrade:
>>
>> 1. Do all brokers need to be on 0.9? Currently we are running 0.8.2. We'd
>> ideally like to convert only a few brokers to 0.9 and only if we don't see
>> problems convert the rest.
>>
>> 2. Is it possible to run Kafka 0.9 clients (specifically the consumer)
>> with Kafka 0.8.2 brokers?
>>
>> Any link to the upgrade path would be really useful.
>>
>> Thanks,
>> Rajiv
>>
>
>
Re: Any gotchas upgrading to 0.9?
Posted by Rajiv Kurian <ra...@signalfx.com>.
Also I remember reading (can't find now) something about default traffic
quotas. I'd hope the default quotas are very large (infinite?) and not
small so that compatibility is maintained. It would be very unfortunate if
some of our traffic was throttled because of the upgrade because of magic
defaults. For example we have a certain cluster dedicated to serving a
single important topic and we'd hate for it to be throttled because of
incorrect defaults.
Thanks,
Rajiv
On Tue, Dec 1, 2015 at 8:54 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> I saw the upgrade path documentation at
> http://kafka.apache.org/documentation.html and that kind of answers (1).
> Not sure if there is anything about client compatibility though.
>
>
> On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
>
>> I plan to upgrade both the server and clients to 0.9. Had a few questions
>> before I went ahead with the upgrade:
>>
>> 1. Do all brokers need to be on 0.9? Currently we are running 0.8.2. We'd
>> ideally like to convert only a few brokers to 0.9 and only if we don't see
>> problems convert the rest.
>>
>> 2. Is it possible to run Kafka 0.9 clients (specifically the consumer)
>> with Kafka 0.8.2 brokers?
>>
>> Any link to the upgrade path would be really useful.
>>
>> Thanks,
>> Rajiv
>>
>
>
Re: Any gotchas upgrading to 0.9?
Posted by Rajiv Kurian <ra...@signalfx.com>.
I saw the upgrade path documentation at
http://kafka.apache.org/documentation.html and that kind of answers (1).
Not sure if there is anything about client compatibility though.
On Tue, Dec 1, 2015 at 8:51 AM, Rajiv Kurian <ra...@signalfx.com> wrote:
> I plan to upgrade both the server and clients to 0.9. Had a few questions
> before I went ahead with the upgrade:
>
> 1. Do all brokers need to be on 0.9? Currently we are running 0.8.2. We'd
> ideally like to convert only a few brokers to 0.9 and only if we don't see
> problems convert the rest.
>
> 2. Is it possible to run Kafka 0.9 clients (specifically the consumer)
> with Kafka 0.8.2 brokers?
>
> Any link to the upgrade path would be really useful.
>
> Thanks,
> Rajiv
>