You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shaun Senecal <sh...@lithium.com> on 2015/10/02 03:09:30 UTC
Shrinking ISR with no load on brokers or incoming messages
Hi
I have noticed that when our brokers have no incoming connections (just connections to other brokers and to the ZK cluster) we get messages about shrinking the ISR for some partitions
[2015-10-02 00:58:31,239] INFO Partition [lia.stage.raw_events,9] on broker 1: Shrinking ISR for partition [lia.stage.raw_events,9] from 1,0,2 to 1 (kafka.cluster.Partition)
...
[2015-10-02 00:58:31,335] INFO Partition [lia.stage.raw_events,9] on broker 1: Expanding ISR for partition [lia.stage.raw_events,9] from 1 to 1,0 (kafka.cluster.Partition)
[2015-10-02 00:58:31,430] INFO Partition [lia.stage.raw_events,9] on broker 1: Expanding ISR for partition [lia.stage.raw_events,9] from 1,0 to 1,0,2 (kafka.cluster.Partition)
It seems weird to me that the ISR would ever change when there is no load on the brokers and no incoming messages at all. Does this indicate a problem with the cluster, or is this normal?
Thanks
Shaun
Re: Shrinking ISR with no load on brokers or incoming messages
Posted by Shaun Senecal <sh...@lithium.com>.
To provide a little more detail into the problem, I have extracted what I think are the relevant log entries
[2015-10-01 23:19:39,834] INFO Logs loading complete. (kafka.log.LogManager)
...
[2015-10-01 23:24:00,310] INFO Partition [lia.stage.bundles,7] on broker 0: Shrinking ISR for partition [lia.stage.bundles,7] from 2,1,0 to 1,0 (kafka.cluster.Partition)
...
[2015-10-01 23:51:30,404] INFO Partition [lia.stage.raw_events,17] on broker 0: Shrinking ISR for partition [lia.stage.raw_events,17] from 1,0,2 to 0,2 (kafka.cluster.Partition)
...
[2015-10-02 00:59:23,182] !!!!!!!! starting feeding messages !!!!!!!
[2015-10-02 00:59:30,357] INFO Partition [lia.stage.raw_events,11] on broker 0: Shrinking ISR for partition [lia.stage.raw_events,11] from 0,1,2 to 0 (kafka.cluster.Partition)
[2015-10-02 00:59:30,671] INFO Partition [lia.stage.raw_events,11] on broker 0: Expanding ISR for partition [lia.stage.raw_events,11] from 0,2 to 0,2,1 (kafka.cluster.Partition)
It looks like ~30mins after the broker had come up, it was still shrinking ISRs. I would expect the ISRs to shrink shortly after the brokers came up, but I would have expected that to happen in 1 shot and not 30mins after the brokers came up.
The second block of ISR adjustments following the "starting feeding messages" happened immediately after I started producing messages into the lia.stage.raw_events topic. This also strikes me as odd, since I thought ISR adjustments should have been more of a rare occurrence.
For reference, here are the config values we use for replication:
default.replication.factor=2
num.replica.fetchers=4
replica.lag.time.max.ms=10000
replica.lag.max.messages=4000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.fetch.max.bytes=52428800
replica.fetch.wait.max.ms=10000
replica.fetch.min.bytes=1024
replica.high.watermark.checkpoint.interval.ms=5000
________________________________________
From: Shaun Senecal <sh...@lithium.com>
Sent: October 1, 2015 6:09 PM
To: users@kafka.apache.org
Subject: Shrinking ISR with no load on brokers or incoming messages
Hi
I have noticed that when our brokers have no incoming connections (just connections to other brokers and to the ZK cluster) we get messages about shrinking the ISR for some partitions
[2015-10-02 00:58:31,239] INFO Partition [lia.stage.raw_events,9] on broker 1: Shrinking ISR for partition [lia.stage.raw_events,9] from 1,0,2 to 1 (kafka.cluster.Partition)
...
[2015-10-02 00:58:31,335] INFO Partition [lia.stage.raw_events,9] on broker 1: Expanding ISR for partition [lia.stage.raw_events,9] from 1 to 1,0 (kafka.cluster.Partition)
[2015-10-02 00:58:31,430] INFO Partition [lia.stage.raw_events,9] on broker 1: Expanding ISR for partition [lia.stage.raw_events,9] from 1,0 to 1,0,2 (kafka.cluster.Partition)
It seems weird to me that the ISR would ever change when there is no load on the brokers and no incoming messages at all. Does this indicate a problem with the cluster, or is this normal?
Thanks
Shaun