You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kiran Nagaraja <ki...@gmail.com> on 2016/03/17 01:17:09 UTC
Not seeing disk writes on all replica brokers
Hi,
I'm running version kafka_2.11-0.9.0.1 in a 2 broker configuration with 10
partitions that are replicated - replication.factor=2
Here's the output from 'kafkat partitions':
Topic Partition Leader Replicas ISRs
bookmarks 0 0 [0, 1] [0, 1]
bookmarks 1 1 [1, 0] [1, 0]
bookmarks 2 0 [0, 1] [0, 1]
bookmarks 3 1 [1, 0] [1, 0]
bookmarks 4 0 [0, 1] [0, 1]
bookmarks 5 1 [1, 0] [1, 0]
bookmarks 6 0 [0, 1] [0, 1]
bookmarks 7 1 [1, 0] [1, 0]
bookmarks 8 0 [0, 1] [0, 1]
bookmarks 9 1 [1, 0] [1, 0]
We monitor the disk writes and I only see writes at broker 0, and broker 1
sees none (not comparable at all). I do see comparable network traffic at
the 2 brokers.
Anything obvious I'm missing.
Thanks,
- Kiran
Re: Not seeing disk writes on all replica brokers
Posted by Kiran Nagaraja <ki...@gmail.com>.
>
> what exactly are you monitoring in terms of the "disk writes"?
Looking at both bytes written and number of IOPs. I do see the batched
writes activity from the OS but they seem to be missing on some replicas.
I looked into the kafka log and found there is frequent change in
cluster/ISR for the partitions:
[2016-03-17 15:02:35,265] INFO Partition [test-rep2-8p,1] on broker 1:
Shrinking ISR for partition [test-rep2-8p,1] from 1,0 to 1
(kafka.cluster.Partition)
[2016-03-17 15:02:35,991] INFO Partition [test-rep2-8p,3] on broker 1:
Shrinking ISR for partition [test-rep2-8p,3] from 1,0 to 1
(kafka.cluster.Partition)
[2016-03-17 15:02:38,662] INFO Partition [test-rep2-8p,7] on broker 1:
Shrinking ISR for partition [test-rep2-8p,7] from 1,0 to 1
(kafka.cluster.Partition)
[2016-03-17 15:02:38,677] INFO Partition [test-rep2-8p,5] on broker 1:
Shrinking ISR for partition [test-rep2-8p,5] from 1,0 to 1
(kafka.cluster.Partition)
[2016-03-17 15:02:40,022] INFO Partition [test-rep2-8p,7] on broker 1:
Expanding ISR for partition [test-rep2-8p,7] from 1 to 1,0
(kafka.cluster.Partition)
[2016-03-17 15:02:40,161] INFO Partition [test-rep2-8p,5] on broker 1:
Expanding ISR for partition [test-rep2-8p,5] from 1 to 1,0
(kafka.cluster.Partition)
[2016-03-17 15:02:40,239] INFO Partition [test-rep2-8p,1] on broker 1:
Expanding ISR for partition [test-rep2-8p,1] from 1 to 1,0
(kafka.cluster.Partition)
[2016-03-17 15:02:40,296] INFO Partition [test-rep2-8p,3] on broker 1:
Expanding ISR for partition [test-rep2-8p,3] from 1 to 1,0
(kafka.cluster.Partition)
[2016-03-17 15:03:02,015
This happens during throughput benchmarking. I suspect Kafka's losing
heartbeats. Anyway to make the broker less sensitive to lost or delayed
heartbeats?
Thanks,
- Kiran
On Wed, Mar 16, 2016 at 7:00 PM, Christian Posta <ch...@gmail.com>
wrote:
> what exactly are you monitoring in terms of the "disk writes"? kafka writes
> to disk but does not fsync. so you may or may not see activity at the disk
> when writing; it'll be up to the OS to write back to the disk.
>
> On Wed, Mar 16, 2016 at 5:17 PM, Kiran Nagaraja <ki...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm running version kafka_2.11-0.9.0.1 in a 2 broker configuration with
> 10
> > partitions that are replicated - replication.factor=2
> >
> > Here's the output from 'kafkat partitions':
> >
> > Topic Partition Leader Replicas ISRs
> >
> > bookmarks 0 0 [0, 1] [0, 1]
> >
> > bookmarks 1 1 [1, 0] [1, 0]
> >
> > bookmarks 2 0 [0, 1] [0, 1]
> >
> > bookmarks 3 1 [1, 0] [1, 0]
> >
> > bookmarks 4 0 [0, 1] [0, 1]
> >
> > bookmarks 5 1 [1, 0] [1, 0]
> >
> > bookmarks 6 0 [0, 1] [0, 1]
> >
> > bookmarks 7 1 [1, 0] [1, 0]
> >
> > bookmarks 8 0 [0, 1] [0, 1]
> >
> > bookmarks 9 1 [1, 0] [1, 0]
> >
> > We monitor the disk writes and I only see writes at broker 0, and broker
> 1
> > sees none (not comparable at all). I do see comparable network traffic at
> > the 2 brokers.
> >
> > Anything obvious I'm missing.
> >
> > Thanks,
> > - Kiran
> >
>
>
>
> --
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
>
Re: Not seeing disk writes on all replica brokers
Posted by Christian Posta <ch...@gmail.com>.
what exactly are you monitoring in terms of the "disk writes"? kafka writes
to disk but does not fsync. so you may or may not see activity at the disk
when writing; it'll be up to the OS to write back to the disk.
On Wed, Mar 16, 2016 at 5:17 PM, Kiran Nagaraja <ki...@gmail.com>
wrote:
> Hi,
>
> I'm running version kafka_2.11-0.9.0.1 in a 2 broker configuration with 10
> partitions that are replicated - replication.factor=2
>
> Here's the output from 'kafkat partitions':
>
> Topic Partition Leader Replicas ISRs
>
> bookmarks 0 0 [0, 1] [0, 1]
>
> bookmarks 1 1 [1, 0] [1, 0]
>
> bookmarks 2 0 [0, 1] [0, 1]
>
> bookmarks 3 1 [1, 0] [1, 0]
>
> bookmarks 4 0 [0, 1] [0, 1]
>
> bookmarks 5 1 [1, 0] [1, 0]
>
> bookmarks 6 0 [0, 1] [0, 1]
>
> bookmarks 7 1 [1, 0] [1, 0]
>
> bookmarks 8 0 [0, 1] [0, 1]
>
> bookmarks 9 1 [1, 0] [1, 0]
>
> We monitor the disk writes and I only see writes at broker 0, and broker 1
> sees none (not comparable at all). I do see comparable network traffic at
> the 2 brokers.
>
> Anything obvious I'm missing.
>
> Thanks,
> - Kiran
>
--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io