You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by nitin sharma <ku...@gmail.com> on 2014/12/17 17:08:08 UTC
Increase in Kafka replication fetcher thread not reducing log replication
Hi All,
I am trying to figure out best configuration for my Kafka brokers so that
in case of restarted, the new node catch up with Leader at quick pace.
My test environment has 2 kafka brokers and 1 Topic with one Partition.
I first ran the test (Test#1) with default setting, i.e.
num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB).
it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec (
37.9*1024/700)
Later I increased the num.replica.fetchers = 5 and
replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the
replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec.
i ran two more test and results got much worse:
Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB.
replication rate = 92.7 MB/sec
Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB.
replication rate = 93.54 MB/sec
Any reason why increasing the replica fetcher thread or increase in fetch
max bytes not increasing my replication rate linearly.
note: in all the test CPU utilization was not more than 45%
Regards,
Nitin Kumar Sharma.
Re: Increase in Kafka replication fetcher thread not reducing log replication
Posted by nitin sharma <ku...@gmail.com>.
HI All -- has any one faced the problem i have posted in my previous email
?
Regards,
Nitin Kumar Sharma.
On Wed, Dec 17, 2014 at 11:08 AM, nitin sharma <ku...@gmail.com>
wrote:
>
> Hi All,
>
> I am trying to figure out best configuration for my Kafka brokers so that
> in case of restarted, the new node catch up with Leader at quick pace.
>
> My test environment has 2 kafka brokers and 1 Topic with one Partition.
>
> I first ran the test (Test#1) with default setting, i.e.
> num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB).
> it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec (
> 37.9*1024/700)
>
> Later I increased the num.replica.fetchers = 5 and
> replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the
> replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec.
>
> i ran two more test and results got much worse:
> Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB.
> replication rate = 92.7 MB/sec
>
> Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB.
> replication rate = 93.54 MB/sec
>
> Any reason why increasing the replica fetcher thread or increase in fetch
> max bytes not increasing my replication rate linearly.
>
> note: in all the test CPU utilization was not more than 45%
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
Re: Increase in Kafka replication fetcher thread not reducing log replication
Posted by svante karlsson <sa...@csi.se>.
What kind of network do you have? gigabit? if so 90 MB/s would make
sense....
Also since you have one partition what's your raw transfer speed to the
disk? 90 MB/s makes sense here as well...
If I were looking for rapid replica catch up I'd have at least 2x Gbit and
partitioned topics spread out over several physical nodes. That would make
it possible to catch up at ~200 MB/s without totally blocking normal
operation of the cluster. For a single partition you would face lower catch
up rate since all data is coming from one physical disk
/svante
2014-12-22 17:26 GMT+01:00 Jun Rao <ju...@confluent.io>:
> Increasing replica.fetch.max.bytes will help, but will only get diminishing
> returns at some point.
>
> Increasing num.replica.fetchers will also help. You need to make sure that
> the leaders are balanced in the cluster. See
> http://kafka.apache.org/documentation.html#basic_ops_leader_balancing
>
> Thanks,
>
> Jun
>
> On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma <kumarsharma.nitin@gmail.com
> >
> wrote:
>
> > Hi All,
> >
> > I am trying to figure out best configuration for my Kafka brokers so that
> > in case of restarted, the new node catch up with Leader at quick pace.
> >
> > My test environment has 2 kafka brokers and 1 Topic with one Partition.
> >
> > I first ran the test (Test#1) with default setting, i.e.
> > num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1
> MB).
> > it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec (
> > 37.9*1024/700)
> >
> > Later I increased the num.replica.fetchers = 5 and
> > replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the
> > replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221
> MB/sec.
> >
> > i ran two more test and results got much worse:
> > Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB.
> > replication rate = 92.7 MB/sec
> >
> > Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB.
> > replication rate = 93.54 MB/sec
> >
> > Any reason why increasing the replica fetcher thread or increase in fetch
> > max bytes not increasing my replication rate linearly.
> >
> > note: in all the test CPU utilization was not more than 45%
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
>
Re: Increase in Kafka replication fetcher thread not reducing log replication
Posted by Jun Rao <ju...@confluent.io>.
Increasing replica.fetch.max.bytes will help, but will only get diminishing
returns at some point.
Increasing num.replica.fetchers will also help. You need to make sure that
the leaders are balanced in the cluster. See
http://kafka.apache.org/documentation.html#basic_ops_leader_balancing
Thanks,
Jun
On Wed, Dec 17, 2014 at 8:08 AM, nitin sharma <ku...@gmail.com>
wrote:
> Hi All,
>
> I am trying to figure out best configuration for my Kafka brokers so that
> in case of restarted, the new node catch up with Leader at quick pace.
>
> My test environment has 2 kafka brokers and 1 Topic with one Partition.
>
> I first ran the test (Test#1) with default setting, i.e.
> num.replica.fetchers =1 and replica.fetch.max.bytes = 1048576 Bytes (1 MB).
> it took 11min and 40 sec to copy the 37.9 GB @ the rate 55.5MB/sec (
> 37.9*1024/700)
>
> Later I increased the num.replica.fetchers = 5 and
> replica.fetch.max.bytes=1MB and ran another test (Test #2). I got the
> replica @ 89 MB/sec. Which is good but i was expecting 4*55 = 221 MB/sec.
>
> i ran two more test and results got much worse:
> Test#3 : replica thread = 5 and replica.fetch.max.bytes = 5MB.
> replication rate = 92.7 MB/sec
>
> Test #4: replication thread = 20 and replica.fetch.max.bytes = 5 MB.
> replication rate = 93.54 MB/sec
>
> Any reason why increasing the replica fetcher thread or increase in fetch
> max bytes not increasing my replication rate linearly.
>
> note: in all the test CPU utilization was not more than 45%
>
>
> Regards,
> Nitin Kumar Sharma.
>