Increase speed of replica fetch in kafka?

[Archie <an...@gmail.com>]

Is there anyway I can speed up the rate at which the slave replicas will
fetch data from master?

I am using bin/kafka-producer-perf-test.sh to test the throughput of my
producer. And I have set a client quota of 50 MBps. Now without any
replicas I am getting throughput ~ 50MBps but when replication factor is
set to 3, it reduces to ~30 MBps.

There is no other traffic in the network so I am not sure why things are
slowing down. Is there some parameter like
replica.socket.receive.buffer.bytes, replica.fetch.min.bytes that needs to
be tuned to achieve high throughput? How can I speed up my replicas?

Thanks,
Archie

Re: Increase speed of replica fetch in kafka?

[Alexander Binzberger <al...@wingcon.com>]

Have you tried using more partitions for the topic?


Am 05.05.2017 um 23:39 schrieb Girish Aher:
> Have you tried increasing num.replica.fetchers on your brokers?
>
> On Fri, May 5, 2017 at 12:59 PM, Archie <an...@gmail.com> wrote:
>
>> Is there anyway I can speed up the rate at which the slave replicas will
>> fetch data from master?
>>
>> I am using bin/kafka-producer-perf-test.sh to test the throughput of my
>> producer. And I have set a client quota of 50 MBps. Now without any
>> replicas I am getting throughput ~ 50MBps but when replication factor is
>> set to 3, it reduces to ~30 MBps.
>>
>> There is no other traffic in the network so I am not sure why things are
>> slowing down. Is there some parameter like
>> replica.socket.receive.buffer.bytes, replica.fetch.min.bytes that needs to
>> be tuned to achieve high throughput? How can I speed up my replicas?
>>
>> Thanks,
>> Archie
>>

-- 
Alexander Binzberger
System Designer - WINGcon AG
Tel. +49 7543 966-119

Sitz der Gesellschaft: Langenargen
Registergericht: ULM, HRB 734260
USt-Id.: DE232931635, WEEE-Id.: DE74015979
Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter), Tobias Treß
Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter), Martin Sauter

Re: Increase speed of replica fetch in kafka?

[Girish Aher <gi...@gmail.com>]

Have you tried increasing num.replica.fetchers on your brokers?

On Fri, May 5, 2017 at 12:59 PM, Archie <an...@gmail.com> wrote:

> Is there anyway I can speed up the rate at which the slave replicas will
> fetch data from master?
>
> I am using bin/kafka-producer-perf-test.sh to test the throughput of my
> producer. And I have set a client quota of 50 MBps. Now without any
> replicas I am getting throughput ~ 50MBps but when replication factor is
> set to 3, it reduces to ~30 MBps.
>
> There is no other traffic in the network so I am not sure why things are
> slowing down. Is there some parameter like
> replica.socket.receive.buffer.bytes, replica.fetch.min.bytes that needs to
> be tuned to achieve high throughput? How can I speed up my replicas?
>
> Thanks,
> Archie
>