Decreasing number of partitions INCREASED write throughput--thoughts?

[John Yost <ho...@gmail.com>]

Hi Everyone,

I discovered yesterday completely by accident. When I went from writing to
10 topics each with 10 partitions to 10 topics each with 2 partitions--all
other config params are the same--my write throughput almost doubled!  I
was not expecting this as I've always thought--and, admittedly, I am still
coming up to speed on Kafka--that increasing parallelism via increasing the
number of partitions and/or topics would increase write throughput. I was
actually experimenting going from 10 partitions per topic to 20 partitions
in an effort to increase throughput but stumbled upon 10X2.

Important notes:

1. Replication factor is 1
2. async producer
3. request.required.acks is 1

Any ideas?

--John

Re: Decreasing number of partitions INCREASED write throughput--thoughts?

[Gerard Klijs <ge...@dizzit.com>]

Interesting,

Do you have any way to look into the amount and size of the actual traffic
being send. I assume because there are more partitions the actual sending
of the messages is separated into more messages, leading to more overhead.

On Sat, Feb 20, 2016 at 12:57 PM John Yost <ho...@gmail.com> wrote:

> Hi Everyone,
>
> I discovered yesterday completely by accident. When I went from writing to
> 10 topics each with 10 partitions to 10 topics each with 2 partitions--all
> other config params are the same--my write throughput almost doubled!  I
> was not expecting this as I've always thought--and, admittedly, I am still
> coming up to speed on Kafka--that increasing parallelism via increasing the
> number of partitions and/or topics would increase write throughput. I was
> actually experimenting going from 10 partitions per topic to 20 partitions
> in an effort to increase throughput but stumbled upon 10X2.
>
> Important notes:
>
> 1. Replication factor is 1
> 2. async producer
> 3. request.required.acks is 1
>
> Any ideas?
>
> --John
>