Number of Kafka producers

[David Yu <da...@optimizely.com>]

Is there a way to control the number of producers? Our Samza job writes a
lot of data to the downstream Kafka topic. I was wondering if there is a
way to optimize concurrency by creating more async producers.

Thanks,
David

Re: Number of Kafka producers

[Jagadish Venkatraman <ja...@gmail.com>]

Hi David,

Have you tried increasing the number of containers? Our benchmarks show
about 1M messages per sec processed on one machine. You should be able to
scale linearly by increasing number of containers. All relevant
Kafka producer properties like batch size, compression and linger can be
passed to Samza.

For concurrency, the upcoming release of Samza includes the multi threading
feature. It will provide built in support for processing multiple tasks in
parallel.

SAMZA-863 and SAMZA-961 are relevant jiras tracking this feature.

Thanks

On Wednesday, July 27, 2016, David Yu <da...@optimizely.com> wrote:

> Is there a way to control the number of producers? Our Samza job writes a
> lot of data to the downstream Kafka topic. I was wondering if there is a
> way to optimize concurrency by creating more async producers.
>
> Thanks,
> David
>


-- 
Sent from my iphone.