Pattern for using kafka producer API

[pushkar priyadarshi <pr...@gmail.com>]

What is the most appropriate design for using kafka producer from
performance view point.I had few in my mind.

1.Since single kafka producer object have synchronization; using single
producer object from multiple thread might not be efficient.so one way
would be to use multiple kafka producer from inside same thread.

2.Have multiple thread each having it's own instance of producer.This has
thread overheads if kafka internally using the same semantics.

It would be great if someone can comment on these approaches or suggest
widely used one.
P.S. im using 0.8.0 and mostly concerned with async producer.

Thanks And Regards,
Pushkar

Re: Pattern for using kafka producer API

[Jun Rao <ju...@gmail.com>]

If you are only worried about throughput, you can use one producer in async
mode. You can tune the batch size and time for better performance.

Thanks,

Jun


On Sun, Feb 9, 2014 at 11:42 PM, pushkar priyadarshi <
priyadarshi.pushkar@gmail.com> wrote:

> What is the most appropriate design for using kafka producer from
> performance view point.I had few in my mind.
>
> 1.Since single kafka producer object have synchronization; using single
> producer object from multiple thread might not be efficient.so one way
> would be to use multiple kafka producer from inside same thread.
>
> 2.Have multiple thread each having it's own instance of producer.This has
> thread overheads if kafka internally using the same semantics.
>
> It would be great if someone can comment on these approaches or suggest
> widely used one.
> P.S. im using 0.8.0 and mostly concerned with async producer.
>
> Thanks And Regards,
> Pushkar
>