Simple KafkaProducer to handle multiple requests or not

[Amit K <am...@gmail.com>]

Hi,

I am using simple kafka producer (java based, version 0.9.0.0) in an
application where I receive lot of hits (about 50 per seconds, in much like
servlet way) on application that has kafka producer. Per request comes
different set of records.

I am using only one instance of kafka producer to push the records to kafka
cluster. Is that good way to use kafka producer? As it is mentioned in
documentation, the kafkaproducer can be shared across multiple threads.

Or should be there one kafka producer created to handle one request?

Is there any best practice documents/guidelines to follow for using simple
java Kafka producer api?

Thanks in advance for your responses.

Thanks,
Amit

Re: Simple KafkaProducer to handle multiple requests or not

[Amit K <am...@gmail.com>]

Thanks Robert Quinlivan,

Got this one clear.
Have been going through additional docs, they seems to point to this though
bit subtly.

Thanks again for your help!

On Mon, Mar 13, 2017 at 10:20 PM, Robert Quinlivan <rq...@signal.co>
wrote:

> There is no need to create a new producer instance for each write request.
> In doing so you lose the advantages of the buffering and batching that the
> producer offers. In your use case I would recommend having a single running
> producer and tuning the batch size and linger.ms settings if you find that
> the producer is using too much memory.
>
> On Mon, Mar 13, 2017 at 5:05 AM, Amit K <am...@gmail.com> wrote:
>
> > Hi,
> >
> > I am using simple kafka producer (java based, version 0.9.0.0) in an
> > application where I receive lot of hits (about 50 per seconds, in much
> like
> > servlet way) on application that has kafka producer. Per request comes
> > different set of records.
> >
> > I am using only one instance of kafka producer to push the records to
> kafka
> > cluster. Is that good way to use kafka producer? As it is mentioned in
> > documentation, the kafkaproducer can be shared across multiple threads.
> >
> > Or should be there one kafka producer created to handle one request?
> >
> > Is there any best practice documents/guidelines to follow for using
> simple
> > java Kafka producer api?
> >
> > Thanks in advance for your responses.
> >
> > Thanks,
> > Amit
> >
>
>
>
> --
> Robert Quinlivan
> Software Engineer, Signal
>

Re: Simple KafkaProducer to handle multiple requests or not

[Robert Quinlivan <rq...@signal.co>]

There is no need to create a new producer instance for each write request.
In doing so you lose the advantages of the buffering and batching that the
producer offers. In your use case I would recommend having a single running
producer and tuning the batch size and linger.ms settings if you find that
the producer is using too much memory.

On Mon, Mar 13, 2017 at 5:05 AM, Amit K <am...@gmail.com> wrote:

> Hi,
>
> I am using simple kafka producer (java based, version 0.9.0.0) in an
> application where I receive lot of hits (about 50 per seconds, in much like
> servlet way) on application that has kafka producer. Per request comes
> different set of records.
>
> I am using only one instance of kafka producer to push the records to kafka
> cluster. Is that good way to use kafka producer? As it is mentioned in
> documentation, the kafkaproducer can be shared across multiple threads.
>
> Or should be there one kafka producer created to handle one request?
>
> Is there any best practice documents/guidelines to follow for using simple
> java Kafka producer api?
>
> Thanks in advance for your responses.
>
> Thanks,
> Amit
>



-- 
Robert Quinlivan
Software Engineer, Signal