You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Devendra Tagare <de...@pubmatic.com> on 2014/11/04 05:32:06 UTC
Issue with async producer
Hi,
We are using an async producer to send data to kafka.
The load the sender handles is around 250 rps ,the size of a message is around 25K.
The configs used in the producer are :
request.required.acks=0
producer.type=async
batch.num.messages=10
topic.metadata.refresh.interval.ms=30000
queue.buffering.max.ms=300
queue.enqueue.timeout.ms=50
While the asnc producer works perfectly fine at 150-175 rps,the invoking method returns under 10 ms.The invoking method takes around 20000ms to return when the load increases to 250rps.
On investigation we noticed QueueFullExceptions in the logs.
On the kafka side the memory utilization was high.Is is because the fsync to memory is not happening fast enough?
The document says the OS level fsync interval should take care of the interval at which writes are happening.Should we expediate the write using
log.flush.interval.messages
and log.flush.interval.ms.
Also, we tried the sync producer but were not able to enforce the timeouts on it.
We have 45 producers & 11 Kafka servers which handle a load of around 500mn events per day.
Some of the server side properties we are using:
log.segment.bytes=536870912
log.retention.check.interval.ms=60000
zookeeper.connection.timeout.ms=1000000
Regards,
Dev
Re: Issue with async producer
Posted by Jun Rao <ju...@gmail.com>.
In 0.8.2 beta, we have a new java producer that's more performing. You can
give it a try.
Thanks,
Jun
On Thu, Nov 6, 2014 at 10:16 AM, Devendra Tagare <
devendra.tagare@pubmatic.com> wrote:
> Hi,
>
> Which version of Kafka are you using?
> 0.8.0
>
> Is the broker I/O or network saturated? - typical writes are around 45 MB
> per broker & there are streaming as well as batch consumers.
>
> We are already sending across compressed packets & the batch size is 100.
>
> Also, we tried the sync producer but were not able to enforce the timeouts
> on it.Is there anything that needs to be taken care of in this case?
>
> Regards,
> Dev
> ________________________________
> From: Devendra Tagare
> Sent: Tuesday, November 04, 2014 10:02 AM
> To: users@kafka.apache.org
> Cc: Devendra Tagare
> Subject: Issue with async producer
>
> Hi,
>
> We are using an async producer to send data to kafka.
>
> The load the sender handles is around 250 rps ,the size of a message is
> around 25K.
>
> The configs used in the producer are :
>
> request.required.acks=0
> producer.type=async
> batch.num.messages=10
> topic.metadata.refresh.interval.ms=30000
> queue.buffering.max.ms=300
> queue.enqueue.timeout.ms=50
>
>
> While the asnc producer works perfectly fine at 150-175 rps,the invoking
> method returns under 10 ms.The invoking method takes around 20000ms to
> return when the load increases to 250rps.
>
> On investigation we noticed QueueFullExceptions in the logs.
>
> On the kafka side the memory utilization was high.Is is because the fsync
> to memory is not happening fast enough?
> The document says the OS level fsync interval should take care of the
> interval at which writes are happening.Should we expediate the write using
>
> log.flush.interval.messages
> and log.flush.interval.ms.
>
> Also, we tried the sync producer but were not able to enforce the timeouts
> on it.
>
> We have 45 producers & 11 Kafka servers which handle a load of around
> 500mn events per day.
>
> Some of the server side properties we are using:
>
> log.segment.bytes=536870912
> log.retention.check.interval.ms=60000
> zookeeper.connection.timeout.ms=1000000
>
> Regards,
> Dev
>
RE: Issue with async producer
Posted by Devendra Tagare <de...@pubmatic.com>.
Hi,
Which version of Kafka are you using?
0.8.0
Is the broker I/O or network saturated? - typical writes are around 45 MB per broker & there are streaming as well as batch consumers.
We are already sending across compressed packets & the batch size is 100.
Also, we tried the sync producer but were not able to enforce the timeouts on it.Is there anything that needs to be taken care of in this case?
Regards,
Dev
________________________________
From: Devendra Tagare
Sent: Tuesday, November 04, 2014 10:02 AM
To: users@kafka.apache.org
Cc: Devendra Tagare
Subject: Issue with async producer
Hi,
We are using an async producer to send data to kafka.
The load the sender handles is around 250 rps ,the size of a message is around 25K.
The configs used in the producer are :
request.required.acks=0
producer.type=async
batch.num.messages=10
topic.metadata.refresh.interval.ms=30000
queue.buffering.max.ms=300
queue.enqueue.timeout.ms=50
While the asnc producer works perfectly fine at 150-175 rps,the invoking method returns under 10 ms.The invoking method takes around 20000ms to return when the load increases to 250rps.
On investigation we noticed QueueFullExceptions in the logs.
On the kafka side the memory utilization was high.Is is because the fsync to memory is not happening fast enough?
The document says the OS level fsync interval should take care of the interval at which writes are happening.Should we expediate the write using
log.flush.interval.messages
and log.flush.interval.ms.
Also, we tried the sync producer but were not able to enforce the timeouts on it.
We have 45 producers & 11 Kafka servers which handle a load of around 500mn events per day.
Some of the server side properties we are using:
log.segment.bytes=536870912
log.retention.check.interval.ms=60000
zookeeper.connection.timeout.ms=1000000
Regards,
Dev
Re: Issue with async producer
Posted by Jun Rao <ju...@gmail.com>.
Which version of Kafka are you using? Is the broker I/O or network
saturated? If so, that will limit the throughput that each producer can
achieve. If not, using a larger number messages per batch and/or enabling
producer side compression typically improves the producer throughput.
Thanks,
Jun
On Mon, Nov 3, 2014 at 8:32 PM, Devendra Tagare <
devendra.tagare@pubmatic.com> wrote:
> Hi,
>
> We are using an async producer to send data to kafka.
>
> The load the sender handles is around 250 rps ,the size of a message is
> around 25K.
>
> The configs used in the producer are :
>
> request.required.acks=0
> producer.type=async
> batch.num.messages=10
> topic.metadata.refresh.interval.ms=30000
> queue.buffering.max.ms=300
> queue.enqueue.timeout.ms=50
>
>
> While the asnc producer works perfectly fine at 150-175 rps,the invoking
> method returns under 10 ms.The invoking method takes around 20000ms to
> return when the load increases to 250rps.
>
> On investigation we noticed QueueFullExceptions in the logs.
>
> On the kafka side the memory utilization was high.Is is because the fsync
> to memory is not happening fast enough?
> The document says the OS level fsync interval should take care of the
> interval at which writes are happening.Should we expediate the write using
>
> log.flush.interval.messages
> and log.flush.interval.ms.
>
> Also, we tried the sync producer but were not able to enforce the timeouts
> on it.
>
> We have 45 producers & 11 Kafka servers which handle a load of around
> 500mn events per day.
>
> Some of the server side properties we are using:
>
> log.segment.bytes=536870912
> log.retention.check.interval.ms=60000
> zookeeper.connection.timeout.ms=1000000
>
> Regards,
> Dev
>