You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Devendra Tagare <de...@pubmatic.com> on 2014/11/20 14:01:08 UTC
Issues with Kafka async producer's enqueue timeout
Hi,
We are using an async producer to send data to a kafka cluster.The event rate at peak is around 250 events/second of size 25KB each.
In the producer code base we have added specific debug statements to capture the time taken to create a producer,create a keyed message with a byte payload & send the message.
We have added the below properties to the ProducerConfig
queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000
We also checked if these properties are being picked.
Based on the documentation, producer.send() queues the message on the async producer's queue.
So, ideally if the queue is full then the enqueue operation should result in an kafka.common.QueueFullException in 20 ms.
The logs indicate that the enqueue operation is taking more than 20ms (takes around 250ms) without throwing any exceptions.
Is there any other property that could conflict with queue.enqueue.timeout.ms which is causing this behavior ?
Or is it possible that the queue is not full & yet the producer.send() call is still taking around 200ms under peak load ?
Also, could you suggest any other alternatives so that we can either enforce a timeout or throw an exception in-case the async producer is taking more than a specified amount of time to queue an event.
Regards,
Dev
Re: Issues with Kafka async producer's enqueue timeout
Posted by Jun Rao <ju...@gmail.com>.
queue.enqueue.timeout.ms only applies if the producer is configured in
async mode.
Thanks,
Jun
On Thu, Nov 20, 2014 at 5:01 AM, Devendra Tagare <
devendra.tagare@pubmatic.com> wrote:
> Hi,
>
> We are using an async producer to send data to a kafka cluster.The event
> rate at peak is around 250 events/second of size 25KB each.
>
> In the producer code base we have added specific debug statements to
> capture the time taken to create a producer,create a keyed message with a
> byte payload & send the message.
>
> We have added the below properties to the ProducerConfig
>
> queue.enqueue.timeout.ms=20
> send.buffer.bytes=1024000
> topic.metadata.refresh.interval.ms=30000
>
> We also checked if these properties are being picked.
>
> Based on the documentation, producer.send() queues the message on the
> async producer's queue.
>
> So, ideally if the queue is full then the enqueue operation should result
> in an kafka.common.QueueFullException in 20 ms.
>
> The logs indicate that the enqueue operation is taking more than 20ms
> (takes around 250ms) without throwing any exceptions.
>
> Is there any other property that could conflict with
> queue.enqueue.timeout.ms which is causing this behavior ?
>
> Or is it possible that the queue is not full & yet the producer.send()
> call is still taking around 200ms under peak load ?
>
> Also, could you suggest any other alternatives so that we can either
> enforce a timeout or throw an exception in-case the async producer is
> taking more than a specified amount of time to queue an event.
>
>
> Regards,
> Dev
>
Re: Issues with Kafka async producer's enqueue timeout
Posted by Jun Rao <ju...@gmail.com>.
Ok, then your understanding is correct. If the queue is full then the
enqueue operation should result in a kafka.common.QueueFullException in 20
ms. Not sure why this is not happening since in the code, we simply call
queue.offer() with the timeout. It could be due to contention. Do you have
lots of threads sending data to the same producer instance? Also, could you
check the queue size JMX to see if it's really full?
Thanks,
Jun
On Thu, Nov 20, 2014 at 9:56 PM, Devendra Tagare <
devendra.tagare@pubmatic.com> wrote:
> Hi,
>
> We are using an async producer.
>
> The producer properties are:
>
> producer.type= async
> queue.enqueue.timeout.ms=20
> send.buffer.bytes=1024000
> topic.metadata.refresh.interval.ms=30000
>
>
> Please find more details in the mail thread below.
>
>
> Regards,
> Dev
> ________________________________
> From: Devendra Tagare
> Sent: Thursday, November 20, 2014 6:31 PM
> To: users@kafka.apache.org
> Cc: Devendra Tagare
> Subject: Issues with Kafka async producer's enqueue timeout
>
> Hi,
>
> We are using an async producer to send data to a kafka cluster.The event
> rate at peak is around 250 events/second of size 25KB each.
>
> In the producer code base we have added specific debug statements to
> capture the time taken to create a producer,create a keyed message with a
> byte payload & send the message.
>
> We have added the below properties to the ProducerConfig
>
> producer.type= async
> queue.enqueue.timeout.ms=20
> send.buffer.bytes=1024000
> topic.metadata.refresh.interval.ms=30000
>
> We also checked if these properties are being picked.
>
> Based on the documentation, producer.send() queues the message on the
> async producer's queue.
>
> So, ideally if the queue is full then the enqueue operation should result
> in an kafka.common.QueueFullException in 20 ms.
>
> The logs indicate that the enqueue operation is taking more than 20ms
> (takes around 250ms) without throwing any exceptions.
>
> Is there any other property that could conflict with
> queue.enqueue.timeout.ms which is causing this behavior ?
>
> Or is it possible that the queue is not full & yet the producer.send()
> call is still taking around 200ms under peak load ?
>
> Also, could you suggest any other alternatives so that we can either
> enforce a timeout or throw an exception in-case the async producer is
> taking more than a specified amount of time to queue an event.
>
>
> Regards,
> Dev
>
Re: Out Of Disk Space Issues
Posted by David Corley <da...@gmail.com>.
See the description of log.retention.bytes here:
https://kafka.apache.org/08/configuration.html
You can set a basic value per log-partition, but you'll need to do some
math to work out an appropriate value based on:
1. The number of partitions per topic
2. The number of topics
3. The capacity of the disks used by the cluster nodes
On Fri, Nov 21, 2014 at 9:58 AM, Nilesh Chhapru <
nilesh.chhapru@ugamsolutions.com> wrote:
> Hi All,
>
> Can anyone give some inputs about retention policy as I am trying to save
> larger data in the topics and hence going out of disk space.
>
> Regards,
> Nilesh Chhapru.
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
Out Of Disk Space Issues
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi All,
Can anyone give some inputs about retention policy as I am trying to save larger data in the topics and hence going out of disk space.
Regards,
Nilesh Chhapru.
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
RE: Issues with Kafka async producer's enqueue timeout
Posted by Devendra Tagare <de...@pubmatic.com>.
Hi,
We are using an async producer.
The producer properties are:
producer.type= async
queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000
Please find more details in the mail thread below.
Regards,
Dev
________________________________
From: Devendra Tagare
Sent: Thursday, November 20, 2014 6:31 PM
To: users@kafka.apache.org
Cc: Devendra Tagare
Subject: Issues with Kafka async producer's enqueue timeout
Hi,
We are using an async producer to send data to a kafka cluster.The event rate at peak is around 250 events/second of size 25KB each.
In the producer code base we have added specific debug statements to capture the time taken to create a producer,create a keyed message with a byte payload & send the message.
We have added the below properties to the ProducerConfig
producer.type= async
queue.enqueue.timeout.ms=20
send.buffer.bytes=1024000
topic.metadata.refresh.interval.ms=30000
We also checked if these properties are being picked.
Based on the documentation, producer.send() queues the message on the async producer's queue.
So, ideally if the queue is full then the enqueue operation should result in an kafka.common.QueueFullException in 20 ms.
The logs indicate that the enqueue operation is taking more than 20ms (takes around 250ms) without throwing any exceptions.
Is there any other property that could conflict with queue.enqueue.timeout.ms which is causing this behavior ?
Or is it possible that the queue is not full & yet the producer.send() call is still taking around 200ms under peak load ?
Also, could you suggest any other alternatives so that we can either enforce a timeout or throw an exception in-case the async producer is taking more than a specified amount of time to queue an event.
Regards,
Dev