Async Producer

[Pankaj Misra <pa...@impetus.co.in>]

Dear All,

I am using async producer to publish event messages to a topic. Each message is about 2KB in size and I intend to publish about 1 million messages, i.e. about 2GB of message volumes.

This publish of event messages happens while consumer is not active. So my expectation was that the messages would be internally flushed to files and would be held till the consumer gets active. However when I ran the async producer to publish 1 million of such messages, after a few thousands of messages (about 30000) I am getting an exception indicating that the event queue is full. I also observed that after this exception if I run the consumer, the consumer is able to consume lesser messages (less than 30000), which also indicates that some messages got lost due to the event queue being full.

While I do understand that this may be happening since the producer is producing event at a higher rate than broker can handle, I was hopeful of tweaking the parameters in the server.properties for messages and log file sizes, but I am not able to figure out what exact parameters will I have to tweak.

I would greatly appreciate community's support here in helping me out.


Thanks and Regards
Pankaj Misra

________________________________

Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.

Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.


NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Re: Async Producer

[Jun Rao <ju...@gmail.com>]

Pankaj,

The queue full exception typically happens when you are trying to send
messages quicker than the broker can handle.

If the producer can't block, you will have to add enough brokers so that
they jointly can handle the load.

If the producer can block, you can set queue.enqueueTimeout.ms in producer
config to -1. If the queue is full, the producer will block instead of
dropping messages.

Thanks,

Jun

On Wed, Oct 31, 2012 at 11:16 PM, Pankaj Misra
<pa...@impetus.co.in>wrote:

> Dear All,
>
> I am using async producer to publish event messages to a topic. Each
> message is about 2KB in size and I intend to publish about 1 million
> messages, i.e. about 2GB of message volumes.
>
> This publish of event messages happens while consumer is not active. So my
> expectation was that the messages would be internally flushed to files and
> would be held till the consumer gets active. However when I ran the async
> producer to publish 1 million of such messages, after a few thousands of
> messages (about 30000) I am getting an exception indicating that the event
> queue is full. I also observed that after this exception if I run the
> consumer, the consumer is able to consume lesser messages (less than
> 30000), which also indicates that some messages got lost due to the event
> queue being full.
>
> While I do understand that this may be happening since the producer is
> producing event at a higher rate than broker can handle, I was hopeful of
> tweaking the parameters in the server.properties for messages and log file
> sizes, but I am not able to figure out what exact parameters will I have to
> tweak.
>
> I would greatly appreciate community's support here in helping me out.
>
>
> Thanks and Regards
> Pankaj Misra
>
> ________________________________
>
> Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
>
> Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor
> Interoperable Systems’ available at http://lf1.me/0E/.
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>

Re: Async Producer

[Xavier Stevens <xs...@mozilla.com>]

If you're using default producer config values you can turn some knobs 
there to get more performance most likely.

|max.message.size| (maybe try setting this to 2MB or 4MB)
|compression.codec (try gzip or snappy compression if you aren't already)
| |batch.size (divide max.message.size by your message size and leave 
some padding in case you have a larger message: try 1000 for starters 
but use a max.message.size > 2MB)
queue.size| (tune this and queue.time if the above doesn't get you where 
you need to be)
||queue.time

Hopefully this helps!

Cheers,

||||||-Xavier
||||||
||
On 10/31/12 11:16 PM, Pankaj Misra wrote:
> Dear All,
>
> I am using async producer to publish event messages to a topic. Each message is about 2KB in size and I intend to publish about 1 million messages, i.e. about 2GB of message volumes.
>
> This publish of event messages happens while consumer is not active. So my expectation was that the messages would be internally flushed to files and would be held till the consumer gets active. However when I ran the async producer to publish 1 million of such messages, after a few thousands of messages (about 30000) I am getting an exception indicating that the event queue is full. I also observed that after this exception if I run the consumer, the consumer is able to consume lesser messages (less than 30000), which also indicates that some messages got lost due to the event queue being full.
>
> While I do understand that this may be happening since the producer is producing event at a higher rate than broker can handle, I was hopeful of tweaking the parameters in the server.properties for messages and log file sizes, but I am not able to figure out what exact parameters will I have to tweak.
>
> I would greatly appreciate community's support here in helping me out.
>
>
> Thanks and Regards
> Pankaj Misra
>
> ________________________________
>
> Impetus Ranked in the Top 50 India’s Best Companies to Work For 2012.
>
> Impetus webcast ‘Designing a Test Automation Framework for Multi-vendor Interoperable Systems’ available at http://lf1.me/0E/.
>
>
> NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.

Re: Async Producer

[howard chen <ho...@gmail.com>]

Hi,


On Thu, Nov 1, 2012 at 2:16 PM, Pankaj Misra <pa...@impetus.co.in> wrote:
> Dear All,
>
> I am using async producer to publish event messages to a topic. Each message is about 2KB in size and I intend to publish about 1 million messages, i.e. about 2GB of message volumes.
>

Our approach is to have the async producer spool to a local file, and
another daemon the read the file and send to kafka. Something similar
to a local SMTP daemon - the only limitation is the local disk space
available.