KafkaLog4jAppender with AsyncProducer?

[Edward Buck <eb...@choicestream.com>]

I see that the KafkaLog4jAppender uses a SyncProducer to send messages. I'm assuming the means that if no broker is available,  the client program will throw an error (rather then enqueue/cache the message locally until a broker is available). Would an AsyncProducer be a good fit for use with a slightly modified KafkaLog4jAppender? If so, this would make the producer code more resilient to network outages.

Thanks,
Ed

________________________________
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.

Re: KafkaLog4jAppender with AsyncProducer?

[Neha Narkhede <ne...@gmail.com>]

Edward,

The kafka log4j in trunk takes in either a zookeeper connection url or
a list of brokers. It just means that the log4j messages will be sent
in a synchronous manner with no buffering. This will mean lower
throughput. But if no broker is available, both sync and async
producers will not be able to send the data at all.

>From a throughput standpoint, it does seem reasonable to change
KafkaLog4jAppender to take in the producer.type config and be able to
pick either the sync or the async producers. Would you mind filing a
JIRA for this ?

Thanks,
Neha

On Mon, Mar 19, 2012 at 12:14 PM, Edward Buck <eb...@choicestream.com> wrote:
> I see that the KafkaLog4jAppender uses a SyncProducer to send messages. I'm assuming the means that if no broker is available,  the client program will throw an error (rather then enqueue/cache the message locally until a broker is available). Would an AsyncProducer be a good fit for use with a slightly modified KafkaLog4jAppender? If so, this would make the producer code more resilient to network outages.
>
> Thanks,
> Ed
>
> ________________________________
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.