How do you handle server down in producer

[Jamie Wang <ja...@actuate.com>]

What is the best way to handle kafka server going down during a message send by a producer? We have seen our producer stuck waiting until kafka server comes back. Is there ways the producer can catch a timeout or ways to test if Kafka server is up so to send the message. Having the producer hang in memory and not do anything indefinitly is not desirable in our scenario.  Thanks for your help.

Jamie 

RE: How do you handle server down in producer

[Jamie Wang <ja...@actuate.com>]

Sorry for the late reply.  Totally understand your point in developing a quality software. But this scenario is a deployment situation where server going down is inevitable (and not so much software is not reliable in this case). What I am really looking for is Kafka producer library (and consumer as well for that matter) to provide ability for handling timeout. In a distributed environment, the ability to handle timeout is common and useful. Thanks.

Jamie

-----Original Message-----
From: Philip O'Toole [mailto:philip@loggly.com] 
Sent: Thursday, April 04, 2013 2:38 PM
To: users@kafka.apache.org
Subject: Re: How do you handle server down in producer

Not much you can do, right? Either buffer the Events in memory (for as much
memory as you have) or drop it.

IMHO if Kafka is a critical piece of your Infrastructure, it's hard to
solve the issue of what to do when it goes down. Instead, engineer your
system so your Producers and Kafka Brokers are reliable and redundant --
and well tested.

Philip

On Thu, Apr 4, 2013 at 2:30 PM, Jamie Wang <ja...@actuate.com> wrote:

> What is the best way to handle kafka server going down during a message
> send by a producer? We have seen our producer stuck waiting until kafka
> server comes back. Is there ways the producer can catch a timeout or ways
> to test if Kafka server is up so to send the message. Having the producer
> hang in memory and not do anything indefinitly is not desirable in our
> scenario.  Thanks for your help.
>
> Jamie
>

Re: How do you handle server down in producer

[Philip O'Toole <ph...@loggly.com>]

Not much you can do, right? Either buffer the Events in memory (for as much
memory as you have) or drop it.

IMHO if Kafka is a critical piece of your Infrastructure, it's hard to
solve the issue of what to do when it goes down. Instead, engineer your
system so your Producers and Kafka Brokers are reliable and redundant --
and well tested.

Philip

On Thu, Apr 4, 2013 at 2:30 PM, Jamie Wang <ja...@actuate.com> wrote:

> What is the best way to handle kafka server going down during a message
> send by a producer? We have seen our producer stuck waiting until kafka
> server comes back. Is there ways the producer can catch a timeout or ways
> to test if Kafka server is up so to send the message. Having the producer
> hang in memory and not do anything indefinitly is not desirable in our
> scenario.  Thanks for your help.
>
> Jamie
>