Producers errors when failing a broker in a replicated 0.8 cluster.

[Bob Jervis <bj...@visibletechnologies.com>]

We are in final testing of Kafka and so far the fail-over tests have been pretty encouraging.  If we kill (-9) one of two kafka brokers, with replication factor=2 we see a flurry of activity as the producer fails and retries its writes (we use a bulk, synchronous send of 1000 messages at a time, each message ~1K long).  Sometimes the library finds the newly elected leader before returning to the application and sometimes it doesn't.  We added retry/backoff logic to our code and we don't seem to be losing content.

However, we have another app in the pipeline that does a fan-out from one Kafka topic to dozens of topics.  We still use a single, synchronous, bulk send.

My question is what are the semantics of a bulk send like that, where one broker dies, but the topic leaders have been spread across both brokers.  Do we get any feedback on which messages went through and which were dropped because the leader just died?  For our own transactioning we can mark messages as 'retries' if we suspect there might have been any hanky-panky, but if we can reliably avoid extra work by not re-sending messages that we know have been delivered we can avoid the extra work on the client side.

Thanks for any insight,

Bob Jervis | Senior Architect

[cid:image001.png@01CE0839.10AC52D0]<http://www.visibletechnologies.com/>
Seattle | Boston | New York | London
Phone: 425.957.6075 | Fax: 781.404.5711

Follow Visibly Intelligent Blog<http://www.visibletechnologies.com/blog/>

[cid:image002.png@01CE0839.10AC52D0]<http://twitter.com/visible>[cid:image003.png@01CE0839.10AC52D0]<http://www.facebook.com/Visible.Technologies> [cid:image004.png@01CE0839.10AC52D0] <http://www.linkedin.com/company/visible-technologies>

Re: Producers errors when failing a broker in a replicated 0.8 cluster.

[Jun Rao <ju...@gmail.com>]

Bob,

In 0.8, if you send a set of messages in sync mode, the producer will throw
back an exception if at least one message can't be sent to the broker after
all retries. The client won't know which messages are sent successfully and
which are not. We do plan to improve the producer API after 0.8 that can
expose more information to the client.

Thanks,

Jun

On Mon, Feb 11, 2013 at 9:27 AM, Bob Jervis <bjervis@visibletechnologies.com
> wrote:

> We are in final testing of Kafka and so far the fail-over tests have been
> pretty encouraging.  If we kill (-9) one of two kafka brokers, with
> replication factor=2 we see a flurry of activity as the producer fails and
> retries its writes (we use a bulk, synchronous send of 1000 messages at a
> time, each message ~1K long).  Sometimes the library finds the newly
> elected leader before returning to the application and sometimes it
> doesn’t.  We added retry/backoff logic to our code and we don’t seem to be
> losing content.****
>
> ** **
>
> However, we have another app in the pipeline that does a fan-out from one
> Kafka topic to dozens of topics.  We still use a single, synchronous, bulk
> send.****
>
> ** **
>
> My question is what are the semantics of a bulk send like that, where one
> broker dies, but the topic leaders have been spread across both brokers.
> Do we get any feedback on which messages went through and which were
> dropped because the leader just died?  For our own transactioning we can
> mark messages as ‘retries’ if we suspect there might have been any
> hanky-panky, but if we can reliably avoid extra work by not re-sending
> messages that we know have been delivered we can avoid the extra work on
> the client side.****
>
> ** **
>
> Thanks for any insight,****
>
> ** **
>
> *Bob Jervis | Senior Architect*
>
>
> *[image: Description: Description: Visible-sm]*<http://www.visibletechnologies.com/>
> **
>
> Seattle *| *Boston* | *New York *|* London****
>
> *Phone:* 425.957.6075* | Fax:* 781.404.5711 ****
>
> ** **
>
> *Follow Visibly Intelligent Blog<http://www.visibletechnologies.com/blog/>
> *
>
> ** **
>
> [image: Description: Description: LinkedIn_Logo60px[1]]<http://twitter.com/visible>[image:
> Description: Description: facebook]<http://www.facebook.com/Visible.Technologies>
>  [image: Description: Description: in]<http://www.linkedin.com/company/visible-technologies>
> ****
>
> ** **
>