You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shafaq <s....@gmail.com> on 2013/10/23 21:12:57 UTC
Questions about failure recovery, load balancing and producer
ack/async mode
Hi,
Following need more elaboration after reading kafka docs:
1- In a scenario during leader fails over, what happens to messages that
are not committed to other followers and to the messages that producer keep
in sending (in async mode) till new leader is elected. Can the producer
buffer these messages till the failover gets completed (Do we see
exceptions. Are these all lost?
2. Broker load distribution or to be exact partition load distribution is
done via hashed key mod no of partition, is that correct? How is it done
at the consumer side where one thread per partition (for high-level
consumer) How is the faliover handlled
3. How do we configure producer for wait for leader ack and wait for
commited act which Jun Rao talks about in his 2013 apachecon presentation
--
Kind Regards,
Shafaq
Fwd: Questions about failure recovery, load balancing and producer
ack/async mode
Posted by Shafaq <s....@gmail.com>.
Hi,
Following need more elaboration after reading kafka docs:
1- In a scenario during leader fails over, what happens to messages that
are not committed to other followers and to the messages that producer keep
in sending (in async mode) till new leader is elected. Can the producer
buffer these messages till the failover gets completed (Do we see
exceptions. Are these all lost?
2. Broker load distribution or to be exact partition load distribution is
done via hashed key mod no of partition, is that correct? How is it done
at the consumer side where one thread per partition (for high-level
consumer) How is the faliover handlled
3. How do we configure producer for wait for leader ack and wait for
commited act which Jun Rao talks about in his 2013 apachecon presentation
--
Kind Regards,
Shafaq
--
Kind Regards,
Shafaq
Re: Questions about failure recovery, load balancing and producer
ack/async mode
Posted by Jun Rao <ju...@gmail.com>.
1,3 Take a look at request.required.acks in
http://kafka.apache.org/documentation.html#producerconfigs
2. The producer does random distribution by default. However, you can
provide a partitioning key and a partitioning function. For details on how
consumer load balancing works, see
http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
Thanks,
Jun
On Wed, Oct 23, 2013 at 12:12 PM, Shafaq <s....@gmail.com> wrote:
> Hi,
>
> Following need more elaboration after reading kafka docs:
>
> 1- In a scenario during leader fails over, what happens to messages that
> are not committed to other followers and to the messages that producer keep
> in sending (in async mode) till new leader is elected. Can the producer
> buffer these messages till the failover gets completed (Do we see
> exceptions. Are these all lost?
>
> 2. Broker load distribution or to be exact partition load distribution is
> done via hashed key mod no of partition, is that correct? How is it done
> at the consumer side where one thread per partition (for high-level
> consumer) How is the faliover handlled
>
> 3. How do we configure producer for wait for leader ack and wait for
> commited act which Jun Rao talks about in his 2013 apachecon presentation
>
> --
> Kind Regards,
> Shafaq
>