You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by ivan mykhailov <iv...@strikead.com> on 2015/09/10 21:07:07 UTC

Can current co-ordination protocol cause data loss or delays in consumption?

Hi,

we plan to use kafka as main message bus and we strongly rely on two
features:
- at-least-once delivery guarantee
- using groups for load balancing between several consumers

But there are some problems described in
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
, like https://issues.apache.org/jira/browse/KAFKA-242

So questions are:
- can it lead to data loss?
-can it lead to delays in consumption? For example if rebalancing left some
partitions without assigned consumers for long time. Even if data is not
entirely lost, but just has not been consumed for about 10 minutes, it also
can break our processing.
- Due to roadmap
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan new
client will be released in 0.8.3(Oct. 2015). Will it include new
co-ordination logic that fix this issue?

Kind Regards,

Ivan