Achiving at least once Delivery on top of Kafka

[Isuru Boyagane <is...@cse.mrt.ac.lk>]

Hi,

We are implementing a use case that needs tight at least once delivery.
Even in the case of failures of nodes, no messages must be lost.

We are trying to find out the least restrictive configurations that can
give us at least once delivery. Following is what we found.


   -

   If we use acks=1, we can't guarantee that messages will not be lost,
   -

   If we use acks= all, we will have a good data safety but unclean leader
   failover may lead to data loss.


As we found, setting acks=all and unclean.leader.election.enable = false
will give us data safety so that no message will be lost (sacrificing the
availability of the system).



Is this the best option we have if we need zero message loss with maximum
availability?

Can anyone clarify?


Thank you. Regards.

Re: Achiving at least once Delivery on top of Kafka

[Isuru Boyagane <is...@cse.mrt.ac.lk>]

Hi,

This is not the complete requirements set we need. Sorry for
any inconvenience occurred.

Thank you. Regards.

On Sun, 29 Sep 2019 at 18:25, Isuru Boyagane <is...@cse.mrt.ac.lk>
wrote:

> Hi,
>
> We are implementing a use case that needs tight at least once delivery.
> Even in the case of failures of nodes, no messages must be lost.
>
> We are trying to find out the least restrictive configurations that can
> give us at least once delivery. Following is what we found.
>
>
>    -
>
>    If we use acks=1, we can't guarantee that messages will not be lost,
>    -
>
>    If we use acks= all, we will have a good data safety but unclean
>    leader failover may lead to data loss.
>
>
> As we found, setting acks=all and unclean.leader.election.enable = false
> will give us data safety so that no message will be lost (sacrificing the
> availability of the system).
>
>
>
> Is this the best option we have if we need zero message loss with maximum
> availability?
>
> Can anyone clarify?
>
>
> Thank you. Regards.
>
>