Minimum replication for Exactly Once semantics

[Rajesh Kalyanasundaram <Ra...@fico.com>]

Hi all,

Kafka Streams application with Exactly once processing semantics, the default replication factor of broker is 3 i.e transaction.state.log.replication.factor
In doc its mentioned that atleast 3 brokers are required for exactly once. What is the minimum replication factor required for exactly once processing?

processing.guarantee
The processing guarantee that should be used. Possible values are "at_least_once" (default) and "exactly_once". Note that if exactly-once processing is enabled, the default for parameter commit.interval.ms changes to 100ms. Additionally, consumers are configured with isolation.level="read_committed" and producers are configured with retries=Integer.MAX_VALUE and enable.idempotence=true per default. Note that "exactly_once" processing requires a cluster of at least three brokers by default, which is the recommended setting for production. For development, you can change this by adjusting the broker settings in both transaction.state.log.replication.factor and transaction.state.log.min.isr to the number of brokers you want to use. To learn more, see Processing Guarantees<https://docs.confluent.io/current/streams/concepts.html#streams-concepts-processing-guarantees>.

Thanks
Regards,
Rajesh
This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.

Re: Minimum replication for Exactly Once semantics

[Colin McCabe <cm...@apache.org>]

Hi Rajesh,

The reason why three brokers are recommended for EOS is because of the default replication factor of some of the internal topics used to implement it.  If you tweak those configurations, you could run with fewer (although that's not necessarily a good idea).

best,
Colin


On Wed, Dec 11, 2019, at 02:26, Rajesh Kalyanasundaram wrote:
> Hi all,
> 
> Kafka Streams application with Exactly once processing semantics, the 
> default replication factor of broker is 3 i.e 
> transaction.state.log.replication.factor
> In doc its mentioned that atleast 3 brokers are required for exactly 
> once. What is the minimum replication factor required for exactly once 
> processing?
> 
> processing.guarantee
> The processing guarantee that should be used. Possible values are 
> "at_least_once" (default) and "exactly_once". Note that if exactly-once 
> processing is enabled, the default for parameter commit.interval.ms 
> changes to 100ms. Additionally, consumers are configured with 
> isolation.level="read_committed" and producers are configured with 
> retries=Integer.MAX_VALUE and enable.idempotence=true per default. Note 
> that "exactly_once" processing requires a cluster of at least three 
> brokers by default, which is the recommended setting for production. 
> For development, you can change this by adjusting the broker settings 
> in both transaction.state.log.replication.factor and 
> transaction.state.log.min.isr to the number of brokers you want to use. 
> To learn more, see Processing 
> Guarantees<https://docs.confluent.io/current/streams/concepts.html#streams-concepts-processing-guarantees>.
> 
> Thanks
> Regards,
> Rajesh
> This email and any files transmitted with it are confidential, 
> proprietary and intended solely for the individual or entity to whom 
> they are addressed. If you have received this email in error please 
> delete it immediately.
>