Exactly Once, Producer configuration

[Th...@t-systems.com]

Hi,

the following article describes how to configure Apache Kafka system to enforce exactly once approach:

https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

One recommendation states that parameter transactional.id should be set in producer.properties file which usually exists only once per broker.

Now I have the following questions. Our system consists of 4 modules which interact in a sequential way by exchanging messages using Kafka topics


A --> B --> C --> D

I.e. modules A, B and C act as message producers and modules B, C and D act as message consumers (and all modules do some application specific useful processing).

1)      Do we need separate producer/consumer configurations for these modules and if so where and how do we have to configure them (as producer configuration files are related to the number of brokers and not to the number of modules)?
2)      Does transcational.id "only" cover technical aspects of Apache Kafka or does it cover application aspects as well (i.e. is my assumption which led to question 1 correct)?
3)      In case it is sufficient to configure transactional.id parameter in producer.properties files and we have more than one broker (and with that more than one producer.properties file). Do the values of the parameter transactional.id have to be equal or do there have to be different (pairwise disjoint) values?

I would be glad to get answers to these questions.

Kind regards,
Thomas Kass

Re: Exactly Once, Producer configuration

["Matthias J. Sax" <ma...@confluent.io>]

Hi,

I don't think that the `producer.properties` file exist once per broker,
because this file should not be at the broker machine at all. It should
be on the application side (ie, your "modules").

For transactions to work correctly, each `Producer` instance needs a
cluster wide unique transactional.id.

I hope this answers your questions.

Btw: you might also be interested in this two blog posts:

- https://www.confluent.io/blog/transactions-apache-kafka/
- https://www.confluent.io/blog/enabling-exactly-kafka-streams/


-Matthias

On 5/14/18 7:26 AM, Thomas.Kass@t-systems.com wrote:
> Hi,
> 
> the following article describes how to configure Apache Kafka system to enforce exactly once approach:
> 
> https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
> 
> One recommendation states that parameter transactional.id should be set in producer.properties file which usually exists only once per broker.
> 
> Now I have the following questions. Our system consists of 4 modules which interact in a sequential way by exchanging messages using Kafka topics
> 
> 
> A --> B --> C --> D
> 
> I.e. modules A, B and C act as message producers and modules B, C and D act as message consumers (and all modules do some application specific useful processing).
> 
> 1)      Do we need separate producer/consumer configurations for these modules and if so where and how do we have to configure them (as producer configuration files are related to the number of brokers and not to the number of modules)?
> 2)      Does transcational.id "only" cover technical aspects of Apache Kafka or does it cover application aspects as well (i.e. is my assumption which led to question 1 correct)?
> 3)      In case it is sufficient to configure transactional.id parameter in producer.properties files and we have more than one broker (and with that more than one producer.properties file). Do the values of the parameter transactional.id have to be equal or do there have to be different (pairwise disjoint) values?
> 
> I would be glad to get answers to these questions.
> 
> Kind regards,
> Thomas Kass
> 
> 
> 
> 
> 
>