Apache Kafka acknowledgement

[selva raj selvaraj <se...@tecnics.com>]

Hi Team!

    I developed a simple kafka producer & consumer module. Just I want to
acknowledge whether the message is delivered or not. I couldn't find any
document, reference or sample code. Could you please guide me on this?


*Development Language : *Python


Thanks!
Raj

Re: Apache Kafka acknowledgement

["Mohammad.R esmailzadeh" <es...@gmail.com>]

Hi Raj,
I think for your proble Rabbitmq RPC usecase is more suitable.
By the way its not recommended to make such tight couple between 2 services
if it's possible.
Regards

On Fri, Jul 29, 2022, 12:06 PM Richard Bosch <ri...@axual.com>
wrote:

> Hi Raj,
>
> The Kafka design is based on fully decoupled producers and consumers, where
> the responsibility of the producer ends after a record has been
> successfully produced.
> Since a producer is not aware of other applications that might be consuming
> the record it can continue with its own flow.
> The same is true for a consumer, they are usually not aware which
> application is producing a record. only where to read a record and how to
> deserialize and interpret the record.
> This allows for new applications to be added as producers or consumers of
> the data.
>
> This is one of the reasons why Kafka does not contain a builtin feature to
> send an acknowledgement to the producer when the record has been consumed.
> It has only built-in support to acknowledge delivery of a record to the
> partition, which is controlled by the acks setting in a producer.
>
> If the design requires an acknowledgement that the record is picked up by a
> consumer then the easiest way I can think of is to create a separate topic
> where the producer will be listening for acknowledgement.
>
> The flow would look something like this for the producing application
> 1. The producer gets a message to send to a topic.
> 2. The producer creates a record with a unique message id in the Kafka
> Header
> 3. The producer part sends the record to the topic, and adds the message id
> to a list of messages waiting for acknowledgement
> 4. The consumer part of the application polls for new records on the
> acknowledgement topic
> 5. The message id is cleared for from the waiting list, and additional
> actions can be taken according to the needs of your application
>
> The flow of the consuming application would look like this
> 1. The consumer polls for records on a specific topic.
> 2. When a record is received and correctly deserialized the message id is
> extracted
> 3. The message id is produced to the acknowledgement topic
> 4. The record is processed by the consuming application
>
> As you can see you're basically recreating a classical queueing mechanism
> with a feedback channel, requiring producing and consuming applications to
> include a lot of additional logic.
>
> Often you can solve this by redesigning the solution to make sure that the
> producer responsibilities end after getting the acknowledgement from the
> broker that the record was produced.
> The data is now available to any consumers, but it is up to the consumers
> responsibility to keep up with the records on the topic.
> This will take of course some time, and your stakeholders and fellow
> developers might need to be convinced of this new way of working.
>
> My apologies for the long explanation, but I hope you can find some use for
> this.
>
> Kind regards,
>
>
> Richard Bosch
>
> Developer Advocate
>
> Axual BV
>
> https://axual.com/
>
>
>
>
> On Thu, Jul 28, 2022 at 4:09 PM selva raj selvaraj <se...@tecnics.com>
> wrote:
>
> > Hi Team!
> >
> >     I developed a simple kafka producer & consumer module. Just I want to
> > acknowledge whether the message is delivered or not. I couldn't find any
> > document, reference or sample code. Could you please guide me on this?
> >
> >
> > *Development Language : *Python
> >
> >
> > Thanks!
> > Raj
> >
>

Re: Apache Kafka acknowledgement

[Richard Bosch <ri...@axual.com>]

Hi Raj,

The Kafka design is based on fully decoupled producers and consumers, where
the responsibility of the producer ends after a record has been
successfully produced.
Since a producer is not aware of other applications that might be consuming
the record it can continue with its own flow.
The same is true for a consumer, they are usually not aware which
application is producing a record. only where to read a record and how to
deserialize and interpret the record.
This allows for new applications to be added as producers or consumers of
the data.

This is one of the reasons why Kafka does not contain a builtin feature to
send an acknowledgement to the producer when the record has been consumed.
It has only built-in support to acknowledge delivery of a record to the
partition, which is controlled by the acks setting in a producer.

If the design requires an acknowledgement that the record is picked up by a
consumer then the easiest way I can think of is to create a separate topic
where the producer will be listening for acknowledgement.

The flow would look something like this for the producing application
1. The producer gets a message to send to a topic.
2. The producer creates a record with a unique message id in the Kafka
Header
3. The producer part sends the record to the topic, and adds the message id
to a list of messages waiting for acknowledgement
4. The consumer part of the application polls for new records on the
acknowledgement topic
5. The message id is cleared for from the waiting list, and additional
actions can be taken according to the needs of your application

The flow of the consuming application would look like this
1. The consumer polls for records on a specific topic.
2. When a record is received and correctly deserialized the message id is
extracted
3. The message id is produced to the acknowledgement topic
4. The record is processed by the consuming application

As you can see you're basically recreating a classical queueing mechanism
with a feedback channel, requiring producing and consuming applications to
include a lot of additional logic.

Often you can solve this by redesigning the solution to make sure that the
producer responsibilities end after getting the acknowledgement from the
broker that the record was produced.
The data is now available to any consumers, but it is up to the consumers
responsibility to keep up with the records on the topic.
This will take of course some time, and your stakeholders and fellow
developers might need to be convinced of this new way of working.

My apologies for the long explanation, but I hope you can find some use for
this.

Kind regards,


Richard Bosch

Developer Advocate

Axual BV

https://axual.com/




On Thu, Jul 28, 2022 at 4:09 PM selva raj selvaraj <se...@tecnics.com>
wrote:

> Hi Team!
>
>     I developed a simple kafka producer & consumer module. Just I want to
> acknowledge whether the message is delivered or not. I couldn't find any
> document, reference or sample code. Could you please guide me on this?
>
>
> *Development Language : *Python
>
>
> Thanks!
> Raj
>