You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by pe...@gmail.com on 2020/10/13 13:37:51 UTC
How to read-your-writes in a side effect
Imagine I have a Kafka cluster with one producer and one consumer, and behind it an API server. The API allows clients to write (PUT requests). When a client writes, it produces an event into Kafka with that data on a stream. Then there is a consumer listening on that stream that will be responsible for updating an external database used for indexing. This is a side-effect of producing that event.
My question is:
Is there a way for the API to be able to respond only once the side-effect of the said event has occurred?
In other words, is it possible for me, at the API level, to wait until a certain consumer is up-to-date?
Re: [External] How to read-your-writes in a side effect
Posted by "Tauzell, Dave" <Da...@surescripts.com>.
When you send a message the returned RecordMetadata has an offset. If you know the name of the Consumer Group that is reading these messages there is an API to get the latest offset of the Consumer Group so you could poll that and wait for the latest Offset to be greater than the offset of your message.
I don't recall what that API is right now.
-Dave
ο»ΏOn 10/13/20, 9:50 AM, "Pedro Teixeira" <pe...@gmail.com> wrote:
I was hoping there was an API for at least knowing the consumer progress..
β
Pedro Teixeira
βοΈ i@pgte.me
π» https://github.com/pgte
π¨π»βπ https://www.linkedin.com/in/pedroteixeira/
βπ» https://metaduck.com
βπ» https://pgte.me
--- original message ---
On 13 October 2020, 3:42 PM WEST dave.tauzell@surescripts.com wrote:
Your consumer will need to put some sort of response message on a different topic and the API will have to subscribe and wait for that response message. You'll need some sort of key so that it can know which one was for that call. I would avoid this type of API, however. Instead I would create an API endpoint to get the "status" of that side effect and have clients periodically poll for that.
-Dave
On 10/13/20, 9:22 AM, "pedro.teixeira@gmail.com" <pe...@gmail.com> wrote:
Imagine I have a Kafka cluster with one producer and one consumer, and behind it an API server. The API allows clients to write (PUT requests). When a client writes, it produces an event into Kafka with that data on a stream. Then there is a consumer listening on that stream that will be responsible for updating an external database used for indexing. This is a side-effect of producing that event.
My question is:
Is there a way for the API to be able to respond only once the side-effect of the said event has occurred?
In other words, is it possible for me, at the API level, to wait until a certain consumer is up-to-date?
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
--- end of original message ---
How to read-your-writes in a side effect
Posted by pe...@gmail.com.
Imagine I have a Kafka cluster with one producer and one consumer, and behind it an API server. The API allows clients to write (PUT requests). When a client writes, it produces an event into Kafka with that data on a stream. Then there is a consumer listening on that stream that will be responsible for updating an external database used for indexing. This is a side-effect of producing that event.
My question is:
Is there a way for the API to be able to respond only once the side-effect of the said event has occurred?
In other words, is it possible for me, at the API level, to wait until a certain consumer is up-to-date?
Re: [External] How to read-your-writes in a side effect
Posted by Pedro Teixeira <pe...@gmail.com>.
I was hoping there was an API for at least knowing the consumer progress..
β
Pedro Teixeira
βοΈ i@pgte.me
π» https://github.com/pgte
π¨π»βπ https://www.linkedin.com/in/pedroteixeira/
βπ» https://metaduck.com
βπ» https://pgte.me
--- original message ---
On 13 October 2020, 3:42 PM WEST dave.tauzell@surescripts.com wrote:
Your consumer will need to put some sort of response message on a different topic and the API will have to subscribe and wait for that response message. You'll need some sort of key so that it can know which one was for that call. I would avoid this type of API, however. Instead I would create an API endpoint to get the "status" of that side effect and have clients periodically poll for that.
-Dave
On 10/13/20, 9:22 AM, "pedro.teixeira@gmail.com" <pe...@gmail.com> wrote:
Imagine I have a Kafka cluster with one producer and one consumer, and behind it an API server. The API allows clients to write (PUT requests). When a client writes, it produces an event into Kafka with that data on a stream. Then there is a consumer listening on that stream that will be responsible for updating an external database used for indexing. This is a side-effect of producing that event.
My question is:
Is there a way for the API to be able to respond only once the side-effect of the said event has occurred?
In other words, is it possible for me, at the API level, to wait until a certain consumer is up-to-date?
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
--- end of original message ---