You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "gkousouris (via GitHub)" <gi...@apache.org> on 2023/09/19 21:59:57 UTC
[GitHub] [kafka] gkousouris commented on pull request #13178: KAFKA-12468, KAFKA-13659, KAFKA-12566: Fix MM2 causing negative downstream lag
gkousouris commented on PR #13178:
URL: https://github.com/apache/kafka/pull/13178#issuecomment-1726599437
Hi @gharris1727, if I understand this PR correctly, this will (almost always) cause duplication of data. Our problem is this:
1. We have a service reading from a Kafka topic and committing offsets for its consumer group.
2. We use MirrorMaker to replicate the topic to a different cluster.
3. We pause the service and check the current offset for the 2 streams (the one in the old cluster and the one in the new cluster).
In step 3, these offsets will be different, specifically, the offset from the old cluster will be the last message the service managed to commit an offset for. And the new topic will have as an offset the value: `last_checkpoint_before_upstream_offset + 1`.
Thus, when we restart the service (and make it consume from the new topic), it will re-process all messages from `last_checkpoint_before_upstream_offset + 1` until `downstream_offset`. Isn't this a problem considering that Mirror Maker is providing Exactly Once Semantics for committing messages ?
This behaviour was verified by looking at the output of the `kafka-consumer-groups` script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org