You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Zach Cox <zc...@gmail.com> on 2019/03/06 17:08:34 UTC
Transaction commit markers and topic partition end offsets
Hi - we've observed that after committing a transaction, commit marker
records are appended to all topic partitions written to in the transaction,
these commit markers are assigned offsets in the partition, and
KafkaConsumer.endOffsets will return the offsets of commit markers if they
are the last records appended to the partition, although the consumer will
never consume records with those commit marker offsets.
This can lead to problems, for example on startup a system gets current end
offsets for a topic's partitions and tracks the current offsets a consumer
has consumed, it will never be "caught up" to the last offsets since they
are the commit marker offsets, until newer records are written to those
partitions. A workaround to this specific problem is to assume lastOffset =
endOffset - 2, but at best this is surprising until you learn about commit
markers.
How have others dealt with this? Are we missing anything important in the
above? Could the docs include more details about commit markers? Should
KafkaConsumer.endOffsets account for commit markers somehow?
Thanks,
Zach
Re: Transaction commit markers and topic partition end offsets
Posted by "Matthias J. Sax" <ma...@confluent.io>.
It's a know problem... (eg:
https://issues.apache.org/jira/browse/KAFKA-6607)
Also `lastOffset = endOffset - 2` does not necessarily do the trick,
because if you have multiple in-flight transactions (from different
producers), there might be multiple commit markers of different
transactions at the end of the topic.
-Matthias
On 3/6/19 9:08 AM, Zach Cox wrote:
> Hi - we've observed that after committing a transaction, commit marker
> records are appended to all topic partitions written to in the transaction,
> these commit markers are assigned offsets in the partition, and
> KafkaConsumer.endOffsets will return the offsets of commit markers if they
> are the last records appended to the partition, although the consumer will
> never consume records with those commit marker offsets.
>
> This can lead to problems, for example on startup a system gets current end
> offsets for a topic's partitions and tracks the current offsets a consumer
> has consumed, it will never be "caught up" to the last offsets since they
> are the commit marker offsets, until newer records are written to those
> partitions. A workaround to this specific problem is to assume lastOffset =
> endOffset - 2, but at best this is surprising until you learn about commit
> markers.
>
> How have others dealt with this? Are we missing anything important in the
> above? Could the docs include more details about commit markers? Should
> KafkaConsumer.endOffsets account for commit markers somehow?
>
> Thanks,
> Zach
>