You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "vbhanuchander-lang (via GitHub)" <gi...@apache.org> on 2026/08/13 23:58:38 UTC
Re: [I] Kafka Connect: duplicate data file commits after task restart (offsets watermark regresses on control topic re-read) [iceberg]
vbhanuchander-lang commented on issue #17340:
URL: https://github.com/apache/iceberg/issues/17340#issuecomment-5287748553
I checked the mechanism against current main and the structural part of your diagnosis holds:
`Channel.consumeAvailable` records the position with a plain `put`, so the map is last-write-wins
rather than a high-water mark:
```java
// Channel.java:126
controlTopicOffsets.put(record.partition(), record.offset() + 1);
```
Nothing in that method compares against the value already stored, so any path that re-reads a
partition from an earlier position — a seek back to committed group offsets on restart, as you
describe — moves the recorded offset **backwards**. The value then flows into
`commitConsumerOffsets()` and into the `kafka.connect.offsets` stamped on the snapshot, which is what
makes the regression durable rather than transient.
Your suggested change is consistent with how the value is used everywhere else, since every consumer
of `controlTopicOffsets()` wants "furthest position consumed", never "most recently seen":
```java
controlTopicOffsets.merge(record.partition(), record.offset() + 1, Long::max);
```
Two things I could not confirm from the code alone, and which I think are the open questions for a
reviewer rather than objections:
1. Whether making the watermark monotonic is sufficient on its own, or whether the replay can also
deliver an envelope whose offset is genuinely above the last committed watermark — in which case
the min-offset filter passes it legitimately and only content-level deduplication would catch it.
Your step 5 suggests the former, but it depends on exactly where the seek lands.
2. Whether the bounded "recently committed locations" set you float is wanted at all. That is a
design call for the maintainers, and worth separating from the one-line fix so the fix is not held
up by it.
I have not opened a PR — you said you can put one up and it is your finding, so it should be yours.
Flagging the verification here in case it helps it move, since this has been open three weeks with
no response and the production evidence you gathered (149 files, ~112k duplicated rows) makes it
worth someone's attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org