Consumer cannot move past missing offset

[Thomas Becker <Th...@xperi.com>]

We are having issues with some of our older consumers getting stuck reading a topic. The issue seems to occur at specific offsets. Here's an excerpt from kafka-dump-log on the topic partition around the offset in question:

baseOffset: 13920966 lastOffset: 13920987 count: 6 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 49 isTransactional: false isControl: false position: 98516844 CreateTime: 1595224747691 size: 4407 magic: 2 compresscodec: NONE crc: 1598305187 isvalid: true
| offset: 13920978 CreateTime: 1595224747691 keysize: 36 valuesize: 681 sequence: -1 headerKeys: []
| offset: 13920979 CreateTime: 1595224747691 keysize: 36 valuesize: 677 sequence: -1 headerKeys: []
| offset: 13920980 CreateTime: 1595224747691 keysize: 36 valuesize: 680 sequence: -1 headerKeys: []
| offset: 13920984 CreateTime: 1595224747691 keysize: 36 valuesize: 681 sequence: -1 headerKeys: []
| offset: 13920985 CreateTime: 1595224747691 keysize: 36 valuesize: 677 sequence: -1 headerKeys: []
| offset: 13920986 CreateTime: 1595224747691 keysize: 36 valuesize: 680 sequence: -1 headerKeys: []

This is the last batch in the segment, and the "bad offset" is 13920987. Note that is listed as the lastOffset contained in the batch, though it doesn't actually exist (this is a compacted topic). When I seek to this offset and begin consumption, different things happen depending on which consumer version is being used:

0.10.0.0 - throws RecordTooLargeException. To me, this looks like a red herring. The old consumer assumes the record must be too large because it got some data back but was not able to parse a single record from it.

0.10.1.0 - gets stuck fetching the same empty batch over and over again.

2.4.1 - works properly, fetching offset 13920991 which is the next valid offset and continues.

We are running Kafka version 2.4.1 on the broker side. I did some searching in JIRA but was unable to find anything to explain this.



--

[https://dts-web-images.s3.amazonaws.com/Images/email+signatures/xperi_117.png]

Tommy Becker
Principal Engineer
Pronouns: he/him/his


O: 919.460.4747
E: thomas.becker@xperi.com



www.xperi.com

________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of Xperi is authorized to conclude any binding agreement on behalf of Xperi by email. Binding agreements with Xperi may only be made by a signed written agreement.