Topic partition reassignment fails because of non-monotonic offsets (Kafka 2.6)

[Joe Ammann <jo...@pyx.ch>]

We did an upgrade from Kafka 2.2 to 2.6, followed by a migration
(through reassign-partitions) from old to new brokers.

As described in
https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume,
all but 1 partition (__consumer_offsets-18) were successfully migrated.
But that one stubbornly refuses to migrate, because new replicas
complain about OutOfOrderOffset. When we looked today at the exception
message more closely, we discovered that really in huge list of offsets,
there are 2 non-monotic offsets

[2020-10-24 15:04:54,528] ERROR [ReplicaFetcher replicaId=10,
leaderId=3, fetcherId=0] Unexpected error occurred while processing data
for partition __consumer_offsets-18 at offset 1545264631
(kafka.server.ReplicaFetcherThread)
kafka.common.OffsetsOutOfOrderException: Out of order offsets found in
append to __consumer_offsets-18: ArrayBuffer(1545264631, 1545264632,
..., 1545271418, 1545271419, 1, 1, 1545271422, 1545271423, ...
1545272005, 1545272006, 1545272007)

Apparently, the leader - when asked for the list of offsets to replicate
- returns that list with the two weird '1' offsets. One would expect
1545271420, 1545271421 in place of the two '1' entries.

Does this mean that the data is really somehow corrupted? That offset
range (probably no by accident) is about when we started the upgrade of
2.2 to 2.6. The last snapshot file written on the leader with the old
2.2 version had offset 1545271411, only 8 messages before the one that
is now apparently causing issue.

Any idea what could have happened here? And more importantly, any idea
how to get out of this mess :-) We would be able to accept the loss of
all data on __consumer_offsets-18. It happens to contain only
"non-essential" consumer groups. So that would also be an option.


CU, Joe

Re: Topic partition reassignment fails because of non-monotonic offsets (Kafka 2.6)

[Joe Ammann <jo...@pyx.ch>]

We were ultimately able to solve this issue - mainly by sitting and waiting

The issue was indeed that somewhen, somehow, the data on the leader of
this __consumer_offset-18 partition got corrupted. This probably
happened during the upgrade from Kafka 2.2 -> 2.6. We were doing this in
a rather dangerous way, as we know now: we simply stopped all brokers,
updated the SW on all, and restarted them. We thought since we can
afford this outage on the weekend, this would be a safe way. But we will
certainly never do that again. At least not unless we know 100% that all
producers and all consumers are really stopped. This was NOT the case
during that upgrade, we overlooked one consumer and left that running,
and that consumer (group) was storing their offsets in the
__consumer_offset-18 partition. So that action - taking all brokers down
and upgrade them - while consumers/producers are still running, did
probably cause the corruption

Lesson learnt: never do that again, always do rolling upgrades, even if
they take a lot longer.

The issue was then actually solved through the nature of the compacting
topic. The default settings for compaction are to run it every week (or
when a segment gets bigger than 1GB, which will not happen in our case).
Compaction of that __consumer_offset-18 partition kicked in yesterday
evening. And by that the 2 corrupted offsets were purged away. After
that it was uphill, the reassign of that partition to the new brokers
then worked like a charm.

We could certainly have speeded up this recovery by setting the topic
parameters in a way that compaction would kick in earlier. But it was
only on Wednesday when we reached the understanding that the problem
could actually be resolved that way. We decided to leave everything as
it was and wait another day.

On 26.10.20 18:41, Joe Ammann wrote:
> We did an upgrade from Kafka 2.2 to 2.6, followed by a migration
> (through reassign-partitions) from old to new brokers.
>
> As described in
> https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume,
> all but 1 partition (__consumer_offsets-18) were successfully migrated.
> But that one stubbornly refuses to migrate, because new replicas
> complain about OutOfOrderOffset. When we looked today at the exception
> message more closely, we discovered that really in huge list of offsets,
> there are 2 non-monotic offsets
>
> [2020-10-24 15:04:54,528] ERROR [ReplicaFetcher replicaId=10,
> leaderId=3, fetcherId=0] Unexpected error occurred while processing data
> for partition __consumer_offsets-18 at offset 1545264631
> (kafka.server.ReplicaFetcherThread)
> kafka.common.OffsetsOutOfOrderException: Out of order offsets found in
> append to __consumer_offsets-18: ArrayBuffer(1545264631, 1545264632,
> ..., 1545271418, 1545271419, 1, 1, 1545271422, 1545271423, ...
> 1545272005, 1545272006, 1545272007)
>
> Apparently, the leader - when asked for the list of offsets to replicate
> - returns that list with the two weird '1' offsets. One would expect
> 1545271420, 1545271421 in place of the two '1' entries.
>
> Does this mean that the data is really somehow corrupted? That offset
> range (probably no by accident) is about when we started the upgrade of
> 2.2 to 2.6. The last snapshot file written on the leader with the old
> 2.2 version had offset 1545271411, only 8 messages before the one that
> is now apparently causing issue.
>
> Any idea what could have happened here? And more importantly, any idea
> how to get out of this mess :-) We would be able to accept the loss of
> all data on __consumer_offsets-18. It happens to contain only
> "non-essential" consumer groups. So that would also be an option.
>