Consumer Offsets partition skew on Kafka 0.10.1.1

[Marcos Juarez <mj...@gmail.com>]

I was investigating some performance issues we're issues in one of our
production clusters, and I ran into extremely unbalanced offset partitions
for the __consumer_offsets topic.  I only pasted the top 8 below, out of 50
total.  As you can see, between the top 5 partitions, those servers have to
handle 83% of the commit volume, and brokers 9 and 10 show up repeatedly on
as leader as well as replicas.


Partition Offsets Percentage Leader Replicas ISR
6 52,761,610,477 34.24% 10 (10,6,7) (7,6,10)
5 46,196,021,230 29.98% 9 (9,5,6) (5,6,9)
42 17,530,298,423 11.38% 10 (10,9,11) (10,11,9)
31 12,927,081,106 8.39% 11 (11,9,10) (10,11,9)
0 8,557,903,671 5.55% 4 (4,12,1) (4,12,1)
2 3,969,232,652 2.58% 6 (6,2,3) (6,3,2)
49 3,555,754,347 2.31% 5 (5,11,7) (5,7,11)
33 2,273,951,745 1.48% 1 (1,11,12) (1,12,11)
Those brokers (9, 10 and 11) also happen to be the ones we're having
performance issues with.  We can't be sure yet if this is the cause of the
performance issues, but it's looking extremely likely.

So, I was wondering, what can be done to "rebalance" these consumer
offsets?  This was something, as far as I know, automatically decided, I
don't believe we ever changed a setting related to this.  I also don't
believe we can influence which partition gets which offsets when
consuming.

It would also be interesting to know what is the algorithm/pattern used to
decide the consumer offset partition, and is this something we can change
or influence?

Thanks,

Marcos Juarez

Re: Consumer Offsets partition skew on Kafka 0.10.1.1

[Marcos Juarez <mj...@gmail.com>]

So, we finally figured out why our __consumer_offsets topics was so skewed
in volume across partitions.  Turns out that the partition is chosen based
on the consumer group_id (which makes sense), and we had a few extremely
over-committing consumers in prod.  A few of them were committing several
thousand times per second, even though they were only consuming one or two
records per second.  So, these consumers were, over time, generating many
hundreds of billions offset commits, and the additional stress on the
brokers with those specific partitions, in turn, was generating URPs across
the entire cluster.

The most egregious offender was Apache Camel running a Kafka client
v0.10.0.1. Upgrading the Kafka clients to 0.10.2.0 seemed to fix the
problem.  Separately, one of the consumers actually had a mechanism where
they were polling every 50ms, across 10 different topics, and committing
after every poll cycle, regardless of if there had been messages consumed
or not.  So, once they dialed that down to commit every 3 seconds per
consumer, the problem went away.

What we have realized is that maybe choosing the __consumer_offset
partition based on the groupId is not ideal.  A lot of teams will use the
same exact group ID for dozens of machines, just consuming from different
topics.  All of those commits will end up on the same exact partition,
hence the same broker, and this might in turn cause performance problems.
Would it be reasonable to change this in the future, to maybe default to
current behavior, but allow a setting that would choose the partition based
on group_ID+topic name?  That would be extremely helpful.

The way we figured out this whole thing was, when going through the client
source code, trying to understand how to decode the binary data in the
__consumer_offsets records, I found the *OffsetsManagerFormatter*, which
allowed me to consume the __consumer_offsets topic directly, deserializing
the consumer/topic that are being committed to.  This is what I was running
to get this information:

*./kafka-console-consumer.sh --bootstrap-server kafka-server:9092 --topic
__consumer_offsets --formatter
"kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" *

I thought this was really helpful, and think this tool should be
mentioned/documented, so other people can find it when diagnosing
performance issues on their Kafka clusters.

Anyway, thought I'd post a follow-up to the original question, since it
might help somebody else in the future.

Marcos

On Mon, Oct 2, 2017 at 6:57 PM, Marcos Juarez <mj...@gmail.com> wrote:

> I apologize for sending this to dev.  Reposting to the Users mailing list.
>
> ---------- Forwarded message ----------
>
> I was investigating some performance issues we're issues in one of our
> production clusters, and I ran into extremely unbalanced offset partitions
> for the __consumer_offsets topic.  I only pasted the top 8 below, out of 50
> total.  As you can see, between the top 5 partitions, those servers have to
> handle 83% of the commit volume, and brokers 9 and 10 show up repeatedly on
> as leader as well as replicas.
>
>
> Partition Offsets Percentage Leader Replicas ISR
> 6 52,761,610,477 34.24% 10 (10,6,7) (7,6,10)
> 5 46,196,021,230 29.98% 9 (9,5,6) (5,6,9)
> 42 17,530,298,423 11.38% 10 (10,9,11) (10,11,9)
> 31 12,927,081,106 8.39% 11 (11,9,10) (10,11,9)
> 0 8,557,903,671 5.55% 4 (4,12,1) (4,12,1)
> 2 3,969,232,652 2.58% 6 (6,2,3) (6,3,2)
> 49 3,555,754,347 2.31% 5 (5,11,7) (5,7,11)
> 33 2,273,951,745 1.48% 1 (1,11,12) (1,12,11)
> Those brokers (9, 10 and 11) also happen to be the ones we're having
> performance issues with.  We can't be sure yet if this is the cause of the
> performance issues, but it's looking extremely likely.
>
> So, I was wondering, what can be done to "rebalance" these consumer
> offsets?  This was something, as far as I know, automatically decided, I
> don't believe we ever changed a setting related to this.  I also don't
> believe we can influence which partition gets which offsets when
> consuming.
>
> It would also be interesting to know what is the algorithm/pattern used to
> decide the consumer offset partition, and is this something we can change
> or influence?
>
> Thanks,
>
> Marcos Juarez
>
>

Fwd: Consumer Offsets partition skew on Kafka 0.10.1.1

[Marcos Juarez <mj...@gmail.com>]

I apologize for sending this to dev.  Reposting to the Users mailing list.

---------- Forwarded message ----------

I was investigating some performance issues we're issues in one of our
production clusters, and I ran into extremely unbalanced offset partitions
for the __consumer_offsets topic.  I only pasted the top 8 below, out of 50
total.  As you can see, between the top 5 partitions, those servers have to
handle 83% of the commit volume, and brokers 9 and 10 show up repeatedly on
as leader as well as replicas.


Partition Offsets Percentage Leader Replicas ISR
6 52,761,610,477 34.24% 10 (10,6,7) (7,6,10)
5 46,196,021,230 29.98% 9 (9,5,6) (5,6,9)
42 17,530,298,423 11.38% 10 (10,9,11) (10,11,9)
31 12,927,081,106 8.39% 11 (11,9,10) (10,11,9)
0 8,557,903,671 5.55% 4 (4,12,1) (4,12,1)
2 3,969,232,652 2.58% 6 (6,2,3) (6,3,2)
49 3,555,754,347 2.31% 5 (5,11,7) (5,7,11)
33 2,273,951,745 1.48% 1 (1,11,12) (1,12,11)
Those brokers (9, 10 and 11) also happen to be the ones we're having
performance issues with.  We can't be sure yet if this is the cause of the
performance issues, but it's looking extremely likely.

So, I was wondering, what can be done to "rebalance" these consumer
offsets?  This was something, as far as I know, automatically decided, I
don't believe we ever changed a setting related to this.  I also don't
believe we can influence which partition gets which offsets when
consuming.

It would also be interesting to know what is the algorithm/pattern used to
decide the consumer offset partition, and is this something we can change
or influence?

Thanks,

Marcos Juarez