Reasonable time to commit offsets?

[Gwilym Evans <gw...@bigcommerce.com>]

Hi folks,

New to the list and new to operating Kafka. I'm trying to find out what a
reasonable turnaround time for committing offsets is.

I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper
nodes, though the cluster has been upgraded from its starting point at
0.10.0.0. The offsets topic is at its default configuration. The partition
I'm consuming and committing offsets for has 150 partitions.

While I can steadily consume around 90,000 messages/sec using 1 consumer
without (synchronously) committing offsets, if I want to do so
synchronously it takes between 1000ms and 2000ms to complete the commit. If
I use multiple consumers to reduce the partition count each is assigned
(I've tested up to 10) then time-to-commit does _not_ change.

Is this reasonable? Or should it be quicker than this?

For now, my use case is simple topic-splitting: read messages, apply basic
filters, discard or resend messages to one or more topics. But I don't want
to advance the offsets until I am certain the messages have been written
back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) commit.

If more information is needed, I'm happy to supply what I can.

Thanks,
Gwilym

Re: Reasonable time to commit offsets?

[Gwilym Evans <gw...@bigcommerce.com>]

Hi Apurva,

When I try producing single messages to a similarly-configured topic (30
partitions, 3x replication, acks=all), and enable full protocol debugging,
I am seeing single-digit ms round-trip-times for the request-response cycle
at a protocol level. But, I see other overheads which might be client-side
which add up to around 900 - 1500 ms.

I'm using the confluent-kafka-go Go bindings for the librdkafka C client
library.

There were some buffering settings I could tune to get Produce calls down
for the Producer, but the same tunings didn't seem to affect Commit calls
of the Consumer.

So this looks like it may be a Go or C lib issue, and I'll try taking my
findings to the developers of those.

Thanks,
Gwilym


On 14 December 2016 at 18:55, Apurva Mehta <ap...@confluent.io> wrote:

> Hi Gwilym,
>
> What is the latency for synchronously producing to this cluster? Is it also
> 1000 to 2000ms?
>
> Thanks,
> Apurva
>
> On Wed, Dec 14, 2016 at 2:17 AM, Gwilym Evans <
> gwilym.evans@bigcommerce.com>
> wrote:
>
> > Hi folks,
> >
> > New to the list and new to operating Kafka. I'm trying to find out what a
> > reasonable turnaround time for committing offsets is.
> >
> > I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper
> > nodes, though the cluster has been upgraded from its starting point at
> > 0.10.0.0. The offsets topic is at its default configuration. The
> partition
> > I'm consuming and committing offsets for has 150 partitions.
> >
> > While I can steadily consume around 90,000 messages/sec using 1 consumer
> > without (synchronously) committing offsets, if I want to do so
> > synchronously it takes between 1000ms and 2000ms to complete the commit.
> If
> > I use multiple consumers to reduce the partition count each is assigned
> > (I've tested up to 10) then time-to-commit does _not_ change.
> >
> > Is this reasonable? Or should it be quicker than this?
> >
> > For now, my use case is simple topic-splitting: read messages, apply
> basic
> > filters, discard or resend messages to one or more topics. But I don't
> want
> > to advance the offsets until I am certain the messages have been written
> > back... hence a cycle of 1) read a batch, 2) process, 3) write, 4)
> commit.
> >
> > If more information is needed, I'm happy to supply what I can.
> >
> > Thanks,
> > Gwilym
> >
>

Re: Reasonable time to commit offsets?

[Apurva Mehta <ap...@confluent.io>]

Hi Gwilym,

What is the latency for synchronously producing to this cluster? Is it also
1000 to 2000ms?

Thanks,
Apurva

On Wed, Dec 14, 2016 at 2:17 AM, Gwilym Evans <gw...@bigcommerce.com>
wrote:

> Hi folks,
>
> New to the list and new to operating Kafka. I'm trying to find out what a
> reasonable turnaround time for committing offsets is.
>
> I'm running a 0.10.1.0 cluster of 17 brokers plus 3 dedicated zookeeper
> nodes, though the cluster has been upgraded from its starting point at
> 0.10.0.0. The offsets topic is at its default configuration. The partition
> I'm consuming and committing offsets for has 150 partitions.
>
> While I can steadily consume around 90,000 messages/sec using 1 consumer
> without (synchronously) committing offsets, if I want to do so
> synchronously it takes between 1000ms and 2000ms to complete the commit. If
> I use multiple consumers to reduce the partition count each is assigned
> (I've tested up to 10) then time-to-commit does _not_ change.
>
> Is this reasonable? Or should it be quicker than this?
>
> For now, my use case is simple topic-splitting: read messages, apply basic
> filters, discard or resend messages to one or more topics. But I don't want
> to advance the offsets until I am certain the messages have been written
> back... hence a cycle of 1) read a batch, 2) process, 3) write, 4) commit.
>
> If more information is needed, I'm happy to supply what I can.
>
> Thanks,
> Gwilym
>