Consumer Ordering

[Tom Amon <ta...@gmail.com>]

Hi All,

I have a question regarding ordering of consumed messages. We timestamp our
messages and send them into Kafka in order. I wrote a simple consumer that
simply consumes the messages and prints out the timestamp. I see messages
for all seven days worth of date being consumed at once.

Our setup:
Kafka 0.8
5 Kafka brokers
1400 partitions

The consumer has 10 threads, simply connects, consumes and prints
timestamps. It is set to the "smallest" offset so that it reads from the
beginning. There are many millions of messages so I think I can rule out
some partitions not having messages for certain days as the cause. I know
that Kafka doesn't guarantee ordering across partitions but I would assume
that with this volume of messages I would see the timestamps for the first
day, followed by the second day, etc. Instead I see them all print at once.

Any ideas what I might be doing wrong?

Re: Consumer Ordering

[Jun Rao <ju...@gmail.com>]

It may be hard to reason about ordering across 1400 partitions. Could you
use the SimpleConsumerShell to consume messages from 1 partition and see if
messages are ordered?

Thanks,

Jun


On Fri, Mar 21, 2014 at 1:04 AM, Tom Amon <ta...@gmail.com> wrote:

> Hi All,
>
> I have a question regarding ordering of consumed messages. We timestamp our
> messages and send them into Kafka in order. I wrote a simple consumer that
> simply consumes the messages and prints out the timestamp. I see messages
> for all seven days worth of date being consumed at once.
>
> Our setup:
> Kafka 0.8
> 5 Kafka brokers
> 1400 partitions
>
> The consumer has 10 threads, simply connects, consumes and prints
> timestamps. It is set to the "smallest" offset so that it reads from the
> beginning. There are many millions of messages so I think I can rule out
> some partitions not having messages for certain days as the cause. I know
> that Kafka doesn't guarantee ordering across partitions but I would assume
> that with this volume of messages I would see the timestamps for the first
> day, followed by the second day, etc. Instead I see them all print at once.
>
> Any ideas what I might be doing wrong?
>