You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ali Akhtar <al...@gmail.com> on 2016/11/30 12:10:16 UTC
Message order different each time stream is replayed?
While I was connected to console-consumer.sh, I posted a few messages to a
Kafka topic, one message at a time, across a few hours.
I'd post a message, see it arrive in console-consumer, a few mins later I'd
post the next message, and so on.
They all arrived in order.
However, when I now try to view console-consumer.sh --from-beginning, each
time I do so, the messages seem to be shown in a shuffled order.
Also the last 2 messages appear to be missing, although its possible I
didn't see them in the shuffled output (the topic has over 10k messages
total which are all dumped to console when I view).
Is this just due to network lag when viewing console-consumer, or should I
expect my actual consumers to also receive messages out of order each time
they replay a topic?
Re: Message order different each time stream is replayed?
Posted by Ali Akhtar <al...@gmail.com>.
Ah, That explains it. I have 3 partitions.
On 1 Dec 2016 5:42 a.m., "Apurva Mehta" <ap...@confluent.io> wrote:
How many partitions do you have in that topic. Kafka only guarantees a
total ordering of messages within a partition, not across partitions of a
topic. If you want total ordering over the entire topic, you need to create
a topic with a single partition.
On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar <al...@gmail.com> wrote:
> While I was connected to console-consumer.sh, I posted a few messages to a
> Kafka topic, one message at a time, across a few hours.
>
> I'd post a message, see it arrive in console-consumer, a few mins later
I'd
> post the next message, and so on.
>
> They all arrived in order.
>
> However, when I now try to view console-consumer.sh --from-beginning, each
> time I do so, the messages seem to be shown in a shuffled order.
>
> Also the last 2 messages appear to be missing, although its possible I
> didn't see them in the shuffled output (the topic has over 10k messages
> total which are all dumped to console when I view).
>
> Is this just due to network lag when viewing console-consumer, or should I
> expect my actual consumers to also receive messages out of order each time
> they replay a topic?
>
Re: Message order different each time stream is replayed?
Posted by Apurva Mehta <ap...@confluent.io>.
How many partitions do you have in that topic. Kafka only guarantees a
total ordering of messages within a partition, not across partitions of a
topic. If you want total ordering over the entire topic, you need to create
a topic with a single partition.
On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar <al...@gmail.com> wrote:
> While I was connected to console-consumer.sh, I posted a few messages to a
> Kafka topic, one message at a time, across a few hours.
>
> I'd post a message, see it arrive in console-consumer, a few mins later I'd
> post the next message, and so on.
>
> They all arrived in order.
>
> However, when I now try to view console-consumer.sh --from-beginning, each
> time I do so, the messages seem to be shown in a shuffled order.
>
> Also the last 2 messages appear to be missing, although its possible I
> didn't see them in the shuffled output (the topic has over 10k messages
> total which are all dumped to console when I view).
>
> Is this just due to network lag when viewing console-consumer, or should I
> expect my actual consumers to also receive messages out of order each time
> they replay a topic?
>