You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by karan alang <ka...@gmail.com> on 2017/06/22 19:33:04 UTC
Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Hi All -
version - kafka 0.10
I'm publishing data into Kafka topic using command line,
and reading the data using kafka console consumer
*Publish command ->*
$KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1
--max-messages 100 --broker-list
localhost:9092,localhost:9093,localhost:9094,localhost:9095
--producer.config $KAFKA_HOME/config/producer.properties
*Console Consumer :*
$KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper localhost:2161
--topic mmtopic1 --from-beginning
What i see is that the Kafka consumer is not reading the data in sequence
i.e. the data on console is seen, but not in order it was published.
Is that expected ?
what do i need to do to ensure the Kafka consumer reads the data in
sequence ?
Re: Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Posted by karan alang <ka...@gmail.com>.
Hey Subhash,
thanks, i was able to test this out with 1 partition topic & verify this.
On Thu, Jun 22, 2017 at 1:39 PM, Subhash Sriram <su...@gmail.com>
wrote:
> Hi Karan,
>
> Yeah, so as to Paolo's point, keep in mind that Kafka does not guarantee
> order across partitions, only within a partition. If you publish messages
> to a topic with 3 partitions, it will only be guaranteed that they are
> consumed in order within the partition.
>
> You can retry your test by publishing to a single partition topic. When you
> consume, you should see that it is all in order.
>
> I hope that helps.
>
> Thanks,
> Subhash
>
> On Thu, Jun 22, 2017 at 4:37 PM, karan alang <ka...@gmail.com>
> wrote:
>
> > Hi Subhash,
> >
> > number of partitions - 3
> >
> > On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram <
> subhash.sriram@gmail.com
> > >
> > wrote:
> >
> > > How many partitions are in your topic?
> > >
> > > On Thu, Jun 22, 2017 at 3:33 PM, karan alang <ka...@gmail.com>
> > > wrote:
> > >
> > > > Hi All -
> > > >
> > > > version - kafka 0.10
> > > > I'm publishing data into Kafka topic using command line,
> > > > and reading the data using kafka console consumer
> > > >
> > > > *Publish command ->*
> > > >
> > > > $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1
> > > > --max-messages 100 --broker-list
> > > > localhost:9092,localhost:9093,localhost:9094,localhost:9095
> > > > --producer.config $KAFKA_HOME/config/producer.properties
> > > >
> > > > *Console Consumer :*
> > > >
> > > > $KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper
> localhost:2161
> > > > --topic mmtopic1 --from-beginning
> > > >
> > > > What i see is that the Kafka consumer is not reading the data in
> > sequence
> > > > i.e. the data on console is seen, but not in order it was published.
> > > >
> > > > Is that expected ?
> > > > what do i need to do to ensure the Kafka consumer reads the data in
> > > > sequence ?
> > > >
> > >
> >
>
Re: Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Posted by Subhash Sriram <su...@gmail.com>.
Hi Karan,
Yeah, so as to Paolo's point, keep in mind that Kafka does not guarantee
order across partitions, only within a partition. If you publish messages
to a topic with 3 partitions, it will only be guaranteed that they are
consumed in order within the partition.
You can retry your test by publishing to a single partition topic. When you
consume, you should see that it is all in order.
I hope that helps.
Thanks,
Subhash
On Thu, Jun 22, 2017 at 4:37 PM, karan alang <ka...@gmail.com> wrote:
> Hi Subhash,
>
> number of partitions - 3
>
> On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram <subhash.sriram@gmail.com
> >
> wrote:
>
> > How many partitions are in your topic?
> >
> > On Thu, Jun 22, 2017 at 3:33 PM, karan alang <ka...@gmail.com>
> > wrote:
> >
> > > Hi All -
> > >
> > > version - kafka 0.10
> > > I'm publishing data into Kafka topic using command line,
> > > and reading the data using kafka console consumer
> > >
> > > *Publish command ->*
> > >
> > > $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1
> > > --max-messages 100 --broker-list
> > > localhost:9092,localhost:9093,localhost:9094,localhost:9095
> > > --producer.config $KAFKA_HOME/config/producer.properties
> > >
> > > *Console Consumer :*
> > >
> > > $KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper localhost:2161
> > > --topic mmtopic1 --from-beginning
> > >
> > > What i see is that the Kafka consumer is not reading the data in
> sequence
> > > i.e. the data on console is seen, but not in order it was published.
> > >
> > > Is that expected ?
> > > what do i need to do to ensure the Kafka consumer reads the data in
> > > sequence ?
> > >
> >
>
Re: Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Posted by karan alang <ka...@gmail.com>.
Hi Subhash,
number of partitions - 3
On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram <su...@gmail.com>
wrote:
> How many partitions are in your topic?
>
> On Thu, Jun 22, 2017 at 3:33 PM, karan alang <ka...@gmail.com>
> wrote:
>
> > Hi All -
> >
> > version - kafka 0.10
> > I'm publishing data into Kafka topic using command line,
> > and reading the data using kafka console consumer
> >
> > *Publish command ->*
> >
> > $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1
> > --max-messages 100 --broker-list
> > localhost:9092,localhost:9093,localhost:9094,localhost:9095
> > --producer.config $KAFKA_HOME/config/producer.properties
> >
> > *Console Consumer :*
> >
> > $KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper localhost:2161
> > --topic mmtopic1 --from-beginning
> >
> > What i see is that the Kafka consumer is not reading the data in sequence
> > i.e. the data on console is seen, but not in order it was published.
> >
> > Is that expected ?
> > what do i need to do to ensure the Kafka consumer reads the data in
> > sequence ?
> >
>
Re: Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Posted by karan alang <ka...@gmail.com>.
got it, thanks!
On Thu, Jun 22, 2017 at 12:40 PM, Paolo Patierno <pp...@live.com> wrote:
> Kafka guarantees messages ordering at partition level not across
> partitions at topic level. Having out of order reading maybe possible If
> your topic has more than one partition.
>
> From: Subhash Sriram
> Sent: Thursday, 22 June, 21:37
> Subject: Re: Kafka 0.10 - kafka console consumer not reading the data in
> order that it was published
> To: users@kafka.apache.org
>
>
> How many partitions are in your topic? On Thu, Jun 22, 2017 at 3:33 PM,
> karan alang wrote: > Hi All - > > version - kafka 0.10 > I'm publishing
> data into Kafka topic using command line, > and reading the data using
> kafka console consumer > > *Publish command ->* > > $KAFKA_HOME/bin/kafka-verifiable-producer.sh
> --topic mmtopic1 > --max-messages 100 --broker-list >
> localhost:9092,localhost:9093,localhost:9094,localhost:9095 >
> --producer.config $KAFKA_HOME/config/producer.properties > > *Console
> Consumer :* > > $KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper
> localhost:2161 > --topic mmtopic1 --from-beginning > > What i see is that
> the Kafka consumer is not reading the data in sequence > i.e. the data on
> console is seen, but not in order it was published. > > Is that expected ?
> > what do i need to do to ensure the Kafka consumer reads the data in >
> sequence ? >
>
>
Re: Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Posted by Paolo Patierno <pp...@live.com>.
Kafka guarantees messages ordering at partition level not across partitions at topic level. Having out of order reading maybe possible If your topic has more than one partition.
From: Subhash Sriram
Sent: Thursday, 22 June, 21:37
Subject: Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published
To: users@kafka.apache.org
How many partitions are in your topic? On Thu, Jun 22, 2017 at 3:33 PM, karan alang wrote: > Hi All - > > version - kafka 0.10 > I'm publishing data into Kafka topic using command line, > and reading the data using kafka console consumer > > *Publish command ->* > > $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1 > --max-messages 100 --broker-list > localhost:9092,localhost:9093,localhost:9094,localhost:9095 > --producer.config $KAFKA_HOME/config/producer.properties > > *Console Consumer :* > > $KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper localhost:2161 > --topic mmtopic1 --from-beginning > > What i see is that the Kafka consumer is not reading the data in sequence > i.e. the data on console is seen, but not in order it was published. > > Is that expected ? > what do i need to do to ensure the Kafka consumer reads the data in > sequence ? >
Re: Kafka 0.10 - kafka console consumer not reading the data in order
that it was published
Posted by Subhash Sriram <su...@gmail.com>.
How many partitions are in your topic?
On Thu, Jun 22, 2017 at 3:33 PM, karan alang <ka...@gmail.com> wrote:
> Hi All -
>
> version - kafka 0.10
> I'm publishing data into Kafka topic using command line,
> and reading the data using kafka console consumer
>
> *Publish command ->*
>
> $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1
> --max-messages 100 --broker-list
> localhost:9092,localhost:9093,localhost:9094,localhost:9095
> --producer.config $KAFKA_HOME/config/producer.properties
>
> *Console Consumer :*
>
> $KAFKA10_HOME/bin/kafka-console-consumer.sh --zookeeper localhost:2161
> --topic mmtopic1 --from-beginning
>
> What i see is that the Kafka consumer is not reading the data in sequence
> i.e. the data on console is seen, but not in order it was published.
>
> Is that expected ?
> what do i need to do to ensure the Kafka consumer reads the data in
> sequence ?
>