You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yang <te...@gmail.com> on 2015/02/24 22:51:09 UTC
latest messages do not show up on consumers
in the following code, I just create a new topic, send some messages onto
the new topic with console producer, then read it with console consumer.
but after the 3rd line, nothing is printed out.
export topic=ttnew2
sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
--topic $topic --partitions 1 --replication-factor 1
echo 'asdfasdfsadffadsfasf' | sudo -u kafka bin/kafka-console-producer.sh
--broker-list localhost:9092 --topic $topic
sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
--topic $topic -group fffff
shouldn't I expect the consumer on the 3rd line to read out the messages?
in fact if I give it --from-beginning, it works fine
this is kafka_2.9.2-0.8.1.1
Thanks
Yang
Re: latest messages do not show up on consumers
Posted by Yang <te...@gmail.com>.
JiangJie:
thanks for the info.
it looks I can change the default behavior and let a new group read from
earliest offset by setting auto.offset.reset=smallest
Yang
On Tue, Feb 24, 2015 at 3:06 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> If a consumer comes from a new consumer group, it will by default consume
> from the log end.
> In you first scenario, because you started consumer after producing data,
> when the consumer starts it will consume from the latest. So you will not
> be able to see the messages even though they are in the broker.
> In your second scenario, because consumer will commit the offsets
> periodically and during rebalance. The first run of consumer will commit
> offset to be 0.
> Then you produced some data and run the second consumer. Because the
> second run of consumer uses the same consumer group, it will pick up the
> committed offset and consume from 0. In this case you will see the
> produced messages. If you use a different consumer group in your second
> run of consumer, you will not be able to see the messages because the
> offset will be reset to latest like in the first scenario.
>
> -Jiangjie Qin
>
> On 2/24/15, 1:56 PM, "Yang" <te...@gmail.com> wrote:
>
> >somehow I have to let the consumer run once BEFORE the messages are
> >produced. then run consumer after messages are produced, then it works:
> >
> >
> >export topic=ttnew21
> > sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
> >--topic $topic --partitions 1 --replication-factor 1
> >echo "first try"
> > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
> >--consumer-timeout-ms 1000 --topic $topic -group fffffxx
> > echo 'asdfasdfsadffadsfasf' | sudo -u kafka
> >bin/kafka-console-producer.sh
> >--broker-list localhost:9092 --topic $topic
> >echo "second try"
> > sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
> >--consumer-timeout-ms 1000 --topic $topic -group fffffxx
> >
> >
> >the first try doesn't read anything out (of course) ; the second one
> >works.
> >but if I remove the first try, the remaining consumer can't read anything
> >either.
> >
> >
> >
> >looks like a bug?
> >
> >On Tue, Feb 24, 2015 at 1:51 PM, Yang <te...@gmail.com> wrote:
> >
> >> in the following code, I just create a new topic, send some messages
> >>onto
> >> the new topic with console producer, then read it with console consumer.
> >> but after the 3rd line, nothing is printed out.
> >>
> >>
> >>
> >> export topic=ttnew2
> >> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
> >> --topic $topic --partitions 1 --replication-factor 1
> >> echo 'asdfasdfsadffadsfasf' | sudo -u kafka
> >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> >>$topic
> >> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
> >> --topic $topic -group fffff
> >>
> >>
> >> shouldn't I expect the consumer on the 3rd line to read out the
> >>messages?
> >> in fact if I give it --from-beginning, it works fine
> >>
> >>
> >>
> >> this is kafka_2.9.2-0.8.1.1
> >>
> >>
> >> Thanks
> >> Yang
> >>
>
>
Re: latest messages do not show up on consumers
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
If a consumer comes from a new consumer group, it will by default consume
from the log end.
In you first scenario, because you started consumer after producing data,
when the consumer starts it will consume from the latest. So you will not
be able to see the messages even though they are in the broker.
In your second scenario, because consumer will commit the offsets
periodically and during rebalance. The first run of consumer will commit
offset to be 0.
Then you produced some data and run the second consumer. Because the
second run of consumer uses the same consumer group, it will pick up the
committed offset and consume from 0. In this case you will see the
produced messages. If you use a different consumer group in your second
run of consumer, you will not be able to see the messages because the
offset will be reset to latest like in the first scenario.
-Jiangjie Qin
On 2/24/15, 1:56 PM, "Yang" <te...@gmail.com> wrote:
>somehow I have to let the consumer run once BEFORE the messages are
>produced. then run consumer after messages are produced, then it works:
>
>
>export topic=ttnew21
> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
>--topic $topic --partitions 1 --replication-factor 1
>echo "first try"
> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
>--consumer-timeout-ms 1000 --topic $topic -group fffffxx
> echo 'asdfasdfsadffadsfasf' | sudo -u kafka
>bin/kafka-console-producer.sh
>--broker-list localhost:9092 --topic $topic
>echo "second try"
> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
>--consumer-timeout-ms 1000 --topic $topic -group fffffxx
>
>
>the first try doesn't read anything out (of course) ; the second one
>works.
>but if I remove the first try, the remaining consumer can't read anything
>either.
>
>
>
>looks like a bug?
>
>On Tue, Feb 24, 2015 at 1:51 PM, Yang <te...@gmail.com> wrote:
>
>> in the following code, I just create a new topic, send some messages
>>onto
>> the new topic with console producer, then read it with console consumer.
>> but after the 3rd line, nothing is printed out.
>>
>>
>>
>> export topic=ttnew2
>> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
>> --topic $topic --partitions 1 --replication-factor 1
>> echo 'asdfasdfsadffadsfasf' | sudo -u kafka
>> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
>>$topic
>> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
>> --topic $topic -group fffff
>>
>>
>> shouldn't I expect the consumer on the 3rd line to read out the
>>messages?
>> in fact if I give it --from-beginning, it works fine
>>
>>
>>
>> this is kafka_2.9.2-0.8.1.1
>>
>>
>> Thanks
>> Yang
>>
Re: latest messages do not show up on consumers
Posted by Yang <te...@gmail.com>.
somehow I have to let the consumer run once BEFORE the messages are
produced. then run consumer after messages are produced, then it works:
export topic=ttnew21
sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
--topic $topic --partitions 1 --replication-factor 1
echo "first try"
sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
--consumer-timeout-ms 1000 --topic $topic -group fffffxx
echo 'asdfasdfsadffadsfasf' | sudo -u kafka bin/kafka-console-producer.sh
--broker-list localhost:9092 --topic $topic
echo "second try"
sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
--consumer-timeout-ms 1000 --topic $topic -group fffffxx
the first try doesn't read anything out (of course) ; the second one works.
but if I remove the first try, the remaining consumer can't read anything
either.
looks like a bug?
On Tue, Feb 24, 2015 at 1:51 PM, Yang <te...@gmail.com> wrote:
> in the following code, I just create a new topic, send some messages onto
> the new topic with console producer, then read it with console consumer.
> but after the 3rd line, nothing is printed out.
>
>
>
> export topic=ttnew2
> sudo -u kafka bin/kafka-topics.sh --zookeeper localhost:7181 --create
> --topic $topic --partitions 1 --replication-factor 1
> echo 'asdfasdfsadffadsfasf' | sudo -u kafka
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic $topic
> sudo -u kafka bin/kafka-console-consumer.sh --zookeeper localhost:7181
> --topic $topic -group fffff
>
>
> shouldn't I expect the consumer on the 3rd line to read out the messages?
> in fact if I give it --from-beginning, it works fine
>
>
>
> this is kafka_2.9.2-0.8.1.1
>
>
> Thanks
> Yang
>