You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tarang Dawer <ta...@gmail.com> on 2013/10/17 15:13:03 UTC
Consumer doesn't start picking existing messages from topic until new
data is added
Hi All
I am facing issues with the kafka 0.8 consumer with the consumer not
picking existing messages in the topic for consumption.
Following is a snippet from my code : -
public void run() {
ConsumerIterator<byte[],byte[]> it = stream.iterator();
System.out.println("************************In Run : before
while loop"); //*1st SYSO*
while (it.hasNext()) {
byte[] message = it.next().message();
System.out.println("Got New Mesage , Processing .. Dump in
hbase table....********************************"); //*2nd SYSO*
}
Steps: -
1. Cleaned zookeeper and kafka logs.
2. Started both again and produced 1000 messages in the topic.
3. Started the consumer. the consumer , i got log till *1st SYSO* but no
progress. it kept on waiting for the stream to get messages
4. Produced 1000 more messages in the topic, now the data consumption
starts , but the messages which i produced earlier in the topic were not
consumed, as the hbase table showed only 1000 messages , which increased
further depending on the additional number of messages i produced further,
but however, the 1st initial deficit is still there.
As it seems to me, the 1st 1000 messages are being lost, although the topic
is getting created in the kafka-log directory , with the log file showing
the initial messages there in the logbefore i start the consumer) , but the
consumer does not seems to be picking them up.
I am using Zookeeper-3.4.5 .
Kafka-Settings are default , except that i the partitions for each topic
are set to 1.
Please help me out.
Thanks
Tarang Dawer
Re: Consumer doesn't start picking existing messages from topic until
new data is added
Posted by Tarang Dawer <ta...@gmail.com>.
Thanks Bruno
that fixed it..!!
On Thu, Oct 17, 2013 at 6:47 PM, Bruno D. Rodrigues <
bruno.rodrigues@litux.org> wrote:
> try this, not sure if it would help though
>
> props.put("auto.offset.reset", "smallest");
>
> A 17/10/2013, às 14:13, Tarang Dawer <ta...@gmail.com> escreveu:
>
> Hi All
>
> I am facing issues with the kafka 0.8 consumer with the consumer not
> picking existing messages in the topic for consumption.
>
>
>
> Following is a snippet from my code : -
>
>
> public void run() {
> ConsumerIterator<byte[],byte[]> it = stream.iterator();
> System.out.println("************************In Run : before
> while loop"); //*1st SYSO*
>
> while (it.hasNext()) {
> byte[] message = it.next().message();
> System.out.println("Got New Mesage , Processing .. Dump in
> hbase table....********************************"); //*2nd SYSO*
>
> }
>
>
> Steps: -
> 1. Cleaned zookeeper and kafka logs.
> 2. Started both again and produced 1000 messages in the topic.
> 3. Started the consumer. the consumer , i got log till *1st SYSO* but no
>
> progress. it kept on waiting for the stream to get messages
> 4. Produced 1000 more messages in the topic, now the data consumption
> starts , but the messages which i produced earlier in the topic were not
> consumed, as the hbase table showed only 1000 messages , which increased
> further depending on the additional number of messages i produced further,
> but however, the 1st initial deficit is still there.
>
> As it seems to me, the 1st 1000 messages are being lost, although the topic
> is getting created in the kafka-log directory , with the log file showing
> the initial messages there in the logbefore i start the consumer) , but the
> consumer does not seems to be picking them up.
>
> I am using Zookeeper-3.4.5 .
>
> Kafka-Settings are default , except that i the partitions for each topic
> are set to 1.
>
> Please help me out.
>
>
> Thanks
> Tarang Dawer
>
>
>
Re: Consumer doesn't start picking existing messages from topic until new data is added
Posted by "Bruno D. Rodrigues" <br...@litux.org>.
try this, not sure if it would help though
props.put("auto.offset.reset", "smallest");
A 17/10/2013, às 14:13, Tarang Dawer <ta...@gmail.com> escreveu:
> Hi All
>
> I am facing issues with the kafka 0.8 consumer with the consumer not
> picking existing messages in the topic for consumption.
>
>
>
> Following is a snippet from my code : -
>
>
> public void run() {
> ConsumerIterator<byte[],byte[]> it = stream.iterator();
> System.out.println("************************In Run : before
> while loop"); //*1st SYSO*
> while (it.hasNext()) {
> byte[] message = it.next().message();
> System.out.println("Got New Mesage , Processing .. Dump in
> hbase table....********************************"); //*2nd SYSO*
> }
>
>
> Steps: -
> 1. Cleaned zookeeper and kafka logs.
> 2. Started both again and produced 1000 messages in the topic.
> 3. Started the consumer. the consumer , i got log till *1st SYSO* but no
> progress. it kept on waiting for the stream to get messages
> 4. Produced 1000 more messages in the topic, now the data consumption
> starts , but the messages which i produced earlier in the topic were not
> consumed, as the hbase table showed only 1000 messages , which increased
> further depending on the additional number of messages i produced further,
> but however, the 1st initial deficit is still there.
>
> As it seems to me, the 1st 1000 messages are being lost, although the topic
> is getting created in the kafka-log directory , with the log file showing
> the initial messages there in the logbefore i start the consumer) , but the
> consumer does not seems to be picking them up.
>
> I am using Zookeeper-3.4.5 .
>
> Kafka-Settings are default , except that i the partitions for each topic
> are set to 1.
>
> Please help me out.
>
>
> Thanks
> Tarang Dawer