kafka-python consumer not receiving messages

[Karthik <kr...@twilio.com>]

Hello,

I am having trouble with KafaConsumer to make it read from the beginning,
or from any other explicit offset.

Running the command line tools for the consumer for the same topic , I do
see messages with the `--from-beginning` option and it hangs otherwise

$ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic
{topic_name} --from-beginning

If I run it through python, it hangs, which I suspect to be caused by
incorrect consumer configs
--------
consumer = KafkaConsumer(topic_name,
                         bootstrap_servers=['localhost:9092'],
                         group_id="test-consumer-group",
                         auto_commit_enable=False,
                         auto_offset_reset='smallest')

print "Consuming messages from the given topic"
for message in consumer:
    print "Message", message
    if message is not None:
        print message.offset, message.value

print "Quit"

Output:
----------
Consuming messages from the given topic
<hangs after that>

I am using kafka-python 0.9.5 and the broker runs kafka 8.2. Not sure what
the exact problem is. Any help would be appreciated.

Thanks,
Karthik

Re: kafka-python consumer not receiving messages

[Dana Powers <da...@rd.io>]

Hi karthik - I usually address kafka-python specific questions via github.
Can you file an issue at github.com/dpkp/kafka-python and I will follow up
there?

My initial reaction is you should leave group_id=None if you want to
duplicate behavior of the console consumer.

-Dana
Hello,

I am having trouble with KafaConsumer to make it read from the beginning,
or from any other explicit offset.

Running the command line tools for the consumer for the same topic , I do
see messages with the `--from-beginning` option and it hangs otherwise

$ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic
{topic_name} --from-beginning

If I run it through python, it hangs, which I suspect to be caused by
incorrect consumer configs
--------
consumer = KafkaConsumer(topic_name,
                         bootstrap_servers=['localhost:9092'],
                         group_id="test-consumer-group",
                         auto_commit_enable=False,
                         auto_offset_reset='smallest')

print "Consuming messages from the given topic"
for message in consumer:
    print "Message", message
    if message is not None:
        print message.offset, message.value

print "Quit"

Output:
----------
Consuming messages from the given topic
<hangs after that>

I am using kafka-python 0.9.5 and the broker runs kafka 8.2. Not sure what
the exact problem is. Any help would be appreciated.

Thanks,
Karthik