You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ilja Golshtein <il...@yandex.ru> on 2016/01/20 11:35:20 UTC
unable to set consumer group
Hello.
I am trying to create consumer using kafka_python-0.9.5.
I expect that several instances of the script
==
consumer = KafkaConsumer('some-topic',
auto_offset_reset='largest',
metadata_broker_list=['localhost:9092'],
group_id='vasya_group',
auto_commit_enable=True)
consumer.set_topic_partitions('some-topic')
while True:
msg=consumer.next()
print(msg)
==
would receive different messages, while in reality every instance receives all messages.
What can be done to achieve
"Consumers label themselves with a consumer group name, and each message published to a topic is delivered to one consumer instance within each subscribing consumer group" as advertised in Kafka Documentation?
I am 100% that messages are distributed among several partitions (namely 10).
Thanks.
--
Best regards
Ilja Golshtein
Re: unable to set consumer group
Posted by Ilja Golshtein <il...@narod.ru>.
Dana,
thank you for comprehensive response.
It seems that we have to be 0.8 compatible.
We've already discovered pykafka and BalancedConsumer is our favorite choice, but it might be that assigning partition explicitly suits our needs better.
20.01.2016, 19:58, "Dana Powers" <da...@gmail.com>:
> version 0.9.5 of kafka-python does not support coordinated consumer groups.
> You can get this feature in the master branch on github (
> https://github.com/dpkp/kafka-python) using kafka 0.9.0.0 brokers. I expect
> to release the updates to pypi soon, but for now you'll have to install
> from source.
>
> Other python alternatives: assign partitions statically to each instance
> via chef/ansible/etc; or try pykafka's BalancedConsumer implementation,
> which uses Zookeeper to coordinate assignment.
>
> -Dana
>
> On Wed, Jan 20, 2016 at 2:35 AM, Ilja Golshtein <il...@yandex.ru> wrote:
>
>> Hello.
>>
>> I am trying to create consumer using kafka_python-0.9.5.
>>
>> I expect that several instances of the script
>>
>> ==
>> consumer = KafkaConsumer('some-topic',
>> auto_offset_reset='largest',
>> metadata_broker_list=['localhost:9092'],
>> group_id='vasya_group',
>> auto_commit_enable=True)
>> consumer.set_topic_partitions('some-topic')
>> while True:
>> msg=consumer.next()
>> print(msg)
>> ==
>> would receive different messages, while in reality every instance receives
>> all messages.
>>
>> What can be done to achieve
>> "Consumers label themselves with a consumer group name, and each message
>> published to a topic is delivered to one consumer instance within each
>> subscribing consumer group" as advertised in Kafka Documentation?
>>
>> I am 100% that messages are distributed among several partitions (namely
>> 10).
>>
>> Thanks.
>>
>> --
>> Best regards
>> Ilja Golshtein
--
Best regards
Ilja Golshtein
Re: unable to set consumer group
Posted by Dana Powers <da...@gmail.com>.
version 0.9.5 of kafka-python does not support coordinated consumer groups.
You can get this feature in the master branch on github (
https://github.com/dpkp/kafka-python) using kafka 0.9.0.0 brokers. I expect
to release the updates to pypi soon, but for now you'll have to install
from source.
Other python alternatives: assign partitions statically to each instance
via chef/ansible/etc; or try pykafka's BalancedConsumer implementation,
which uses Zookeeper to coordinate assignment.
-Dana
On Wed, Jan 20, 2016 at 2:35 AM, Ilja Golshtein <il...@yandex.ru> wrote:
> Hello.
>
> I am trying to create consumer using kafka_python-0.9.5.
>
> I expect that several instances of the script
>
> ==
> consumer = KafkaConsumer('some-topic',
> auto_offset_reset='largest',
> metadata_broker_list=['localhost:9092'],
> group_id='vasya_group',
> auto_commit_enable=True)
> consumer.set_topic_partitions('some-topic')
> while True:
> msg=consumer.next()
> print(msg)
> ==
> would receive different messages, while in reality every instance receives
> all messages.
>
> What can be done to achieve
> "Consumers label themselves with a consumer group name, and each message
> published to a topic is delivered to one consumer instance within each
> subscribing consumer group" as advertised in Kafka Documentation?
>
> I am 100% that messages are distributed among several partitions (namely
> 10).
>
> Thanks.
>
> --
> Best regards
> Ilja Golshtein
>