You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Рябков Алексей Николаевич <a....@ntc-vulkan.ru> on 2015/10/10 12:11:06 UTC
RE: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
Hello!
>> Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends on the settings in two properties: offset.storage and >>dual.commit.enabled (here <http://kafka.apache.org/documentation.html#consumerconfigs>). Currently ZK is the default.
Thanks for explanation.... but I have more questions...
1. As far as I understand when I use offset.storage=kafka and dual.commit.enabled in consumer (I mean java-based) then when I call commit method commit then internally client commit offset to kafka and then to zk.... Is it correct?
2. if first is correct ...then what happened if client crash between this 2 operations (commit 2 kafka and commit 2 zookeeper)?
3. What happened if broker crash.... And my client support only zookeeper? (I guess that I read message from beggining) Is it correct? Or broker explicitly sync kafka commit with zookeeper when started?
Thanks in advance, Aleksey
-----Original Message-----
From: Ben Stopford [mailto:ben@confluent.io]
Sent: Friday, October 9, 2015 5:54 PM
To: users@kafka.apache.org
Subject: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
Hi Alexey
Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends on the settings in two properties: offset.storage and dual.commit.enabled (here <http://kafka.apache.org/documentation.html#consumerconfigs>). Currently ZK is the default.
Commits happen either periodically (auto.commit.enable=true && auto.commit.interval.ms) or explicitly. For example the high level consumer has a commitOffsets method. I expect the python bindings have a similar function. The rest proxy offers this function too.
Hope that helps
B
> On 6 Oct 2015, at 23:06, Рябков Алексей Николаевич <a....@ntc-vulkan.ru> wrote:
>
> Hello!
>
> Can somebody explain me how to use multiple consumers with different commit storage...
> For example, java-based consumers use kafka commit storage...
> python-based consumers use zookeeper commit storage
>
> My question is:
> Is it true that when one consumer commit to kafka, server also commit
> automatically to zookeeper (so all consumers see the same commit) Is
> it true that when one consumer commit to zookeeper, server also
> commit automatically to kafka (so all consumers see the same commit)
> Is it true that after server restart it read commits from kafka
> storage and publish (sync) commit to zookeeper (so after server
> restart all consumers can see also the same commit)
>
> Thanks in advance, Aleksey
Re: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
Posted by Ben Stopford <be...@confluent.io>.
Dual commits are only really designed to aid migration of topics that were started using ZK and want to move offset storage to Kafka. They are completely separate offset storage mechanisms. I suggest you pick one and use that. I’d suggest Kafka.
In the crash case you describe, offsets would be different in ZK & Kafka. The consumer would pick the larger of the two offsets to use.
> On 10 Oct 2015, at 11:11, Рябков Алексей Николаевич <a....@ntc-vulkan.ru> wrote:
>
> Hello!
>
>>> Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends on the settings in two properties: offset.storage and >>dual.commit.enabled (here <http://kafka.apache.org/documentation.html#consumerconfigs>). Currently ZK is the default.
>
> Thanks for explanation.... but I have more questions...
> 1. As far as I understand when I use offset.storage=kafka and dual.commit.enabled in consumer (I mean java-based) then when I call commit method commit then internally client commit offset to kafka and then to zk.... Is it correct?
> 2. if first is correct ...then what happened if client crash between this 2 operations (commit 2 kafka and commit 2 zookeeper)?
> 3. What happened if broker crash.... And my client support only zookeeper? (I guess that I read message from beggining) Is it correct? Or broker explicitly sync kafka commit with zookeeper when started?
>
> Thanks in advance, Aleksey
>
> -----Original Message-----
> From: Ben Stopford [mailto:ben@confluent.io]
> Sent: Friday, October 9, 2015 5:54 PM
> To: users@kafka.apache.org
> Subject: [!!Mass Mail]Re: Dual commit with zookeeper and kafka
>
> Hi Alexey
>
> Whether you commit offsets to Kafka itself (stored in offsets topic) or ZK or both depends on the settings in two properties: offset.storage and dual.commit.enabled (here <http://kafka.apache.org/documentation.html#consumerconfigs>). Currently ZK is the default.
>
> Commits happen either periodically (auto.commit.enable=true && auto.commit.interval.ms) or explicitly. For example the high level consumer has a commitOffsets method. I expect the python bindings have a similar function. The rest proxy offers this function too.
>
> Hope that helps
>
> B
>
>
>> On 6 Oct 2015, at 23:06, Рябков Алексей Николаевич <a....@ntc-vulkan.ru> wrote:
>>
>> Hello!
>>
>> Can somebody explain me how to use multiple consumers with different commit storage...
>> For example, java-based consumers use kafka commit storage...
>> python-based consumers use zookeeper commit storage
>>
>> My question is:
>> Is it true that when one consumer commit to kafka, server also commit
>> automatically to zookeeper (so all consumers see the same commit) Is
>> it true that when one consumer commit to zookeeper, server also
>> commit automatically to kafka (so all consumers see the same commit)
>> Is it true that after server restart it read commits from kafka
>> storage and publish (sync) commit to zookeeper (so after server
>> restart all consumers can see also the same commit)
>>
>> Thanks in advance, Aleksey
>