You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Neeraj Vaidya <ne...@yahoo.co.in.INVALID> on 2021/10/24 11:23:26 UTC
Apache Kafka mirrormaker2 : sync.group.offsets.enabled and
sync.group.offsets.interval.seconds
When replicating data from one datacentre (DC1) to another (DC2) using MM2, I can see that the consumer offsets are actually getting synchronized from source to target cluster in realtime. And not really at the default sync frequency time of 60 seconds.
I have enabled both the above config options (in the subject of this post) in MM2 config.
The way I tested this is by making consumers listen to the topic , say T1 in source cluster as well as DC2.T1 in target cluster.
If I produce a message to T1 in DC1, then the consumer in DC1 consumes the message, but the consumer in DC2 which is listening on DC2.T1 does not process the message. I am guessing because the message was processed by consumer in DC1 so it does not need to be processed by the consumer in DC2 as the offsets are being synchronized.
This is actually good for me because I can then run consumers in both DC's in parallel and make use of parallelism as well as geo-redundancy.
But I would like to understand if this is expected behaviour.
Re: Apache Kafka mirrormaker2 : sync.group.offsets.enabled and sync.group.offsets.interval.seconds
Posted by Neeraj Vaidya <ne...@yahoo.co.in.INVALID>.
Hello All,
Any inputs on my question below ?
Regards,
Neeraj
Sent from my iPhone
> On 26 Oct 2021, at 10:29 am, Neeraj Vaidya <ne...@yahoo.co.in.invalid> wrote:
>
> Anyone has inputs on my question below ?
>
> Regards,
> Neeraj
>
> Sent from my iPhone
>
>> On 24 Oct 2021, at 10:23 pm, Neeraj Vaidya <ne...@yahoo.co.in.invalid> wrote:
>>
>> When replicating data from one datacentre (DC1) to another (DC2) using MM2, I can see that the consumer offsets are actually getting synchronized from source to target cluster in realtime. And not really at the default sync frequency time of 60 seconds.
>>
>> I have enabled both the above config options (in the subject of this post) in MM2 config.
>>
>> The way I tested this is by making consumers listen to the topic , say T1 in source cluster as well as DC2.T1 in target cluster.
>>
>> If I produce a message to T1 in DC1, then the consumer in DC1 consumes the message, but the consumer in DC2 which is listening on DC2.T1 does not process the message. I am guessing because the message was processed by consumer in DC1 so it does not need to be processed by the consumer in DC2 as the offsets are being synchronized.
>>
>> This is actually good for me because I can then run consumers in both DC's in parallel and make use of parallelism as well as geo-redundancy.
>>
>> But I would like to understand if this is expected behaviour.
>
Re: Apache Kafka mirrormaker2 : sync.group.offsets.enabled and sync.group.offsets.interval.seconds
Posted by Neeraj Vaidya <ne...@yahoo.co.in.INVALID>.
Anyone has inputs on my question below ?
Regards,
Neeraj
Sent from my iPhone
> On 24 Oct 2021, at 10:23 pm, Neeraj Vaidya <ne...@yahoo.co.in.invalid> wrote:
>
> When replicating data from one datacentre (DC1) to another (DC2) using MM2, I can see that the consumer offsets are actually getting synchronized from source to target cluster in realtime. And not really at the default sync frequency time of 60 seconds.
>
> I have enabled both the above config options (in the subject of this post) in MM2 config.
>
> The way I tested this is by making consumers listen to the topic , say T1 in source cluster as well as DC2.T1 in target cluster.
>
> If I produce a message to T1 in DC1, then the consumer in DC1 consumes the message, but the consumer in DC2 which is listening on DC2.T1 does not process the message. I am guessing because the message was processed by consumer in DC1 so it does not need to be processed by the consumer in DC2 as the offsets are being synchronized.
>
> This is actually good for me because I can then run consumers in both DC's in parallel and make use of parallelism as well as geo-redundancy.
>
> But I would like to understand if this is expected behaviour.