Inter Mirror maker processes offset sync

[tao xiao <xi...@gmail.com>]

Hi all,

I have two mirror maker processes running on two different machines
fetching messages from same topic from one data center to another data
center. These two processes are assigned to the same consumer group. If I
want no data loss or data duplication even when one of the mirror maker
processes die I need to find a way to inform another process the offset the
last successfully sent message by the dead process but I know the offset
stored in zookeeper is the offset the last consumed message by mirror maker
not the offset last successfully sent message, is there a way to configure
mirror maker to achieve exact one message semantic?

-- 
Regards,
Tao

Re: Inter Mirror maker processes offset sync

[Guozhang Wang <wa...@gmail.com>]

Hi Tao,

You may want to take a look at the current progress we are making at MM:

https://issues.apache.org/jira/browse/KAFKA-1650

which makes a zero data-loss MM, and:

https://issues.apache.org/jira/browse/KAFKA-1839

attempts to remove possible duplicates at MM.

Guozhang

On Wed, Jan 21, 2015 at 7:47 AM, tao xiao <xi...@gmail.com> wrote:

> Hi all,
>
> I have two mirror maker processes running on two different machines
> fetching messages from same topic from one data center to another data
> center. These two processes are assigned to the same consumer group. If I
> want no data loss or data duplication even when one of the mirror maker
> processes die I need to find a way to inform another process the offset the
> last successfully sent message by the dead process but I know the offset
> stored in zookeeper is the offset the last consumed message by mirror maker
> not the offset last successfully sent message, is there a way to configure
> mirror maker to achieve exact one message semantic?
>
> --
> Regards,
> Tao
>



-- 
-- Guozhang