mirrormaker's configuration to minimize/prevent data loss

[Andrey Yegorov <an...@gmail.com>]

As I read, consumer and producer in mirrormaker are independent and use
queue to communicate. Therefore consumers keep on consuming/commiting
offsets to zk even if producer is failing. Is it still the way it works in
0.8.0, any plans to change?

Is there any way to minimize data loss in this case? I am ok with not using
async mode on producer, but will it help? Can I configure mirrormaker to
exit immediately if producer fails? If this should be a responsibility of
an external process, what should I monitor log for to kill the mirroring
process in case of error?

----------
Andrey Yegorov

Re: mirrormaker's configuration to minimize/prevent data loss

[Jun Rao <ju...@gmail.com>]

Currently, mirrormaker only logs the error if the producer fails. You can
potentially increase # retries to deal with producer failures.

Thanks,

Jun


On Mon, Jun 16, 2014 at 3:53 PM, Andrey Yegorov <an...@gmail.com>
wrote:

> As I read, consumer and producer in mirrormaker are independent and use
> queue to communicate. Therefore consumers keep on consuming/commiting
> offsets to zk even if producer is failing. Is it still the way it works in
> 0.8.0, any plans to change?
>
> Is there any way to minimize data loss in this case? I am ok with not using
> async mode on producer, but will it help? Can I configure mirrormaker to
> exit immediately if producer fails? If this should be a responsibility of
> an external process, what should I monitor log for to kill the mirroring
> process in case of error?
>
> ----------
> Andrey Yegorov
>