You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jun Rao <ju...@gmail.com> on 2013/07/01 05:51:59 UTC

Re: Is it possible to get latest offset from kafka server?

If you use MirrorMaker to mirror the data from the primary cluster to the
backup cluster, currently the offsets are not preserved btw the two
clusters. In other words, the offsets for the same message in the two
clusters are not necessarily the same.

Thanks,

Jun


On Sat, Jun 29, 2013 at 10:56 AM, Yu, Libo <li...@citi.com> wrote:

> Hi,
>
> I know kafka.tools.ConsumerOffsetChecker can find out the latest offset for
> a topic from log files internally. Is there any public API that users can
> use for that?
>
> This would be useful for the failover scenario. When one cluster is down,
> the
> publisher will publish to the backup cluster. But the backup cluster
> always lags
> behind. If it is possible for publisher to find out the latest offset from
> the backup,
> it can always publish from that offset assuming publisher has a large
> enough
> buffer.  This guarantees no message loss. Thanks.
>
>
> Libo
>
>

RE: Is it possible to get latest offset from kafka server?

Posted by "Yu, Libo " <li...@citi.com>.
Thanks for your comment, Jun. Your last sentence is really helpful.

Regards,

Libo


-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Sunday, June 30, 2013 11:52 PM
To: users@kafka.apache.org
Subject: Re: Is it possible to get latest offset from kafka server?

If you use MirrorMaker to mirror the data from the primary cluster to the backup cluster, currently the offsets are not preserved btw the two clusters. In other words, the offsets for the same message in the two clusters are not necessarily the same.

Thanks,

Jun


On Sat, Jun 29, 2013 at 10:56 AM, Yu, Libo <li...@citi.com> wrote:

> Hi,
>
> I know kafka.tools.ConsumerOffsetChecker can find out the latest 
> offset for a topic from log files internally. Is there any public API 
> that users can use for that?
>
> This would be useful for the failover scenario. When one cluster is 
> down, the publisher will publish to the backup cluster. But the backup 
> cluster always lags behind. If it is possible for publisher to find 
> out the latest offset from the backup, it can always publish from that 
> offset assuming publisher has a large enough buffer.  This guarantees 
> no message loss. Thanks.
>
>
> Libo
>
>