You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Surendranauth Hiraman <su...@velos.io> on 2015/02/18 17:16:22 UTC
Resetting Offsets
We are using the High Level Consumer API to interact with Kafka.
However, on restart in the case of failures, we want to be able to manually
reset offsets in certain situations.
What is the recommended way to do this?
Should we use the Simple Consumer API just for this restart case?
Ideally, it would be great to use the same approach in 0.8.1 and in 0.8.2.
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
54 West 40th Street, 3RD FLOOR
NEW YORK, NY 10018
T: @suren_h
E: suren.hiraman@v <su...@sociocast.com>elos.io
W: www.velos.io
Re: Resetting Offsets
Posted by Suren <ym...@yahoo.com.INVALID>.
Reading offsets looks like it's compatible across 0.8.1 and 0.8.2.
However, we cannot use the update logic in ImportZkOffsets, since we want to store offsets in the broker in 0.8.2.
It looks like SimpleConsumer.commitOffsets() would work with either version. Is there a better way?
-Suren
On Wednesday, February 18, 2015 11:49 AM, Michal Michalski <mi...@boxever.com> wrote:
See https://cwiki.apache.org/confluence/display/KAFKA/System+Tools
and check the following:
GetOffsetShell (not very accurate - will set your offsets to much smaller
values than you really need; we log offsets frequently in application logs
and get it from there)
ImportZkOffsets
Kind regards,
Michał Michalski,
michal.michalski@boxever.com
On 18 February 2015 at 16:16, Surendranauth Hiraman <su...@velos.io>
wrote:
> We are using the High Level Consumer API to interact with Kafka.
>
> However, on restart in the case of failures, we want to be able to manually
> reset offsets in certain situations.
>
> What is the recommended way to do this?
>
> Should we use the Simple Consumer API just for this restart case?
>
> Ideally, it would be great to use the same approach in 0.8.1 and in 0.8.2.
>
>
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
>
> 54 West 40th Street, 3RD FLOOR
> NEW YORK, NY 10018
> T: @suren_h
> E: suren.hiraman@v <su...@sociocast.com>elos.io
> W: www.velos.io
>
Re: Resetting Offsets
Posted by Michal Michalski <mi...@boxever.com>.
See https://cwiki.apache.org/confluence/display/KAFKA/System+Tools
and check the following:
GetOffsetShell (not very accurate - will set your offsets to much smaller
values than you really need; we log offsets frequently in application logs
and get it from there)
ImportZkOffsets
Kind regards,
Michał Michalski,
michal.michalski@boxever.com
On 18 February 2015 at 16:16, Surendranauth Hiraman <su...@velos.io>
wrote:
> We are using the High Level Consumer API to interact with Kafka.
>
> However, on restart in the case of failures, we want to be able to manually
> reset offsets in certain situations.
>
> What is the recommended way to do this?
>
> Should we use the Simple Consumer API just for this restart case?
>
> Ideally, it would be great to use the same approach in 0.8.1 and in 0.8.2.
>
>
>
> SUREN HIRAMAN, VP TECHNOLOGY
> Velos
> Accelerating Machine Learning
>
> 54 West 40th Street, 3RD FLOOR
> NEW YORK, NY 10018
> T: @suren_h
> E: suren.hiraman@v <su...@sociocast.com>elos.io
> W: www.velos.io
>