You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Pariksheet Barapatre <pa...@gmail.com> on 2016/07/20 11:57:16 UTC

Kafka Streaming Question : reset offset

Hi Experts,

I have written Kafka Streaming app that just filters rows based on some
condition and load it to MongoDB.

The streaming process is working fine but due to some flaw in my code, I
want to reprocess whole data again.

One way is to do this -  kill streaming app , change consumer group id ,
delete data from mongo and rerun the app.

How to achieve this scenario without changing consumer group id.

Many Thanks
Pari

Re: Kafka Streaming Question : reset offset

Posted by Pariksheet Barapatre <pa...@gmail.com>.

Many Thanks Matthias for update.

Regards
Pari

On 20 July 2016 at 17:48, Matthias J. Sax <ma...@confluent.io> wrote:

> Hi Pari,
>
> currently, changing the application ID is the best way to follow.
> Cleaning up the application state correctly, is a little bit tricky. We
> are currently working on an improvement for this -- should be available
> soon.
>
> See https://issues.apache.org/jira/browse/KAFKA-3185
>
>
> -Matthias
>
> On 07/20/2016 01:57 PM, Pariksheet Barapatre wrote:
> > Hi Experts,
> >
> > I have written Kafka Streaming app that just filters rows based on some
> > condition and load it to MongoDB.
> >
> > The streaming process is working fine but due to some flaw in my code, I
> > want to reprocess whole data again.
> >
> > One way is to do this -  kill streaming app , change consumer group id ,
> > delete data from mongo and rerun the app.
> >
> > How to achieve this scenario without changing consumer group id.
> >
> > Many Thanks
> > Pari
> >
>
>

Re: Kafka Streaming Question : reset offset

Posted by "Matthias J. Sax" <ma...@confluent.io>.

Hi Pari,

currently, changing the application ID is the best way to follow.
Cleaning up the application state correctly, is a little bit tricky. We
are currently working on an improvement for this -- should be available
soon.

See https://issues.apache.org/jira/browse/KAFKA-3185


-Matthias

On 07/20/2016 01:57 PM, Pariksheet Barapatre wrote:
> Hi Experts,
> 
> I have written Kafka Streaming app that just filters rows based on some
> condition and load it to MongoDB.
> 
> The streaming process is working fine but due to some flaw in my code, I
> want to reprocess whole data again.
> 
> One way is to do this -  kill streaming app , change consumer group id ,
> delete data from mongo and rerun the app.
> 
> How to achieve this scenario without changing consumer group id.
> 
> Many Thanks
> Pari
>