You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by John Hayles <jh...@etcc.com> on 2016/11/18 14:50:16 UTC

Change current offset with KStream

Is there offset controls when using KStream?   Before starting stream, can the current offset be changed to current offset - 5000.

I see KafkaConsumer has seek method, but I don't see how this helps when using KStream.  Could KafkaConsumer be used to change offset on a topic, then use KStream to process it?

Thanks for feedback...

Regards,
John

Re: Change current offset with KStream

Posted by Eno Thereska <en...@gmail.com>.

Hi John,

Perhaps you can tell us a bit more on what kind of fault tolerance you are looking for. I ask because Kafka Streams is fault tolerant and highly available by default and perhaps you don't need to do anything extra for your application.

Thanks
Eno


> On 18 Nov 2016, at 15:21, John Hayles <jh...@etcc.com> wrote:
> 
> Eno, thanks!
> 
> In my case I think I need more functionality for fault tolerance, so will look at KafkaConsumer.  
> 
> Thanks,
> John 
> 
> -----Original Message-----
> From: Eno Thereska [mailto:eno.thereska@gmail.com] 
> Sent: Friday, November 18, 2016 9:02 AM
> To: users@kafka.apache.org
> Subject: Re: Change current offset with KStream
> 
> Hi John,
> 
> Currently you can only change the following global configuration by using "earliest" or "latest", as shown in the code snippet below. As the Javadoc mentions: "What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted)":
> 
> ...
> props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); ...
> return new KafkaStreams(builder, props)
> 
> 
> 
> In addition, there is a tool to reset the offsets of all topics to the beginning. This is useful for reprocessing: https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/
> 
> However, there is no option currently for resetting the offset to an arbitrary offset.
> 
> Thanks
> Eno
> 
> 
>> On 18 Nov 2016, at 14:50, John Hayles <jh...@etcc.com> wrote:
>> 
>> Is there offset controls when using KStream?   Before starting stream, can the current offset be changed to current offset - 5000.
>> 
>> I see KafkaConsumer has seek method, but I don't see how this helps when using KStream.  Could KafkaConsumer be used to change offset on a topic, then use KStream to process it?
>> 
>> Thanks for feedback...
>> 
>> Regards,
>> John
>> 
>> 
>> 
>

RE: Change current offset with KStream

Posted by John Hayles <jh...@etcc.com>.

Eno, thanks!

In my case I think I need more functionality for fault tolerance, so will look at KafkaConsumer.  

Thanks,
John 

-----Original Message-----
From: Eno Thereska [mailto:eno.thereska@gmail.com] 
Sent: Friday, November 18, 2016 9:02 AM
To: users@kafka.apache.org
Subject: Re: Change current offset with KStream

Hi John,

Currently you can only change the following global configuration by using "earliest" or "latest", as shown in the code snippet below. As the Javadoc mentions: "What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted)":

...
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); ...
return new KafkaStreams(builder, props)

In addition, there is a tool to reset the offsets of all topics to the beginning. This is useful for reprocessing: https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

However, there is no option currently for resetting the offset to an arbitrary offset.

Thanks
Eno

> On 18 Nov 2016, at 14:50, John Hayles <jh...@etcc.com> wrote:
> 
> Is there offset controls when using KStream?   Before starting stream, can the current offset be changed to current offset - 5000.
> 
> I see KafkaConsumer has seek method, but I don't see how this helps when using KStream.  Could KafkaConsumer be used to change offset on a topic, then use KStream to process it?
> 
> Thanks for feedback...
> 
> Regards,
> John
> 
> 
>

Re: Change current offset with KStream

Posted by Eno Thereska <en...@gmail.com>.

Hi John,

Currently you can only change the following global configuration by using "earliest" or "latest", as shown in the code snippet below. As the Javadoc mentions: "What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted)":

...
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
...
return new KafkaStreams(builder, props)

In addition, there is a tool to reset the offsets of all topics to the beginning. This is useful for reprocessing: https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

However, there is no option currently for resetting the offset to an arbitrary offset.

Thanks
Eno

> On 18 Nov 2016, at 14:50, John Hayles <jh...@etcc.com> wrote:
> 
> Is there offset controls when using KStream?   Before starting stream, can the current offset be changed to current offset - 5000.
> 
> I see KafkaConsumer has seek method, but I don't see how this helps when using KStream.  Could KafkaConsumer be used to change offset on a topic, then use KStream to process it?
> 
> Thanks for feedback...
> 
> Regards,
> John
> 
> 
>