Specfic offset processing in Kafka Streams

[Sameer Kumar <sa...@gmail.com>]

If I want to reprocess only a section(i.e. from-to offsets) of the topic
through Kafka Streams, what do you think could be way to achieve it.

I want the data to be stored in same state stores, this I think would be a
common scenario in a typical production environment.

-Sameer.

Re: Specfic offset processing in Kafka Streams

["Matthias J. Sax" <ma...@confluent.io>]

I don't think that a "from-to" pattern would a common scenario -- Kafka
is about stream processing, not batch processing.

I guess you can to a hand crafted solution though.

1) use bin/kafka-consumer-groups.sh to seek to the corresponding start
offset for the group.id/application.id of your Streams app

2) use a custom `transform()` in you Streams app as first operator to
receive the offset of the currently processed message via the provided
context object

2a) if offset exceeds your end offset, discard the message and set a
flag to stop processing

3) in you main application, use a while loop to regularly check if the
stop-flag was set to stop your application



-Matthias

On 11/22/17 4:21 AM, Sameer Kumar wrote:
> If I want to reprocess only a section(i.e. from-to offsets) of the topic
> through Kafka Streams, what do you think could be way to achieve it.
> 
> I want the data to be stored in same state stores, this I think would be a
> common scenario in a typical production environment.
> 
> -Sameer.
>