Exiting a streams app at end of stream?

[Thomas Becker <to...@Tivo.com>]

We have had a number of situations where we need to migrate data in a
Kafka topic to a new topic that is keyed differently. Stream processing
is a good fit for this use-case with one exception: there is no easy
way to know when your "migration job" is finished. Has any thought been
given to adding an "end of stream" notion to Kafka Streams, and a
corresponding mode to exit the application when all input streams have
hit it?

--


    Tommy Becker

    Senior Software Engineer

    O +1 919.460.4747

    tivo.com


________________________________

This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.

Re: Exiting a streams app at end of stream?

["Matthias J. Sax" <ma...@confluent.io>]

Hi,

there is KIP for this idea, but nobody or working on it actively atm:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-95%3A+Incremental+Batch+Processing+for+Kafka+Streams


Right now, you can only check the committed offsets and terminate the
app accordingly.


-Matthias

On 5/3/17 5:32 AM, Thomas Becker wrote:
> We have had a number of situations where we need to migrate data in a
> Kafka topic to a new topic that is keyed differently. Stream processing
> is a good fit for this use-case with one exception: there is no easy
> way to know when your "migration job" is finished. Has any thought been
> given to adding an "end of stream" notion to Kafka Streams, and a
> corresponding mode to exit the application when all input streams have
> hit it?
> 
> --
> 
> 
>     Tommy Becker
> 
>     Senior Software Engineer
> 
>     O +1 919.460.4747
> 
>     tivo.com
> 
> 
> ________________________________
> 
> This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
>