pause and resume streaming app

[Shushant Arora <sh...@gmail.com>]

Is it possible to pause and resume a streaming app?

I have a streaming app which reads events from kafka and post to some
external source. I want to pause the app when external source is down and
 resume it automatically when it comes back ?

Is it possible to pause the app and is it possible to pause only processing
part let receiver work and brong data in memory+disk, once external source
is up process whole data in one go.

Thanks
Shushant

Re: pause and resume streaming app

[Tathagata Das <td...@databricks.com>]

Currently the only way to pause it is to stop it. The way I would do this
is use the Direct Kafka API to access the Kafka offsets, and save them to a
data store as batches finish. If you see a batch job failing because
downstream is down, stop the context. When it comes back up, start a new
streaming context using the saved offsets as starting points.

TD

On Wed, Jul 8, 2015 at 11:52 AM, Shushant Arora <sh...@gmail.com>
wrote:

> Is it possible to pause and resume a streaming app?
>
> I have a streaming app which reads events from kafka and post to some
> external source. I want to pause the app when external source is down and
>  resume it automatically when it comes back ?
>
> Is it possible to pause the app and is it possible to pause only
> processing part let receiver work and brong data in memory+disk, once
> external source is up process whole data in one go.
>
> Thanks
> Shushant
>