Recovering from receiver failure in Spark streaming

[Russell Cardullo <ru...@gmail.com>]

Hello All,

When running Spark streaming 24/7, we will occasionally get errors in our receivers which causes the Spark driver program to de-register that receiver, yet continue running.

For instance, in the example 'spark.streaming.examples.NetworkWordCount' program, if the source for the network receiver is temporarily unavailable the driver will de-register the receiver but the driver will continue to run indefinitely.  If the source later becomes available again the receiver will not re-register.

Is there some easy means to catch this and re-register the receiver if it becomes available later?

Alternatively is there some way to catch the case where the receiver is lost and call stop() on the streaming context?

Cheers,
--Russell

Re: Recovering from receiver failure in Spark streaming

[Vadim Chekan <ko...@gmail.com>]

Perhaps receiver needs to be failure-resistant itself. Spark does not know
what your application requirements are and what should be done in case of
network stream error. I would implement akka actor which switches into
connection retry mode upon network error and switches back into receiving
mode when connection restored.

Vadim.


On Fri, Sep 20, 2013 at 9:07 AM, Russell Cardullo <russellcardullo@gmail.com
> wrote:

> Hello All,
>
> When running Spark streaming 24/7, we will occasionally get errors in our
> receivers which causes the Spark driver program to de-register that
> receiver, yet continue running.
>
> For instance, in the example 'spark.streaming.examples.NetworkWordCount'
> program, if the source for the network receiver is temporarily unavailable
> the driver will de-register the receiver but the driver will continue to
> run indefinitely.  If the source later becomes available again the receiver
> will not re-register.
>
> Is there some easy means to catch this and re-register the receiver if it
> becomes available later?
>
> Alternatively is there some way to catch the case where the receiver is
> lost and call stop() on the streaming context?
>
> Cheers,
> --Russell




-- 
>From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is
explicitly specified