persistence and fault tolerance in Spark Streaming

[Diana Carroll <dc...@cloudera.com>]

As I understand it, Spark streaming automatically persists (replication =
2) windowed dstreams, but not regular dstreams.

So my question is, what happens in the case of worker node failure with a
non-windowed dstream whose data source is a network stream?

Say I'm getting a feed of log data, and one of my workers drops out halfway
through an operation.  What happens?  Non-streaming RDDs are resilient
because they can be recomputed from the source file, but in this case there
is no source file.  If the original data from the stream wasn't replicated,
does that mean it's just lost?  Will the task just fail?  Will the job fail?

Also, I tried testing on a cluster with two workers and a windowed dstream.
 The "Storage" tab in the app UI does show the data being persisted, but
only with single replication.  Is that because my cluster is too small?

[image: Inline image 1]

Thanks,
Diana

Re: persistence and fault tolerance in Spark Streaming

[Chris Fregly <ch...@fregly.com>]

@TD:  could you provide some guidance on this?  these same types of
questions come up a lot in the field - and i'd like to have a solid answer
for folks.

thanks so much!

-chris


On Wed, May 28, 2014 at 10:45 AM, Diana Carroll <dc...@cloudera.com>
wrote:

> As I understand it, Spark streaming automatically persists (replication =
> 2) windowed dstreams, but not regular dstreams.
>
> So my question is, what happens in the case of worker node failure with a
> non-windowed dstream whose data source is a network stream?
>
> Say I'm getting a feed of log data, and one of my workers drops out
> halfway through an operation.  What happens?  Non-streaming RDDs are
> resilient because they can be recomputed from the source file, but in this
> case there is no source file.  If the original data from the stream wasn't
> replicated, does that mean it's just lost?  Will the task just fail?  Will
> the job fail?
>
> Also, I tried testing on a cluster with two workers and a windowed
> dstream.  The "Storage" tab in the app UI does show the data being
> persisted, but only with single replication.  Is that because my cluster is
> too small?
>
> [image: Inline image 1]
>
> Thanks,
> Diana
>