You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Bryan Jeffrey <br...@gmail.com> on 2015/09/08 15:46:40 UTC
Getting Started with Spark
Hello. We're getting started with Spark Streaming. We're working to build
some unit/acceptance testing around functions that consume DStreams. The
current method for creating DStreams is to populate the data by creating an
InputDStream:
val input = Array(TestDataFactory.CreateEvent(123 notFoundData))
val queue =
scala.collection.mutable.Queue(ssc.sparkContext.parallelize(input))
val events: InputDStream[MyEvent] = ssc.queueStream(queue)
The 'events' InputDStream can then be fed into functions. However, the
stream does not allow checkpointing. This means that we're unable to use
this to feed methods/classes that execute stateful actions like
'updateStateByKey'.
Does anyone have a simple, contained method to create DStreams that allow
for checkpointing? I looked at the Spark unit test framework, but that
seems to require access to a bunch of spark internals (requiring that
you're within the spark package, etc.)
Re: Getting Started with Spark
Posted by Tathagata Das <td...@databricks.com>.
This is a known issue introduced in Spark 1.4.1 and 1.5.0 and will be fixed
in Spark 1.5.1. In the mean time, you could prototype in Spark 1.4.0 and
wait for Spark 1.5.1/1.4.2 to come out. You could also download the source
code and compile the Spark master branch.
https://issues.apache.org/jira/browse/SPARK-10071
On Tue, Sep 8, 2015 at 6:46 AM, Bryan Jeffrey <br...@gmail.com>
wrote:
> Hello. We're getting started with Spark Streaming. We're working to build
> some unit/acceptance testing around functions that consume DStreams. The
> current method for creating DStreams is to populate the data by creating an
> InputDStream:
>
> val input = Array(TestDataFactory.CreateEvent(123 notFoundData))
> val queue =
> scala.collection.mutable.Queue(ssc.sparkContext.parallelize(input))
> val events: InputDStream[MyEvent] = ssc.queueStream(queue)
>
> The 'events' InputDStream can then be fed into functions. However, the
> stream does not allow checkpointing. This means that we're unable to use
> this to feed methods/classes that execute stateful actions like
> 'updateStateByKey'.
>
> Does anyone have a simple, contained method to create DStreams that allow
> for checkpointing? I looked at the Spark unit test framework, but that
> seems to require access to a bunch of spark internals (requiring that
> you're within the spark package, etc.)
>
>