You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anfernee Xu <an...@gmail.com> on 2015/10/24 01:13:10 UTC
Spark Streaming: how to StreamingContext.queueStream
Hi,
Here's my situation, I have some kind of offline dataset, but I want to
form a virtual data stream feeding to Spark Streaming, my code looks like
this
// sort offline data by time
1) JavaRDD sortedByTime = offlineDataRDD.sortBy( );
// compute a list of JavaRDD, each element JavaRDD is hosting the data
in the same time
// bucket.
2) List<JavaRDD> virtualStreamRdd = ?
Queue<JavaRDD<Row>> queue = Queues.newLinkedBlockingQueue();
queue.addAll(virtualStreamRdd);
/*
* Create DStream from the queue
*/
3) final JavaDStream<Row> rowDStream =
streamingContext.queueStream(queue);
Currently I'm stucking in 2), any suggestion is appreciated.
Thanks
--
--Anfernee
Re: Spark Streaming: how to StreamingContext.queueStream
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can do something like this:
val rddQueue = scala.collection.mutable.Queue(rdd1,rdd2,rdd3)
val qDstream = ssc.queueStream(rddQueue)
Thanks
Best Regards
On Sat, Oct 24, 2015 at 4:43 AM, Anfernee Xu <an...@gmail.com> wrote:
> Hi,
>
> Here's my situation, I have some kind of offline dataset, but I want to
> form a virtual data stream feeding to Spark Streaming, my code looks like
> this
>
>
> // sort offline data by time
> 1) JavaRDD sortedByTime = offlineDataRDD.sortBy( );
>
> // compute a list of JavaRDD, each element JavaRDD is hosting the data
> in the same time
> // bucket.
> 2) List<JavaRDD> virtualStreamRdd = ?
>
> Queue<JavaRDD<Row>> queue = Queues.newLinkedBlockingQueue();
> queue.addAll(virtualStreamRdd);
>
> /*
> * Create DStream from the queue
> */
>
> 3) final JavaDStream<Row> rowDStream =
> streamingContext.queueStream(queue);
>
>
> Currently I'm stucking in 2), any suggestion is appreciated.
>
> Thanks
>
> --
> --Anfernee
>