You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anfernee Xu <an...@gmail.com> on 2015/10/24 01:13:10 UTC

Spark Streaming: how to StreamingContext.queueStream

Hi,

Here's my situation, I have some kind of offline dataset, but I want to
form a virtual data stream feeding to Spark Streaming, my code looks like
this


   // sort offline data by time
 1)  JavaRDD sortedByTime = offlineDataRDD.sortBy( );

   // compute a list of JavaRDD,  each element JavaRDD is hosting the data
in the same time
   // bucket.
  2) List<JavaRDD> virtualStreamRdd = ?

    Queue<JavaRDD<Row>> queue = Queues.newLinkedBlockingQueue();
    queue.addAll(virtualStreamRdd);

    /*
     * Create DStream from the queue
     */

    3) final JavaDStream<Row> rowDStream =
streamingContext.queueStream(queue);


Currently I'm stucking in 2), any suggestion is appreciated.

Thanks

-- 
--Anfernee

Re: Spark Streaming: how to StreamingContext.queueStream

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can do something like this:


val rddQueue = scala.collection.mutable.Queue(rdd1,rdd2,rdd3)

val qDstream = ssc.queueStream(rddQueue)




Thanks
Best Regards

On Sat, Oct 24, 2015 at 4:43 AM, Anfernee Xu <an...@gmail.com> wrote:

> Hi,
>
> Here's my situation, I have some kind of offline dataset, but I want to
> form a virtual data stream feeding to Spark Streaming, my code looks like
> this
>
>
>    // sort offline data by time
>  1)  JavaRDD sortedByTime = offlineDataRDD.sortBy( );
>
>    // compute a list of JavaRDD,  each element JavaRDD is hosting the data
> in the same time
>    // bucket.
>   2) List<JavaRDD> virtualStreamRdd = ?
>
>     Queue<JavaRDD<Row>> queue = Queues.newLinkedBlockingQueue();
>     queue.addAll(virtualStreamRdd);
>
>     /*
>      * Create DStream from the queue
>      */
>
>     3) final JavaDStream<Row> rowDStream =
> streamingContext.queueStream(queue);
>
>
> Currently I'm stucking in 2), any suggestion is appreciated.
>
> Thanks
>
> --
> --Anfernee
>