You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Nicholas Pritchard <ni...@falkonry.com> on 2013/10/02 00:37:52 UTC

Join on DStream and RDD

Hi,

I have DStream and RDD, both of type (key, value), that I would like to
perform a join operation on (and get a DStream result). It doesn't seem
possible to do this currently, but I feel like its a logical operation
since DStream are essentially made up of RDDs. Are there any plans to
include this functionality or any possible workarounds? Maybe creating a
DStream object that generates the same RDD each timestep...

I appreciate any suggestions.

Thanks,
Nick

Re: Join on DStream and RDD

Posted by Nicholas Pritchard <ni...@falkonry.com>.

Nevermind, now I see the "transform" method so I can do:

myDStream.transform(rdd => rdd.join(myRDD))


On Tue, Oct 1, 2013 at 3:37 PM, Nicholas Pritchard <
nicholas.pritchard@falkonry.com> wrote:

> Hi,
>
> I have DStream and RDD, both of type (key, value), that I would like to
> perform a join operation on (and get a DStream result). It doesn't seem
> possible to do this currently, but I feel like its a logical operation
> since DStream are essentially made up of RDDs. Are there any plans to
> include this functionality or any possible workarounds? Maybe creating a
> DStream object that generates the same RDD each timestep...
>
> I appreciate any suggestions.
>
> Thanks,
> Nick
>