You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Stephen Boesch <ja...@gmail.com> on 2014/05/15 06:25:34 UTC
Equivalent of collect() on DStream
Given that collect() does not exist on DStream apparently my mental model
of Streaming RDD (DStream) needs correction/refinement. So what is the
means to convert DStream data into a JVM in-memory representation. All of
the methods on DStream i.e. filter, map, transform, reduce, etc generate
other DStream's, and not an in memory data structure.
Re: Equivalent of collect() on DStream
Posted by Stephen Boesch <ja...@gmail.com>.
It seems the concept I had been missing is to invoke the DStream "foreach"
method. This method takes a function expecting an RDD and applies the
function to each RDD within the DStream.
2014-05-14 21:33 GMT-07:00 Stephen Boesch <ja...@gmail.com>:
> Looking further it appears the functionality I am seeking is in the
> following *private[spark] * class ForEachdStream
>
> (version 0.8.1 , yes we are presently using an older release..)
>
> private[streaming]
> class ForEachDStream[T: ClassManifest] (
> parent: DStream[T],
> * foreachFunc: (RDD[T], Time) => Unit*
> ) extends DStream[Unit](parent.ssc) {
>
> I would like to have access to this structure - particularly the ability
> to define an "foreachFunc" that gets applied to each RDD within the
> DStream. Is there a means to do so?
>
>
>
> 2014-05-14 21:25 GMT-07:00 Stephen Boesch <ja...@gmail.com>:
>
>
>> Given that collect() does not exist on DStream apparently my mental model
>> of Streaming RDD (DStream) needs correction/refinement. So what is the
>> means to convert DStream data into a JVM in-memory representation. All of
>> the methods on DStream i.e. filter, map, transform, reduce, etc generate
>> other DStream's, and not an in memory data structure.
>>
>>
>>
>>
>
Re: Equivalent of collect() on DStream
Posted by Sean Owen <so...@cloudera.com>.
Are you not just looking for the foreachRDD() method on DStream?
http://spark.apache.org/docs/0.9.1/streaming-programming-guide.html#output-operations
It gives you an RDD that you can do what you want with, including collect() it.
On Thu, May 15, 2014 at 5:33 AM, Stephen Boesch <ja...@gmail.com> wrote:
> Looking further it appears the functionality I am seeking is in the
> following private[spark] class ForEachdStream
>
> (version 0.8.1 , yes we are presently using an older release..)
>
> private[streaming]
> class ForEachDStream[T: ClassManifest] (
> parent: DStream[T],
> foreachFunc: (RDD[T], Time) => Unit
> ) extends DStream[Unit](parent.ssc) {
>
> I would like to have access to this structure - particularly the ability to
> define an "foreachFunc" that gets applied to each RDD within the DStream.
> Is there a means to do so?
>
>
>
> 2014-05-14 21:25 GMT-07:00 Stephen Boesch <ja...@gmail.com>:
>>
>>
>> Given that collect() does not exist on DStream apparently my mental model
>> of Streaming RDD (DStream) needs correction/refinement. So what is the
>> means to convert DStream data into a JVM in-memory representation. All of
>> the methods on DStream i.e. filter, map, transform, reduce, etc generate
>> other DStream's, and not an in memory data structure.
>>
>>
>>
>
Re: Equivalent of collect() on DStream
Posted by Tathagata Das <ta...@gmail.com>.
Doesnt DStream.foreach() suffice?
anyDStream.foreach { rdd =>
// do something with rdd
}
On Wed, May 14, 2014 at 9:33 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Looking further it appears the functionality I am seeking is in the
> following *private[spark] * class ForEachdStream
>
> (version 0.8.1 , yes we are presently using an older release..)
>
> private[streaming]
> class ForEachDStream[T: ClassManifest] (
> parent: DStream[T],
> * foreachFunc: (RDD[T], Time) => Unit*
> ) extends DStream[Unit](parent.ssc) {
>
> I would like to have access to this structure - particularly the ability
> to define an "foreachFunc" that gets applied to each RDD within the
> DStream. Is there a means to do so?
>
>
>
> 2014-05-14 21:25 GMT-07:00 Stephen Boesch <ja...@gmail.com>:
>
>>
>> Given that collect() does not exist on DStream apparently my mental model
>> of Streaming RDD (DStream) needs correction/refinement. So what is the
>> means to convert DStream data into a JVM in-memory representation. All of
>> the methods on DStream i.e. filter, map, transform, reduce, etc generate
>> other DStream's, and not an in memory data structure.
>>
>>
>>
>>
>
Re: Equivalent of collect() on DStream
Posted by Stephen Boesch <ja...@gmail.com>.
Looking further it appears the functionality I am seeking is in the
following *private[spark] * class ForEachdStream
(version 0.8.1 , yes we are presently using an older release..)
private[streaming]
class ForEachDStream[T: ClassManifest] (
parent: DStream[T],
* foreachFunc: (RDD[T], Time) => Unit*
) extends DStream[Unit](parent.ssc) {
I would like to have access to this structure - particularly the ability to
define an "foreachFunc" that gets applied to each RDD within the DStream.
Is there a means to do so?
2014-05-14 21:25 GMT-07:00 Stephen Boesch <ja...@gmail.com>:
>
> Given that collect() does not exist on DStream apparently my mental model
> of Streaming RDD (DStream) needs correction/refinement. So what is the
> means to convert DStream data into a JVM in-memory representation. All of
> the methods on DStream i.e. filter, map, transform, reduce, etc generate
> other DStream's, and not an in memory data structure.
>
>
>
>