You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Wolfinger, Fred" <fw...@cyberpointllc.com> on 2014/06/12 17:06:33 UTC

Access DStream content?

Good morning.

I have a question related to Spark Streaming. I have reduced some data down
to a simple count value (by window), and would like to take immediate action
on that value before storing in HDFS. I don't see any DStream member
functions that would allow me to access its contents. Is what I am trying to
do not in the spirit of Spark Streaming? If it's not, what would be the best
practice for doing so?

Thanks so much!

Fred

Re: Access DStream content?

Posted by Gianluca Privitera <gi...@studio.unibo.it>.

You can use ForeachRDD then access RDD data.
Hope this works for you.

Gianluca

On 12 Jun 2014, at 10:06, Wolfinger, Fred <fw...@cyberpointllc.com>> wrote:

Good morning.

I have a question related to Spark Streaming. I have reduced some data down to a simple count value (by window), and would like to take immediate action on that value before storing in HDFS. I don't see any DStream member functions that would allow me to access its contents. Is what I am trying to do not in the spirit of Spark Streaming? If it's not, what would be the best practice for doing so?

Thanks so much!

Fred