You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Madabhattula Rajesh Kumar <mr...@gmail.com> on 2014/07/09 15:28:46 UTC

how to convert JavaDStream to JavaRDD

Hi Team,

Could you please help me to resolve below query.

My use case is :

I'm using JavaStreamingContext to read text files from Hadoop - HDFS
directory

JavaDStream<String> lines_2 =
ssc.textFileStream("hdfs://localhost:9000/user/rajesh/EventsDirectory/");

How to convert JavaDStream<String> result to JavaRDD<String>? if we can
convert. I can use collect() method on JavaRDD and process my textfile.

I'm not able to find collect method on JavaRDD<String>.

Thank you very much in advance.

Regards,
Rajesh

Re: how to convert JavaDStream to JavaRDD

Posted by Laeeq Ahmed <la...@yahoo.com>.

Hi,

First use foreachrdd and then use collect as


DStream.foreachRDD(rdd => {
       rdd.collect.foreach({

Also its better to use scala. Less verbose. 


Regards,
Laeeq



On Wednesday, July 9, 2014 3:29 PM, Madabhattula Rajesh Kumar <mr...@gmail.com> wrote:
 


Hi Team,

Could you please help me to resolve below query.

My use case is :

I'm using JavaStreamingContext to read text files from Hadoop - HDFS directory

JavaDStream<String> lines_2 = ssc.textFileStream("hdfs://localhost:9000/user/rajesh/EventsDirectory/");

How to convert JavaDStream<String> result to JavaRDD<String>? 
if we can convert. I can use collect() method on JavaRDD and process my 
textfile.

I'm not able to find collect method on JavaRDD<String>.

Thank you very much in advance.

Regards,
Rajesh