You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by SK <sk...@gmail.com> on 2014/06/14 03:08:57 UTC

convert List to RDD

Hi,

I have a List[ (String, Int, Int) ] that I would liek to convert to an RDD.
I tried to use sc.parallelize and sc.makeRDD, but in each case the original
order of items in the List gets modified. Is there a simple way to convert a
List to RDD without using SparkContext? 

thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: convert List to RDD

Posted by Zongheng Yang <zo...@gmail.com>.

Sorry I wasn't being clear. The idea off the top of my head was that
you could append an original position index to each element (using the
line above), and modified what ever processing functions you have in
mind to make them aware of these indices. And I think you are right
that RDD collections are unordered by default.

On Fri, Jun 13, 2014 at 6:33 PM, SK <sk...@gmail.com> wrote:
> Thanks. But that did not work.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606p7609.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: convert List to RDD

Posted by SK <sk...@gmail.com>.

Thanks. But that did not work.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606p7609.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: convert List to RDD

Posted by Zongheng Yang <zo...@gmail.com>.

I may be wrong, but I think RDDs must be created inside a
SparkContext. To somehow preserve the order of the list, perhaps you
could try something like:

sc.parallelize((1 to xs.size).zip(xs))

On Fri, Jun 13, 2014 at 6:08 PM, SK <sk...@gmail.com> wrote:
> Hi,
>
> I have a List[ (String, Int, Int) ] that I would liek to convert to an RDD.
> I tried to use sc.parallelize and sc.makeRDD, but in each case the original
> order of items in the List gets modified. Is there a simple way to convert a
> List to RDD without using SparkContext?
>
> thanks
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.