You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Deep Pradhan <pr...@gmail.com> on 2014/08/26 06:55:40 UTC

Pair RDD

Hi,
I have an input file of a graph in the format <source_node   dest_node>
When I use sc.textFile, it will change the entire text file into an RDD.
How can I transform the file into key, value pair and then eventually into
paired RDDs.
Thank You

Re: Pair RDD

Posted by Yanbo Liang <ya...@gmail.com>.
val node = textFile.map(line => {
      val fileds = line.split("\\s+")
      (fileds(1),fileds(2))
    })

then you can manipulate node RDD with PairRDD function.


2014-08-26 12:55 GMT+08:00 Deep Pradhan <pr...@gmail.com>:

> Hi,
> I have an input file of a graph in the format <source_node   dest_node>
> When I use sc.textFile, it will change the entire text file into an RDD.
> How can I transform the file into key, value pair and then eventually into
> paired RDDs.
> Thank You
>