Need help with coalesce

[Madhura <da...@gmail.com>]

Hi,

I have a file called out with random numbers where each number in on one
line in the file. I am loading the complete file into a RDD and I want to
create partitions with the help of coalesce function.
This is my code snippet.
import scala.math.Ordered
import org.apache.spark.rdd.CoalescedRDD
import org.apache.spark.api.java.JavaPairRDD
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.api.java.function.Function
import org.apache.spark.api.java.function.PairFunction

val dRDD = sc.textFile("hdfs://master:54310/out",10)

      val keyval=dRDD.coalesce(100,true).mapPartitionsWithIndex{(ind,iter)
=> iter.map(x => process(ind,x.trim().split(' ').map(_.toDouble),q,m,r))}

However, I am getting this error. I tried looking at various other links but
I always got this error.
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.rdd.RDD.coalesce(IZ)Lorg/apache/spark/rdd/RDD;
        at SimpleApp$.main(SimpleApp.scala:432)
        at SimpleApp.main(SimpleApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

A follow up, is there any way I can access the elements in a RDD with the
help of an index and create partitions? Like for example I have a RDD with
values like 1,2,3,...,100. I would like to create partitions which look like
this:
part1: 1,2,3..,10
part2: 8,9,10,...,20
part3: 18,19,20,...,30 and so on...

Thanks and regards,
Madhura




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Need-help-with-coalesce-tp10243.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.