RangePartitioning

[ddav <da...@gmail.com>]

Hi,

I have the following pair RDD created in java. 

JavaPairRDD<Integer, ProgramDataRef> progRef =
sc.textFile(programReferenceDataFile, 12).filter(
				(String s) -> !s.startsWith("#")).mapToPair(
				(String s) -> {
						ProgramDataRef ref = new ProgramDataRef(s);
						return new Tuple2<Integer, ProgramDataRef>(ref.startTime, ref);
					}
				);

I need to partition this RDD using the provided RangePartitioner in the
framework. However, I am unable to determine what the input parameters are
in order to create the partitioner. 

new RangePartitioner(12, ?, true, ?, ?);

I am specifically looking for info on the 2nd parameter - RDD<? extends
scala.Product2<K,V>> rdd - which doesn't match my pair RDD. How do I create
an RDD of this type from the RDD I have (or the input text file). 
Also, the 4th and 5th parameter - scala.math.Ordering<K> evidence$1,
scala.reflect.ClassTag<K> evidence$2. How do I create these classes based on
my key type of Integer. 

Finally once I have my RDD partitioned I want to call
OrderedRDDFunctions.filterByRange(...). I believe once I have the answers to
the above question I will have the input parameters in order to create an
instance of OrderedRDDFunctions.   

Thanks,
Dave.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/RangePartitioning-tp26010.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org