how to split one big RDD (about 100G) into several small ones?

[Joe L <se...@yahoo.com>]

I want to split a single big rdd into small rdds without reading too much
from disk (hdfs). what is the best way to do that?

this is my current code: 
 subclass_pairs    = schema_triples.filter(lambda (s, p, o): p ==
PROPERTIES['subClassOf']).map(lambda (s, p, o): (s, o))
    subproperty_pairs = schema_triples.filter(lambda (s, p, o): p ==
PROPERTIES['subPropertyOf']).map(lambda (s, p, o): (s, o)).cache()
    domain_pairs      = schema_triples.filter(lambda (s, p, o): p ==
PROPERTIES['domain']).map(lambda (s, p, o): (s, o))
    range_pairs       = schema_triples.filter(lambda (s, p, o): p ==
PROPERTIES['range']).map(lambda (s, p, o): (s, o))
    total_triples     = instance_triples.union(schema_triples)
    type_pairs        = total_triples.filter(lambda (s, p, o):  p ==
PROPERTIES['type']).map(lambda (s, p, o): (s, o)).distinct().cache()
    triples           = total_triples.filter(lambda (s, p, o): 
isUserDefined(p)).map(lambda (s, p, o): (s, p, o)).distinct().cache()  



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-split-one-big-RDD-about-100G-into-several-small-ones-tp4450.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.