reduce only removes duplicates, cannot be arbitrary function

[GlennStrycker <gl...@gmail.com>]

I am attempting to write a mapreduce job on a graph object to take an edge
list and return a new edge list.  Unfortunately I find that the current
function is

def reduce(f: (T, T) => T): T

not

def reduce(f: (T1, T2) => T3): T


I see this because the following 2 commands give different results for the
final number, which should be the same (tempMappedRDD is a MappedRDD of the
form (Edge,1), and I found the the A and B here are (1,4) and (7,3) )

tempMappedRDD.reduce( (A,B) => (Edge(A._1.srcId, A._1.dstId,
A._1.dstId.toInt), 1) )  // (Edge(1,4,4),1)
tempMappedRDD.reduce( (A,B) => (Edge(A._1.srcId, B._1.dstId,
A._1.dstId.toInt), 1) )  // (Edge(1,3,3),1)

why is the 3rd digit above a '3' in the second line, and not a '4'?  Does it
have something to do with toInt?

the really weird thing is that it is only for A, since the following
commands work correctly:

tempMappedRDD.reduce( (A,B) => (Edge(B._1.srcId, B._1.dstId,
B._1.dstId.toInt), 1) )  // (Edge(7,3,3),1)
tempMappedRDD.reduce( (A,B) => (Edge(B._1.srcId, A._1.dstId,
B._1.dstId.toInt), 1) )  // (Edge(7,4,3),1)




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/reduce-only-removes-duplicates-cannot-be-arbitrary-function-tp6606.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.