[jira] [Created] (SPARK-3364) zip equal-length but unequally-partition

["Kevin Jung (JIRA)" <ji...@apache.org>]

Kevin Jung created SPARK-3364:
---------------------------------

             Summary: zip equal-length but unequally-partition
                 Key: SPARK-3364
                 URL: https://issues.apache.org/jira/browse/SPARK-3364
             Project: Spark
          Issue Type: Bug
            Reporter: Kevin Jung
            Priority: Critical


ZippedRDD losts some elements after zipping RDDs with equal numbers of partitions but unequal numbers of elements in their each partitions.
This can happen when a user create RDD by sc.textFile(path,partitionNumbers) with physically unbalanced HDFS file.

var x = sc.parallelize(1 to 9,3)
var y = sc.parallelize(Array(1,1,1,1,1,2,2,3,3),3).keyBy(i=>i)
var z = y.partitionBy(new RangePartitioner(3,y))

expected
x.zip(y).count()
9
x.zip(y).collect()
Array[(Int, (Int, Int))] = Array((1,(1,1)), (2,(1,1)), (3,(1,1)), (4,(1,1)), (5,(1,1)), (6,(2,2)), (7,(2,2)), (8,(3,3)), (9,(3,3)))

unexpected
x.zip(z).count()
7
x.zip(z).collect()
Array[(Int, (Int, Int))] = Array((1,(1,1)), (2,(1,1)), (3,(1,1)), (4,(2,2)), (5,(2,2)), (7,(3,3)), (8,(3,3)))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org