[jira] [Created] (SPARK-17817) PySpark RDD Repartitioning Results in Highly Skewed Partition Sizes

["Mike Dusenberry (JIRA)" <ji...@apache.org>]

Mike Dusenberry created SPARK-17817:
---------------------------------------

             Summary: PySpark RDD Repartitioning Results in Highly Skewed Partition Sizes
                 Key: SPARK-17817
                 URL: https://issues.apache.org/jira/browse/SPARK-17817
             Project: Spark
          Issue Type: Bug
    Affects Versions: 2.0.1, 2.0.0, 1.6.2, 1.6.1
            Reporter: Mike Dusenberry


Calling {{repartition}} on a PySpark RDD to increase the number of partitions results in highly skewed partition sizes, with most having 0 rows.  The {{repartition}} method should evenly spread out the rows across the partitions, and this behavior is correctly seen on the Scala side.

Please reference the following code for a reproducible example of this issue:

{code}
# Python
num_partitions = 20000
a = sc.parallelize(range(int(1e6)), 2)  # start with 2 even partitions
l = a.repartition(num_partitions).glom().map(len).collect()  # get length of each partition
min(l), max(l), sum(l)/len(l), len(l)  # skewed!

# Scala
val numPartitions = 20000
val a = sc.parallelize(0 until 1e6.toInt, 2)  # start with 2 even partitions
val l = a.repartition(numPartitions).glom().map(_.length).collect()  # get length of each partition
print(l.min, l.max, l.sum/l.length, l.length)  # even!
{code}

The issue here is that highly skewed partitions can result in severe memory pressure in subsequent steps of a processing pipeline, resulting in OOM errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org