[jira] [Created] (SPARK-6922) RDD.cartesian is much slower than join

["David Tolnay (JIRA)" <ji...@apache.org>]

David Tolnay created SPARK-6922:
-----------------------------------

             Summary: RDD.cartesian is much slower than join
                 Key: SPARK-6922
                 URL: https://issues.apache.org/jira/browse/SPARK-6922
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.3.0
         Environment: Ubuntu 12.04.5, Spark 1.3.0 CDH 4 binary, standalone
            Reporter: David Tolnay


Cartesian takes 3 minutes to join 500x500 partitions. Join with a constant key takes only 4 seconds. Here is a deterministic repro:

{code}
val lst = List.fill(500)(Tuple1(0))
val df = sqlContext.createDataFrame(lst).repartition(500)
df.select($"_1".as("a")).saveAsParquetFile("file:///tmp/parquet/left")
df.select($"_1".as("b")).saveAsParquetFile("file:///tmp/parquet/right")
val left = sqlContext.parquetFile("file:///tmp/parquet/left")
val right = sqlContext.parquetFile("file:///tmp/parquet/right")

def time[A](f: => A) = {
  val start = System.nanoTime
  f
  (System.nanoTime-start)/1e6
}
time { left.rdd.cartesian(right.rdd).count } // 3 minutes
time { left.rdd.keyBy(_=>0).join(right.rdd.keyBy(_=>0)).count } // 4 seconds
{code}

Possibly related to SPARK-6307 in which cartesian causes the block manager to fetch the same blocks over and over.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org