[jira] [Resolved] (SPARK-6922) RDD.cartesian is much slower than join

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-6922.
------------------------------
    Resolution: Duplicate

Yes, a likely subset of SPARK-6307. I'm closing this since I don't see it's likely that there will be separate activity here.

> RDD.cartesian is much slower than join
> --------------------------------------
>
>                 Key: SPARK-6922
>                 URL: https://issues.apache.org/jira/browse/SPARK-6922
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.3.0
>         Environment: Ubuntu 12.04.5, Spark 1.3.0 CDH 4 binary, standalone
>            Reporter: David Tolnay
>
> Cartesian takes 3 minutes to join 500x500 partitions. Join with a constant key takes only 4 seconds. Here is a deterministic repro:
> {code}
> val lst = List.fill(500)(Tuple1(0))
> val df = sqlContext.createDataFrame(lst).repartition(500)
> df.select($"_1".as("a")).saveAsParquetFile("file:///tmp/parquet/left")
> df.select($"_1".as("b")).saveAsParquetFile("file:///tmp/parquet/right")
> val left = sqlContext.parquetFile("file:///tmp/parquet/left")
> val right = sqlContext.parquetFile("file:///tmp/parquet/right")
> def time[A](f: => A) = {
>   val start = System.nanoTime
>   f
>   (System.nanoTime-start)/1e6
> }
> time { left.rdd.cartesian(right.rdd).count } // 3 minutes
> time { left.rdd.keyBy(_=>0).join(right.rdd.keyBy(_=>0)).count } // 4 seconds
> {code}
> Possibly related to SPARK-6307 in which cartesian causes the block manager to fetch the same blocks over and over.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org