[jira] [Commented] (SPARK-13307) TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1

["Davies Liu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-13307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174473#comment-15174473 ] 

Davies Liu commented on SPARK-13307:
------------------------------------

In the plan, I saw that the column pruning does not work well, it's fixed recently in master. I ran it locally with scale factor 10, it took 35 seconds (with 1 CPUs), all joins are BroadcastHashJoin.

Could you checkout master, it should be faster than 1.4

It's true that SortMergeJoin could be slower than ShuffleHashJoin, we may revisit that later.

> TPCDS query 66 degraded by 30% in 1.6.0 compared to 1.4.1
> ---------------------------------------------------------
>
>                 Key: SPARK-13307
>                 URL: https://issues.apache.org/jira/browse/SPARK-13307
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: JESSE CHEN
>
> Majority of the TPCDS queries ran faster in 1.6.0 than in 1.4.1, average about 9% faster. There are a few degraded, and one that is definitely not within error margin is query 66.
> Query 66 in 1.4.1: 699 seconds
> Query 66 in 1.6.0: 918 seconds
> 30% worse.
> Collected the physical plans from both versions - drastic difference maybe partially from using Tungsten in 1.6, but anything else at play here?
> Please see plans here:
> https://ibm.box.com/spark-sql-q66-debug-160plan
> https://ibm.box.com/spark-sql-q66-debug-141plan



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org