You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Toth (Jira)" <ji...@apache.org> on 2020/03/04 09:07:00 UTC
[jira] [Commented] (SPARK-30563) Regressions in Join benchmarks
[ https://issues.apache.org/jira/browse/SPARK-30563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051014#comment-17051014 ]
Peter Toth commented on SPARK-30563:
------------------------------------
[~maxgekk], [~dongjoon], [~hyukjin.kwon] it looks like the change in the {{JoinBenchmark}} ([https://github.com/apache/spark/commit/f5118f81e395bde0cd8253dbef6a9e6455c3958a#diff-da1033f4d10b6046046202dd8f85e3f7L49-R49]) causes this regression. If we used {{df.groupBy().count().noop()}} and measure the same as previously there won't be any regression in this suite. Please see the results running the fixed benchmark on my machine: [https://github.com/peter-toth/spark/commit/207d15d1801cfcf9c40635a481d4aa7192911548]
This is because lots of rows are returned and we spend a lot of time in [this loop|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala#L438-L442] even if {{NoopWriter}} does nothing.
Another very minor improvement regarding `NoOp` datasource could be to turn off using commit coordinator in {{NoopBatchWrite}}.
Shall I open a PR with these changes (excluding the non-official benchmark result)?
> Regressions in Join benchmarks
> ------------------------------
>
> Key: SPARK-30563
> URL: https://issues.apache.org/jira/browse/SPARK-30563
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Minor
>
> Regenerated benchmark results in the https://github.com/apache/spark/pull/27078 shows many regressions in JoinBenchmark. The benchmarked queries slowed down by up to 3 times, see
> old results:
> https://github.com/apache/spark/pull/27078/files#diff-d5cbaab2b49ee9fddfa0e229de8f607dL10
> new results:
> https://github.com/apache/spark/pull/27078/files#diff-d5cbaab2b49ee9fddfa0e229de8f607dR10
> One of the difference in queries is using the `NoOp` datasource in new queries.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org