You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/15 12:08:16 UTC
[GitHub] [spark] wangyum commented on pull request #35806: [WIP][SPARK-38505][SQL] Make partial aggregation adaptive
wangyum commented on pull request #35806:
URL: https://github.com/apache/spark/pull/35806#issuecomment-1067911655
```scala
import org.apache.spark.benchmark.Benchmark
val numRows = 1024 * 1024 * 50
spark.sql(s"CREATE TABLE t1 using parquet AS SELECT id AS a, id % ${numRows / 10000} AS b, id % ${numRows / 10000} AS c, id AS d FROM range(1, ${numRows}L, 1, 10)")
val benchmark = new Benchmark("Benchmark WholeStageCodegenExec", numRows, minNumIters = 2)
Seq(0, 10000).foreach { threshold =>
benchmark.addCase(s"SELECT a, c, sum(b), sum(d) FROM t1 where a > 100 group by a, c and partialAggThreshold=$threshold") { _ =>
withSQLConf("spark.sql.aggregate.adaptivePartialAggregationThreshold" -> threshold.toString) {
spark.sql("SELECT a, c, sum(b), sum(d) FROM t1 where a > 100 group by a, c").write.format("noop").mode("Overwrite").save()
}
}
}
benchmark.run()
```
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_281-b09 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark WholeStageCodegenExec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SELECT a, c, sum(b), sum(d) FROM t1 where a > 100 group by a, c and partialAggThreshold=0 56519 57012 697 0.9 1078.0 1.0X
SELECT a, c, sum(b), sum(d) FROM t1 where a > 100 group by a, c and partialAggThreshold=10000 41908 42369 653 1.3 799.3 1.3X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org