[GitHub] [spark] wangyum edited a comment on pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

[GitBox <gi...@apache.org>]

wangyum edited a comment on pull request #31691:
URL: https://github.com/apache/spark/pull/31691#issuecomment-792288495


   Benchmark there is a window function with a different orderSpec:
   ```scala
   import org.apache.spark.benchmark.Benchmark
   val numRows = 50000000L
   val benchmark = new Benchmark("Benchmark bloom filter write", numRows, minNumIters = 3)
   spark.sql(s"CREATE TABLE T1 USING PARQUET AS SELECT ID AS A, ID AS B, ID AS C FROM range(${numRows})")
   Seq("org.apache.spark.sql.catalyst.optimizer.LimitPushDown", "").foreach { rule =>
     benchmark.addCase(s"Benchmark excludedRules: ${rule}") { _ =>
       withSQLConf("spark.sql.optimizer.excludedRules" -> rule) {
         spark.sql("SELECT *, ROW_NUMBER() OVER(ORDER BY a) AS rn, RANK() OVER(ORDER BY b DESC) AS rk FROM t1 LIMIT 5").show()
       }
     }
   }
   benchmark.run()
   ```
   Benchmark results:
   ```
   Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.15.7
   Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
   Benchmark bloom filter write:                                                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   --------------------------------------------------------------------------------------------------------------------------------------------------------------
   Benchmark excludedRules: org.apache.spark.sql.catalyst.optimizer.LimitPushDown          98110         107339        1965          0.5        1962.2       1.0X
   Benchmark excludedRules:                                                                53945          65014        1566          0.9        1078.9       1.8X
   
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org