You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/04/01 06:55:46 UTC
[GitHub] [spark] cloud-fan commented on a change in pull request #28076:
[SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource
cloud-fan commented on a change in pull request #28076: [SPARK-31311][SQL][TESTS] Benchmark date-time rebasing in ORC datasource
URL: https://github.com/apache/spark/pull/28076#discussion_r401393426
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -6,48 +6,89 @@ OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save dates to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 9304 9304 0 10.7 93.0 1.0X
-before 1582, noop 9187 9187 0 10.9 91.9 1.0X
-after 1582, rebase off 22054 22054 0 4.5 220.5 0.4X
-after 1582, rebase on 20361 20361 0 4.9 203.6 0.5X
-before 1582, rebase off 20286 20286 0 4.9 202.9 0.5X
-before 1582, rebase on 22230 22230 0 4.5 222.3 0.4X
+after 1582, noop 9299 9299 0 10.8 93.0 1.0X
+before 1582, noop 9220 9220 0 10.8 92.2 1.0X
+after 1582, rebase off 20390 20390 0 4.9 203.9 0.5X
+after 1582, rebase on 20378 20378 0 4.9 203.8 0.5X
+before 1582, rebase off 20069 20069 0 5.0 200.7 0.5X
+before 1582, rebase on 20637 20637 0 4.8 206.4 0.5X
OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load dates from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase off 12773 12866 129 7.8 127.7 1.0X
-after 1582, vec off, rebase on 13063 13086 39 7.7 130.6 1.0X
-after 1582, vec on, rebase off 3678 3719 61 27.2 36.8 3.5X
-after 1582, vec on, rebase on 5078 5121 52 19.7 50.8 2.5X
-before 1582, vec off, rebase off 12942 12972 42 7.7 129.4 1.0X
-before 1582, vec off, rebase on 13866 13904 58 7.2 138.7 0.9X
-before 1582, vec on, rebase off 3678 3711 43 27.2 36.8 3.5X
-before 1582, vec on, rebase on 5621 5657 44 17.8 56.2 2.3X
+after 1582, vec off, rebase off 12927 13017 78 7.7 129.3 1.0X
+after 1582, vec off, rebase on 13127 13176 50 7.6 131.3 1.0X
+after 1582, vec on, rebase off 3725 3779 91 26.8 37.3 3.5X
+after 1582, vec on, rebase on 5134 5221 99 19.5 51.3 2.5X
+before 1582, vec off, rebase off 13049 13061 16 7.7 130.5 1.0X
+before 1582, vec off, rebase on 13877 13916 51 7.2 138.8 0.9X
+before 1582, vec on, rebase off 3702 3736 56 27.0 37.0 3.5X
+before 1582, vec on, rebase on 5567 5637 78 18.0 55.7 2.3X
OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save timestamps to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 2983 2983 0 33.5 29.8 1.0X
-before 1582, noop 2979 2979 0 33.6 29.8 1.0X
-after 1582, rebase off 17452 17452 0 5.7 174.5 0.2X
-after 1582, rebase on 70193 70193 0 1.4 701.9 0.0X
-before 1582, rebase off 17784 17784 0 5.6 177.8 0.2X
-before 1582, rebase on 83498 83498 0 1.2 835.0 0.0X
+after 1582, noop 2988 2988 0 33.5 29.9 1.0X
+before 1582, noop 3000 3000 0 33.3 30.0 1.0X
+after 1582, rebase off 16163 16163 0 6.2 161.6 0.2X
+after 1582, rebase on 68399 68399 0 1.5 684.0 0.0X
+before 1582, rebase off 16921 16921 0 5.9 169.2 0.2X
+before 1582, rebase on 74425 74425 0 1.3 744.3 0.0X
OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load timestamps from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase off 15114 15151 32 6.6 151.1 1.0X
-after 1582, vec off, rebase on 45804 45912 126 2.2 458.0 0.3X
-after 1582, vec on, rebase off 4900 4947 56 20.4 49.0 3.1X
-after 1582, vec on, rebase on 34599 34650 45 2.9 346.0 0.4X
-before 1582, vec off, rebase off 15093 15174 70 6.6 150.9 1.0X
-before 1582, vec off, rebase on 47367 47472 121 2.1 473.7 0.3X
-before 1582, vec on, rebase off 4884 4952 80 20.5 48.8 3.1X
-before 1582, vec on, rebase on 35831 35883 59 2.8 358.3 0.4X
+after 1582, vec off, rebase off 15147 15258 97 6.6 151.5 1.0X
+after 1582, vec off, rebase on 45035 45101 60 2.2 450.3 0.3X
+after 1582, vec on, rebase off 4934 5012 100 20.3 49.3 3.1X
+after 1582, vec on, rebase on 34263 34360 88 2.9 342.6 0.4X
+before 1582, vec off, rebase off 15177 15220 37 6.6 151.8 1.0X
+before 1582, vec off, rebase on 46754 46761 12 2.1 467.5 0.3X
+before 1582, vec on, rebase off 4892 4956 61 20.4 48.9 3.1X
+before 1582, vec on, rebase on 35989 36014 22 2.8 359.9 0.4X
+
+
+================================================================================================
+Rebasing dates/timestamps in ORC datasource
+================================================================================================
+
+OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
+Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Save dates to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+after 1582, noop 9295 9295 0 10.8 93.0 1.0X
+before 1582, noop 9352 9352 0 10.7 93.5 1.0X
+after 1582 17112 17112 0 5.8 171.1 0.5X
+before 1582 17979 17979 0 5.6 179.8 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
+Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Load dates from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+after 1582, vec off 20874 20905 38 4.8 208.7 1.0X
+after 1582, vec on 3813 3844 28 26.2 38.1 5.5X
+before 1582, vec off 25912 25949 38 3.9 259.1 0.8X
+before 1582, vec on 4322 4343 19 23.1 43.2 4.8X
+
+OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
+Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Save timestamps to ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+after 1582, noop 3003 3003 0 33.3 30.0 1.0X
+before 1582, noop 3012 3012 0 33.2 30.1 1.0X
+after 1582 41031 41031 0 2.4 410.3 0.1X
+before 1582 44436 44436 0 2.3 444.4 0.1X
+
+OpenJDK 64-Bit Server VM 11.0.6+10-post-Ubuntu-1ubuntu118.04.1 on Linux 4.15.0-1063-aws
+Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Load timestamps from ORC: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
Review comment:
why the rebasing overhead in read is much smaller than write side?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org