You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/10/21 10:30:40 UTC
[GitHub] [spark] MaxGekk opened a new pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
MaxGekk opened a new pull request #30118:
URL: https://github.com/apache/spark/pull/30118
### What changes were proposed in this pull request?
1. Turn off/on the SQL config `spark.sql.legacy.parquet.int96RebaseModeInWrite` which was added by https://github.com/apache/spark/pull/30056 in `DateTimeRebaseBenchmark`. The parquet readers should infer correct rebasing mode automatically from metadata.
2. Regenerate benchmark results of `DateTimeRebaseBenchmark` in the environment:
| Item | Description |
| ---- | ----|
| Region | us-west-2 (Oregon) |
| Instance | r3.xlarge (spot instance) |
| AMI | ami-06f2f779464715dc5 (ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1) |
| Java | OpenJDK8/11 installed by`sudo add-apt-repository ppa:openjdk-r/ppa` & `sudo apt install openjdk-11-jdk`|
### Why are the changes needed?
To have up-to date info about INT96 performance which is the default type for Catalyst's timestamp type.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By updating benchmark results:
```
$ SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.DateTimeRebaseBenchmark"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30118:
URL: https://github.com/apache/spark/pull/30118#discussion_r509170757
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -2,153 +2,153 @@
Rebasing dates/timestamps in Parquet datasource
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 20023 20023 0 5.0 200.2 1.0X
-before 1582, noop 10729 10729 0 9.3 107.3 1.9X
-after 1582, rebase EXCEPTION 31834 31834 0 3.1 318.3 0.6X
-after 1582, rebase LEGACY 31997 31997 0 3.1 320.0 0.6X
-after 1582, rebase CORRECTED 31712 31712 0 3.2 317.1 0.6X
-before 1582, rebase LEGACY 23663 23663 0 4.2 236.6 0.8X
-before 1582, rebase CORRECTED 22749 22749 0 4.4 227.5 0.9X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, noop 21041 21041 0 4.8 210.4 1.0X
+before 1582, noop 11202 11202 0 8.9 112.0 1.9X
+after 1582, rebase EXCEPTION 32810 32810 0 3.0 328.1 0.6X
+after 1582, rebase LEGACY 32530 32530 0 3.1 325.3 0.6X
+after 1582, rebase CORRECTED 32849 32849 0 3.0 328.5 0.6X
+before 1582, rebase LEGACY 23537 23537 0 4.2 235.4 0.9X
+before 1582, rebase CORRECTED 22870 22870 0 4.4 228.7 0.9X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load DATE from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase EXCEPTION 12984 13262 257 7.7 129.8 1.0X
-after 1582, vec off, rebase LEGACY 13278 13330 50 7.5 132.8 1.0X
-after 1582, vec off, rebase CORRECTED 13202 13255 50 7.6 132.0 1.0X
-after 1582, vec on, rebase EXCEPTION 3823 3853 40 26.2 38.2 3.4X
-after 1582, vec on, rebase LEGACY 3846 3876 27 26.0 38.5 3.4X
-after 1582, vec on, rebase CORRECTED 3775 3838 62 26.5 37.7 3.4X
-before 1582, vec off, rebase LEGACY 13671 13692 26 7.3 136.7 0.9X
-before 1582, vec off, rebase CORRECTED 13387 13476 106 7.5 133.9 1.0X
-before 1582, vec on, rebase LEGACY 4477 4484 7 22.3 44.8 2.9X
-before 1582, vec on, rebase CORRECTED 3729 3773 50 26.8 37.3 3.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, vec off, rebase EXCEPTION 13114 13225 104 7.6 131.1 1.0X
+after 1582, vec off, rebase LEGACY 13175 13189 15 7.6 131.8 1.0X
+after 1582, vec off, rebase CORRECTED 13080 13115 34 7.6 130.8 1.0X
+after 1582, vec on, rebase EXCEPTION 3698 3726 29 27.0 37.0 3.5X
+after 1582, vec on, rebase LEGACY 3730 3745 17 26.8 37.3 3.5X
+after 1582, vec on, rebase CORRECTED 3714 3758 75 26.9 37.1 3.5X
+before 1582, vec off, rebase LEGACY 13519 13575 63 7.4 135.2 1.0X
+before 1582, vec off, rebase CORRECTED 13210 13309 108 7.6 132.1 1.0X
+before 1582, vec on, rebase LEGACY 4459 4488 44 22.4 44.6 2.9X
+before 1582, vec on, rebase CORRECTED 3661 3718 88 27.3 36.6 3.6X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save TIMESTAMP_INT96 to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, noop 3020 3020 0 33.1 30.2 1.0X
-before 1900, noop 3013 3013 0 33.2 30.1 1.0X
-after 1900, rebase EXCEPTION 28796 28796 0 3.5 288.0 0.1X
-after 1900, rebase LEGACY 28869 28869 0 3.5 288.7 0.1X
-after 1900, rebase CORRECTED 28522 28522 0 3.5 285.2 0.1X
-before 1900, rebase LEGACY 30594 30594 0 3.3 305.9 0.1X
-before 1900, rebase CORRECTED 30743 30743 0 3.3 307.4 0.1X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, noop 2900 2900 0 34.5 29.0 1.0X
+before 1900, noop 2848 2848 0 35.1 28.5 1.0X
+after 1900, rebase EXCEPTION 27623 27623 0 3.6 276.2 0.1X
+after 1900, rebase LEGACY 27305 27305 0 3.7 273.0 0.1X
+after 1900, rebase CORRECTED 27715 27715 0 3.6 277.2 0.1X
+before 1900, rebase LEGACY 30911 30911 0 3.2 309.1 0.1X
+before 1900, rebase CORRECTED 27944 27944 0 3.6 279.4 0.1X
Review comment:
Parquet writer without rebasing is ~10% faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713631507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713509083
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713630134
**[Test build #130090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130090/testReport)** for PR 30118 at commit [`c6d5b5c`](https://github.com/apache/spark/commit/c6d5b5ce8e06b9aff395bbe5b51cfddd03f30cd7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-714105137
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713473461
@HyukjinKwon @cloud-fan @tomvanbussel @ala @mswit-databricks @bart-samwel Please, review this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30118:
URL: https://github.com/apache/spark/pull/30118#discussion_r509170757
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -2,153 +2,153 @@
Rebasing dates/timestamps in Parquet datasource
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 20023 20023 0 5.0 200.2 1.0X
-before 1582, noop 10729 10729 0 9.3 107.3 1.9X
-after 1582, rebase EXCEPTION 31834 31834 0 3.1 318.3 0.6X
-after 1582, rebase LEGACY 31997 31997 0 3.1 320.0 0.6X
-after 1582, rebase CORRECTED 31712 31712 0 3.2 317.1 0.6X
-before 1582, rebase LEGACY 23663 23663 0 4.2 236.6 0.8X
-before 1582, rebase CORRECTED 22749 22749 0 4.4 227.5 0.9X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, noop 21041 21041 0 4.8 210.4 1.0X
+before 1582, noop 11202 11202 0 8.9 112.0 1.9X
+after 1582, rebase EXCEPTION 32810 32810 0 3.0 328.1 0.6X
+after 1582, rebase LEGACY 32530 32530 0 3.1 325.3 0.6X
+after 1582, rebase CORRECTED 32849 32849 0 3.0 328.5 0.6X
+before 1582, rebase LEGACY 23537 23537 0 4.2 235.4 0.9X
+before 1582, rebase CORRECTED 22870 22870 0 4.4 228.7 0.9X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load DATE from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase EXCEPTION 12984 13262 257 7.7 129.8 1.0X
-after 1582, vec off, rebase LEGACY 13278 13330 50 7.5 132.8 1.0X
-after 1582, vec off, rebase CORRECTED 13202 13255 50 7.6 132.0 1.0X
-after 1582, vec on, rebase EXCEPTION 3823 3853 40 26.2 38.2 3.4X
-after 1582, vec on, rebase LEGACY 3846 3876 27 26.0 38.5 3.4X
-after 1582, vec on, rebase CORRECTED 3775 3838 62 26.5 37.7 3.4X
-before 1582, vec off, rebase LEGACY 13671 13692 26 7.3 136.7 0.9X
-before 1582, vec off, rebase CORRECTED 13387 13476 106 7.5 133.9 1.0X
-before 1582, vec on, rebase LEGACY 4477 4484 7 22.3 44.8 2.9X
-before 1582, vec on, rebase CORRECTED 3729 3773 50 26.8 37.3 3.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, vec off, rebase EXCEPTION 13114 13225 104 7.6 131.1 1.0X
+after 1582, vec off, rebase LEGACY 13175 13189 15 7.6 131.8 1.0X
+after 1582, vec off, rebase CORRECTED 13080 13115 34 7.6 130.8 1.0X
+after 1582, vec on, rebase EXCEPTION 3698 3726 29 27.0 37.0 3.5X
+after 1582, vec on, rebase LEGACY 3730 3745 17 26.8 37.3 3.5X
+after 1582, vec on, rebase CORRECTED 3714 3758 75 26.9 37.1 3.5X
+before 1582, vec off, rebase LEGACY 13519 13575 63 7.4 135.2 1.0X
+before 1582, vec off, rebase CORRECTED 13210 13309 108 7.6 132.1 1.0X
+before 1582, vec on, rebase LEGACY 4459 4488 44 22.4 44.6 2.9X
+before 1582, vec on, rebase CORRECTED 3661 3718 88 27.3 36.6 3.6X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save TIMESTAMP_INT96 to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, noop 3020 3020 0 33.1 30.2 1.0X
-before 1900, noop 3013 3013 0 33.2 30.1 1.0X
-after 1900, rebase EXCEPTION 28796 28796 0 3.5 288.0 0.1X
-after 1900, rebase LEGACY 28869 28869 0 3.5 288.7 0.1X
-after 1900, rebase CORRECTED 28522 28522 0 3.5 285.2 0.1X
-before 1900, rebase LEGACY 30594 30594 0 3.3 305.9 0.1X
-before 1900, rebase CORRECTED 30743 30743 0 3.3 307.4 0.1X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, noop 2900 2900 0 34.5 29.0 1.0X
+before 1900, noop 2848 2848 0 35.1 28.5 1.0X
+after 1900, rebase EXCEPTION 27623 27623 0 3.6 276.2 0.1X
+after 1900, rebase LEGACY 27305 27305 0 3.7 273.0 0.1X
+after 1900, rebase CORRECTED 27715 27715 0 3.6 277.2 0.1X
+before 1900, rebase LEGACY 30911 30911 0 3.2 309.1 0.1X
+before 1900, rebase CORRECTED 27944 27944 0 3.6 279.4 0.1X
Review comment:
Before: no diff
After: ~10%
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713473988
**[Test build #130090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130090/testReport)** for PR 30118 at commit [`c6d5b5c`](https://github.com/apache/spark/commit/c6d5b5ce8e06b9aff395bbe5b51cfddd03f30cd7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713631507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713473988
**[Test build #130090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130090/testReport)** for PR 30118 at commit [`c6d5b5c`](https://github.com/apache/spark/commit/c6d5b5ce8e06b9aff395bbe5b51cfddd03f30cd7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713509083
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713497057
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34699/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30118:
URL: https://github.com/apache/spark/pull/30118#discussion_r509172026
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -2,153 +2,153 @@
Rebasing dates/timestamps in Parquet datasource
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 20023 20023 0 5.0 200.2 1.0X
-before 1582, noop 10729 10729 0 9.3 107.3 1.9X
-after 1582, rebase EXCEPTION 31834 31834 0 3.1 318.3 0.6X
-after 1582, rebase LEGACY 31997 31997 0 3.1 320.0 0.6X
-after 1582, rebase CORRECTED 31712 31712 0 3.2 317.1 0.6X
-before 1582, rebase LEGACY 23663 23663 0 4.2 236.6 0.8X
-before 1582, rebase CORRECTED 22749 22749 0 4.4 227.5 0.9X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, noop 21041 21041 0 4.8 210.4 1.0X
+before 1582, noop 11202 11202 0 8.9 112.0 1.9X
+after 1582, rebase EXCEPTION 32810 32810 0 3.0 328.1 0.6X
+after 1582, rebase LEGACY 32530 32530 0 3.1 325.3 0.6X
+after 1582, rebase CORRECTED 32849 32849 0 3.0 328.5 0.6X
+before 1582, rebase LEGACY 23537 23537 0 4.2 235.4 0.9X
+before 1582, rebase CORRECTED 22870 22870 0 4.4 228.7 0.9X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load DATE from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase EXCEPTION 12984 13262 257 7.7 129.8 1.0X
-after 1582, vec off, rebase LEGACY 13278 13330 50 7.5 132.8 1.0X
-after 1582, vec off, rebase CORRECTED 13202 13255 50 7.6 132.0 1.0X
-after 1582, vec on, rebase EXCEPTION 3823 3853 40 26.2 38.2 3.4X
-after 1582, vec on, rebase LEGACY 3846 3876 27 26.0 38.5 3.4X
-after 1582, vec on, rebase CORRECTED 3775 3838 62 26.5 37.7 3.4X
-before 1582, vec off, rebase LEGACY 13671 13692 26 7.3 136.7 0.9X
-before 1582, vec off, rebase CORRECTED 13387 13476 106 7.5 133.9 1.0X
-before 1582, vec on, rebase LEGACY 4477 4484 7 22.3 44.8 2.9X
-before 1582, vec on, rebase CORRECTED 3729 3773 50 26.8 37.3 3.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, vec off, rebase EXCEPTION 13114 13225 104 7.6 131.1 1.0X
+after 1582, vec off, rebase LEGACY 13175 13189 15 7.6 131.8 1.0X
+after 1582, vec off, rebase CORRECTED 13080 13115 34 7.6 130.8 1.0X
+after 1582, vec on, rebase EXCEPTION 3698 3726 29 27.0 37.0 3.5X
+after 1582, vec on, rebase LEGACY 3730 3745 17 26.8 37.3 3.5X
+after 1582, vec on, rebase CORRECTED 3714 3758 75 26.9 37.1 3.5X
+before 1582, vec off, rebase LEGACY 13519 13575 63 7.4 135.2 1.0X
+before 1582, vec off, rebase CORRECTED 13210 13309 108 7.6 132.1 1.0X
+before 1582, vec on, rebase LEGACY 4459 4488 44 22.4 44.6 2.9X
+before 1582, vec on, rebase CORRECTED 3661 3718 88 27.3 36.6 3.6X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save TIMESTAMP_INT96 to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, noop 3020 3020 0 33.1 30.2 1.0X
-before 1900, noop 3013 3013 0 33.2 30.1 1.0X
-after 1900, rebase EXCEPTION 28796 28796 0 3.5 288.0 0.1X
-after 1900, rebase LEGACY 28869 28869 0 3.5 288.7 0.1X
-after 1900, rebase CORRECTED 28522 28522 0 3.5 285.2 0.1X
-before 1900, rebase LEGACY 30594 30594 0 3.3 305.9 0.1X
-before 1900, rebase CORRECTED 30743 30743 0 3.3 307.4 0.1X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, noop 2900 2900 0 34.5 29.0 1.0X
+before 1900, noop 2848 2848 0 35.1 28.5 1.0X
+after 1900, rebase EXCEPTION 27623 27623 0 3.6 276.2 0.1X
+after 1900, rebase LEGACY 27305 27305 0 3.7 273.0 0.1X
+after 1900, rebase CORRECTED 27715 27715 0 3.6 277.2 0.1X
+before 1900, rebase LEGACY 30911 30911 0 3.2 309.1 0.1X
+before 1900, rebase CORRECTED 27944 27944 0 3.6 279.4 0.1X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load TIMESTAMP_INT96 from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, vec off, rebase EXCEPTION 19325 19468 135 5.2 193.3 1.0X
-after 1900, vec off, rebase LEGACY 19568 19602 30 5.1 195.7 1.0X
-after 1900, vec off, rebase CORRECTED 19532 19538 6 5.1 195.3 1.0X
-after 1900, vec on, rebase EXCEPTION 9884 9990 94 10.1 98.8 2.0X
-after 1900, vec on, rebase LEGACY 9933 9985 49 10.1 99.3 1.9X
-after 1900, vec on, rebase CORRECTED 9967 10043 76 10.0 99.7 1.9X
-before 1900, vec off, rebase LEGACY 24162 24198 37 4.1 241.6 0.8X
-before 1900, vec off, rebase CORRECTED 24034 24056 20 4.2 240.3 0.8X
-before 1900, vec on, rebase LEGACY 12548 12625 72 8.0 125.5 1.5X
-before 1900, vec on, rebase CORRECTED 12580 12660 115 7.9 125.8 1.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, vec off, rebase EXCEPTION 16853 16885 41 5.9 168.5 1.0X
+after 1900, vec off, rebase LEGACY 16804 16816 21 6.0 168.0 1.0X
+after 1900, vec off, rebase CORRECTED 16985 17020 58 5.9 169.9 1.0X
+after 1900, vec on, rebase EXCEPTION 7044 7063 19 14.2 70.4 2.4X
+after 1900, vec on, rebase LEGACY 7183 7255 94 13.9 71.8 2.3X
+after 1900, vec on, rebase CORRECTED 7047 7137 86 14.2 70.5 2.4X
+before 1900, vec off, rebase LEGACY 20371 20458 81 4.9 203.7 0.8X
+before 1900, vec off, rebase CORRECTED 17484 17541 54 5.7 174.8 1.0X
Review comment:
Parquet-MR speed up ~15%
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30118:
URL: https://github.com/apache/spark/pull/30118#discussion_r509171281
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -2,153 +2,153 @@
Rebasing dates/timestamps in Parquet datasource
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 20023 20023 0 5.0 200.2 1.0X
-before 1582, noop 10729 10729 0 9.3 107.3 1.9X
-after 1582, rebase EXCEPTION 31834 31834 0 3.1 318.3 0.6X
-after 1582, rebase LEGACY 31997 31997 0 3.1 320.0 0.6X
-after 1582, rebase CORRECTED 31712 31712 0 3.2 317.1 0.6X
-before 1582, rebase LEGACY 23663 23663 0 4.2 236.6 0.8X
-before 1582, rebase CORRECTED 22749 22749 0 4.4 227.5 0.9X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, noop 21041 21041 0 4.8 210.4 1.0X
+before 1582, noop 11202 11202 0 8.9 112.0 1.9X
+after 1582, rebase EXCEPTION 32810 32810 0 3.0 328.1 0.6X
+after 1582, rebase LEGACY 32530 32530 0 3.1 325.3 0.6X
+after 1582, rebase CORRECTED 32849 32849 0 3.0 328.5 0.6X
+before 1582, rebase LEGACY 23537 23537 0 4.2 235.4 0.9X
+before 1582, rebase CORRECTED 22870 22870 0 4.4 228.7 0.9X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load DATE from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase EXCEPTION 12984 13262 257 7.7 129.8 1.0X
-after 1582, vec off, rebase LEGACY 13278 13330 50 7.5 132.8 1.0X
-after 1582, vec off, rebase CORRECTED 13202 13255 50 7.6 132.0 1.0X
-after 1582, vec on, rebase EXCEPTION 3823 3853 40 26.2 38.2 3.4X
-after 1582, vec on, rebase LEGACY 3846 3876 27 26.0 38.5 3.4X
-after 1582, vec on, rebase CORRECTED 3775 3838 62 26.5 37.7 3.4X
-before 1582, vec off, rebase LEGACY 13671 13692 26 7.3 136.7 0.9X
-before 1582, vec off, rebase CORRECTED 13387 13476 106 7.5 133.9 1.0X
-before 1582, vec on, rebase LEGACY 4477 4484 7 22.3 44.8 2.9X
-before 1582, vec on, rebase CORRECTED 3729 3773 50 26.8 37.3 3.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, vec off, rebase EXCEPTION 13114 13225 104 7.6 131.1 1.0X
+after 1582, vec off, rebase LEGACY 13175 13189 15 7.6 131.8 1.0X
+after 1582, vec off, rebase CORRECTED 13080 13115 34 7.6 130.8 1.0X
+after 1582, vec on, rebase EXCEPTION 3698 3726 29 27.0 37.0 3.5X
+after 1582, vec on, rebase LEGACY 3730 3745 17 26.8 37.3 3.5X
+after 1582, vec on, rebase CORRECTED 3714 3758 75 26.9 37.1 3.5X
+before 1582, vec off, rebase LEGACY 13519 13575 63 7.4 135.2 1.0X
+before 1582, vec off, rebase CORRECTED 13210 13309 108 7.6 132.1 1.0X
+before 1582, vec on, rebase LEGACY 4459 4488 44 22.4 44.6 2.9X
+before 1582, vec on, rebase CORRECTED 3661 3718 88 27.3 36.6 3.6X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save TIMESTAMP_INT96 to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, noop 3020 3020 0 33.1 30.2 1.0X
-before 1900, noop 3013 3013 0 33.2 30.1 1.0X
-after 1900, rebase EXCEPTION 28796 28796 0 3.5 288.0 0.1X
-after 1900, rebase LEGACY 28869 28869 0 3.5 288.7 0.1X
-after 1900, rebase CORRECTED 28522 28522 0 3.5 285.2 0.1X
-before 1900, rebase LEGACY 30594 30594 0 3.3 305.9 0.1X
-before 1900, rebase CORRECTED 30743 30743 0 3.3 307.4 0.1X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, noop 2900 2900 0 34.5 29.0 1.0X
+before 1900, noop 2848 2848 0 35.1 28.5 1.0X
+after 1900, rebase EXCEPTION 27623 27623 0 3.6 276.2 0.1X
+after 1900, rebase LEGACY 27305 27305 0 3.7 273.0 0.1X
+after 1900, rebase CORRECTED 27715 27715 0 3.6 277.2 0.1X
+before 1900, rebase LEGACY 30911 30911 0 3.2 309.1 0.1X
+before 1900, rebase CORRECTED 27944 27944 0 3.6 279.4 0.1X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load TIMESTAMP_INT96 from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, vec off, rebase EXCEPTION 19325 19468 135 5.2 193.3 1.0X
-after 1900, vec off, rebase LEGACY 19568 19602 30 5.1 195.7 1.0X
-after 1900, vec off, rebase CORRECTED 19532 19538 6 5.1 195.3 1.0X
-after 1900, vec on, rebase EXCEPTION 9884 9990 94 10.1 98.8 2.0X
-after 1900, vec on, rebase LEGACY 9933 9985 49 10.1 99.3 1.9X
-after 1900, vec on, rebase CORRECTED 9967 10043 76 10.0 99.7 1.9X
-before 1900, vec off, rebase LEGACY 24162 24198 37 4.1 241.6 0.8X
-before 1900, vec off, rebase CORRECTED 24034 24056 20 4.2 240.3 0.8X
-before 1900, vec on, rebase LEGACY 12548 12625 72 8.0 125.5 1.5X
-before 1900, vec on, rebase CORRECTED 12580 12660 115 7.9 125.8 1.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, vec off, rebase EXCEPTION 16853 16885 41 5.9 168.5 1.0X
+after 1900, vec off, rebase LEGACY 16804 16816 21 6.0 168.0 1.0X
+after 1900, vec off, rebase CORRECTED 16985 17020 58 5.9 169.9 1.0X
+after 1900, vec on, rebase EXCEPTION 7044 7063 19 14.2 70.4 2.4X
+after 1900, vec on, rebase LEGACY 7183 7255 94 13.9 71.8 2.3X
+after 1900, vec on, rebase CORRECTED 7047 7137 86 14.2 70.5 2.4X
+before 1900, vec off, rebase LEGACY 20371 20458 81 4.9 203.7 0.8X
+before 1900, vec off, rebase CORRECTED 17484 17541 54 5.7 174.8 1.0X
+before 1900, vec on, rebase LEGACY 10284 10327 45 9.7 102.8 1.6X
+before 1900, vec on, rebase CORRECTED 7044 7073 37 14.2 70.4 2.4X
Review comment:
Vectorized Reader speed up: ~30%
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713509066
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34699/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #30118:
URL: https://github.com/apache/spark/pull/30118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30118:
URL: https://github.com/apache/spark/pull/30118#issuecomment-713509092
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34699/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30118: [SPARK-33160][SQL][FOLLOWUP] Update benchmarks of INT96 type rebasing
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30118:
URL: https://github.com/apache/spark/pull/30118#discussion_r509171281
##########
File path: sql/core/benchmarks/DateTimeRebaseBenchmark-jdk11-results.txt
##########
@@ -2,153 +2,153 @@
Rebasing dates/timestamps in Parquet datasource
================================================================================================
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 20023 20023 0 5.0 200.2 1.0X
-before 1582, noop 10729 10729 0 9.3 107.3 1.9X
-after 1582, rebase EXCEPTION 31834 31834 0 3.1 318.3 0.6X
-after 1582, rebase LEGACY 31997 31997 0 3.1 320.0 0.6X
-after 1582, rebase CORRECTED 31712 31712 0 3.2 317.1 0.6X
-before 1582, rebase LEGACY 23663 23663 0 4.2 236.6 0.8X
-before 1582, rebase CORRECTED 22749 22749 0 4.4 227.5 0.9X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, noop 21041 21041 0 4.8 210.4 1.0X
+before 1582, noop 11202 11202 0 8.9 112.0 1.9X
+after 1582, rebase EXCEPTION 32810 32810 0 3.0 328.1 0.6X
+after 1582, rebase LEGACY 32530 32530 0 3.1 325.3 0.6X
+after 1582, rebase CORRECTED 32849 32849 0 3.0 328.5 0.6X
+before 1582, rebase LEGACY 23537 23537 0 4.2 235.4 0.9X
+before 1582, rebase CORRECTED 22870 22870 0 4.4 228.7 0.9X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load DATE from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, vec off, rebase EXCEPTION 12984 13262 257 7.7 129.8 1.0X
-after 1582, vec off, rebase LEGACY 13278 13330 50 7.5 132.8 1.0X
-after 1582, vec off, rebase CORRECTED 13202 13255 50 7.6 132.0 1.0X
-after 1582, vec on, rebase EXCEPTION 3823 3853 40 26.2 38.2 3.4X
-after 1582, vec on, rebase LEGACY 3846 3876 27 26.0 38.5 3.4X
-after 1582, vec on, rebase CORRECTED 3775 3838 62 26.5 37.7 3.4X
-before 1582, vec off, rebase LEGACY 13671 13692 26 7.3 136.7 0.9X
-before 1582, vec off, rebase CORRECTED 13387 13476 106 7.5 133.9 1.0X
-before 1582, vec on, rebase LEGACY 4477 4484 7 22.3 44.8 2.9X
-before 1582, vec on, rebase CORRECTED 3729 3773 50 26.8 37.3 3.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1582, vec off, rebase EXCEPTION 13114 13225 104 7.6 131.1 1.0X
+after 1582, vec off, rebase LEGACY 13175 13189 15 7.6 131.8 1.0X
+after 1582, vec off, rebase CORRECTED 13080 13115 34 7.6 130.8 1.0X
+after 1582, vec on, rebase EXCEPTION 3698 3726 29 27.0 37.0 3.5X
+after 1582, vec on, rebase LEGACY 3730 3745 17 26.8 37.3 3.5X
+after 1582, vec on, rebase CORRECTED 3714 3758 75 26.9 37.1 3.5X
+before 1582, vec off, rebase LEGACY 13519 13575 63 7.4 135.2 1.0X
+before 1582, vec off, rebase CORRECTED 13210 13309 108 7.6 132.1 1.0X
+before 1582, vec on, rebase LEGACY 4459 4488 44 22.4 44.6 2.9X
+before 1582, vec on, rebase CORRECTED 3661 3718 88 27.3 36.6 3.6X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Save TIMESTAMP_INT96 to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, noop 3020 3020 0 33.1 30.2 1.0X
-before 1900, noop 3013 3013 0 33.2 30.1 1.0X
-after 1900, rebase EXCEPTION 28796 28796 0 3.5 288.0 0.1X
-after 1900, rebase LEGACY 28869 28869 0 3.5 288.7 0.1X
-after 1900, rebase CORRECTED 28522 28522 0 3.5 285.2 0.1X
-before 1900, rebase LEGACY 30594 30594 0 3.3 305.9 0.1X
-before 1900, rebase CORRECTED 30743 30743 0 3.3 307.4 0.1X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, noop 2900 2900 0 34.5 29.0 1.0X
+before 1900, noop 2848 2848 0 35.1 28.5 1.0X
+after 1900, rebase EXCEPTION 27623 27623 0 3.6 276.2 0.1X
+after 1900, rebase LEGACY 27305 27305 0 3.7 273.0 0.1X
+after 1900, rebase CORRECTED 27715 27715 0 3.6 277.2 0.1X
+before 1900, rebase LEGACY 30911 30911 0 3.2 309.1 0.1X
+before 1900, rebase CORRECTED 27944 27944 0 3.6 279.4 0.1X
+
+OpenJDK 64-Bit Server VM 11.0.8+10-post-Ubuntu-0ubuntu118.04.1 on Linux 5.3.0-1034-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Load TIMESTAMP_INT96 from parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1900, vec off, rebase EXCEPTION 19325 19468 135 5.2 193.3 1.0X
-after 1900, vec off, rebase LEGACY 19568 19602 30 5.1 195.7 1.0X
-after 1900, vec off, rebase CORRECTED 19532 19538 6 5.1 195.3 1.0X
-after 1900, vec on, rebase EXCEPTION 9884 9990 94 10.1 98.8 2.0X
-after 1900, vec on, rebase LEGACY 9933 9985 49 10.1 99.3 1.9X
-after 1900, vec on, rebase CORRECTED 9967 10043 76 10.0 99.7 1.9X
-before 1900, vec off, rebase LEGACY 24162 24198 37 4.1 241.6 0.8X
-before 1900, vec off, rebase CORRECTED 24034 24056 20 4.2 240.3 0.8X
-before 1900, vec on, rebase LEGACY 12548 12625 72 8.0 125.5 1.5X
-before 1900, vec on, rebase CORRECTED 12580 12660 115 7.9 125.8 1.5X
-
-OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 on Linux 4.15.0-1063-aws
+after 1900, vec off, rebase EXCEPTION 16853 16885 41 5.9 168.5 1.0X
+after 1900, vec off, rebase LEGACY 16804 16816 21 6.0 168.0 1.0X
+after 1900, vec off, rebase CORRECTED 16985 17020 58 5.9 169.9 1.0X
+after 1900, vec on, rebase EXCEPTION 7044 7063 19 14.2 70.4 2.4X
+after 1900, vec on, rebase LEGACY 7183 7255 94 13.9 71.8 2.3X
+after 1900, vec on, rebase CORRECTED 7047 7137 86 14.2 70.5 2.4X
+before 1900, vec off, rebase LEGACY 20371 20458 81 4.9 203.7 0.8X
+before 1900, vec off, rebase CORRECTED 17484 17541 54 5.7 174.8 1.0X
+before 1900, vec on, rebase LEGACY 10284 10327 45 9.7 102.8 1.6X
+before 1900, vec on, rebase CORRECTED 7044 7073 37 14.2 70.4 2.4X
Review comment:
Before: no diff
After: ~30%
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org