You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/11/14 00:38:10 UTC
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter...
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/23027
[SPARK-26049][SQL][TEST] Add InMemoryTable filter benchmark to FilterPushdownBenchmark
## What changes were proposed in this pull request?
Add InMemoryTable filter benchmark to `FilterPushdownBenchmark`.
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-26049
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23027.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23027
----
commit d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7
Author: Yuming Wang <yu...@...>
Date: 2018-11-14T00:35:35Z
Add InMemoryTable filter benchmark
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r233689556
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X
+Parquet Vectorized 7823 / 7996 2.0 497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X
+InMemoryTable Vectorized 6 / 6 2859.1 0.3 1422.0X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3023.0 0.3 1503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row (value = '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11878 / 11888 1.3 755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0 40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1 466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3 33.0 22.9X
+Parquet Vectorized 8322 / 8386 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 434 / 441 36.2 27.6 19.2X
+Native ORC Vectorized 5659 / 5944 2.8 359.8 1.5X
+Native ORC Vectorized (Pushdown) 535 / 567 29.4 34.0 15.6X
+InMemoryTable Vectorized 4784 / 4879 3.3 304.1 1.7X
+InMemoryTable Vectorized (Pushdown) 1950 / 1985 8.1 124.0 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row (value <=> '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11423 / 11440 1.4 726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2 39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2 465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0 32.2 22.5X
+Parquet Vectorized 8377 / 8458 1.9 532.6 1.0X
+Parquet Vectorized (Pushdown) 449 / 457 35.1 28.5 18.7X
+Native ORC Vectorized 5664 / 5786 2.8 360.1 1.5X
+Native ORC Vectorized (Pushdown) 527 / 560 29.9 33.5 15.9X
+InMemoryTable Vectorized 4743 / 4813 3.3 301.5 1.8X
+InMemoryTable Vectorized (Pushdown) 1876 / 1963 8.4 119.3 4.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11440 / 11478 1.4 727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8 40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2 464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4 32.8 22.1X
+Parquet Vectorized 8120 / 8293 1.9 516.2 1.0X
+Parquet Vectorized (Pushdown) 423 / 436 37.2 26.9 19.2X
+Native ORC Vectorized 5490 / 5833 2.9 349.0 1.5X
+Native ORC Vectorized (Pushdown) 538 / 562 29.3 34.2 15.1X
+InMemoryTable Vectorized 4722 / 4774 3.3 300.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1923 / 1954 8.2 122.2 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 20750 / 20872 0.8 1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7 1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9 1062.6 1.2X
-Native ORC Vectorized (Pushdown) 16926 / 16965 0.9 1076.1 1.2X
+Parquet Vectorized 15437 / 15682 1.0 981.4 1.0X
+Parquet Vectorized (Pushdown) 15748 / 15811 1.0 1001.2 1.0X
+Native ORC Vectorized 13111 / 13245 1.2 833.6 1.2X
+Native ORC Vectorized (Pushdown) 12931 / 13216 1.2 822.1 1.2X
+InMemoryTable Vectorized 9661 / 9908 1.6 614.2 1.6X
+InMemoryTable Vectorized (Pushdown) 9838 / 9908 1.6 625.5 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 int row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10510 / 10532 1.5 668.2 1.0X
-Parquet Vectorized (Pushdown) 642 / 665 24.5 40.8 16.4X
-Native ORC Vectorized 6609 / 6618 2.4 420.2 1.6X
-Native ORC Vectorized (Pushdown) 502 / 512 31.4 31.9 21.0X
+Parquet Vectorized 7549 / 7625 2.1 479.9 1.0X
+Parquet Vectorized (Pushdown) 427 / 450 36.9 27.1 17.7X
+Native ORC Vectorized 5028 / 5221 3.1 319.7 1.5X
+Native ORC Vectorized (Pushdown) 530 / 548 29.7 33.7 14.3X
+InMemoryTable Vectorized 5 / 6 3221.5 0.3 1546.1X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3303.5 0.3 1585.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 int row (7864320 < value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10505 / 10514 1.5 667.9 1.0X
-Parquet Vectorized (Pushdown) 659 / 673 23.9 41.9 15.9X
-Native ORC Vectorized 6634 / 6641 2.4 421.8 1.6X
-Native ORC Vectorized (Pushdown) 513 / 526 30.7 32.6 20.5X
+Parquet Vectorized 7542 / 7584 2.1 479.5 1.0X
+Parquet Vectorized (Pushdown) 458 / 468 34.3 29.1 16.5X
+Native ORC Vectorized 5035 / 5533 3.1 320.1 1.5X
+Native ORC Vectorized (Pushdown) 538 / 560 29.2 34.2 14.0X
+InMemoryTable Vectorized 3866 / 3925 4.1 245.8 2.0X
+InMemoryTable Vectorized (Pushdown) 1766 / 1792 8.9 112.3 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10555 / 10570 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 651 / 668 24.2 41.4 16.2X
-Native ORC Vectorized 6721 / 6728 2.3 427.3 1.6X
-Native ORC Vectorized (Pushdown) 508 / 519 31.0 32.3 20.8X
+Parquet Vectorized 7540 / 7660 2.1 479.3 1.0X
+Parquet Vectorized (Pushdown) 444 / 454 35.5 28.2 17.0X
+Native ORC Vectorized 5155 / 5310 3.1 327.7 1.5X
+Native ORC Vectorized (Pushdown) 524 / 541 30.0 33.3 14.4X
+InMemoryTable Vectorized 3910 / 3999 4.0 248.6 1.9X
+InMemoryTable Vectorized (Pushdown) 1843 / 1868 8.5 117.2 4.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (value <=> 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10566 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 647 / 654 24.3 41.1 16.3X
-Native ORC Vectorized 6716 / 6728 2.3 427.0 1.6X
-Native ORC Vectorized (Pushdown) 510 / 521 30.9 32.4 20.7X
+Parquet Vectorized 7568 / 7679 2.1 481.1 1.0X
+Parquet Vectorized (Pushdown) 446 / 460 35.2 28.4 17.0X
+Native ORC Vectorized 5083 / 5244 3.1 323.2 1.5X
+Native ORC Vectorized (Pushdown) 529 / 542 29.7 33.6 14.3X
+InMemoryTable Vectorized 4033 / 4068 3.9 256.4 1.9X
+InMemoryTable Vectorized (Pushdown) 1818 / 1848 8.6 115.6 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (7864320 <= value <= 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10565 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 649 / 654 24.2 41.3 16.3X
-Native ORC Vectorized 6700 / 6712 2.3 426.0 1.6X
-Native ORC Vectorized (Pushdown) 509 / 520 30.9 32.3 20.8X
+Parquet Vectorized 7446 / 7622 2.1 473.4 1.0X
+Parquet Vectorized (Pushdown) 440 / 455 35.8 28.0 16.9X
+Native ORC Vectorized 5155 / 5347 3.1 327.8 1.4X
+Native ORC Vectorized (Pushdown) 509 / 538 30.9 32.3 14.6X
+InMemoryTable Vectorized 3952 / 3977 4.0 251.3 1.9X
+InMemoryTable Vectorized (Pushdown) 1832 / 1887 8.6 116.5 4.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (7864319 < value < 7864321): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10547 / 10566 1.5 670.5 1.0X
-Parquet Vectorized (Pushdown) 649 / 653 24.2 41.3 16.3X
-Native ORC Vectorized 6703 / 6713 2.3 426.2 1.6X
-Native ORC Vectorized (Pushdown) 510 / 520 30.8 32.5 20.7X
+Parquet Vectorized 7576 / 7711 2.1 481.7 1.0X
+Parquet Vectorized (Pushdown) 467 / 477 33.7 29.7 16.2X
+Native ORC Vectorized 5147 / 5412 3.1 327.3 1.5X
+Native ORC Vectorized (Pushdown) 521 / 551 30.2 33.1 14.6X
+InMemoryTable Vectorized 3882 / 3964 4.1 246.8 2.0X
+InMemoryTable Vectorized (Pushdown) 1805 / 1845 8.7 114.7 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% int rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11478 / 11525 1.4 729.7 1.0X
-Parquet Vectorized (Pushdown) 2576 / 2587 6.1 163.8 4.5X
-Native ORC Vectorized 7633 / 7657 2.1 485.3 1.5X
-Native ORC Vectorized (Pushdown) 2076 / 2096 7.6 132.0 5.5X
+Parquet Vectorized 8439 / 8541 1.9 536.5 1.0X
+Parquet Vectorized (Pushdown) 1952 / 1993 8.1 124.1 4.3X
+Native ORC Vectorized 5864 / 6092 2.7 372.8 1.4X
+Native ORC Vectorized (Pushdown) 1786 / 1805 8.8 113.6 4.7X
+InMemoryTable Vectorized 4463 / 4527 3.5 283.8 1.9X
+InMemoryTable Vectorized (Pushdown) 2559 / 2596 6.1 162.7 3.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% int rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14785 / 14802 1.1 940.0 1.0X
-Parquet Vectorized (Pushdown) 9971 / 9977 1.6 633.9 1.5X
-Native ORC Vectorized 11082 / 11107 1.4 704.6 1.3X
-Native ORC Vectorized (Pushdown) 8061 / 8073 2.0 512.5 1.8X
+Parquet Vectorized 11106 / 11248 1.4 706.1 1.0X
+Parquet Vectorized (Pushdown) 7656 / 7697 2.1 486.7 1.5X
+Native ORC Vectorized 8593 / 8892 1.8 546.4 1.3X
+Native ORC Vectorized (Pushdown) 6474 / 6567 2.4 411.6 1.7X
+InMemoryTable Vectorized 6623 / 6731 2.4 421.1 1.7X
+InMemoryTable Vectorized (Pushdown) 5613 / 5747 2.8 356.9 2.0X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% int rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 18174 / 18214 0.9 1155.5 1.0X
-Parquet Vectorized (Pushdown) 17387 / 17403 0.9 1105.5 1.0X
-Native ORC Vectorized 14465 / 14492 1.1 919.7 1.3X
-Native ORC Vectorized (Pushdown) 14024 / 14041 1.1 891.6 1.3X
+Parquet Vectorized 13716 / 13810 1.1 872.1 1.0X
+Parquet Vectorized (Pushdown) 13044 / 13233 1.2 829.3 1.1X
+Native ORC Vectorized 11795 / 11899 1.3 749.9 1.2X
+Native ORC Vectorized (Pushdown) 11074 / 11265 1.4 704.1 1.2X
+InMemoryTable Vectorized 9023 / 9166 1.7 573.7 1.5X
+InMemoryTable Vectorized (Pushdown) 8929 / 9015 1.8 567.7 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19004 / 19014 0.8 1208.2 1.0X
-Parquet Vectorized (Pushdown) 19219 / 19232 0.8 1221.9 1.0X
-Native ORC Vectorized 15266 / 15290 1.0 970.6 1.2X
-Native ORC Vectorized (Pushdown) 15469 / 15482 1.0 983.5 1.2X
+Parquet Vectorized 14319 / 14511 1.1 910.4 1.0X
+Parquet Vectorized (Pushdown) 14433 / 14610 1.1 917.6 1.0X
+Native ORC Vectorized 11961 / 12449 1.3 760.4 1.2X
+Native ORC Vectorized (Pushdown) 12243 / 12382 1.3 778.4 1.2X
+InMemoryTable Vectorized 7885 / 8006 2.0 501.3 1.8X
+InMemoryTable Vectorized (Pushdown) 7776 / 7909 2.0 494.4 1.8X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value > -1): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19036 / 19052 0.8 1210.3 1.0X
-Parquet Vectorized (Pushdown) 19287 / 19306 0.8 1226.2 1.0X
-Native ORC Vectorized 15311 / 15371 1.0 973.5 1.2X
-Native ORC Vectorized (Pushdown) 15517 / 15590 1.0 986.5 1.2X
+Parquet Vectorized 14275 / 14442 1.1 907.6 1.0X
+Parquet Vectorized (Pushdown) 14291 / 14586 1.1 908.6 1.0X
+Native ORC Vectorized 11957 / 12244 1.3 760.2 1.2X
+Native ORC Vectorized (Pushdown) 11871 / 12044 1.3 754.7 1.2X
+InMemoryTable Vectorized 9404 / 9540 1.7 597.9 1.5X
+InMemoryTable Vectorized (Pushdown) 9429 / 9558 1.7 599.5 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value != -1): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19072 / 19102 0.8 1212.6 1.0X
-Parquet Vectorized (Pushdown) 19288 / 19318 0.8 1226.3 1.0X
-Native ORC Vectorized 15277 / 15293 1.0 971.3 1.2X
-Native ORC Vectorized (Pushdown) 15479 / 15499 1.0 984.1 1.2X
+Parquet Vectorized 13937 / 14019 1.1 886.1 1.0X
+Parquet Vectorized (Pushdown) 14197 / 14239 1.1 902.6 1.0X
+Native ORC Vectorized 12177 / 12524 1.3 774.2 1.1X
+Native ORC Vectorized (Pushdown) 11866 / 12013 1.3 754.4 1.2X
+InMemoryTable Vectorized 9194 / 9383 1.7 584.6 1.5X
+InMemoryTable Vectorized (Pushdown) 9240 / 9410 1.7 587.5 1.5X
================================================================================================
Pushdown for few distinct value case (use dictionary encoding)
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 distinct string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10250 / 10274 1.5 651.7 1.0X
-Parquet Vectorized (Pushdown) 571 / 576 27.5 36.3 17.9X
-Native ORC Vectorized 8651 / 8660 1.8 550.0 1.2X
-Native ORC Vectorized (Pushdown) 909 / 933 17.3 57.8 11.3X
+Parquet Vectorized 6978 / 7046 2.3 443.7 1.0X
+Parquet Vectorized (Pushdown) 367 / 378 42.9 23.3 19.0X
+Native ORC Vectorized 6320 / 6444 2.5 401.8 1.1X
+Native ORC Vectorized (Pushdown) 972 / 992 16.2 61.8 7.2X
+InMemoryTable Vectorized 4289 / 4365 3.7 272.7 1.6X
+InMemoryTable Vectorized (Pushdown) 1594 / 1665 9.9 101.3 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 distinct string row ('100' < value < '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10420 / 10426 1.5 662.5 1.0X
-Parquet Vectorized (Pushdown) 574 / 579 27.4 36.5 18.2X
-Native ORC Vectorized 8973 / 8982 1.8 570.5 1.2X
-Native ORC Vectorized (Pushdown) 916 / 955 17.2 58.2 11.4X
+Parquet Vectorized 7119 / 7216 2.2 452.6 1.0X
+Parquet Vectorized (Pushdown) 367 / 379 42.9 23.3 19.4X
+Native ORC Vectorized 6449 / 6574 2.4 410.0 1.1X
+Native ORC Vectorized (Pushdown) 945 / 956 16.6 60.1 7.5X
+InMemoryTable Vectorized 4390 / 4538 3.6 279.1 1.6X
+InMemoryTable Vectorized (Pushdown) 1628 / 1647 9.7 103.5 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row (value = '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10428 / 10441 1.5 663.0 1.0X
-Parquet Vectorized (Pushdown) 789 / 809 19.9 50.2 13.2X
-Native ORC Vectorized 9042 / 9055 1.7 574.9 1.2X
-Native ORC Vectorized (Pushdown) 1130 / 1145 13.9 71.8 9.2X
+Parquet Vectorized 7027 / 7252 2.2 446.8 1.0X
+Parquet Vectorized (Pushdown) 536 / 557 29.4 34.1 13.1X
+Native ORC Vectorized 6569 / 6818 2.4 417.7 1.1X
+Native ORC Vectorized (Pushdown) 1141 / 1187 13.8 72.6 6.2X
+InMemoryTable Vectorized 4557 / 4650 3.5 289.8 1.5X
+InMemoryTable Vectorized (Pushdown) 1754 / 1785 9.0 111.5 4.0X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row (value <=> '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10402 / 10416 1.5 661.3 1.0X
-Parquet Vectorized (Pushdown) 791 / 806 19.9 50.3 13.2X
-Native ORC Vectorized 9042 / 9055 1.7 574.9 1.2X
-Native ORC Vectorized (Pushdown) 1112 / 1145 14.1 70.7 9.4X
+Parquet Vectorized 7200 / 7303 2.2 457.7 1.0X
+Parquet Vectorized (Pushdown) 530 / 551 29.7 33.7 13.6X
+Native ORC Vectorized 6563 / 6667 2.4 417.2 1.1X
+Native ORC Vectorized (Pushdown) 1128 / 1144 13.9 71.7 6.4X
+InMemoryTable Vectorized 4561 / 4646 3.4 290.0 1.6X
+InMemoryTable Vectorized (Pushdown) 1731 / 1792 9.1 110.1 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row ('100' <= value <= '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10548 / 10563 1.5 670.6 1.0X
-Parquet Vectorized (Pushdown) 790 / 796 19.9 50.2 13.4X
-Native ORC Vectorized 9144 / 9153 1.7 581.3 1.2X
-Native ORC Vectorized (Pushdown) 1117 / 1148 14.1 71.0 9.4X
+Parquet Vectorized 7028 / 7212 2.2 446.8 1.0X
+Parquet Vectorized (Pushdown) 539 / 552 29.2 34.3 13.0X
+Native ORC Vectorized 6491 / 6600 2.4 412.7 1.1X
+Native ORC Vectorized (Pushdown) 1102 / 1146 14.3 70.0 6.4X
+InMemoryTable Vectorized 4604 / 4725 3.4 292.7 1.5X
+InMemoryTable Vectorized (Pushdown) 1806 / 1854 8.7 114.8 3.9X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all distinct string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 20445 / 20469 0.8 1299.8 1.0X
-Parquet Vectorized (Pushdown) 20686 / 20699 0.8 1315.2 1.0X
-Native ORC Vectorized 18851 / 18953 0.8 1198.5 1.1X
-Native ORC Vectorized (Pushdown) 19255 / 19268 0.8 1224.2 1.1X
+Parquet Vectorized 15379 / 15520 1.0 977.8 1.0X
+Parquet Vectorized (Pushdown) 15319 / 15503 1.0 973.9 1.0X
+Native ORC Vectorized 14291 / 14421 1.1 908.6 1.1X
+Native ORC Vectorized (Pushdown) 14542 / 14769 1.1 924.6 1.1X
+InMemoryTable Vectorized 11200 / 11431 1.4 712.1 1.4X
+InMemoryTable Vectorized (Pushdown) 11262 / 11429 1.4 716.0 1.4X
================================================================================================
Pushdown benchmark for StringStartsWith
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '10%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14265 / 15213 1.1 907.0 1.0X
-Parquet Vectorized (Pushdown) 4228 / 4870 3.7 268.8 3.4X
-Native ORC Vectorized 10116 / 10977 1.6 643.2 1.4X
-Native ORC Vectorized (Pushdown) 10653 / 11376 1.5 677.3 1.3X
+Parquet Vectorized 9439 / 10391 1.7 600.1 1.0X
+Parquet Vectorized (Pushdown) 2669 / 3183 5.9 169.7 3.5X
+Native ORC Vectorized 7250 / 9963 2.2 460.9 1.3X
+Native ORC Vectorized (Pushdown) 7493 / 8724 2.1 476.4 1.3X
+InMemoryTable Vectorized 6214 / 6641 2.5 395.1 1.5X
+InMemoryTable Vectorized (Pushdown) 6303 / 6775 2.5 400.8 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '1000%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11499 / 11539 1.4 731.1 1.0X
-Parquet Vectorized (Pushdown) 669 / 672 23.5 42.5 17.2X
-Native ORC Vectorized 7343 / 7363 2.1 466.8 1.6X
-Native ORC Vectorized (Pushdown) 7559 / 7568 2.1 480.6 1.5X
+Parquet Vectorized 7906 / 8069 2.0 502.7 1.0X
+Parquet Vectorized (Pushdown) 439 / 440 35.9 27.9 18.0X
+Native ORC Vectorized 5484 / 5697 2.9 348.7 1.4X
+Native ORC Vectorized (Pushdown) 5507 / 5634 2.9 350.1 1.4X
+InMemoryTable Vectorized 4286 / 4382 3.7 272.5 1.8X
+InMemoryTable Vectorized (Pushdown) 4236 / 4339 3.7 269.3 1.9X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '786432%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11463 / 11468 1.4 728.8 1.0X
-Parquet Vectorized (Pushdown) 647 / 651 24.3 41.1 17.7X
-Native ORC Vectorized 7322 / 7338 2.1 465.5 1.6X
-Native ORC Vectorized (Pushdown) 7533 / 7544 2.1 478.9 1.5X
+Parquet Vectorized 7775 / 7965 2.0 494.3 1.0X
+Parquet Vectorized (Pushdown) 414 / 441 38.0 26.3 18.8X
+Native ORC Vectorized 5366 / 5595 2.9 341.1 1.4X
+Native ORC Vectorized (Pushdown) 5413 / 5615 2.9 344.2 1.4X
+InMemoryTable Vectorized 4320 / 4416 3.6 274.6 1.8X
+InMemoryTable Vectorized (Pushdown) 4260 / 4341 3.7 270.9 1.8X
================================================================================================
Pushdown benchmark for decimal
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(9, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 5543 / 5564 2.8 352.4 1.0X
-Parquet Vectorized (Pushdown) 168 / 174 93.7 10.7 33.0X
-Native ORC Vectorized 4992 / 5052 3.2 317.4 1.1X
-Native ORC Vectorized (Pushdown) 840 / 850 18.7 53.4 6.6X
+Parquet Vectorized 3661 / 3785 4.3 232.7 1.0X
+Parquet Vectorized (Pushdown) 107 / 113 146.8 6.8 34.2X
+Native ORC Vectorized 3480 / 3528 4.5 221.3 1.1X
+Native ORC Vectorized (Pushdown) 614 / 628 25.6 39.0 6.0X
+InMemoryTable Vectorized 2915 / 3008 5.4 185.3 1.3X
+InMemoryTable Vectorized (Pushdown) 874 / 919 18.0 55.6 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(9, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7312 / 7358 2.2 464.9 1.0X
-Parquet Vectorized (Pushdown) 3008 / 3078 5.2 191.2 2.4X
-Native ORC Vectorized 6775 / 6798 2.3 430.7 1.1X
-Native ORC Vectorized (Pushdown) 6819 / 6832 2.3 433.5 1.1X
+Parquet Vectorized 5167 / 5304 3.0 328.5 1.0X
+Parquet Vectorized (Pushdown) 2232 / 2295 7.0 141.9 2.3X
+Native ORC Vectorized 4883 / 4999 3.2 310.5 1.1X
+Native ORC Vectorized (Pushdown) 4982 / 5045 3.2 316.7 1.0X
+InMemoryTable Vectorized 3980 / 4111 4.0 253.0 1.3X
+InMemoryTable Vectorized (Pushdown) 2385 / 2463 6.6 151.6 2.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(9, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 13232 / 13241 1.2 841.3 1.0X
-Parquet Vectorized (Pushdown) 12555 / 12569 1.3 798.2 1.1X
-Native ORC Vectorized 12597 / 12627 1.2 800.9 1.1X
-Native ORC Vectorized (Pushdown) 12677 / 12711 1.2 806.0 1.0X
+Parquet Vectorized 9996 / 10052 1.6 635.5 1.0X
+Parquet Vectorized (Pushdown) 9331 / 9501 1.7 593.2 1.1X
+Native ORC Vectorized 9632 / 9770 1.6 612.4 1.0X
+Native ORC Vectorized (Pushdown) 9700 / 9788 1.6 616.7 1.0X
+InMemoryTable Vectorized 7858 / 7999 2.0 499.6 1.3X
+InMemoryTable Vectorized (Pushdown) 7506 / 7565 2.1 477.2 1.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(9, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14725 / 14729 1.1 936.2 1.0X
-Parquet Vectorized (Pushdown) 14781 / 14800 1.1 939.7 1.0X
-Native ORC Vectorized 15360 / 15453 1.0 976.5 1.0X
-Native ORC Vectorized (Pushdown) 15444 / 15466 1.0 981.9 1.0X
+Parquet Vectorized 10776 / 10969 1.5 685.1 1.0X
+Parquet Vectorized (Pushdown) 10931 / 11012 1.4 695.0 1.0X
+Native ORC Vectorized 10849 / 10910 1.4 689.7 1.0X
+Native ORC Vectorized (Pushdown) 11030 / 11112 1.4 701.2 1.0X
+InMemoryTable Vectorized 8675 / 8822 1.8 551.6 1.2X
+InMemoryTable Vectorized (Pushdown) 8668 / 8741 1.8 551.1 1.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(18, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 5746 / 5763 2.7 365.3 1.0X
-Parquet Vectorized (Pushdown) 166 / 169 94.8 10.6 34.6X
-Native ORC Vectorized 5007 / 5023 3.1 318.3 1.1X
-Native ORC Vectorized (Pushdown) 2629 / 2640 6.0 167.1 2.2X
+Parquet Vectorized 3886 / 3932 4.0 247.0 1.0X
+Parquet Vectorized (Pushdown) 104 / 115 151.0 6.6 37.3X
+Native ORC Vectorized 3476 / 3527 4.5 221.0 1.1X
+Native ORC Vectorized (Pushdown) 1804 / 1839 8.7 114.7 2.2X
+InMemoryTable Vectorized 2956 / 3002 5.3 187.9 1.3X
+InMemoryTable Vectorized (Pushdown) 865 / 941 18.2 55.0 4.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(18, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 6827 / 6864 2.3 434.0 1.0X
-Parquet Vectorized (Pushdown) 1809 / 1827 8.7 115.0 3.8X
-Native ORC Vectorized 6287 / 6296 2.5 399.7 1.1X
-Native ORC Vectorized (Pushdown) 6364 / 6377 2.5 404.6 1.1X
+Parquet Vectorized 4498 / 4642 3.5 286.0 1.0X
+Parquet Vectorized (Pushdown) 1201 / 1242 13.1 76.3 3.7X
+Native ORC Vectorized 4218 / 4291 3.7 268.1 1.1X
+Native ORC Vectorized (Pushdown) 4262 / 4311 3.7 270.9 1.1X
+InMemoryTable Vectorized 3509 / 3599 4.5 223.1 1.3X
+InMemoryTable Vectorized (Pushdown) 1640 / 1702 9.6 104.3 2.7X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(18, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11315 / 11342 1.4 719.4 1.0X
-Parquet Vectorized (Pushdown) 8431 / 8450 1.9 536.0 1.3X
-Native ORC Vectorized 11591 / 11611 1.4 736.9 1.0X
-Native ORC Vectorized (Pushdown) 11424 / 11475 1.4 726.3 1.0X
+Parquet Vectorized 7445 / 7525 2.1 473.3 1.0X
+Parquet Vectorized (Pushdown) 5609 / 5732 2.8 356.6 1.3X
+Native ORC Vectorized 7056 / 7187 2.2 448.6 1.1X
+Native ORC Vectorized (Pushdown) 7226 / 7330 2.2 459.4 1.0X
+InMemoryTable Vectorized 5707 / 5801 2.8 362.8 1.3X
+InMemoryTable Vectorized (Pushdown) 4694 / 4810 3.4 298.4 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(18, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 15703 / 15712 1.0 998.4 1.0X
-Parquet Vectorized (Pushdown) 14982 / 15009 1.0 952.5 1.0X
-Native ORC Vectorized 16887 / 16955 0.9 1073.7 0.9X
-Native ORC Vectorized (Pushdown) 16518 / 16530 1.0 1050.2 1.0X
+Parquet Vectorized 10324 / 10427 1.5 656.4 1.0X
+Parquet Vectorized (Pushdown) 9916 / 10093 1.6 630.4 1.0X
+Native ORC Vectorized 10422 / 10539 1.5 662.6 1.0X
+Native ORC Vectorized (Pushdown) 10099 / 10261 1.6 642.1 1.0X
+InMemoryTable Vectorized 8220 / 8284 1.9 522.6 1.3X
+InMemoryTable Vectorized (Pushdown) 7809 / 7963 2.0 496.5 1.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(38, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 8101 / 8130 1.9 515.1 1.0X
-Parquet Vectorized (Pushdown) 184 / 187 85.6 11.7 44.1X
-Native ORC Vectorized 4998 / 5027 3.1 317.8 1.6X
-Native ORC Vectorized (Pushdown) 165 / 168 95.6 10.5 49.2X
+Parquet Vectorized 5346 / 5526 2.9 339.9 1.0X
+Parquet Vectorized (Pushdown) 116 / 121 136.2 7.3 46.3X
+Native ORC Vectorized 3483 / 3551 4.5 221.4 1.5X
+Native ORC Vectorized (Pushdown) 152 / 166 103.7 9.6 35.2X
+InMemoryTable Vectorized 4312 / 4344 3.6 274.2 1.2X
+InMemoryTable Vectorized (Pushdown) 878 / 930 17.9 55.8 6.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(38, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 9405 / 9447 1.7 597.9 1.0X
-Parquet Vectorized (Pushdown) 2269 / 2275 6.9 144.2 4.1X
-Native ORC Vectorized 6167 / 6203 2.6 392.1 1.5X
-Native ORC Vectorized (Pushdown) 1783 / 1787 8.8 113.3 5.3X
+Parquet Vectorized 6229 / 6430 2.5 396.0 1.0X
+Parquet Vectorized (Pushdown) 1530 / 1560 10.3 97.3 4.1X
+Native ORC Vectorized 4407 / 4469 3.6 280.2 1.4X
+Native ORC Vectorized (Pushdown) 1361 / 1399 11.6 86.5 4.6X
+InMemoryTable Vectorized 5006 / 5129 3.1 318.3 1.2X
+InMemoryTable Vectorized (Pushdown) 1995 / 2037 7.9 126.8 3.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(38, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14700 / 14707 1.1 934.6 1.0X
-Parquet Vectorized (Pushdown) 10699 / 10712 1.5 680.2 1.4X
-Native ORC Vectorized 10687 / 10703 1.5 679.5 1.4X
-Native ORC Vectorized (Pushdown) 8364 / 8415 1.9 531.8 1.8X
+Parquet Vectorized 10057 / 10179 1.6 639.4 1.0X
+Parquet Vectorized (Pushdown) 7411 / 7518 2.1 471.2 1.4X
+Native ORC Vectorized 7911 / 8161 2.0 503.0 1.3X
+Native ORC Vectorized (Pushdown) 6199 / 6354 2.5 394.1 1.6X
+InMemoryTable Vectorized 8000 / 8128 2.0 508.6 1.3X
+InMemoryTable Vectorized (Pushdown) 6330 / 6413 2.5 402.4 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(38, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19780 / 19894 0.8 1257.6 1.0X
-Parquet Vectorized (Pushdown) 19003 / 19025 0.8 1208.1 1.0X
-Native ORC Vectorized 15385 / 15404 1.0 978.2 1.3X
-Native ORC Vectorized (Pushdown) 15032 / 15060 1.0 955.7 1.3X
+Parquet Vectorized 13756 / 13841 1.1 874.6 1.0X
+Parquet Vectorized (Pushdown) 12922 / 13129 1.2 821.6 1.1X
+Native ORC Vectorized 11386 / 11548 1.4 723.9 1.2X
+Native ORC Vectorized (Pushdown) 11170 / 11345 1.4 710.1 1.2X
+InMemoryTable Vectorized 11079 / 11194 1.4 704.4 1.2X
+InMemoryTable Vectorized (Pushdown) 10712 / 10793 1.5 681.0 1.3X
================================================================================================
Pushdown benchmark for InSet -> InFilters
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10521 / 10534 1.5 668.9 1.0X
-Parquet Vectorized (Pushdown) 677 / 691 23.2 43.1 15.5X
-Native ORC Vectorized 6768 / 6776 2.3 430.3 1.6X
-Native ORC Vectorized (Pushdown) 501 / 512 31.4 31.8 21.0X
+Parquet Vectorized 7187 / 7325 2.2 456.9 1.0X
+Parquet Vectorized (Pushdown) 447 / 460 35.2 28.4 16.1X
+Native ORC Vectorized 4850 / 4976 3.2 308.3 1.5X
+Native ORC Vectorized (Pushdown) 508 / 527 30.9 32.3 14.1X
+InMemoryTable Vectorized 3758 / 3828 4.2 238.9 1.9X
+InMemoryTable Vectorized (Pushdown) 1704 / 1761 9.2 108.4 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 50): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10531 / 10538 1.5 669.5 1.0X
-Parquet Vectorized (Pushdown) 677 / 718 23.2 43.0 15.6X
-Native ORC Vectorized 6765 / 6773 2.3 430.1 1.6X
-Native ORC Vectorized (Pushdown) 499 / 507 31.5 31.7 21.1X
+Parquet Vectorized 7145 / 7191 2.2 454.3 1.0X
+Parquet Vectorized (Pushdown) 442 / 461 35.6 28.1 16.2X
+Native ORC Vectorized 4911 / 5138 3.2 312.2 1.5X
+Native ORC Vectorized (Pushdown) 503 / 522 31.3 32.0 14.2X
+InMemoryTable Vectorized 3693 / 3781 4.3 234.8 1.9X
+InMemoryTable Vectorized (Pushdown) 1647 / 1733 9.6 104.7 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 90): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10540 / 10553 1.5 670.1 1.0X
-Parquet Vectorized (Pushdown) 678 / 710 23.2 43.1 15.5X
-Native ORC Vectorized 6787 / 6794 2.3 431.5 1.6X
-Native ORC Vectorized (Pushdown) 501 / 509 31.4 31.9 21.0X
+Parquet Vectorized 7216 / 7350 2.2 458.8 1.0X
+Parquet Vectorized (Pushdown) 430 / 451 36.6 27.3 16.8X
+Native ORC Vectorized 4900 / 5100 3.2 311.5 1.5X
+Native ORC Vectorized (Pushdown) 520 / 542 30.2 33.1 13.9X
+InMemoryTable Vectorized 3639 / 3689 4.3 231.4 2.0X
+InMemoryTable Vectorized (Pushdown) 1647 / 1732 9.5 104.7 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10551 / 10559 1.5 670.8 1.0X
-Parquet Vectorized (Pushdown) 703 / 708 22.4 44.7 15.0X
-Native ORC Vectorized 6791 / 6802 2.3 431.7 1.6X
-Native ORC Vectorized (Pushdown) 519 / 526 30.3 33.0 20.3X
+Parquet Vectorized 7195 / 7290 2.2 457.4 1.0X
+Parquet Vectorized (Pushdown) 446 / 478 35.3 28.3 16.1X
+Native ORC Vectorized 4976 / 5229 3.2 316.4 1.4X
+Native ORC Vectorized (Pushdown) 520 / 543 30.2 33.1 13.8X
+InMemoryTable Vectorized 3635 / 3744 4.3 231.1 2.0X
+InMemoryTable Vectorized (Pushdown) 1684 / 1766 9.3 107.1 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 50): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10561 / 10565 1.5 671.4 1.0X
-Parquet Vectorized (Pushdown) 711 / 716 22.1 45.2 14.9X
-Native ORC Vectorized 6791 / 6806 2.3 431.8 1.6X
-Native ORC Vectorized (Pushdown) 529 / 537 29.8 33.6 20.0X
+Parquet Vectorized 7246 / 7375 2.2 460.7 1.0X
+Parquet Vectorized (Pushdown) 477 / 496 33.0 30.3 15.2X
+Native ORC Vectorized 4828 / 5158 3.3 307.0 1.5X
+Native ORC Vectorized (Pushdown) 524 / 540 30.0 33.3 13.8X
+InMemoryTable Vectorized 3695 / 3785 4.3 234.9 2.0X
+InMemoryTable Vectorized (Pushdown) 1702 / 1745 9.2 108.2 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 90): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10572 / 10590 1.5 672.1 1.0X
-Parquet Vectorized (Pushdown) 713 / 716 22.1 45.3 14.8X
-Native ORC Vectorized 6808 / 6815 2.3 432.9 1.6X
-Native ORC Vectorized (Pushdown) 530 / 541 29.7 33.7 19.9X
+Parquet Vectorized 7303 / 7388 2.2 464.3 1.0X
+Parquet Vectorized (Pushdown) 471 / 477 33.4 29.9 15.5X
+Native ORC Vectorized 4848 / 5005 3.2 308.2 1.5X
+Native ORC Vectorized (Pushdown) 518 / 550 30.4 32.9 14.1X
+InMemoryTable Vectorized 3665 / 3790 4.3 233.0 2.0X
+InMemoryTable Vectorized (Pushdown) 1689 / 1717 9.3 107.4 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 50, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10871 / 10882 1.4 691.2 1.0X
-Parquet Vectorized (Pushdown) 11104 / 11110 1.4 706.0 1.0X
-Native ORC Vectorized 7088 / 7104 2.2 450.7 1.5X
-Native ORC Vectorized (Pushdown) 665 / 677 23.6 42.3 16.3X
+Parquet Vectorized 7474 / 7577 2.1 475.2 1.0X
+Parquet Vectorized (Pushdown) 7649 / 7707 2.1 486.3 1.0X
+Native ORC Vectorized 5167 / 5457 3.0 328.5 1.4X
+Native ORC Vectorized (Pushdown) 637
--- End diff --
I don't know yet, but I am very interested in this case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #99007 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99007/testReport)** for PR 23027 at commit [`b8c54ea`](https://github.com/apache/spark/commit/b8c54ea5048524f7df0b750a11a8fb109b43f479).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #99104 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99104/testReport)** for PR 23027 at commit [`b8c54ea`](https://github.com/apache/spark/commit/b8c54ea5048524f7df0b750a11a8fb109b43f479).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23027
Looks fine. @dongjoon-hyun WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r234521489
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala ---
@@ -104,6 +107,10 @@ object FilterPushdownBenchmark extends BenchmarkBase with SQLHelper {
df.write.mode("overwrite")
.option("parquet.block.size", blockSize).parquet(parquetPath)
spark.read.parquet(parquetPath).createOrReplaceTempView("parquetTable")
+
+ df.write.mode("overwrite").save(inMemoryTablePath)
--- End diff --
Cache `inMemoryTable` from file to avoid the performance issue: https://github.com/apache/spark/pull/23027#pullrequestreview-175054485
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r233687968
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X
+Parquet Vectorized 7823 / 7996 2.0 497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X
+InMemoryTable Vectorized 6 / 6 2859.1 0.3 1422.0X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3023.0 0.3 1503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X
--- End diff --
Yes. This is the current benchmark result. I plan to improve it step by step. Example: [SPARK-26004](https://issues.apache.org/jira/browse/SPARK-26004)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum closed the pull request at:
https://github.com/apache/spark/pull/23027
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99104/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r233600662
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X
+Parquet Vectorized 7823 / 7996 2.0 497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X
+InMemoryTable Vectorized 6 / 6 2859.1 0.3 1422.0X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3023.0 0.3 1503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row (value = '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11878 / 11888 1.3 755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0 40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1 466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3 33.0 22.9X
+Parquet Vectorized 8322 / 8386 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 434 / 441 36.2 27.6 19.2X
+Native ORC Vectorized 5659 / 5944 2.8 359.8 1.5X
+Native ORC Vectorized (Pushdown) 535 / 567 29.4 34.0 15.6X
+InMemoryTable Vectorized 4784 / 4879 3.3 304.1 1.7X
+InMemoryTable Vectorized (Pushdown) 1950 / 1985 8.1 124.0 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row (value <=> '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11423 / 11440 1.4 726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2 39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2 465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0 32.2 22.5X
+Parquet Vectorized 8377 / 8458 1.9 532.6 1.0X
+Parquet Vectorized (Pushdown) 449 / 457 35.1 28.5 18.7X
+Native ORC Vectorized 5664 / 5786 2.8 360.1 1.5X
+Native ORC Vectorized (Pushdown) 527 / 560 29.9 33.5 15.9X
+InMemoryTable Vectorized 4743 / 4813 3.3 301.5 1.8X
+InMemoryTable Vectorized (Pushdown) 1876 / 1963 8.4 119.3 4.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11440 / 11478 1.4 727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8 40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2 464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4 32.8 22.1X
+Parquet Vectorized 8120 / 8293 1.9 516.2 1.0X
+Parquet Vectorized (Pushdown) 423 / 436 37.2 26.9 19.2X
+Native ORC Vectorized 5490 / 5833 2.9 349.0 1.5X
+Native ORC Vectorized (Pushdown) 538 / 562 29.3 34.2 15.1X
+InMemoryTable Vectorized 4722 / 4774 3.3 300.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1923 / 1954 8.2 122.2 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 20750 / 20872 0.8 1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7 1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9 1062.6 1.2X
-Native ORC Vectorized (Pushdown) 16926 / 16965 0.9 1076.1 1.2X
+Parquet Vectorized 15437 / 15682 1.0 981.4 1.0X
+Parquet Vectorized (Pushdown) 15748 / 15811 1.0 1001.2 1.0X
+Native ORC Vectorized 13111 / 13245 1.2 833.6 1.2X
+Native ORC Vectorized (Pushdown) 12931 / 13216 1.2 822.1 1.2X
+InMemoryTable Vectorized 9661 / 9908 1.6 614.2 1.6X
+InMemoryTable Vectorized (Pushdown) 9838 / 9908 1.6 625.5 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 int row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10510 / 10532 1.5 668.2 1.0X
-Parquet Vectorized (Pushdown) 642 / 665 24.5 40.8 16.4X
-Native ORC Vectorized 6609 / 6618 2.4 420.2 1.6X
-Native ORC Vectorized (Pushdown) 502 / 512 31.4 31.9 21.0X
+Parquet Vectorized 7549 / 7625 2.1 479.9 1.0X
+Parquet Vectorized (Pushdown) 427 / 450 36.9 27.1 17.7X
+Native ORC Vectorized 5028 / 5221 3.1 319.7 1.5X
+Native ORC Vectorized (Pushdown) 530 / 548 29.7 33.7 14.3X
+InMemoryTable Vectorized 5 / 6 3221.5 0.3 1546.1X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3303.5 0.3 1585.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 int row (7864320 < value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10505 / 10514 1.5 667.9 1.0X
-Parquet Vectorized (Pushdown) 659 / 673 23.9 41.9 15.9X
-Native ORC Vectorized 6634 / 6641 2.4 421.8 1.6X
-Native ORC Vectorized (Pushdown) 513 / 526 30.7 32.6 20.5X
+Parquet Vectorized 7542 / 7584 2.1 479.5 1.0X
+Parquet Vectorized (Pushdown) 458 / 468 34.3 29.1 16.5X
+Native ORC Vectorized 5035 / 5533 3.1 320.1 1.5X
+Native ORC Vectorized (Pushdown) 538 / 560 29.2 34.2 14.0X
+InMemoryTable Vectorized 3866 / 3925 4.1 245.8 2.0X
+InMemoryTable Vectorized (Pushdown) 1766 / 1792 8.9 112.3 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10555 / 10570 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 651 / 668 24.2 41.4 16.2X
-Native ORC Vectorized 6721 / 6728 2.3 427.3 1.6X
-Native ORC Vectorized (Pushdown) 508 / 519 31.0 32.3 20.8X
+Parquet Vectorized 7540 / 7660 2.1 479.3 1.0X
+Parquet Vectorized (Pushdown) 444 / 454 35.5 28.2 17.0X
+Native ORC Vectorized 5155 / 5310 3.1 327.7 1.5X
+Native ORC Vectorized (Pushdown) 524 / 541 30.0 33.3 14.4X
+InMemoryTable Vectorized 3910 / 3999 4.0 248.6 1.9X
+InMemoryTable Vectorized (Pushdown) 1843 / 1868 8.5 117.2 4.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (value <=> 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10566 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 647 / 654 24.3 41.1 16.3X
-Native ORC Vectorized 6716 / 6728 2.3 427.0 1.6X
-Native ORC Vectorized (Pushdown) 510 / 521 30.9 32.4 20.7X
+Parquet Vectorized 7568 / 7679 2.1 481.1 1.0X
+Parquet Vectorized (Pushdown) 446 / 460 35.2 28.4 17.0X
+Native ORC Vectorized 5083 / 5244 3.1 323.2 1.5X
+Native ORC Vectorized (Pushdown) 529 / 542 29.7 33.6 14.3X
+InMemoryTable Vectorized 4033 / 4068 3.9 256.4 1.9X
+InMemoryTable Vectorized (Pushdown) 1818 / 1848 8.6 115.6 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (7864320 <= value <= 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10565 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 649 / 654 24.2 41.3 16.3X
-Native ORC Vectorized 6700 / 6712 2.3 426.0 1.6X
-Native ORC Vectorized (Pushdown) 509 / 520 30.9 32.3 20.8X
+Parquet Vectorized 7446 / 7622 2.1 473.4 1.0X
+Parquet Vectorized (Pushdown) 440 / 455 35.8 28.0 16.9X
+Native ORC Vectorized 5155 / 5347 3.1 327.8 1.4X
+Native ORC Vectorized (Pushdown) 509 / 538 30.9 32.3 14.6X
+InMemoryTable Vectorized 3952 / 3977 4.0 251.3 1.9X
+InMemoryTable Vectorized (Pushdown) 1832 / 1887 8.6 116.5 4.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (7864319 < value < 7864321): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10547 / 10566 1.5 670.5 1.0X
-Parquet Vectorized (Pushdown) 649 / 653 24.2 41.3 16.3X
-Native ORC Vectorized 6703 / 6713 2.3 426.2 1.6X
-Native ORC Vectorized (Pushdown) 510 / 520 30.8 32.5 20.7X
+Parquet Vectorized 7576 / 7711 2.1 481.7 1.0X
+Parquet Vectorized (Pushdown) 467 / 477 33.7 29.7 16.2X
+Native ORC Vectorized 5147 / 5412 3.1 327.3 1.5X
+Native ORC Vectorized (Pushdown) 521 / 551 30.2 33.1 14.6X
+InMemoryTable Vectorized 3882 / 3964 4.1 246.8 2.0X
+InMemoryTable Vectorized (Pushdown) 1805 / 1845 8.7 114.7 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% int rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11478 / 11525 1.4 729.7 1.0X
-Parquet Vectorized (Pushdown) 2576 / 2587 6.1 163.8 4.5X
-Native ORC Vectorized 7633 / 7657 2.1 485.3 1.5X
-Native ORC Vectorized (Pushdown) 2076 / 2096 7.6 132.0 5.5X
+Parquet Vectorized 8439 / 8541 1.9 536.5 1.0X
+Parquet Vectorized (Pushdown) 1952 / 1993 8.1 124.1 4.3X
+Native ORC Vectorized 5864 / 6092 2.7 372.8 1.4X
+Native ORC Vectorized (Pushdown) 1786 / 1805 8.8 113.6 4.7X
+InMemoryTable Vectorized 4463 / 4527 3.5 283.8 1.9X
+InMemoryTable Vectorized (Pushdown) 2559 / 2596 6.1 162.7 3.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% int rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14785 / 14802 1.1 940.0 1.0X
-Parquet Vectorized (Pushdown) 9971 / 9977 1.6 633.9 1.5X
-Native ORC Vectorized 11082 / 11107 1.4 704.6 1.3X
-Native ORC Vectorized (Pushdown) 8061 / 8073 2.0 512.5 1.8X
+Parquet Vectorized 11106 / 11248 1.4 706.1 1.0X
+Parquet Vectorized (Pushdown) 7656 / 7697 2.1 486.7 1.5X
+Native ORC Vectorized 8593 / 8892 1.8 546.4 1.3X
+Native ORC Vectorized (Pushdown) 6474 / 6567 2.4 411.6 1.7X
+InMemoryTable Vectorized 6623 / 6731 2.4 421.1 1.7X
+InMemoryTable Vectorized (Pushdown) 5613 / 5747 2.8 356.9 2.0X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% int rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 18174 / 18214 0.9 1155.5 1.0X
-Parquet Vectorized (Pushdown) 17387 / 17403 0.9 1105.5 1.0X
-Native ORC Vectorized 14465 / 14492 1.1 919.7 1.3X
-Native ORC Vectorized (Pushdown) 14024 / 14041 1.1 891.6 1.3X
+Parquet Vectorized 13716 / 13810 1.1 872.1 1.0X
+Parquet Vectorized (Pushdown) 13044 / 13233 1.2 829.3 1.1X
+Native ORC Vectorized 11795 / 11899 1.3 749.9 1.2X
+Native ORC Vectorized (Pushdown) 11074 / 11265 1.4 704.1 1.2X
+InMemoryTable Vectorized 9023 / 9166 1.7 573.7 1.5X
+InMemoryTable Vectorized (Pushdown) 8929 / 9015 1.8 567.7 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19004 / 19014 0.8 1208.2 1.0X
-Parquet Vectorized (Pushdown) 19219 / 19232 0.8 1221.9 1.0X
-Native ORC Vectorized 15266 / 15290 1.0 970.6 1.2X
-Native ORC Vectorized (Pushdown) 15469 / 15482 1.0 983.5 1.2X
+Parquet Vectorized 14319 / 14511 1.1 910.4 1.0X
+Parquet Vectorized (Pushdown) 14433 / 14610 1.1 917.6 1.0X
+Native ORC Vectorized 11961 / 12449 1.3 760.4 1.2X
+Native ORC Vectorized (Pushdown) 12243 / 12382 1.3 778.4 1.2X
+InMemoryTable Vectorized 7885 / 8006 2.0 501.3 1.8X
+InMemoryTable Vectorized (Pushdown) 7776 / 7909 2.0 494.4 1.8X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value > -1): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19036 / 19052 0.8 1210.3 1.0X
-Parquet Vectorized (Pushdown) 19287 / 19306 0.8 1226.2 1.0X
-Native ORC Vectorized 15311 / 15371 1.0 973.5 1.2X
-Native ORC Vectorized (Pushdown) 15517 / 15590 1.0 986.5 1.2X
+Parquet Vectorized 14275 / 14442 1.1 907.6 1.0X
+Parquet Vectorized (Pushdown) 14291 / 14586 1.1 908.6 1.0X
+Native ORC Vectorized 11957 / 12244 1.3 760.2 1.2X
+Native ORC Vectorized (Pushdown) 11871 / 12044 1.3 754.7 1.2X
+InMemoryTable Vectorized 9404 / 9540 1.7 597.9 1.5X
+InMemoryTable Vectorized (Pushdown) 9429 / 9558 1.7 599.5 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value != -1): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19072 / 19102 0.8 1212.6 1.0X
-Parquet Vectorized (Pushdown) 19288 / 19318 0.8 1226.3 1.0X
-Native ORC Vectorized 15277 / 15293 1.0 971.3 1.2X
-Native ORC Vectorized (Pushdown) 15479 / 15499 1.0 984.1 1.2X
+Parquet Vectorized 13937 / 14019 1.1 886.1 1.0X
+Parquet Vectorized (Pushdown) 14197 / 14239 1.1 902.6 1.0X
+Native ORC Vectorized 12177 / 12524 1.3 774.2 1.1X
+Native ORC Vectorized (Pushdown) 11866 / 12013 1.3 754.4 1.2X
+InMemoryTable Vectorized 9194 / 9383 1.7 584.6 1.5X
+InMemoryTable Vectorized (Pushdown) 9240 / 9410 1.7 587.5 1.5X
================================================================================================
Pushdown for few distinct value case (use dictionary encoding)
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 distinct string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10250 / 10274 1.5 651.7 1.0X
-Parquet Vectorized (Pushdown) 571 / 576 27.5 36.3 17.9X
-Native ORC Vectorized 8651 / 8660 1.8 550.0 1.2X
-Native ORC Vectorized (Pushdown) 909 / 933 17.3 57.8 11.3X
+Parquet Vectorized 6978 / 7046 2.3 443.7 1.0X
+Parquet Vectorized (Pushdown) 367 / 378 42.9 23.3 19.0X
+Native ORC Vectorized 6320 / 6444 2.5 401.8 1.1X
+Native ORC Vectorized (Pushdown) 972 / 992 16.2 61.8 7.2X
+InMemoryTable Vectorized 4289 / 4365 3.7 272.7 1.6X
+InMemoryTable Vectorized (Pushdown) 1594 / 1665 9.9 101.3 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 distinct string row ('100' < value < '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10420 / 10426 1.5 662.5 1.0X
-Parquet Vectorized (Pushdown) 574 / 579 27.4 36.5 18.2X
-Native ORC Vectorized 8973 / 8982 1.8 570.5 1.2X
-Native ORC Vectorized (Pushdown) 916 / 955 17.2 58.2 11.4X
+Parquet Vectorized 7119 / 7216 2.2 452.6 1.0X
+Parquet Vectorized (Pushdown) 367 / 379 42.9 23.3 19.4X
+Native ORC Vectorized 6449 / 6574 2.4 410.0 1.1X
+Native ORC Vectorized (Pushdown) 945 / 956 16.6 60.1 7.5X
+InMemoryTable Vectorized 4390 / 4538 3.6 279.1 1.6X
+InMemoryTable Vectorized (Pushdown) 1628 / 1647 9.7 103.5 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row (value = '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10428 / 10441 1.5 663.0 1.0X
-Parquet Vectorized (Pushdown) 789 / 809 19.9 50.2 13.2X
-Native ORC Vectorized 9042 / 9055 1.7 574.9 1.2X
-Native ORC Vectorized (Pushdown) 1130 / 1145 13.9 71.8 9.2X
+Parquet Vectorized 7027 / 7252 2.2 446.8 1.0X
+Parquet Vectorized (Pushdown) 536 / 557 29.4 34.1 13.1X
+Native ORC Vectorized 6569 / 6818 2.4 417.7 1.1X
+Native ORC Vectorized (Pushdown) 1141 / 1187 13.8 72.6 6.2X
+InMemoryTable Vectorized 4557 / 4650 3.5 289.8 1.5X
+InMemoryTable Vectorized (Pushdown) 1754 / 1785 9.0 111.5 4.0X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row (value <=> '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10402 / 10416 1.5 661.3 1.0X
-Parquet Vectorized (Pushdown) 791 / 806 19.9 50.3 13.2X
-Native ORC Vectorized 9042 / 9055 1.7 574.9 1.2X
-Native ORC Vectorized (Pushdown) 1112 / 1145 14.1 70.7 9.4X
+Parquet Vectorized 7200 / 7303 2.2 457.7 1.0X
+Parquet Vectorized (Pushdown) 530 / 551 29.7 33.7 13.6X
+Native ORC Vectorized 6563 / 6667 2.4 417.2 1.1X
+Native ORC Vectorized (Pushdown) 1128 / 1144 13.9 71.7 6.4X
+InMemoryTable Vectorized 4561 / 4646 3.4 290.0 1.6X
+InMemoryTable Vectorized (Pushdown) 1731 / 1792 9.1 110.1 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row ('100' <= value <= '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10548 / 10563 1.5 670.6 1.0X
-Parquet Vectorized (Pushdown) 790 / 796 19.9 50.2 13.4X
-Native ORC Vectorized 9144 / 9153 1.7 581.3 1.2X
-Native ORC Vectorized (Pushdown) 1117 / 1148 14.1 71.0 9.4X
+Parquet Vectorized 7028 / 7212 2.2 446.8 1.0X
+Parquet Vectorized (Pushdown) 539 / 552 29.2 34.3 13.0X
+Native ORC Vectorized 6491 / 6600 2.4 412.7 1.1X
+Native ORC Vectorized (Pushdown) 1102 / 1146 14.3 70.0 6.4X
+InMemoryTable Vectorized 4604 / 4725 3.4 292.7 1.5X
+InMemoryTable Vectorized (Pushdown) 1806 / 1854 8.7 114.8 3.9X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all distinct string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 20445 / 20469 0.8 1299.8 1.0X
-Parquet Vectorized (Pushdown) 20686 / 20699 0.8 1315.2 1.0X
-Native ORC Vectorized 18851 / 18953 0.8 1198.5 1.1X
-Native ORC Vectorized (Pushdown) 19255 / 19268 0.8 1224.2 1.1X
+Parquet Vectorized 15379 / 15520 1.0 977.8 1.0X
+Parquet Vectorized (Pushdown) 15319 / 15503 1.0 973.9 1.0X
+Native ORC Vectorized 14291 / 14421 1.1 908.6 1.1X
+Native ORC Vectorized (Pushdown) 14542 / 14769 1.1 924.6 1.1X
+InMemoryTable Vectorized 11200 / 11431 1.4 712.1 1.4X
+InMemoryTable Vectorized (Pushdown) 11262 / 11429 1.4 716.0 1.4X
================================================================================================
Pushdown benchmark for StringStartsWith
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '10%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14265 / 15213 1.1 907.0 1.0X
-Parquet Vectorized (Pushdown) 4228 / 4870 3.7 268.8 3.4X
-Native ORC Vectorized 10116 / 10977 1.6 643.2 1.4X
-Native ORC Vectorized (Pushdown) 10653 / 11376 1.5 677.3 1.3X
+Parquet Vectorized 9439 / 10391 1.7 600.1 1.0X
+Parquet Vectorized (Pushdown) 2669 / 3183 5.9 169.7 3.5X
+Native ORC Vectorized 7250 / 9963 2.2 460.9 1.3X
+Native ORC Vectorized (Pushdown) 7493 / 8724 2.1 476.4 1.3X
+InMemoryTable Vectorized 6214 / 6641 2.5 395.1 1.5X
+InMemoryTable Vectorized (Pushdown) 6303 / 6775 2.5 400.8 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '1000%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11499 / 11539 1.4 731.1 1.0X
-Parquet Vectorized (Pushdown) 669 / 672 23.5 42.5 17.2X
-Native ORC Vectorized 7343 / 7363 2.1 466.8 1.6X
-Native ORC Vectorized (Pushdown) 7559 / 7568 2.1 480.6 1.5X
+Parquet Vectorized 7906 / 8069 2.0 502.7 1.0X
+Parquet Vectorized (Pushdown) 439 / 440 35.9 27.9 18.0X
+Native ORC Vectorized 5484 / 5697 2.9 348.7 1.4X
+Native ORC Vectorized (Pushdown) 5507 / 5634 2.9 350.1 1.4X
+InMemoryTable Vectorized 4286 / 4382 3.7 272.5 1.8X
+InMemoryTable Vectorized (Pushdown) 4236 / 4339 3.7 269.3 1.9X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '786432%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11463 / 11468 1.4 728.8 1.0X
-Parquet Vectorized (Pushdown) 647 / 651 24.3 41.1 17.7X
-Native ORC Vectorized 7322 / 7338 2.1 465.5 1.6X
-Native ORC Vectorized (Pushdown) 7533 / 7544 2.1 478.9 1.5X
+Parquet Vectorized 7775 / 7965 2.0 494.3 1.0X
+Parquet Vectorized (Pushdown) 414 / 441 38.0 26.3 18.8X
+Native ORC Vectorized 5366 / 5595 2.9 341.1 1.4X
+Native ORC Vectorized (Pushdown) 5413 / 5615 2.9 344.2 1.4X
+InMemoryTable Vectorized 4320 / 4416 3.6 274.6 1.8X
+InMemoryTable Vectorized (Pushdown) 4260 / 4341 3.7 270.9 1.8X
================================================================================================
Pushdown benchmark for decimal
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(9, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 5543 / 5564 2.8 352.4 1.0X
-Parquet Vectorized (Pushdown) 168 / 174 93.7 10.7 33.0X
-Native ORC Vectorized 4992 / 5052 3.2 317.4 1.1X
-Native ORC Vectorized (Pushdown) 840 / 850 18.7 53.4 6.6X
+Parquet Vectorized 3661 / 3785 4.3 232.7 1.0X
+Parquet Vectorized (Pushdown) 107 / 113 146.8 6.8 34.2X
+Native ORC Vectorized 3480 / 3528 4.5 221.3 1.1X
+Native ORC Vectorized (Pushdown) 614 / 628 25.6 39.0 6.0X
+InMemoryTable Vectorized 2915 / 3008 5.4 185.3 1.3X
+InMemoryTable Vectorized (Pushdown) 874 / 919 18.0 55.6 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(9, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7312 / 7358 2.2 464.9 1.0X
-Parquet Vectorized (Pushdown) 3008 / 3078 5.2 191.2 2.4X
-Native ORC Vectorized 6775 / 6798 2.3 430.7 1.1X
-Native ORC Vectorized (Pushdown) 6819 / 6832 2.3 433.5 1.1X
+Parquet Vectorized 5167 / 5304 3.0 328.5 1.0X
+Parquet Vectorized (Pushdown) 2232 / 2295 7.0 141.9 2.3X
+Native ORC Vectorized 4883 / 4999 3.2 310.5 1.1X
+Native ORC Vectorized (Pushdown) 4982 / 5045 3.2 316.7 1.0X
+InMemoryTable Vectorized 3980 / 4111 4.0 253.0 1.3X
+InMemoryTable Vectorized (Pushdown) 2385 / 2463 6.6 151.6 2.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(9, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 13232 / 13241 1.2 841.3 1.0X
-Parquet Vectorized (Pushdown) 12555 / 12569 1.3 798.2 1.1X
-Native ORC Vectorized 12597 / 12627 1.2 800.9 1.1X
-Native ORC Vectorized (Pushdown) 12677 / 12711 1.2 806.0 1.0X
+Parquet Vectorized 9996 / 10052 1.6 635.5 1.0X
+Parquet Vectorized (Pushdown) 9331 / 9501 1.7 593.2 1.1X
+Native ORC Vectorized 9632 / 9770 1.6 612.4 1.0X
+Native ORC Vectorized (Pushdown) 9700 / 9788 1.6 616.7 1.0X
+InMemoryTable Vectorized 7858 / 7999 2.0 499.6 1.3X
+InMemoryTable Vectorized (Pushdown) 7506 / 7565 2.1 477.2 1.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(9, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14725 / 14729 1.1 936.2 1.0X
-Parquet Vectorized (Pushdown) 14781 / 14800 1.1 939.7 1.0X
-Native ORC Vectorized 15360 / 15453 1.0 976.5 1.0X
-Native ORC Vectorized (Pushdown) 15444 / 15466 1.0 981.9 1.0X
+Parquet Vectorized 10776 / 10969 1.5 685.1 1.0X
+Parquet Vectorized (Pushdown) 10931 / 11012 1.4 695.0 1.0X
+Native ORC Vectorized 10849 / 10910 1.4 689.7 1.0X
+Native ORC Vectorized (Pushdown) 11030 / 11112 1.4 701.2 1.0X
+InMemoryTable Vectorized 8675 / 8822 1.8 551.6 1.2X
+InMemoryTable Vectorized (Pushdown) 8668 / 8741 1.8 551.1 1.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(18, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 5746 / 5763 2.7 365.3 1.0X
-Parquet Vectorized (Pushdown) 166 / 169 94.8 10.6 34.6X
-Native ORC Vectorized 5007 / 5023 3.1 318.3 1.1X
-Native ORC Vectorized (Pushdown) 2629 / 2640 6.0 167.1 2.2X
+Parquet Vectorized 3886 / 3932 4.0 247.0 1.0X
+Parquet Vectorized (Pushdown) 104 / 115 151.0 6.6 37.3X
+Native ORC Vectorized 3476 / 3527 4.5 221.0 1.1X
+Native ORC Vectorized (Pushdown) 1804 / 1839 8.7 114.7 2.2X
+InMemoryTable Vectorized 2956 / 3002 5.3 187.9 1.3X
+InMemoryTable Vectorized (Pushdown) 865 / 941 18.2 55.0 4.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(18, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 6827 / 6864 2.3 434.0 1.0X
-Parquet Vectorized (Pushdown) 1809 / 1827 8.7 115.0 3.8X
-Native ORC Vectorized 6287 / 6296 2.5 399.7 1.1X
-Native ORC Vectorized (Pushdown) 6364 / 6377 2.5 404.6 1.1X
+Parquet Vectorized 4498 / 4642 3.5 286.0 1.0X
+Parquet Vectorized (Pushdown) 1201 / 1242 13.1 76.3 3.7X
+Native ORC Vectorized 4218 / 4291 3.7 268.1 1.1X
+Native ORC Vectorized (Pushdown) 4262 / 4311 3.7 270.9 1.1X
+InMemoryTable Vectorized 3509 / 3599 4.5 223.1 1.3X
+InMemoryTable Vectorized (Pushdown) 1640 / 1702 9.6 104.3 2.7X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(18, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11315 / 11342 1.4 719.4 1.0X
-Parquet Vectorized (Pushdown) 8431 / 8450 1.9 536.0 1.3X
-Native ORC Vectorized 11591 / 11611 1.4 736.9 1.0X
-Native ORC Vectorized (Pushdown) 11424 / 11475 1.4 726.3 1.0X
+Parquet Vectorized 7445 / 7525 2.1 473.3 1.0X
+Parquet Vectorized (Pushdown) 5609 / 5732 2.8 356.6 1.3X
+Native ORC Vectorized 7056 / 7187 2.2 448.6 1.1X
+Native ORC Vectorized (Pushdown) 7226 / 7330 2.2 459.4 1.0X
+InMemoryTable Vectorized 5707 / 5801 2.8 362.8 1.3X
+InMemoryTable Vectorized (Pushdown) 4694 / 4810 3.4 298.4 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(18, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 15703 / 15712 1.0 998.4 1.0X
-Parquet Vectorized (Pushdown) 14982 / 15009 1.0 952.5 1.0X
-Native ORC Vectorized 16887 / 16955 0.9 1073.7 0.9X
-Native ORC Vectorized (Pushdown) 16518 / 16530 1.0 1050.2 1.0X
+Parquet Vectorized 10324 / 10427 1.5 656.4 1.0X
+Parquet Vectorized (Pushdown) 9916 / 10093 1.6 630.4 1.0X
+Native ORC Vectorized 10422 / 10539 1.5 662.6 1.0X
+Native ORC Vectorized (Pushdown) 10099 / 10261 1.6 642.1 1.0X
+InMemoryTable Vectorized 8220 / 8284 1.9 522.6 1.3X
+InMemoryTable Vectorized (Pushdown) 7809 / 7963 2.0 496.5 1.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(38, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 8101 / 8130 1.9 515.1 1.0X
-Parquet Vectorized (Pushdown) 184 / 187 85.6 11.7 44.1X
-Native ORC Vectorized 4998 / 5027 3.1 317.8 1.6X
-Native ORC Vectorized (Pushdown) 165 / 168 95.6 10.5 49.2X
+Parquet Vectorized 5346 / 5526 2.9 339.9 1.0X
+Parquet Vectorized (Pushdown) 116 / 121 136.2 7.3 46.3X
+Native ORC Vectorized 3483 / 3551 4.5 221.4 1.5X
+Native ORC Vectorized (Pushdown) 152 / 166 103.7 9.6 35.2X
+InMemoryTable Vectorized 4312 / 4344 3.6 274.2 1.2X
+InMemoryTable Vectorized (Pushdown) 878 / 930 17.9 55.8 6.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(38, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 9405 / 9447 1.7 597.9 1.0X
-Parquet Vectorized (Pushdown) 2269 / 2275 6.9 144.2 4.1X
-Native ORC Vectorized 6167 / 6203 2.6 392.1 1.5X
-Native ORC Vectorized (Pushdown) 1783 / 1787 8.8 113.3 5.3X
+Parquet Vectorized 6229 / 6430 2.5 396.0 1.0X
+Parquet Vectorized (Pushdown) 1530 / 1560 10.3 97.3 4.1X
+Native ORC Vectorized 4407 / 4469 3.6 280.2 1.4X
+Native ORC Vectorized (Pushdown) 1361 / 1399 11.6 86.5 4.6X
+InMemoryTable Vectorized 5006 / 5129 3.1 318.3 1.2X
+InMemoryTable Vectorized (Pushdown) 1995 / 2037 7.9 126.8 3.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(38, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14700 / 14707 1.1 934.6 1.0X
-Parquet Vectorized (Pushdown) 10699 / 10712 1.5 680.2 1.4X
-Native ORC Vectorized 10687 / 10703 1.5 679.5 1.4X
-Native ORC Vectorized (Pushdown) 8364 / 8415 1.9 531.8 1.8X
+Parquet Vectorized 10057 / 10179 1.6 639.4 1.0X
+Parquet Vectorized (Pushdown) 7411 / 7518 2.1 471.2 1.4X
+Native ORC Vectorized 7911 / 8161 2.0 503.0 1.3X
+Native ORC Vectorized (Pushdown) 6199 / 6354 2.5 394.1 1.6X
+InMemoryTable Vectorized 8000 / 8128 2.0 508.6 1.3X
+InMemoryTable Vectorized (Pushdown) 6330 / 6413 2.5 402.4 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(38, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19780 / 19894 0.8 1257.6 1.0X
-Parquet Vectorized (Pushdown) 19003 / 19025 0.8 1208.1 1.0X
-Native ORC Vectorized 15385 / 15404 1.0 978.2 1.3X
-Native ORC Vectorized (Pushdown) 15032 / 15060 1.0 955.7 1.3X
+Parquet Vectorized 13756 / 13841 1.1 874.6 1.0X
+Parquet Vectorized (Pushdown) 12922 / 13129 1.2 821.6 1.1X
+Native ORC Vectorized 11386 / 11548 1.4 723.9 1.2X
+Native ORC Vectorized (Pushdown) 11170 / 11345 1.4 710.1 1.2X
+InMemoryTable Vectorized 11079 / 11194 1.4 704.4 1.2X
+InMemoryTable Vectorized (Pushdown) 10712 / 10793 1.5 681.0 1.3X
================================================================================================
Pushdown benchmark for InSet -> InFilters
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10521 / 10534 1.5 668.9 1.0X
-Parquet Vectorized (Pushdown) 677 / 691 23.2 43.1 15.5X
-Native ORC Vectorized 6768 / 6776 2.3 430.3 1.6X
-Native ORC Vectorized (Pushdown) 501 / 512 31.4 31.8 21.0X
+Parquet Vectorized 7187 / 7325 2.2 456.9 1.0X
+Parquet Vectorized (Pushdown) 447 / 460 35.2 28.4 16.1X
+Native ORC Vectorized 4850 / 4976 3.2 308.3 1.5X
+Native ORC Vectorized (Pushdown) 508 / 527 30.9 32.3 14.1X
+InMemoryTable Vectorized 3758 / 3828 4.2 238.9 1.9X
+InMemoryTable Vectorized (Pushdown) 1704 / 1761 9.2 108.4 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 50): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10531 / 10538 1.5 669.5 1.0X
-Parquet Vectorized (Pushdown) 677 / 718 23.2 43.0 15.6X
-Native ORC Vectorized 6765 / 6773 2.3 430.1 1.6X
-Native ORC Vectorized (Pushdown) 499 / 507 31.5 31.7 21.1X
+Parquet Vectorized 7145 / 7191 2.2 454.3 1.0X
+Parquet Vectorized (Pushdown) 442 / 461 35.6 28.1 16.2X
+Native ORC Vectorized 4911 / 5138 3.2 312.2 1.5X
+Native ORC Vectorized (Pushdown) 503 / 522 31.3 32.0 14.2X
+InMemoryTable Vectorized 3693 / 3781 4.3 234.8 1.9X
+InMemoryTable Vectorized (Pushdown) 1647 / 1733 9.6 104.7 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 90): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10540 / 10553 1.5 670.1 1.0X
-Parquet Vectorized (Pushdown) 678 / 710 23.2 43.1 15.5X
-Native ORC Vectorized 6787 / 6794 2.3 431.5 1.6X
-Native ORC Vectorized (Pushdown) 501 / 509 31.4 31.9 21.0X
+Parquet Vectorized 7216 / 7350 2.2 458.8 1.0X
+Parquet Vectorized (Pushdown) 430 / 451 36.6 27.3 16.8X
+Native ORC Vectorized 4900 / 5100 3.2 311.5 1.5X
+Native ORC Vectorized (Pushdown) 520 / 542 30.2 33.1 13.9X
+InMemoryTable Vectorized 3639 / 3689 4.3 231.4 2.0X
+InMemoryTable Vectorized (Pushdown) 1647 / 1732 9.5 104.7 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10551 / 10559 1.5 670.8 1.0X
-Parquet Vectorized (Pushdown) 703 / 708 22.4 44.7 15.0X
-Native ORC Vectorized 6791 / 6802 2.3 431.7 1.6X
-Native ORC Vectorized (Pushdown) 519 / 526 30.3 33.0 20.3X
+Parquet Vectorized 7195 / 7290 2.2 457.4 1.0X
+Parquet Vectorized (Pushdown) 446 / 478 35.3 28.3 16.1X
+Native ORC Vectorized 4976 / 5229 3.2 316.4 1.4X
+Native ORC Vectorized (Pushdown) 520 / 543 30.2 33.1 13.8X
+InMemoryTable Vectorized 3635 / 3744 4.3 231.1 2.0X
+InMemoryTable Vectorized (Pushdown) 1684 / 1766 9.3 107.1 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 50): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10561 / 10565 1.5 671.4 1.0X
-Parquet Vectorized (Pushdown) 711 / 716 22.1 45.2 14.9X
-Native ORC Vectorized 6791 / 6806 2.3 431.8 1.6X
-Native ORC Vectorized (Pushdown) 529 / 537 29.8 33.6 20.0X
+Parquet Vectorized 7246 / 7375 2.2 460.7 1.0X
+Parquet Vectorized (Pushdown) 477 / 496 33.0 30.3 15.2X
+Native ORC Vectorized 4828 / 5158 3.3 307.0 1.5X
+Native ORC Vectorized (Pushdown) 524 / 540 30.0 33.3 13.8X
+InMemoryTable Vectorized 3695 / 3785 4.3 234.9 2.0X
+InMemoryTable Vectorized (Pushdown) 1702 / 1745 9.2 108.2 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 90): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10572 / 10590 1.5 672.1 1.0X
-Parquet Vectorized (Pushdown) 713 / 716 22.1 45.3 14.8X
-Native ORC Vectorized 6808 / 6815 2.3 432.9 1.6X
-Native ORC Vectorized (Pushdown) 530 / 541 29.7 33.7 19.9X
+Parquet Vectorized 7303 / 7388 2.2 464.3 1.0X
+Parquet Vectorized (Pushdown) 471 / 477 33.4 29.9 15.5X
+Native ORC Vectorized 4848 / 5005 3.2 308.2 1.5X
+Native ORC Vectorized (Pushdown) 518 / 550 30.4 32.9 14.1X
+InMemoryTable Vectorized 3665 / 3790 4.3 233.0 2.0X
+InMemoryTable Vectorized (Pushdown) 1689 / 1717 9.3 107.4 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 50, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10871 / 10882 1.4 691.2 1.0X
-Parquet Vectorized (Pushdown) 11104 / 11110 1.4 706.0 1.0X
-Native ORC Vectorized 7088 / 7104 2.2 450.7 1.5X
-Native ORC Vectorized (Pushdown) 665 / 677 23.6 42.3 16.3X
+Parquet Vectorized 7474 / 7577 2.1 475.2 1.0X
+Parquet Vectorized (Pushdown) 7649 / 7707 2.1 486.3 1.0X
+Native ORC Vectorized 5167 / 5457 3.0 328.5 1.4X
+Native ORC Vectorized (Pushdown) 637
--- End diff --
Do you know why this is so slow?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/23027
cc @HyukjinKwon @dongjoon-hyun
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5218/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter benchm...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #98806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98806/testReport)** for PR 23027 at commit [`d0a2a3f`](https://github.com/apache/spark/commit/d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r234482766
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X
+Parquet Vectorized 7823 / 7996 2.0 497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X
+InMemoryTable Vectorized 6 / 6 2859.1 0.3 1422.0X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3023.0 0.3 1503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row (value = '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11878 / 11888 1.3 755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0 40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1 466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3 33.0 22.9X
+Parquet Vectorized 8322 / 8386 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 434 / 441 36.2 27.6 19.2X
+Native ORC Vectorized 5659 / 5944 2.8 359.8 1.5X
+Native ORC Vectorized (Pushdown) 535 / 567 29.4 34.0 15.6X
+InMemoryTable Vectorized 4784 / 4879 3.3 304.1 1.7X
+InMemoryTable Vectorized (Pushdown) 1950 / 1985 8.1 124.0 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row (value <=> '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11423 / 11440 1.4 726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2 39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2 465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0 32.2 22.5X
+Parquet Vectorized 8377 / 8458 1.9 532.6 1.0X
+Parquet Vectorized (Pushdown) 449 / 457 35.1 28.5 18.7X
+Native ORC Vectorized 5664 / 5786 2.8 360.1 1.5X
+Native ORC Vectorized (Pushdown) 527 / 560 29.9 33.5 15.9X
+InMemoryTable Vectorized 4743 / 4813 3.3 301.5 1.8X
+InMemoryTable Vectorized (Pushdown) 1876 / 1963 8.4 119.3 4.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11440 / 11478 1.4 727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8 40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2 464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4 32.8 22.1X
+Parquet Vectorized 8120 / 8293 1.9 516.2 1.0X
+Parquet Vectorized (Pushdown) 423 / 436 37.2 26.9 19.2X
+Native ORC Vectorized 5490 / 5833 2.9 349.0 1.5X
+Native ORC Vectorized (Pushdown) 538 / 562 29.3 34.2 15.1X
+InMemoryTable Vectorized 4722 / 4774 3.3 300.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1923 / 1954 8.2 122.2 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 20750 / 20872 0.8 1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7 1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9 1062.6 1.2X
-Native ORC Vectorized (Pushdown) 16926 / 16965 0.9 1076.1 1.2X
+Parquet Vectorized 15437 / 15682 1.0 981.4 1.0X
+Parquet Vectorized (Pushdown) 15748 / 15811 1.0 1001.2 1.0X
+Native ORC Vectorized 13111 / 13245 1.2 833.6 1.2X
+Native ORC Vectorized (Pushdown) 12931 / 13216 1.2 822.1 1.2X
+InMemoryTable Vectorized 9661 / 9908 1.6 614.2 1.6X
+InMemoryTable Vectorized (Pushdown) 9838 / 9908 1.6 625.5 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 int row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10510 / 10532 1.5 668.2 1.0X
-Parquet Vectorized (Pushdown) 642 / 665 24.5 40.8 16.4X
-Native ORC Vectorized 6609 / 6618 2.4 420.2 1.6X
-Native ORC Vectorized (Pushdown) 502 / 512 31.4 31.9 21.0X
+Parquet Vectorized 7549 / 7625 2.1 479.9 1.0X
+Parquet Vectorized (Pushdown) 427 / 450 36.9 27.1 17.7X
+Native ORC Vectorized 5028 / 5221 3.1 319.7 1.5X
+Native ORC Vectorized (Pushdown) 530 / 548 29.7 33.7 14.3X
+InMemoryTable Vectorized 5 / 6 3221.5 0.3 1546.1X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3303.5 0.3 1585.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 int row (7864320 < value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10505 / 10514 1.5 667.9 1.0X
-Parquet Vectorized (Pushdown) 659 / 673 23.9 41.9 15.9X
-Native ORC Vectorized 6634 / 6641 2.4 421.8 1.6X
-Native ORC Vectorized (Pushdown) 513 / 526 30.7 32.6 20.5X
+Parquet Vectorized 7542 / 7584 2.1 479.5 1.0X
+Parquet Vectorized (Pushdown) 458 / 468 34.3 29.1 16.5X
+Native ORC Vectorized 5035 / 5533 3.1 320.1 1.5X
+Native ORC Vectorized (Pushdown) 538 / 560 29.2 34.2 14.0X
+InMemoryTable Vectorized 3866 / 3925 4.1 245.8 2.0X
+InMemoryTable Vectorized (Pushdown) 1766 / 1792 8.9 112.3 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10555 / 10570 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 651 / 668 24.2 41.4 16.2X
-Native ORC Vectorized 6721 / 6728 2.3 427.3 1.6X
-Native ORC Vectorized (Pushdown) 508 / 519 31.0 32.3 20.8X
+Parquet Vectorized 7540 / 7660 2.1 479.3 1.0X
+Parquet Vectorized (Pushdown) 444 / 454 35.5 28.2 17.0X
+Native ORC Vectorized 5155 / 5310 3.1 327.7 1.5X
+Native ORC Vectorized (Pushdown) 524 / 541 30.0 33.3 14.4X
+InMemoryTable Vectorized 3910 / 3999 4.0 248.6 1.9X
+InMemoryTable Vectorized (Pushdown) 1843 / 1868 8.5 117.2 4.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (value <=> 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10566 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 647 / 654 24.3 41.1 16.3X
-Native ORC Vectorized 6716 / 6728 2.3 427.0 1.6X
-Native ORC Vectorized (Pushdown) 510 / 521 30.9 32.4 20.7X
+Parquet Vectorized 7568 / 7679 2.1 481.1 1.0X
+Parquet Vectorized (Pushdown) 446 / 460 35.2 28.4 17.0X
+Native ORC Vectorized 5083 / 5244 3.1 323.2 1.5X
+Native ORC Vectorized (Pushdown) 529 / 542 29.7 33.6 14.3X
+InMemoryTable Vectorized 4033 / 4068 3.9 256.4 1.9X
+InMemoryTable Vectorized (Pushdown) 1818 / 1848 8.6 115.6 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (7864320 <= value <= 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10556 / 10565 1.5 671.1 1.0X
-Parquet Vectorized (Pushdown) 649 / 654 24.2 41.3 16.3X
-Native ORC Vectorized 6700 / 6712 2.3 426.0 1.6X
-Native ORC Vectorized (Pushdown) 509 / 520 30.9 32.3 20.8X
+Parquet Vectorized 7446 / 7622 2.1 473.4 1.0X
+Parquet Vectorized (Pushdown) 440 / 455 35.8 28.0 16.9X
+Native ORC Vectorized 5155 / 5347 3.1 327.8 1.4X
+Native ORC Vectorized (Pushdown) 509 / 538 30.9 32.3 14.6X
+InMemoryTable Vectorized 3952 / 3977 4.0 251.3 1.9X
+InMemoryTable Vectorized (Pushdown) 1832 / 1887 8.6 116.5 4.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 int row (7864319 < value < 7864321): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10547 / 10566 1.5 670.5 1.0X
-Parquet Vectorized (Pushdown) 649 / 653 24.2 41.3 16.3X
-Native ORC Vectorized 6703 / 6713 2.3 426.2 1.6X
-Native ORC Vectorized (Pushdown) 510 / 520 30.8 32.5 20.7X
+Parquet Vectorized 7576 / 7711 2.1 481.7 1.0X
+Parquet Vectorized (Pushdown) 467 / 477 33.7 29.7 16.2X
+Native ORC Vectorized 5147 / 5412 3.1 327.3 1.5X
+Native ORC Vectorized (Pushdown) 521 / 551 30.2 33.1 14.6X
+InMemoryTable Vectorized 3882 / 3964 4.1 246.8 2.0X
+InMemoryTable Vectorized (Pushdown) 1805 / 1845 8.7 114.7 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% int rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11478 / 11525 1.4 729.7 1.0X
-Parquet Vectorized (Pushdown) 2576 / 2587 6.1 163.8 4.5X
-Native ORC Vectorized 7633 / 7657 2.1 485.3 1.5X
-Native ORC Vectorized (Pushdown) 2076 / 2096 7.6 132.0 5.5X
+Parquet Vectorized 8439 / 8541 1.9 536.5 1.0X
+Parquet Vectorized (Pushdown) 1952 / 1993 8.1 124.1 4.3X
+Native ORC Vectorized 5864 / 6092 2.7 372.8 1.4X
+Native ORC Vectorized (Pushdown) 1786 / 1805 8.8 113.6 4.7X
+InMemoryTable Vectorized 4463 / 4527 3.5 283.8 1.9X
+InMemoryTable Vectorized (Pushdown) 2559 / 2596 6.1 162.7 3.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% int rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14785 / 14802 1.1 940.0 1.0X
-Parquet Vectorized (Pushdown) 9971 / 9977 1.6 633.9 1.5X
-Native ORC Vectorized 11082 / 11107 1.4 704.6 1.3X
-Native ORC Vectorized (Pushdown) 8061 / 8073 2.0 512.5 1.8X
+Parquet Vectorized 11106 / 11248 1.4 706.1 1.0X
+Parquet Vectorized (Pushdown) 7656 / 7697 2.1 486.7 1.5X
+Native ORC Vectorized 8593 / 8892 1.8 546.4 1.3X
+Native ORC Vectorized (Pushdown) 6474 / 6567 2.4 411.6 1.7X
+InMemoryTable Vectorized 6623 / 6731 2.4 421.1 1.7X
+InMemoryTable Vectorized (Pushdown) 5613 / 5747 2.8 356.9 2.0X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% int rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 18174 / 18214 0.9 1155.5 1.0X
-Parquet Vectorized (Pushdown) 17387 / 17403 0.9 1105.5 1.0X
-Native ORC Vectorized 14465 / 14492 1.1 919.7 1.3X
-Native ORC Vectorized (Pushdown) 14024 / 14041 1.1 891.6 1.3X
+Parquet Vectorized 13716 / 13810 1.1 872.1 1.0X
+Parquet Vectorized (Pushdown) 13044 / 13233 1.2 829.3 1.1X
+Native ORC Vectorized 11795 / 11899 1.3 749.9 1.2X
+Native ORC Vectorized (Pushdown) 11074 / 11265 1.4 704.1 1.2X
+InMemoryTable Vectorized 9023 / 9166 1.7 573.7 1.5X
+InMemoryTable Vectorized (Pushdown) 8929 / 9015 1.8 567.7 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19004 / 19014 0.8 1208.2 1.0X
-Parquet Vectorized (Pushdown) 19219 / 19232 0.8 1221.9 1.0X
-Native ORC Vectorized 15266 / 15290 1.0 970.6 1.2X
-Native ORC Vectorized (Pushdown) 15469 / 15482 1.0 983.5 1.2X
+Parquet Vectorized 14319 / 14511 1.1 910.4 1.0X
+Parquet Vectorized (Pushdown) 14433 / 14610 1.1 917.6 1.0X
+Native ORC Vectorized 11961 / 12449 1.3 760.4 1.2X
+Native ORC Vectorized (Pushdown) 12243 / 12382 1.3 778.4 1.2X
+InMemoryTable Vectorized 7885 / 8006 2.0 501.3 1.8X
+InMemoryTable Vectorized (Pushdown) 7776 / 7909 2.0 494.4 1.8X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value > -1): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19036 / 19052 0.8 1210.3 1.0X
-Parquet Vectorized (Pushdown) 19287 / 19306 0.8 1226.2 1.0X
-Native ORC Vectorized 15311 / 15371 1.0 973.5 1.2X
-Native ORC Vectorized (Pushdown) 15517 / 15590 1.0 986.5 1.2X
+Parquet Vectorized 14275 / 14442 1.1 907.6 1.0X
+Parquet Vectorized (Pushdown) 14291 / 14586 1.1 908.6 1.0X
+Native ORC Vectorized 11957 / 12244 1.3 760.2 1.2X
+Native ORC Vectorized (Pushdown) 11871 / 12044 1.3 754.7 1.2X
+InMemoryTable Vectorized 9404 / 9540 1.7 597.9 1.5X
+InMemoryTable Vectorized (Pushdown) 9429 / 9558 1.7 599.5 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all int rows (value != -1): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19072 / 19102 0.8 1212.6 1.0X
-Parquet Vectorized (Pushdown) 19288 / 19318 0.8 1226.3 1.0X
-Native ORC Vectorized 15277 / 15293 1.0 971.3 1.2X
-Native ORC Vectorized (Pushdown) 15479 / 15499 1.0 984.1 1.2X
+Parquet Vectorized 13937 / 14019 1.1 886.1 1.0X
+Parquet Vectorized (Pushdown) 14197 / 14239 1.1 902.6 1.0X
+Native ORC Vectorized 12177 / 12524 1.3 774.2 1.1X
+Native ORC Vectorized (Pushdown) 11866 / 12013 1.3 754.4 1.2X
+InMemoryTable Vectorized 9194 / 9383 1.7 584.6 1.5X
+InMemoryTable Vectorized (Pushdown) 9240 / 9410 1.7 587.5 1.5X
================================================================================================
Pushdown for few distinct value case (use dictionary encoding)
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 distinct string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10250 / 10274 1.5 651.7 1.0X
-Parquet Vectorized (Pushdown) 571 / 576 27.5 36.3 17.9X
-Native ORC Vectorized 8651 / 8660 1.8 550.0 1.2X
-Native ORC Vectorized (Pushdown) 909 / 933 17.3 57.8 11.3X
+Parquet Vectorized 6978 / 7046 2.3 443.7 1.0X
+Parquet Vectorized (Pushdown) 367 / 378 42.9 23.3 19.0X
+Native ORC Vectorized 6320 / 6444 2.5 401.8 1.1X
+Native ORC Vectorized (Pushdown) 972 / 992 16.2 61.8 7.2X
+InMemoryTable Vectorized 4289 / 4365 3.7 272.7 1.6X
+InMemoryTable Vectorized (Pushdown) 1594 / 1665 9.9 101.3 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 distinct string row ('100' < value < '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10420 / 10426 1.5 662.5 1.0X
-Parquet Vectorized (Pushdown) 574 / 579 27.4 36.5 18.2X
-Native ORC Vectorized 8973 / 8982 1.8 570.5 1.2X
-Native ORC Vectorized (Pushdown) 916 / 955 17.2 58.2 11.4X
+Parquet Vectorized 7119 / 7216 2.2 452.6 1.0X
+Parquet Vectorized (Pushdown) 367 / 379 42.9 23.3 19.4X
+Native ORC Vectorized 6449 / 6574 2.4 410.0 1.1X
+Native ORC Vectorized (Pushdown) 945 / 956 16.6 60.1 7.5X
+InMemoryTable Vectorized 4390 / 4538 3.6 279.1 1.6X
+InMemoryTable Vectorized (Pushdown) 1628 / 1647 9.7 103.5 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row (value = '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10428 / 10441 1.5 663.0 1.0X
-Parquet Vectorized (Pushdown) 789 / 809 19.9 50.2 13.2X
-Native ORC Vectorized 9042 / 9055 1.7 574.9 1.2X
-Native ORC Vectorized (Pushdown) 1130 / 1145 13.9 71.8 9.2X
+Parquet Vectorized 7027 / 7252 2.2 446.8 1.0X
+Parquet Vectorized (Pushdown) 536 / 557 29.4 34.1 13.1X
+Native ORC Vectorized 6569 / 6818 2.4 417.7 1.1X
+Native ORC Vectorized (Pushdown) 1141 / 1187 13.8 72.6 6.2X
+InMemoryTable Vectorized 4557 / 4650 3.5 289.8 1.5X
+InMemoryTable Vectorized (Pushdown) 1754 / 1785 9.0 111.5 4.0X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row (value <=> '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10402 / 10416 1.5 661.3 1.0X
-Parquet Vectorized (Pushdown) 791 / 806 19.9 50.3 13.2X
-Native ORC Vectorized 9042 / 9055 1.7 574.9 1.2X
-Native ORC Vectorized (Pushdown) 1112 / 1145 14.1 70.7 9.4X
+Parquet Vectorized 7200 / 7303 2.2 457.7 1.0X
+Parquet Vectorized (Pushdown) 530 / 551 29.7 33.7 13.6X
+Native ORC Vectorized 6563 / 6667 2.4 417.2 1.1X
+Native ORC Vectorized (Pushdown) 1128 / 1144 13.9 71.7 6.4X
+InMemoryTable Vectorized 4561 / 4646 3.4 290.0 1.6X
+InMemoryTable Vectorized (Pushdown) 1731 / 1792 9.1 110.1 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 distinct string row ('100' <= value <= '100'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10548 / 10563 1.5 670.6 1.0X
-Parquet Vectorized (Pushdown) 790 / 796 19.9 50.2 13.4X
-Native ORC Vectorized 9144 / 9153 1.7 581.3 1.2X
-Native ORC Vectorized (Pushdown) 1117 / 1148 14.1 71.0 9.4X
+Parquet Vectorized 7028 / 7212 2.2 446.8 1.0X
+Parquet Vectorized (Pushdown) 539 / 552 29.2 34.3 13.0X
+Native ORC Vectorized 6491 / 6600 2.4 412.7 1.1X
+Native ORC Vectorized (Pushdown) 1102 / 1146 14.3 70.0 6.4X
+InMemoryTable Vectorized 4604 / 4725 3.4 292.7 1.5X
+InMemoryTable Vectorized (Pushdown) 1806 / 1854 8.7 114.8 3.9X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select all distinct string rows (value IS NOT NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 20445 / 20469 0.8 1299.8 1.0X
-Parquet Vectorized (Pushdown) 20686 / 20699 0.8 1315.2 1.0X
-Native ORC Vectorized 18851 / 18953 0.8 1198.5 1.1X
-Native ORC Vectorized (Pushdown) 19255 / 19268 0.8 1224.2 1.1X
+Parquet Vectorized 15379 / 15520 1.0 977.8 1.0X
+Parquet Vectorized (Pushdown) 15319 / 15503 1.0 973.9 1.0X
+Native ORC Vectorized 14291 / 14421 1.1 908.6 1.1X
+Native ORC Vectorized (Pushdown) 14542 / 14769 1.1 924.6 1.1X
+InMemoryTable Vectorized 11200 / 11431 1.4 712.1 1.4X
+InMemoryTable Vectorized (Pushdown) 11262 / 11429 1.4 716.0 1.4X
================================================================================================
Pushdown benchmark for StringStartsWith
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '10%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14265 / 15213 1.1 907.0 1.0X
-Parquet Vectorized (Pushdown) 4228 / 4870 3.7 268.8 3.4X
-Native ORC Vectorized 10116 / 10977 1.6 643.2 1.4X
-Native ORC Vectorized (Pushdown) 10653 / 11376 1.5 677.3 1.3X
+Parquet Vectorized 9439 / 10391 1.7 600.1 1.0X
+Parquet Vectorized (Pushdown) 2669 / 3183 5.9 169.7 3.5X
+Native ORC Vectorized 7250 / 9963 2.2 460.9 1.3X
+Native ORC Vectorized (Pushdown) 7493 / 8724 2.1 476.4 1.3X
+InMemoryTable Vectorized 6214 / 6641 2.5 395.1 1.5X
+InMemoryTable Vectorized (Pushdown) 6303 / 6775 2.5 400.8 1.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '1000%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11499 / 11539 1.4 731.1 1.0X
-Parquet Vectorized (Pushdown) 669 / 672 23.5 42.5 17.2X
-Native ORC Vectorized 7343 / 7363 2.1 466.8 1.6X
-Native ORC Vectorized (Pushdown) 7559 / 7568 2.1 480.6 1.5X
+Parquet Vectorized 7906 / 8069 2.0 502.7 1.0X
+Parquet Vectorized (Pushdown) 439 / 440 35.9 27.9 18.0X
+Native ORC Vectorized 5484 / 5697 2.9 348.7 1.4X
+Native ORC Vectorized (Pushdown) 5507 / 5634 2.9 350.1 1.4X
+InMemoryTable Vectorized 4286 / 4382 3.7 272.5 1.8X
+InMemoryTable Vectorized (Pushdown) 4236 / 4339 3.7 269.3 1.9X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
StringStartsWith filter: (value like '786432%'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11463 / 11468 1.4 728.8 1.0X
-Parquet Vectorized (Pushdown) 647 / 651 24.3 41.1 17.7X
-Native ORC Vectorized 7322 / 7338 2.1 465.5 1.6X
-Native ORC Vectorized (Pushdown) 7533 / 7544 2.1 478.9 1.5X
+Parquet Vectorized 7775 / 7965 2.0 494.3 1.0X
+Parquet Vectorized (Pushdown) 414 / 441 38.0 26.3 18.8X
+Native ORC Vectorized 5366 / 5595 2.9 341.1 1.4X
+Native ORC Vectorized (Pushdown) 5413 / 5615 2.9 344.2 1.4X
+InMemoryTable Vectorized 4320 / 4416 3.6 274.6 1.8X
+InMemoryTable Vectorized (Pushdown) 4260 / 4341 3.7 270.9 1.8X
================================================================================================
Pushdown benchmark for decimal
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(9, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 5543 / 5564 2.8 352.4 1.0X
-Parquet Vectorized (Pushdown) 168 / 174 93.7 10.7 33.0X
-Native ORC Vectorized 4992 / 5052 3.2 317.4 1.1X
-Native ORC Vectorized (Pushdown) 840 / 850 18.7 53.4 6.6X
+Parquet Vectorized 3661 / 3785 4.3 232.7 1.0X
+Parquet Vectorized (Pushdown) 107 / 113 146.8 6.8 34.2X
+Native ORC Vectorized 3480 / 3528 4.5 221.3 1.1X
+Native ORC Vectorized (Pushdown) 614 / 628 25.6 39.0 6.0X
+InMemoryTable Vectorized 2915 / 3008 5.4 185.3 1.3X
+InMemoryTable Vectorized (Pushdown) 874 / 919 18.0 55.6 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(9, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 7312 / 7358 2.2 464.9 1.0X
-Parquet Vectorized (Pushdown) 3008 / 3078 5.2 191.2 2.4X
-Native ORC Vectorized 6775 / 6798 2.3 430.7 1.1X
-Native ORC Vectorized (Pushdown) 6819 / 6832 2.3 433.5 1.1X
+Parquet Vectorized 5167 / 5304 3.0 328.5 1.0X
+Parquet Vectorized (Pushdown) 2232 / 2295 7.0 141.9 2.3X
+Native ORC Vectorized 4883 / 4999 3.2 310.5 1.1X
+Native ORC Vectorized (Pushdown) 4982 / 5045 3.2 316.7 1.0X
+InMemoryTable Vectorized 3980 / 4111 4.0 253.0 1.3X
+InMemoryTable Vectorized (Pushdown) 2385 / 2463 6.6 151.6 2.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(9, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 13232 / 13241 1.2 841.3 1.0X
-Parquet Vectorized (Pushdown) 12555 / 12569 1.3 798.2 1.1X
-Native ORC Vectorized 12597 / 12627 1.2 800.9 1.1X
-Native ORC Vectorized (Pushdown) 12677 / 12711 1.2 806.0 1.0X
+Parquet Vectorized 9996 / 10052 1.6 635.5 1.0X
+Parquet Vectorized (Pushdown) 9331 / 9501 1.7 593.2 1.1X
+Native ORC Vectorized 9632 / 9770 1.6 612.4 1.0X
+Native ORC Vectorized (Pushdown) 9700 / 9788 1.6 616.7 1.0X
+InMemoryTable Vectorized 7858 / 7999 2.0 499.6 1.3X
+InMemoryTable Vectorized (Pushdown) 7506 / 7565 2.1 477.2 1.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(9, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14725 / 14729 1.1 936.2 1.0X
-Parquet Vectorized (Pushdown) 14781 / 14800 1.1 939.7 1.0X
-Native ORC Vectorized 15360 / 15453 1.0 976.5 1.0X
-Native ORC Vectorized (Pushdown) 15444 / 15466 1.0 981.9 1.0X
+Parquet Vectorized 10776 / 10969 1.5 685.1 1.0X
+Parquet Vectorized (Pushdown) 10931 / 11012 1.4 695.0 1.0X
+Native ORC Vectorized 10849 / 10910 1.4 689.7 1.0X
+Native ORC Vectorized (Pushdown) 11030 / 11112 1.4 701.2 1.0X
+InMemoryTable Vectorized 8675 / 8822 1.8 551.6 1.2X
+InMemoryTable Vectorized (Pushdown) 8668 / 8741 1.8 551.1 1.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(18, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 5746 / 5763 2.7 365.3 1.0X
-Parquet Vectorized (Pushdown) 166 / 169 94.8 10.6 34.6X
-Native ORC Vectorized 5007 / 5023 3.1 318.3 1.1X
-Native ORC Vectorized (Pushdown) 2629 / 2640 6.0 167.1 2.2X
+Parquet Vectorized 3886 / 3932 4.0 247.0 1.0X
+Parquet Vectorized (Pushdown) 104 / 115 151.0 6.6 37.3X
+Native ORC Vectorized 3476 / 3527 4.5 221.0 1.1X
+Native ORC Vectorized (Pushdown) 1804 / 1839 8.7 114.7 2.2X
+InMemoryTable Vectorized 2956 / 3002 5.3 187.9 1.3X
+InMemoryTable Vectorized (Pushdown) 865 / 941 18.2 55.0 4.5X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(18, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 6827 / 6864 2.3 434.0 1.0X
-Parquet Vectorized (Pushdown) 1809 / 1827 8.7 115.0 3.8X
-Native ORC Vectorized 6287 / 6296 2.5 399.7 1.1X
-Native ORC Vectorized (Pushdown) 6364 / 6377 2.5 404.6 1.1X
+Parquet Vectorized 4498 / 4642 3.5 286.0 1.0X
+Parquet Vectorized (Pushdown) 1201 / 1242 13.1 76.3 3.7X
+Native ORC Vectorized 4218 / 4291 3.7 268.1 1.1X
+Native ORC Vectorized (Pushdown) 4262 / 4311 3.7 270.9 1.1X
+InMemoryTable Vectorized 3509 / 3599 4.5 223.1 1.3X
+InMemoryTable Vectorized (Pushdown) 1640 / 1702 9.6 104.3 2.7X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(18, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11315 / 11342 1.4 719.4 1.0X
-Parquet Vectorized (Pushdown) 8431 / 8450 1.9 536.0 1.3X
-Native ORC Vectorized 11591 / 11611 1.4 736.9 1.0X
-Native ORC Vectorized (Pushdown) 11424 / 11475 1.4 726.3 1.0X
+Parquet Vectorized 7445 / 7525 2.1 473.3 1.0X
+Parquet Vectorized (Pushdown) 5609 / 5732 2.8 356.6 1.3X
+Native ORC Vectorized 7056 / 7187 2.2 448.6 1.1X
+Native ORC Vectorized (Pushdown) 7226 / 7330 2.2 459.4 1.0X
+InMemoryTable Vectorized 5707 / 5801 2.8 362.8 1.3X
+InMemoryTable Vectorized (Pushdown) 4694 / 4810 3.4 298.4 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(18, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 15703 / 15712 1.0 998.4 1.0X
-Parquet Vectorized (Pushdown) 14982 / 15009 1.0 952.5 1.0X
-Native ORC Vectorized 16887 / 16955 0.9 1073.7 0.9X
-Native ORC Vectorized (Pushdown) 16518 / 16530 1.0 1050.2 1.0X
+Parquet Vectorized 10324 / 10427 1.5 656.4 1.0X
+Parquet Vectorized (Pushdown) 9916 / 10093 1.6 630.4 1.0X
+Native ORC Vectorized 10422 / 10539 1.5 662.6 1.0X
+Native ORC Vectorized (Pushdown) 10099 / 10261 1.6 642.1 1.0X
+InMemoryTable Vectorized 8220 / 8284 1.9 522.6 1.3X
+InMemoryTable Vectorized (Pushdown) 7809 / 7963 2.0 496.5 1.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 1 decimal(38, 2) row (value = 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 8101 / 8130 1.9 515.1 1.0X
-Parquet Vectorized (Pushdown) 184 / 187 85.6 11.7 44.1X
-Native ORC Vectorized 4998 / 5027 3.1 317.8 1.6X
-Native ORC Vectorized (Pushdown) 165 / 168 95.6 10.5 49.2X
+Parquet Vectorized 5346 / 5526 2.9 339.9 1.0X
+Parquet Vectorized (Pushdown) 116 / 121 136.2 7.3 46.3X
+Native ORC Vectorized 3483 / 3551 4.5 221.4 1.5X
+Native ORC Vectorized (Pushdown) 152 / 166 103.7 9.6 35.2X
+InMemoryTable Vectorized 4312 / 4344 3.6 274.2 1.2X
+InMemoryTable Vectorized (Pushdown) 878 / 930 17.9 55.8 6.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 10% decimal(38, 2) rows (value < 1572864): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 9405 / 9447 1.7 597.9 1.0X
-Parquet Vectorized (Pushdown) 2269 / 2275 6.9 144.2 4.1X
-Native ORC Vectorized 6167 / 6203 2.6 392.1 1.5X
-Native ORC Vectorized (Pushdown) 1783 / 1787 8.8 113.3 5.3X
+Parquet Vectorized 6229 / 6430 2.5 396.0 1.0X
+Parquet Vectorized (Pushdown) 1530 / 1560 10.3 97.3 4.1X
+Native ORC Vectorized 4407 / 4469 3.6 280.2 1.4X
+Native ORC Vectorized (Pushdown) 1361 / 1399 11.6 86.5 4.6X
+InMemoryTable Vectorized 5006 / 5129 3.1 318.3 1.2X
+InMemoryTable Vectorized (Pushdown) 1995 / 2037 7.9 126.8 3.1X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 50% decimal(38, 2) rows (value < 7864320): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 14700 / 14707 1.1 934.6 1.0X
-Parquet Vectorized (Pushdown) 10699 / 10712 1.5 680.2 1.4X
-Native ORC Vectorized 10687 / 10703 1.5 679.5 1.4X
-Native ORC Vectorized (Pushdown) 8364 / 8415 1.9 531.8 1.8X
+Parquet Vectorized 10057 / 10179 1.6 639.4 1.0X
+Parquet Vectorized (Pushdown) 7411 / 7518 2.1 471.2 1.4X
+Native ORC Vectorized 7911 / 8161 2.0 503.0 1.3X
+Native ORC Vectorized (Pushdown) 6199 / 6354 2.5 394.1 1.6X
+InMemoryTable Vectorized 8000 / 8128 2.0 508.6 1.3X
+InMemoryTable Vectorized (Pushdown) 6330 / 6413 2.5 402.4 1.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 90% decimal(38, 2) rows (value < 14155776): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 19780 / 19894 0.8 1257.6 1.0X
-Parquet Vectorized (Pushdown) 19003 / 19025 0.8 1208.1 1.0X
-Native ORC Vectorized 15385 / 15404 1.0 978.2 1.3X
-Native ORC Vectorized (Pushdown) 15032 / 15060 1.0 955.7 1.3X
+Parquet Vectorized 13756 / 13841 1.1 874.6 1.0X
+Parquet Vectorized (Pushdown) 12922 / 13129 1.2 821.6 1.1X
+Native ORC Vectorized 11386 / 11548 1.4 723.9 1.2X
+Native ORC Vectorized (Pushdown) 11170 / 11345 1.4 710.1 1.2X
+InMemoryTable Vectorized 11079 / 11194 1.4 704.4 1.2X
+InMemoryTable Vectorized (Pushdown) 10712 / 10793 1.5 681.0 1.3X
================================================================================================
Pushdown benchmark for InSet -> InFilters
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10521 / 10534 1.5 668.9 1.0X
-Parquet Vectorized (Pushdown) 677 / 691 23.2 43.1 15.5X
-Native ORC Vectorized 6768 / 6776 2.3 430.3 1.6X
-Native ORC Vectorized (Pushdown) 501 / 512 31.4 31.8 21.0X
+Parquet Vectorized 7187 / 7325 2.2 456.9 1.0X
+Parquet Vectorized (Pushdown) 447 / 460 35.2 28.4 16.1X
+Native ORC Vectorized 4850 / 4976 3.2 308.3 1.5X
+Native ORC Vectorized (Pushdown) 508 / 527 30.9 32.3 14.1X
+InMemoryTable Vectorized 3758 / 3828 4.2 238.9 1.9X
+InMemoryTable Vectorized (Pushdown) 1704 / 1761 9.2 108.4 4.2X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 50): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10531 / 10538 1.5 669.5 1.0X
-Parquet Vectorized (Pushdown) 677 / 718 23.2 43.0 15.6X
-Native ORC Vectorized 6765 / 6773 2.3 430.1 1.6X
-Native ORC Vectorized (Pushdown) 499 / 507 31.5 31.7 21.1X
+Parquet Vectorized 7145 / 7191 2.2 454.3 1.0X
+Parquet Vectorized (Pushdown) 442 / 461 35.6 28.1 16.2X
+Native ORC Vectorized 4911 / 5138 3.2 312.2 1.5X
+Native ORC Vectorized (Pushdown) 503 / 522 31.3 32.0 14.2X
+InMemoryTable Vectorized 3693 / 3781 4.3 234.8 1.9X
+InMemoryTable Vectorized (Pushdown) 1647 / 1733 9.6 104.7 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 5, distribution: 90): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10540 / 10553 1.5 670.1 1.0X
-Parquet Vectorized (Pushdown) 678 / 710 23.2 43.1 15.5X
-Native ORC Vectorized 6787 / 6794 2.3 431.5 1.6X
-Native ORC Vectorized (Pushdown) 501 / 509 31.4 31.9 21.0X
+Parquet Vectorized 7216 / 7350 2.2 458.8 1.0X
+Parquet Vectorized (Pushdown) 430 / 451 36.6 27.3 16.8X
+Native ORC Vectorized 4900 / 5100 3.2 311.5 1.5X
+Native ORC Vectorized (Pushdown) 520 / 542 30.2 33.1 13.9X
+InMemoryTable Vectorized 3639 / 3689 4.3 231.4 2.0X
+InMemoryTable Vectorized (Pushdown) 1647 / 1732 9.5 104.7 4.4X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10551 / 10559 1.5 670.8 1.0X
-Parquet Vectorized (Pushdown) 703 / 708 22.4 44.7 15.0X
-Native ORC Vectorized 6791 / 6802 2.3 431.7 1.6X
-Native ORC Vectorized (Pushdown) 519 / 526 30.3 33.0 20.3X
+Parquet Vectorized 7195 / 7290 2.2 457.4 1.0X
+Parquet Vectorized (Pushdown) 446 / 478 35.3 28.3 16.1X
+Native ORC Vectorized 4976 / 5229 3.2 316.4 1.4X
+Native ORC Vectorized (Pushdown) 520 / 543 30.2 33.1 13.8X
+InMemoryTable Vectorized 3635 / 3744 4.3 231.1 2.0X
+InMemoryTable Vectorized (Pushdown) 1684 / 1766 9.3 107.1 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 50): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10561 / 10565 1.5 671.4 1.0X
-Parquet Vectorized (Pushdown) 711 / 716 22.1 45.2 14.9X
-Native ORC Vectorized 6791 / 6806 2.3 431.8 1.6X
-Native ORC Vectorized (Pushdown) 529 / 537 29.8 33.6 20.0X
+Parquet Vectorized 7246 / 7375 2.2 460.7 1.0X
+Parquet Vectorized (Pushdown) 477 / 496 33.0 30.3 15.2X
+Native ORC Vectorized 4828 / 5158 3.3 307.0 1.5X
+Native ORC Vectorized (Pushdown) 524 / 540 30.0 33.3 13.8X
+InMemoryTable Vectorized 3695 / 3785 4.3 234.9 2.0X
+InMemoryTable Vectorized (Pushdown) 1702 / 1745 9.2 108.2 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 10, distribution: 90): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10572 / 10590 1.5 672.1 1.0X
-Parquet Vectorized (Pushdown) 713 / 716 22.1 45.3 14.8X
-Native ORC Vectorized 6808 / 6815 2.3 432.9 1.6X
-Native ORC Vectorized (Pushdown) 530 / 541 29.7 33.7 19.9X
+Parquet Vectorized 7303 / 7388 2.2 464.3 1.0X
+Parquet Vectorized (Pushdown) 471 / 477 33.4 29.9 15.5X
+Native ORC Vectorized 4848 / 5005 3.2 308.2 1.5X
+Native ORC Vectorized (Pushdown) 518 / 550 30.4 32.9 14.1X
+InMemoryTable Vectorized 3665 / 3790 4.3 233.0 2.0X
+InMemoryTable Vectorized (Pushdown) 1689 / 1717 9.3 107.4 4.3X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
InSet -> InFilters (values count: 50, distribution: 10): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 10871 / 10882 1.4 691.2 1.0X
-Parquet Vectorized (Pushdown) 11104 / 11110 1.4 706.0 1.0X
-Native ORC Vectorized 7088 / 7104 2.2 450.7 1.5X
-Native ORC Vectorized (Pushdown) 665 / 677 23.6 42.3 16.3X
+Parquet Vectorized 7474 / 7577 2.1 475.2 1.0X
+Parquet Vectorized (Pushdown) 7649 / 7707 2.1 486.3 1.0X
+Native ORC Vectorized 5167 / 5457 3.0 328.5 1.4X
+Native ORC Vectorized (Pushdown) 637
--- End diff --
The reason is that Parquet/ORC [set all nullability fields to true](https://github.com/apache/spark/blob/2d085c13b7f715dbff23dd1f81af45ff903d1a79/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L504-L505), but InMemoryTable does't. The `InferFiltersFromConstraints` optimizer will [add `isNotNull` to all columns](https://github.com/apache/spark/blob/d87d30e4fe9c9e91c462351e9f744a830db8d6fc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala#L93-L95).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter benchm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #98806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98806/testReport)** for PR 23027 at commit [`d0a2a3f`](https://github.com/apache/spark/commit/d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #98993 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98993/testReport)** for PR 23027 at commit [`01d01e7`](https://github.com/apache/spark/commit/01d01e7995ecb72705d0d610892dc99a6c3f4621).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/23027
I like new benchmark, but I still hope we have a separate suite for in-memory benchmark.
The current one looks misleading and inconsistent in this file.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98806/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum closed the pull request at:
https://github.com/apache/spark/pull/23027
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r235259340
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X
+Parquet Vectorized 7823 / 7996 2.0 497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X
+InMemoryTable Vectorized 6 / 6 2859.1 0.3 1422.0X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3023.0 0.3 1503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X
--- End diff --
I think the reason is [SPARK-22599](https://issues.apache.org/jira/browse/SPARK-22599). But if we cached all data to memory, the result is:
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
Parquet Vectorized 6491 / 6716 2.4 412.7 1.0X
Parquet Vectorized (Pushdown) 491 / 496 32.0 31.2 13.2X
Native ORC Vectorized 5849 / 6103 2.7 371.9 1.1X
Native ORC Vectorized (Pushdown) 533 / 572 29.5 33.9 12.2X
InMemoryTable Vectorized 2788 / 2854 5.6 177.2 2.3X
InMemoryTable Vectorized (Pushdown) 370 / 408 42.5 23.5 17.5X
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #99104 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99104/testReport)** for PR 23027 at commit [`b8c54ea`](https://github.com/apache/spark/commit/b8c54ea5048524f7df0b750a11a8fb109b43f479).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] Add InMemoryTable filter benchm...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5008/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99007/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5141/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #99007 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99007/testReport)** for PR 23027 at commit [`b8c54ea`](https://github.com/apache/spark/commit/b8c54ea5048524f7df0b750a11a8fb109b43f479).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
GitHub user wangyum reopened a pull request:
https://github.com/apache/spark/pull/23027
[SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InMemoryTable case
## What changes were proposed in this pull request?
`FilterPushdownBenchmark` add InMemoryTable case.
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-26049
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23027.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23027
----
commit d0a2a3f4af492fbf69f7774e03d67d4af39cd5c7
Author: Yuming Wang <yu...@...>
Date: 2018-11-14T00:35:35Z
Add InMemoryTable filter benchmark
commit 01d01e7995ecb72705d0d610892dc99a6c3f4621
Author: Yuming Wang <yu...@...>
Date: 2018-11-19T07:59:19Z
cache inMemoryTable from file
commit b8c54ea5048524f7df0b750a11a8fb109b43f479
Author: Yuming Wang <yu...@...>
Date: 2018-11-19T12:44:46Z
Fix path
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r233600002
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
================================================================================================
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X
+Parquet Vectorized 7823 / 7996 2.0 497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X
+InMemoryTable Vectorized 6 / 6 2859.1 0.3 1422.0X
+InMemoryTable Vectorized (Pushdown) 5 / 6 3023.0 0.3 1503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
-Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X
--- End diff --
Oh, it's slower than Orc/Parquet file in this case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r233603036
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala ---
@@ -131,6 +134,15 @@ object FilterPushdownBenchmark extends BenchmarkBase with SQLHelper {
}
}
+ Seq(false, true).foreach { pushDownEnabled =>
+ val name = s"InMemoryTable Vectorized ${if (pushDownEnabled) s"(Pushdown)" else ""}"
+ benchmark.addCase(name) { _ =>
+ withSQLConf(SQLConf.IN_MEMORY_PARTITION_PRUNING.key -> s"$pushDownEnabled") {
--- End diff --
@wangyum . `FilterPushdownBenchmark` is not related to `Partition Pruning`, isn't it?
This benchmark case will be misleading. I'd like to have another benchmark for this `IN_MEMORY_PARTITION_PRUNING `.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23027
**[Test build #98993 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98993/testReport)** for PR 23027 at commit [`01d01e7`](https://github.com/apache/spark/commit/01d01e7995ecb72705d0d610892dc99a6c3f4621).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5133/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23027
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98993/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23027
@wangyum, why did you close this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r233686986
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala ---
@@ -131,6 +134,15 @@ object FilterPushdownBenchmark extends BenchmarkBase with SQLHelper {
}
}
+ Seq(false, true).foreach { pushDownEnabled =>
+ val name = s"InMemoryTable Vectorized ${if (pushDownEnabled) s"(Pushdown)" else ""}"
+ benchmark.addCase(name) { _ =>
+ withSQLConf(SQLConf.IN_MEMORY_PARTITION_PRUNING.key -> s"$pushDownEnabled") {
--- End diff --
I think the InMemoryTable's partition same to Parquet RowGroup(@kiszk please correct if I'm wrong). We put them together and it's easy to compare performance.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org