You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sameeragarwal <gi...@git.apache.org> on 2016/04/14 03:35:28 UTC
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
GitHub user sameeragarwal opened a pull request:
https://github.com/apache/spark/pull/12379
[SPARK-14620][SQL][WIP] Use/benchmark a better hash in AggregateHashMap
## What changes were proposed in this pull request?
This PR uses a better hashing algorithm while probing the AggregateHashMap:
```java
long h = 0
h = (h << 5) - h + key_1
h = (h << 5) - h + key_2
...
h = (h << 5) - h + key_n
return h
```
Depends on: https://github.com/apache/spark/pull/12345
## How was this patch tested?
Java HotSpot(TM) 64-Bit Server VM 1.8.0_73-b02 on Mac OS X 10.11.4
Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
Aggregate w keys: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
codegen = F 2417 / 2457 8.7 115.2 1.0X
codegen = T hashmap = F 1554 / 1581 13.5 74.1 1.6X
codegen = T hashmap = T 877 / 929 23.9 41.8 2.8X
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sameeragarwal/spark hash
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12379.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12379
----
commit 7c158bd137f057453d17ef360906e5be90bf5004
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-03-31T21:15:34Z
[SPARK-14394]
commit ebaea6a87b704afedd47bdd2dd17c92c3ffc6e8e
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-07T00:37:08Z
Integrating AggregateHashMap for Aggregates with Group By
commit cee7e65b3cf7569b4e46941158f164c2130c3981
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-12T17:33:42Z
Add SQLConf
commit 8c9e17a1d40e3014e39b1d04f3a458aa129784f8
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-12T23:01:03Z
20ns
commit 3379294b76d91a55dbe86e31efb9812c8d37768c
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-12T23:18:36Z
generated code
commit 4ee56873764d62efdaf8c47cb74aa399f2194fde
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-13T01:23:27Z
benchmark
commit c2fc38584dd073036a1f04f7cd7da9fcf50739e8
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-13T01:58:01Z
fix comment
commit fc6b8cb337e11d3d92f0f13828b8a0b85e30929c
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-13T18:48:56Z
enable conf by default for testing
commit ececd5770e0c5410cf3da463f3b52db467d1f5ca
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-13T22:12:16Z
fix tests
commit 0ca0db17130bb6a1e59cdbc699f5abd946821d44
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-13T22:31:24Z
review comments
commit 555bcd2c9ef31818141f1d1434d17f42b8ff8acb
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-14T00:02:50Z
hash
commit bbb966338a311dd2bf4b0dd7194bd29ebb04ce48
Author: Sameer Agarwal <sa...@databricks.com>
Date: 2016-04-14T01:27:41Z
hash function
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210298508
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210298511
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55896/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210246419
**[Test build #55877 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55877/consoleFull)** for PR 12379 at commit [`1fc03d1`](https://github.com/apache/spark/commit/1fc03d15004cc98008678ab381e538375be67bfa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-209715914
**[Test build #55781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55781/consoleFull)** for PR 12379 at commit [`bbb9663`](https://github.com/apache/spark/commit/bbb966338a311dd2bf4b0dd7194bd29ebb04ce48).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by sameeragarwal <gi...@git.apache.org>.
Github user sameeragarwal commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210289458
cc @nongli
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12379
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-209731555
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12379#discussion_r59904543
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala ---
@@ -180,9 +180,47 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
Aggregate w keys: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
- codegen = F 2219 / 2392 9.4 105.8 1.0X
- codegen = T hashmap = F 1330 / 1466 15.8 63.4 1.7X
- codegen = T hashmap = T 384 / 518 54.7 18.3 5.8X
+ codegen = F 2323 / 2567 9.0 110.8 1.0X
+ codegen = T hashmap = F 1182 / 1246 17.7 56.4 2.0X
+ codegen = T hashmap = T 381 / 489 55.0 18.2 6.1X
+ */
+ }
+
+ ignore("aggregate with randomized keys") {
+ val N = 20 << 20
+
+ val benchmark = new Benchmark("Aggregate w keys", N)
+ sqlContext.range(N).selectExpr("id", "floor(rand() * 10000) as k").registerTempTable("test")
+
+ def f(): Unit = sqlContext.sql("select k, k, sum(id) from test group by k, k").collect()
--- End diff --
what does this plan look like?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210246891
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210673678
**[Test build #55962 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55962/consoleFull)** for PR 12379 at commit [`1562d46`](https://github.com/apache/spark/commit/1562d46af39b8e78d8265f046f52f1753cb484a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210298083
**[Test build #55896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55896/consoleFull)** for PR 12379 at commit [`657edf5`](https://github.com/apache/spark/commit/657edf5c5ce16df1e35ef593d115c6e6205d0951).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210649310
**[Test build #55962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55962/consoleFull)** for PR 12379 at commit [`1562d46`](https://github.com/apache/spark/commit/1562d46af39b8e78d8265f046f52f1753cb484a7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-209731561
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55781/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210220435
**[Test build #55877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55877/consoleFull)** for PR 12379 at commit [`1fc03d1`](https://github.com/apache/spark/commit/1fc03d15004cc98008678ab381e538375be67bfa).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210246894
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55877/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-209731471
**[Test build #55781 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55781/consoleFull)** for PR 12379 at commit [`bbb9663`](https://github.com/apache/spark/commit/bbb966338a311dd2bf4b0dd7194bd29ebb04ce48).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210673919
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by sameeragarwal <gi...@git.apache.org>.
Github user sameeragarwal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12379#discussion_r59913017
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala ---
@@ -180,9 +180,47 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
Aggregate w keys: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
- codegen = F 2219 / 2392 9.4 105.8 1.0X
- codegen = T hashmap = F 1330 / 1466 15.8 63.4 1.7X
- codegen = T hashmap = T 384 / 518 54.7 18.3 5.8X
+ codegen = F 2323 / 2567 9.0 110.8 1.0X
+ codegen = T hashmap = F 1182 / 1246 17.7 56.4 2.0X
+ codegen = T hashmap = T 381 / 489 55.0 18.2 6.1X
+ */
+ }
+
+ ignore("aggregate with randomized keys") {
+ val N = 20 << 20
+
+ val benchmark = new Benchmark("Aggregate w keys", N)
+ sqlContext.range(N).selectExpr("id", "floor(rand() * 10000) as k").registerTempTable("test")
+
+ def f(): Unit = sqlContext.sql("select k, k, sum(id) from test group by k, k").collect()
--- End diff --
```
WholeStageCodegen
: +- TungstenAggregate(key=[k#3L,k#3L], functions=[(sum(id#0L),mode=Final,isDistinct=false)], output=[k#3L,k#3L,sum(id)#183L])
: +- INPUT
+- Exchange hashpartitioning(k#3L, k#3L, 1), None
+- WholeStageCodegen
: +- TungstenAggregate(key=[k#3L,k#3L], functions=[(sum(id#0L),mode=Partial,isDistinct=false)], output=[k#3L,k#3L,sum#185L])
: +- Project [id#0L,FLOOR((rand(-9053518532274118725) * 10000.0)) AS k#3L]
: +- Range 0, 1, 1, 20971520, [id#0L]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210673921
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55962/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on a diff in the pull request:
https://github.com/apache/spark/pull/12379#discussion_r59921018
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala ---
@@ -180,9 +180,47 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
Aggregate w keys: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
- codegen = F 2219 / 2392 9.4 105.8 1.0X
- codegen = T hashmap = F 1330 / 1466 15.8 63.4 1.7X
- codegen = T hashmap = T 384 / 518 54.7 18.3 5.8X
+ codegen = F 2323 / 2567 9.0 110.8 1.0X
+ codegen = T hashmap = F 1182 / 1246 17.7 56.4 2.0X
+ codegen = T hashmap = T 381 / 489 55.0 18.2 6.1X
+ */
+ }
+
+ ignore("aggregate with randomized keys") {
+ val N = 20 << 20
+
+ val benchmark = new Benchmark("Aggregate w keys", N)
+ sqlContext.range(N).selectExpr("id", "floor(rand() * 10000) as k").registerTempTable("test")
+
+ def f(): Unit = sqlContext.sql("select k, k, sum(id) from test group by k, k").collect()
--- End diff --
I feel you're spending a lot of time in rand
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by nongli <gi...@git.apache.org>.
Github user nongli commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210667378
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL] Use/benchmark a better hash...
Posted by sameeragarwal <gi...@git.apache.org>.
Github user sameeragarwal commented on a diff in the pull request:
https://github.com/apache/spark/pull/12379#discussion_r59921348
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala ---
@@ -180,9 +180,47 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
Intel(R) Core(TM) i7-4960HQ CPU @ 2.60GHz
Aggregate w keys: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
-------------------------------------------------------------------------------------------
- codegen = F 2219 / 2392 9.4 105.8 1.0X
- codegen = T hashmap = F 1330 / 1466 15.8 63.4 1.7X
- codegen = T hashmap = T 384 / 518 54.7 18.3 5.8X
+ codegen = F 2323 / 2567 9.0 110.8 1.0X
+ codegen = T hashmap = F 1182 / 1246 17.7 56.4 2.0X
+ codegen = T hashmap = T 381 / 489 55.0 18.2 6.1X
+ */
+ }
+
+ ignore("aggregate with randomized keys") {
+ val N = 20 << 20
+
+ val benchmark = new Benchmark("Aggregate w keys", N)
+ sqlContext.range(N).selectExpr("id", "floor(rand() * 10000) as k").registerTempTable("test")
+
+ def f(): Unit = sqlContext.sql("select k, k, sum(id) from test group by k, k").collect()
--- End diff --
I see. Doesn't `registerTempTable` materialize the table before registering it? If not, is there a way to do that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14620][SQL][WIP] Use/benchmark a better...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12379#issuecomment-210282632
**[Test build #55896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55896/consoleFull)** for PR 12379 at commit [`657edf5`](https://github.com/apache/spark/commit/657edf5c5ce16df1e35ef593d115c6e6205d0951).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org