You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/09/20 08:41:59 UTC
[GitHub] spark pull request #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBen...
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/22486
[SPARK-25478][TEST] Refactor CompressionSchemeBenchmark to use main method
## What changes were proposed in this pull request?
Refactor `CompressionSchemeBenchmark` to use main method.
To gererate benchmark result:
```sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.columnar.compression.CompressionSchemeBenchmark"
```
## How was this patch tested?
manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-25478
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22486.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22486
----
commit 4dc46ad21e32784d42ab4b052ba73e31a050efb8
Author: Yuming Wang <yu...@...>
Date: 2018-09-20T08:38:04Z
Refactor CompressionSchemeBenchmark
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3393/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22486
@dongjoon-hyun Thanks a lot.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSche...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22486
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96339/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22486
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22486
**[Test build #96494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96494/testReport)** for PR 22486 at commit [`9494afd`](https://github.com/apache/spark/commit/9494afd9e649751188b52fee5ac30d745985a03c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3369/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBen...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22486#discussion_r219711018
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/compression/CompressionSchemeBenchmark.scala ---
@@ -318,28 +229,17 @@ object CompressionSchemeBenchmark extends AllCompressionSchemes {
}
testData.rewind()
- // Intel(R) Core(TM) i7-4578U CPU @ 3.00GHz
- // STRING Encode: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
- // -------------------------------------------------------------------------------------------
- // PassThrough(1.000) 56 / 57 1197.9 0.8 1.0X
- // RunLengthEncoding(0.893) 4892 / 4937 13.7 72.9 0.0X
- // DictionaryEncoding(0.167) 2968 / 2992 22.6 44.2 0.0X
runEncodeBenchmark("STRING Encode", iters, count, STRING, testData)
-
- // Intel(R) Core(TM) i7-4578U CPU @ 3.00GHz
- // STRING Decode: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
- // -------------------------------------------------------------------------------------------
- // PassThrough 2422 / 2449 27.7 36.1 1.0X
- // RunLengthEncoding 2885 / 3018 23.3 43.0 0.8X
- // DictionaryEncoding 2716 / 2752 24.7 40.5 0.9X
runDecodeBenchmark("STRING Decode", iters, count, STRING, testData)
}
- def main(args: Array[String]): Unit = {
- bitEncodingBenchmark(1024)
- shortEncodingBenchmark(1024)
- intEncodingBenchmark(1024)
- longEncodingBenchmark(1024)
- stringEncodingBenchmark(1024)
+ override def benchmark(): Unit = {
+ runBenchmark("encoding benchmark") {
--- End diff --
How about `Compression Scheme Benchmark`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22486
Could you add `[SQL]` at the title?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22486
**[Test build #96339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96339/testReport)** for PR 22486 at commit [`4dc46ad`](https://github.com/apache/spark/commit/4dc46ad21e32784d42ab4b052ba73e31a050efb8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22486
**[Test build #96339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96339/testReport)** for PR 22486 at commit [`4dc46ad`](https://github.com/apache/spark/commit/4dc46ad21e32784d42ab4b052ba73e31a050efb8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22486
**[Test build #96462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96462/testReport)** for PR 22486 at commit [`69bde3a`](https://github.com/apache/spark/commit/69bde3a6e8a81d2a90f3cccae16a1eb83ee71aab).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/22486
Hi, @wangyum . If you don't mind, could you review and merge [my PR](https://github.com/wangyum/spark/pull/10) to your branch? The BM title is updated and latest OpenJDK on AWS is used.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96494/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][SQL][TEST] Refactor CompressionSchemeBench...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22486
**[Test build #96494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96494/testReport)** for PR 22486 at commit [`9494afd`](https://github.com/apache/spark/commit/9494afd9e649751188b52fee5ac30d745985a03c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96462/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22486
**[Test build #96462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96462/testReport)** for PR 22486 at commit [`69bde3a`](https://github.com/apache/spark/commit/69bde3a6e8a81d2a90f3cccae16a1eb83ee71aab).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22486: [SPARK-25478][TEST] Refactor CompressionSchemeBenchmark ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3291/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org