You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/27 19:42:49 UTC
[GitHub] [spark] MyeongKim opened a new pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
MyeongKim opened a new pull request #33544:
URL: https://github.com/apache/spark/pull/33544
### What changes were proposed in this pull request?
This PR adds a new option(run-tcp-ds-query-benchmark) in 'Run benchmarks' workflow of Github actions in order to support TPCDSQueryBenchmark.
When this option is true, the job will populate a specific matrix with `org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark` class and `"--data-location tpcds-sf-1"` argument which then executes the TPCDSQueryBenchmark class with the specific argument. Other benchmark classes runs in parallel from separate jobs.
To set up TPC-DS (SF=1) table data, a previous [merged PR](https://github.com/apache/spark/pull/31886/files) was referred
(NOTE: Also, we need to update 'Running benchmarks in your forked repository' section of https://spark.apache.org/developer-tools.html to address the new option.)
### Why are the changes needed?
To support a new benchmark class(TPCDSQueryBenchmark.scala) via Github Actions. Currently [this class is excluded](https://github.com/apache/spark/blob/71effba5f2f6f6e4755d0af3411d8d15d0e24577/core/src/test/scala/org/apache/spark/benchmark/Benchmarks.scala#L100) from this feature because the current structure doesn't take an argument for a specific benchmark class.
### Does this PR introduce _any_ user-facing change?
No. It's only for developers.
### How was this patch tested?
1. Manually tested in my fork repository
- [without TPCDSQueryBenchmark(the new option value equals to 'false')](https://github.com/MyeongKim/spark/actions/runs/1065162581)
- [with TPCDSQueryBenchmark(the new option value equals to 'true')](https://github.com/MyeongKim/spark/actions/runs/1065365313)
2. More test cases with the combinations below
| input class string | run-tcp-ds-query-benchmark flag | GA url |
| ------------- | ------------- | ------------- |
| org.apache.spark.ml.linalg.BLASBenchmark | false | https://github.com/MyeongKim/spark/actions/runs/1065165135 |
| org.apache.spark.ml.linalg.BLASBenchmark | true | https://github.com/MyeongKim/spark/actions/runs/1065369228 |
| org.apache.spark.sql.* | false | https://github.com/MyeongKim/spark/actions/runs/1066119336 |
| org.apache.spark.sql.* | true | https://github.com/MyeongKim/spark/actions/runs/1066398591 |
3. GA produces an expected `TPCDSQueryBenchmark-results.txt` result file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890592936
**[Test build #141933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141933/testReport)** for PR 33544 at commit [`88d0f8b`](https://github.com/apache/spark/commit/88d0f8b883cbb00ce505ec8d6af50fafbf35d2f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888199756
**[Test build #141770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141770/testReport)** for PR 33544 at commit [`e2e17c1`](https://github.com/apache/spark/commit/e2e17c16731c6b4f2059146cd34acdb64cd6b5d4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678765692
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ # Any TPC-DS related updates on this job need to be applied to tpcds-1g job of build_and_test.yml as well
+ generate-tpc-ds-dataset:
+ name: "Generate an input dataset for TPCDSQueryBenchmark"
+ if: github.event.inputs.run-tcp-ds-query-benchmark == 'true'
+ runs-on: ubuntu-20.04
+ env:
+ SPARK_GENERATE_BENCHMARK_FILES: 1
+ SPARK_LOCAL_IP: localhost
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ # In order to get diff files
+ with:
+ fetch-depth: 0
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/coursier
+ key: benchmark-coursier-${{ github.event.inputs.jdk }}-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ benchmark-coursier-${{ github.event.inputs.jdk }}
+ - name: Install Java ${{ github.event.inputs.jdk }}
+ uses: actions/setup-java@v1
+ with:
+ java-version: ${{ github.event.inputs.jdk }}
+ - name: Cache TPC-DS generated data
+ id: cache-tpcds-sf-1
+ uses: actions/cache@v2
+ with:
+ path: ./tpcds-sf-1
+ key: tpcds-${{ hashFiles('.github/workflows/benchmark.yml', 'sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala') }}
+ - name: Checkout tpcds-kit repository
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ uses: actions/checkout@v2
+ with:
+ repository: databricks/tpcds-kit
+ ref: 2a5078a782192ddb6efbcead8de9973d6ab4f069
+ path: ./tpcds-kit
+ - name: Build tpcds-kit
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: cd tpcds-kit/tools && make OS=LINUX
+ - name: Generate TPC-DS (SF=1) table data
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: build/sbt "sql/test:runMain org.apache.spark.sql.GenTPCDSData --dsdgenDir `pwd`/tpcds-kit/tools --location `pwd`/tpcds-sf-1 --scaleFactor 1 --numPartitions 1 --overwrite"
benchmark:
- name: "Run benchmarks: ${{ github.event.inputs.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
- needs: matrix-gen
+ name: "Run benchmarks: ${{ matrix.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
+ if: always()
+ needs: [matrix-gen, generate-tpc-ds-dataset]
# Ubuntu 20.04 is the latest LTS. The next LTS is 22.04.
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
- split: ${{fromJSON(needs.matrix-gen.outputs.matrix)}}
+ class: ${{ fromJSON(needs.matrix-gen.outputs.class) }}
Review comment:
e.g., you can do sth like: `class: ${{ github.event.inputs.class }}`.
BTW, why do we define another dimension in the matrix here? I think we can just:
1. Define one environment variable for the data location (e.g., `TPC_DS_DATA_LOC`) when `github.event.inputs.run-tcp-ds-query-benchmark` is true.
2. Enable TPC-DS benchmark here (https://github.com/apache/spark/pull/33544/files#diff-e50c749ae234bd48c56241408518145a665df366e62fc9978ad383a7b4afefddL99-L100) by appending the data location argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678765692
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ # Any TPC-DS related updates on this job need to be applied to tpcds-1g job of build_and_test.yml as well
+ generate-tpc-ds-dataset:
+ name: "Generate an input dataset for TPCDSQueryBenchmark"
+ if: github.event.inputs.run-tcp-ds-query-benchmark == 'true'
+ runs-on: ubuntu-20.04
+ env:
+ SPARK_GENERATE_BENCHMARK_FILES: 1
+ SPARK_LOCAL_IP: localhost
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ # In order to get diff files
+ with:
+ fetch-depth: 0
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/coursier
+ key: benchmark-coursier-${{ github.event.inputs.jdk }}-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ benchmark-coursier-${{ github.event.inputs.jdk }}
+ - name: Install Java ${{ github.event.inputs.jdk }}
+ uses: actions/setup-java@v1
+ with:
+ java-version: ${{ github.event.inputs.jdk }}
+ - name: Cache TPC-DS generated data
+ id: cache-tpcds-sf-1
+ uses: actions/cache@v2
+ with:
+ path: ./tpcds-sf-1
+ key: tpcds-${{ hashFiles('.github/workflows/benchmark.yml', 'sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala') }}
+ - name: Checkout tpcds-kit repository
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ uses: actions/checkout@v2
+ with:
+ repository: databricks/tpcds-kit
+ ref: 2a5078a782192ddb6efbcead8de9973d6ab4f069
+ path: ./tpcds-kit
+ - name: Build tpcds-kit
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: cd tpcds-kit/tools && make OS=LINUX
+ - name: Generate TPC-DS (SF=1) table data
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: build/sbt "sql/test:runMain org.apache.spark.sql.GenTPCDSData --dsdgenDir `pwd`/tpcds-kit/tools --location `pwd`/tpcds-sf-1 --scaleFactor 1 --numPartitions 1 --overwrite"
benchmark:
- name: "Run benchmarks: ${{ github.event.inputs.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
- needs: matrix-gen
+ name: "Run benchmarks: ${{ matrix.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
+ if: always()
+ needs: [matrix-gen, generate-tpc-ds-dataset]
# Ubuntu 20.04 is the latest LTS. The next LTS is 22.04.
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
- split: ${{fromJSON(needs.matrix-gen.outputs.matrix)}}
+ class: ${{ fromJSON(needs.matrix-gen.outputs.class) }}
Review comment:
e.g., you can do sth like: `class: ${{ github.event.inputs.class }}`.
BTW, why do we define the matrix here? I think we can just:
1. Define one environment variable for the data location (e.g., `TPC_DS_DATA_LOC`) when `github.event.inputs.run-tcp-ds-query-benchmark` is true.
2. Enable TPC-DS benchmark here (https://github.com/apache/spark/pull/33544/files#diff-e50c749ae234bd48c56241408518145a665df366e62fc9978ad383a7b4afefddL99-L100) by appending the data location argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888285403
**[Test build #141770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141770/testReport)** for PR 33544 at commit [`e2e17c1`](https://github.com/apache/spark/commit/e2e17c16731c6b4f2059146cd34acdb64cd6b5d4).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class: $`
* ` SPARK_BENCHMARK_CLASS: $`
* ` name: \"Run benchmarks: $`
* ` SPARK_BENCHMARK_CLASS: $`
* ` (matrix.class == 'org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark' &&`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678701487
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,107 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ generate-TCP-DS-dataset:
Review comment:
thank you for your comment! Addressed in an additional commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888685465
**[Test build #141792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141792/testReport)** for PR 33544 at commit [`7efd0b2`](https://github.com/apache/spark/commit/7efd0b22c6906ba729ab585e76bf2e355a5aea02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893873597
**[Test build #142114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142114/testReport)** for PR 33544 at commit [`8389a0f`](https://github.com/apache/spark/commit/8389a0f7c7febe1d3c9489ce18b61a1501f6f0d1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888266510
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46283/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888266510
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46283/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678762666
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
Review comment:
I think you won't have to pass it via GitHub outputs. `matrix` is being passed to parse a multi element JSON array but for `class` and `num-split`, we could just directly access it by `${{ steps.set-matrix.outputs.class }}` or `${{ steps.set-matrix.outputs.num-split }}`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678762666
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
Review comment:
I think you won't have to pass it via GitHub outputs. `matrix` is being passed to parse a multi element JSON array but for `class` and `num-split`, we could just directly access it by `${{ github.event.inputs.class }}` or `${{ github.event.inputs.num-splits }}`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888738551
**[Test build #141792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141792/testReport)** for PR 33544 at commit [`7efd0b2`](https://github.com/apache/spark/commit/7efd0b22c6906ba729ab585e76bf2e355a5aea02).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887809867
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893875824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887809867
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890592936
**[Test build #141933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141933/testReport)** for PR 33544 at commit [`88d0f8b`](https://github.com/apache/spark/commit/88d0f8b883cbb00ce505ec8d6af50fafbf35d2f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890753661
**[Test build #141933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141933/testReport)** for PR 33544 at commit [`88d0f8b`](https://github.com/apache/spark/commit/88d0f8b883cbb00ce505ec8d6af50fafbf35d2f1).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class RetryingBlockTransferor `
* `public final class Aggregation implements Serializable `
* `public final class Count implements AggregateFunc `
* `public final class CountStar implements AggregateFunc `
* `public final class Max implements AggregateFunc `
* `public final class Min implements AggregateFunc `
* `public final class Sum implements AggregateFunc `
* `trait AlterTableColumnCommand extends UnaryCommand `
* `case class AlterTableAddColumns(`
* `case class AlterTableReplaceColumns(`
* `case class CoalescedMapperPartitionSpec(`
* `trait AQEShuffleReadRule extends Rule[SparkPlan] `
* `case class CoalesceShufflePartitions(session: SparkSession) extends AQEShuffleReadRule `
* `case class ScanBuilderHolder(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887980018
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887937423
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888231014
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46283/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890608225
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46443/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893875824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142114/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893909889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #33544:
URL: https://github.com/apache/spark/pull/33544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r683832027
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
Review comment:
yes. right I think your approach is simpler. removed the matrix dimension
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893873597
**[Test build #142114 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142114/testReport)** for PR 33544 at commit [`8389a0f`](https://github.com/apache/spark/commit/8389a0f7c7febe1d3c9489ce18b61a1501f6f0d1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890591913
Thank you so much for your comments. Let me test the code with your approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888307417
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141770/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893866484
retriggered test cases with the combinations below
| input class string | run-tcp-ds-query-benchmark flag | GA url |
| ------------- | ------------- | ------------- |
| org.apache.spark.sql.* | true | https://github.com/MyeongKim/spark/actions/runs/1103168229 |
| org.apache.spark.sql.* | false | https://github.com/MyeongKim/spark/actions/runs/1103166487 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim edited a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim edited a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-898587171
@HyukjinKwon Hello Hyukjin, I've updated the yml file to follow your approach. Could you please advice me on how to proceed? Is there anything I should modify? Thank you in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887939284
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888685465
**[Test build #141792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141792/testReport)** for PR 33544 at commit [`7efd0b2`](https://github.com/apache/spark/commit/7efd0b22c6906ba729ab585e76bf2e355a5aea02).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887919508
**[Test build #141742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141742/testReport)** for PR 33544 at commit [`e2e17c1`](https://github.com/apache/spark/commit/e2e17c16731c6b4f2059146cd34acdb64cd6b5d4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890765818
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141933/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887919508
**[Test build #141742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141742/testReport)** for PR 33544 at commit [`e2e17c1`](https://github.com/apache/spark/commit/e2e17c16731c6b4f2059146cd34acdb64cd6b5d4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888685575
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888704286
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888307417
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141770/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678766666
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ # Any TPC-DS related updates on this job need to be applied to tpcds-1g job of build_and_test.yml as well
+ generate-tpc-ds-dataset:
+ name: "Generate an input dataset for TPCDSQueryBenchmark"
+ if: github.event.inputs.run-tcp-ds-query-benchmark == 'true'
+ runs-on: ubuntu-20.04
+ env:
+ SPARK_GENERATE_BENCHMARK_FILES: 1
+ SPARK_LOCAL_IP: localhost
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ # In order to get diff files
+ with:
+ fetch-depth: 0
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/coursier
+ key: benchmark-coursier-${{ github.event.inputs.jdk }}-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ benchmark-coursier-${{ github.event.inputs.jdk }}
+ - name: Install Java ${{ github.event.inputs.jdk }}
+ uses: actions/setup-java@v1
+ with:
+ java-version: ${{ github.event.inputs.jdk }}
+ - name: Cache TPC-DS generated data
+ id: cache-tpcds-sf-1
+ uses: actions/cache@v2
+ with:
+ path: ./tpcds-sf-1
+ key: tpcds-${{ hashFiles('.github/workflows/benchmark.yml', 'sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala') }}
+ - name: Checkout tpcds-kit repository
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ uses: actions/checkout@v2
+ with:
+ repository: databricks/tpcds-kit
+ ref: 2a5078a782192ddb6efbcead8de9973d6ab4f069
+ path: ./tpcds-kit
+ - name: Build tpcds-kit
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: cd tpcds-kit/tools && make OS=LINUX
+ - name: Generate TPC-DS (SF=1) table data
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: build/sbt "sql/test:runMain org.apache.spark.sql.GenTPCDSData --dsdgenDir `pwd`/tpcds-kit/tools --location `pwd`/tpcds-sf-1 --scaleFactor 1 --numPartitions 1 --overwrite"
benchmark:
- name: "Run benchmarks: ${{ github.event.inputs.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
- needs: matrix-gen
+ name: "Run benchmarks: ${{ matrix.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
+ if: always()
+ needs: [matrix-gen, generate-tpc-ds-dataset]
# Ubuntu 20.04 is the latest LTS. The next LTS is 22.04.
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
- split: ${{fromJSON(needs.matrix-gen.outputs.matrix)}}
+ class: ${{ fromJSON(needs.matrix-gen.outputs.class) }}
Review comment:
Let's also don't forget to update `object Benchmarks`'s docuentation. We can, for example, if you define `TPC_DS_DATA_LOC` environment variable with a proper data location, that triggers TPC-DS as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890598299
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46443/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888707197
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888199756
**[Test build #141770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141770/testReport)** for PR 33544 at commit [`e2e17c1`](https://github.com/apache/spark/commit/e2e17c16731c6b4f2059146cd34acdb64cd6b5d4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887980011
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141742/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-898587171
@HyukjinKwon Hello Hyukjin, Could you please advice me on how to proceed? Is there anything I should modify? Thank you in advance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890608225
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46443/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890603099
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46443/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888791842
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888750209
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141792/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888707197
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46305/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888791842
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887980011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888750209
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141792/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893892231
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888252860
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46283/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r683832157
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,108 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ # Any TPC-DS related updates on this job need to be applied to tpcds-1g job of build_and_test.yml as well
+ generate-tpc-ds-dataset:
+ name: "Generate an input dataset for TPCDSQueryBenchmark"
+ if: github.event.inputs.run-tcp-ds-query-benchmark == 'true'
+ runs-on: ubuntu-20.04
+ env:
+ SPARK_GENERATE_BENCHMARK_FILES: 1
+ SPARK_LOCAL_IP: localhost
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v2
+ # In order to get diff files
+ with:
+ fetch-depth: 0
+ - name: Cache Scala, SBT and Maven
+ uses: actions/cache@v2
+ with:
+ path: |
+ build/apache-maven-*
+ build/scala-*
+ build/*.jar
+ ~/.sbt
+ key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
+ restore-keys: |
+ build-
+ - name: Cache Coursier local repository
+ uses: actions/cache@v2
+ with:
+ path: ~/.cache/coursier
+ key: benchmark-coursier-${{ github.event.inputs.jdk }}-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ restore-keys: |
+ benchmark-coursier-${{ github.event.inputs.jdk }}
+ - name: Install Java ${{ github.event.inputs.jdk }}
+ uses: actions/setup-java@v1
+ with:
+ java-version: ${{ github.event.inputs.jdk }}
+ - name: Cache TPC-DS generated data
+ id: cache-tpcds-sf-1
+ uses: actions/cache@v2
+ with:
+ path: ./tpcds-sf-1
+ key: tpcds-${{ hashFiles('.github/workflows/benchmark.yml', 'sql/core/src/test/scala/org/apache/spark/sql/TPCDSSchema.scala') }}
+ - name: Checkout tpcds-kit repository
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ uses: actions/checkout@v2
+ with:
+ repository: databricks/tpcds-kit
+ ref: 2a5078a782192ddb6efbcead8de9973d6ab4f069
+ path: ./tpcds-kit
+ - name: Build tpcds-kit
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: cd tpcds-kit/tools && make OS=LINUX
+ - name: Generate TPC-DS (SF=1) table data
+ if: steps.cache-tpcds-sf-1.outputs.cache-hit != 'true'
+ run: build/sbt "sql/test:runMain org.apache.spark.sql.GenTPCDSData --dsdgenDir `pwd`/tpcds-kit/tools --location `pwd`/tpcds-sf-1 --scaleFactor 1 --numPartitions 1 --overwrite"
benchmark:
- name: "Run benchmarks: ${{ github.event.inputs.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
- needs: matrix-gen
+ name: "Run benchmarks: ${{ matrix.class }} (JDK ${{ github.event.inputs.jdk }}, ${{ matrix.split }} out of ${{ github.event.inputs.num-splits }} splits)"
+ if: always()
+ needs: [matrix-gen, generate-tpc-ds-dataset]
# Ubuntu 20.04 is the latest LTS. The next LTS is 22.04.
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
- split: ${{fromJSON(needs.matrix-gen.outputs.matrix)}}
+ class: ${{ fromJSON(needs.matrix-gen.outputs.class) }}
Review comment:
thank you. removed the matrix dimension and added a comment in Benchmarks object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893909871
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r677886174
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,107 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ generate-TCP-DS-dataset:
Review comment:
Can you leave a cross comment in build_and_test.yml so we don't forgot to update together?
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,107 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ generate-TCP-DS-dataset:
Review comment:
```suggestion
generate-tpc-ds-dataset:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MyeongKim commented on a change in pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
MyeongKim commented on a change in pull request #33544:
URL: https://github.com/apache/spark/pull/33544#discussion_r678701587
##########
File path: .github/workflows/benchmark.yml
##########
@@ -19,32 +19,107 @@ on:
description: 'Number of job splits'
required: true
default: '1'
+ run-tcp-ds-query-benchmark:
+ description: 'Run TPCDSQueryBenchmark: true or false'
+ required: false
+ default: 'false'
jobs:
matrix-gen:
name: Generate matrix for job splits
runs-on: ubuntu-20.04
outputs:
- matrix: ${{ steps.set-matrix.outputs.matrix }}
+ class: ${{ steps.set-matrix.outputs.class }}
+ split: ${{ steps.set-matrix.outputs.split }}
+ num-split: ${{ steps.set-matrix.outputs.num-split }}
env:
+ SPARK_BENCHMARK_CLASS: ${{ github.event.inputs.class }}
SPARK_BENCHMARK_NUM_SPLITS: ${{ github.event.inputs.num-splits }}
steps:
- name: Generate matrix
id: set-matrix
- run: echo "::set-output name=matrix::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ run: |
+ echo "::set-output name=class::['"$SPARK_BENCHMARK_CLASS"']"
+ echo "::set-output name=split::["`seq -s, 1 $SPARK_BENCHMARK_NUM_SPLITS`"]"
+ echo "::set-output name=num-split::['"$SPARK_BENCHMARK_NUM_SPLITS"']"
+
+ generate-TCP-DS-dataset:
Review comment:
Thanks! Addressed in an additional commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-888185918
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-974927271
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887967794
**[Test build #141742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141742/testReport)** for PR 33544 at commit [`e2e17c1`](https://github.com/apache/spark/commit/e2e17c16731c6b4f2059146cd34acdb64cd6b5d4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class: $`
* ` SPARK_BENCHMARK_CLASS: $`
* ` name: \"Run benchmarks: $`
* ` SPARK_BENCHMARK_CLASS: $`
* ` (matrix.class == 'org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark' &&`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887939284
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46255/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-887918626
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893909889
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46626/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-890765818
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141933/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33544: [SPARK-34927][INFRA] Support TPCDSQueryBenchmark in Benchmarks
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33544:
URL: https://github.com/apache/spark/pull/33544#issuecomment-893875804
**[Test build #142114 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142114/testReport)** for PR 33544 at commit [`8389a0f`](https://github.com/apache/spark/commit/8389a0f7c7febe1d3c9489ce18b61a1501f6f0d1).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` name: \"Run benchmarks: $`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org