[GitHub] [spark] HyukjinKwon commented on a change in pull request #31303: [SPARK-34211][SQL][TEST] Benchmark TPC-DS with 1GB scale factor

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #31303:
URL: https://github.com/apache/spark/pull/31303#discussion_r563230324



##########
File path: .github/workflows/build_and_test.yml
##########
@@ -430,3 +430,38 @@ jobs:
     - name: Build with SBT
       run: |
         ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Phadoop-2.7 compile test:compile
+
+  tpcds1g:
+    name: Benchmark TPC-DS with 1GB scale factor
+    runs-on: ubuntu-20.04
+    continue-on-error: true
+    steps:
+      - name: Checkout Spark repository
+        uses: actions/checkout@v2
+      - name: Checkout tpcds-kit repository
+        uses: actions/checkout@v2
+        with:
+          repository: databricks/tpcds-kit
+          path: ./tpcds-kit
+      - name: Checkout spark-sql-perf repository
+        uses: actions/checkout@v2
+        with:
+          repository: wangyum/spark-sql-perf
+          path: ./spark-sql-perf
+      - name: Install Java 8
+        uses: actions/setup-java@v1
+        with:
+          java-version: 8
+      - name: Build tpcds-kit
+        run: cd tpcds-kit/tools && make OS=LINUX
+      - name: Gen TPCDS Data
+        run: cd spark-sql-perf && build/sbt "test:runMain com.databricks.spark.sql.GenTPCDSData `pwd`/../tpcds-kit/tools 1 `pwd`/../tpcds1g parquet"
+      - name: Run TPCDSQueryBenchmark
+        run: |
+          SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location `pwd`/tpcds1g --cbo"

Review comment:
       Do we enable CBO by default? I think we should run it with the default conf.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org