You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Nam0101 (via GitHub)" <gi...@apache.org> on 2026/07/13 07:00:49 UTC

[PR] fix: resolve #1845 — Add TPC-DS workflow [datafusion-ballista]

Nam0101 opened a new pull request, #2022:
URL: https://github.com/apache/datafusion-ballista/pull/2022

   ## Summary
   
   fix: resolve #1845 — Add TPC-DS workflow
   
   ## Problem
   
   **Severity**: `High` | **File**: `benchmarks/tpcds/generate_data.sh (or same pattern as TPCH data gen)`
   
   TPCH uses dbgen (or downloads). TPC-DS uses dsdgen from TPC tools. CI must produce parquet (or CSV then convert) under a stable path the workflow mounts.
   
   ## Solution
   
   Copy TPCH generate script. Swap tool to dsdgen; SF=1; output parquet preferred (Ballista/DataFusion path). Document tool download URL/version pin. If building dsdgen in CI is heavy, cache tarball or use prebuilt like TPCH does. Tables: full TPC-DS schema for SF1. `ponytail: SF1 only; SF10 via input when CI budget allows.`
   
   ## Changes
   
   - `benchmarks/tpcds/generate_data.sh` (new)
   
   ## Testing
   
   - [ ] Existing tests pass
   - [ ] Manual review completed
   - [ ] No new warnings/errors introduced
   
   ---
   _Note: this change was drafted with AI assistance and reviewed locally before submission._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: resolve #1845 — Add TPC-DS workflow [datafusion-ballista]

Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on code in PR #2022:
URL: https://github.com/apache/datafusion-ballista/pull/2022#discussion_r3583992458


##########
benchmarks/tpcds/generate_data.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Generate TPC-DS SF1 data as Parquet for Ballista benchmarks.
+# Requires dsdgen from TPC-DS tools (https://www.tpc.org/tpcds/).

Review Comment:
   It can also output to parquet directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: resolve #1845 — Add TPC-DS workflow [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #2022: fix: resolve #1845 — Add TPC-DS workflow
URL: https://github.com/apache/datafusion-ballista/pull/2022


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: resolve #1845 — Add TPC-DS workflow [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2022:
URL: https://github.com/apache/datafusion-ballista/pull/2022#discussion_r3571065680


##########
benchmarks/tpcds/generate_data.sh:
##########
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+# Generate TPC-DS SF1 data as Parquet for Ballista benchmarks.
+# Requires dsdgen from TPC-DS tools (https://www.tpc.org/tpcds/).

Review Comment:
   I'd recommend using https://github.com/clflushopt/tpchgen-rs instead. It supports generating both TPC-H and TPC-DS and is much faster than dsdgen



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: resolve #1845 — Add TPC-DS workflow [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2022:
URL: https://github.com/apache/datafusion-ballista/pull/2022#issuecomment-5034136263

   Thanks for starting this @Nam0101 but we now have TPC-DS enabled in CI, so will close this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org