(datafusion) branch main updated: bench: make wide_schema honor DATA_DIR like the other sql_benchmarks (#22836)

[gi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion.git


The following commit(s) were added to refs/heads/main by this push:
     new 9b81ff86d1 bench: make wide_schema honor DATA_DIR like the other sql_benchmarks (#22836)
9b81ff86d1 is described below

commit 9b81ff86d1adf35d953329eb05b4e6d3e0904ea8
Author: Adrian Garcia Badaracco <17...@users.noreply.github.com>
AuthorDate: Mon Jun 8 22:54:16 2026 -0400

    bench: make wide_schema honor DATA_DIR like the other sql_benchmarks (#22836)
    
    ## Which issue does this PR close?
    
    - Relates to #21968 (wide-schema benchmark work).
    
    ## Rationale for this change
    
    Every `sql_benchmarks` suite declares its external-table `LOCATION` as
    `${DATA_DIR:-data}/...`, and the matching `bench.sh run_*` forwards
    `DATA_DIR="${DATA_DIR}"` to the benchmark process — e.g. `clickbench`,
    `imdb`, `tpch`, `push_down_topk`, `h2o` all do this.
    
    `wide_schema` (added in #21970) diverged on **both** counts:
    - `wide_schema/init/load.sql` hardcoded `LOCATION
    'data/wide_schema/...'` (no `${DATA_DIR}`)
    - `run_wide_schema` did not forward `DATA_DIR`
    
    Run from inside the repo this happens to work, because the process CWD
    is the repo's `benchmarks/` and the relative `data/` path resolves. But
    it breaks whenever the data directory is not under the current directory
    — for example the CI benchmark runner builds/runs the benchmark from a
    separate source checkout while staging the dataset elsewhere and
    pointing at it via `DATA_DIR`. There, wide_schema fails at load:
    
    ```
    initialization failed: No files found at file:///.../benchmarks/data/wide_schema/wide/.
    Cannot infer schema from an empty location; either add data files or declare an explicit schema for the table.
    ```
    
    (The `bench.sh` variable `DATA_DIR` is set but not exported, so it only
    reaches the benchmark when a `run_*` function passes it explicitly —
    which `run_tpch` etc. do and `run_wide_schema` did not.)
    
    ## What changes are included in this PR?
    
    Two one-line fixes bringing `wide_schema` in line with the convention
    used by every other suite:
    - `load.sql`: `data/wide_schema/...` →
    `${DATA_DIR:-data}/wide_schema/...`
    - `run_wide_schema`: forward `DATA_DIR="${DATA_DIR}"` for both the
    `wide` and `narrow` subgroups
    
    ## Are these changes tested?
    
    Benchmark tooling only. The substitution path is the same one every
    other suite already relies on: the SQL harness's `process_replacements`
    resolves `${DATA_DIR:-data}` from the env (falling back to `data`), and
    `run_wide_schema` now forwards `DATA_DIR` exactly as `run_tpch` does.
    Verified `bench.sh` still parses (`bash -n`). Default in-repo behavior
    is unchanged (CWD-relative `data/`).
    
    ## Are there any user-facing changes?
    
    No. Benchmark harness only.
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    Co-authored-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
 benchmarks/bench.sh                                 | 2 ++
 benchmarks/sql_benchmarks/wide_schema/init/load.sql | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/benchmarks/bench.sh b/benchmarks/bench.sh
index 16b96da477..0e28beadf8 100755
--- a/benchmarks/bench.sh
+++ b/benchmarks/bench.sh
@@ -786,12 +786,14 @@ data_wide_schema() {
 run_wide_schema() {
     echo "Running wide_schema benchmark (wide subgroup)..."
     debug_run env BENCH_NAME=wide_schema BENCH_SUBGROUP=wide \
+      DATA_DIR="${DATA_DIR}" \
       SIMULATE_LATENCY="${SIMULATE_LATENCY}" \
       ${QUERY:+BENCH_QUERY="${QUERY}"}  \
       bash -c "$SQL_CARGO_COMMAND"
 
     echo "Running wide_schema benchmark (narrow baseline subgroup)..."
     debug_run env BENCH_NAME=wide_schema BENCH_SUBGROUP=narrow \
+      DATA_DIR="${DATA_DIR}" \
       SIMULATE_LATENCY="${SIMULATE_LATENCY}" \
       ${QUERY:+BENCH_QUERY="${QUERY}"}  \
       bash -c "$SQL_CARGO_COMMAND"
diff --git a/benchmarks/sql_benchmarks/wide_schema/init/load.sql b/benchmarks/sql_benchmarks/wide_schema/init/load.sql
index 4fbcda1d58..7248610677 100644
--- a/benchmarks/sql_benchmarks/wide_schema/init/load.sql
+++ b/benchmarks/sql_benchmarks/wide_schema/init/load.sql
@@ -3,4 +3,4 @@
 --   BENCH_SUBGROUP=wide   → 1024-col synthetic dataset (the actual benchmark)
 --   BENCH_SUBGROUP=narrow → 8-col baseline (companion only — meaningful
 --                           only when compared to the wide numbers)
-CREATE EXTERNAL TABLE events STORED AS PARQUET LOCATION 'data/wide_schema/${BENCH_SUBGROUP:-wide}/';
+CREATE EXTERNAL TABLE events STORED AS PARQUET LOCATION '${DATA_DIR:-data}/wide_schema/${BENCH_SUBGROUP:-wide}/';


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org