You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Omega359 (via GitHub)" <gi...@apache.org> on 2026/08/30 21:25:21 UTC

[PR] Prepare SQL runner for native benchmark migration [datafusion]

Omega359 opened a new pull request, #24806:
URL: https://github.com/apache/datafusion/pull/24806

   ## Which issue does this PR close?
     
     - Part of #21706.
     - Prepares for #24311 and #24312.
     - This PR does not close these issues.
   
   ## Rationale for this change
   
   The remaining migration from native Rust benchmarks to SQL benchmarks will make `benchmark_runner` the primary path used by `bench.sh`. Before making that switch, the SQL runner needs to preserve the reporting behaviour of the native runner and keep Criterion results from different benchmark configurations separate.
   
     Two issues currently prevent that:
   
     1. In fixed-iteration mode, a SQL benchmark with cleanup DDL after its result-producing query can report the DDL row count instead of the query row count. This would produce incorrect row counts in the JSON output used by the benchmark comparison tools.
     2. Criterion derives its artifact paths from benchmark group and function names. `bench.sh` runs some suites under several configurations, such as TPC-H with different formats and scale factors or H2O with different formats and sizes. These runs can share the same SQL group and query names, causing their Criterion artifacts to collide.
   
     Fixing these issues allows the final migration to retain fixed-iteration reporting while offering Criterion as an optional mode for every migrated SQL benchmark.
   
     ## What changes are included in this PR?
   
     ### Preserve the result-producing query row count
   
     The streaming SQL runner now updates the reported row count only for `SELECT` and `WITH` statements. It still executes subsequent DDL, but that DDL no longer replaces the row count from the last result-producing query.
   
     The buffered and streaming paths now use the same result-statement classification.
   
     This matches the behaviour needed by the native-to-SQL migration: fixed-iteration benchmark output continues to report the number of rows produced by the workload rather than zero from a trailing cleanup statement.
   
     ### Add Criterion result namespaces
   
     The SQL Criterion runner accepts an optional namespace and appends it to the Criterion group name. The standalone runner exposes this through:
   
     - `--criterion-namespace <NAMESPACE>`
     - `BENCH_NAMESPACE`
   
     This lets future `bench.sh` Criterion runs assign distinct groups such as `tpch__parquet_sf1` and `tpch__memory_sf1`, preventing one configuration from overwriting another configuration's measurements.
   
     Namespaces are restricted to lowercase ASCII letters, digits, `_`, and `-`. Lowercase-only names avoid collisions on case-insensitive filesystems, while the character restrictions prevent a namespace from introducing extra path components.
   
     The combined Criterion group and namespace must not exceed 64 bytes. Criterion truncates longer directory components, so accepting longer names could make otherwise distinct namespaces resolve to the same artifact directory.
   
     The runner validates all selected group names before starting any benchmarks. An invalid namespace therefore fails the invocation before Criterion writes partial results.
   
     ### Share direct Criterion harness configuration
   
     The direct `cargo bench --bench sql` harness now obtains its `BENCH_*` configuration from `sql_benchmark_runner`.
   
     Keeping this parsing with the shared runner ensures that both Criterion entry points support the same namespace behaviour:
   
     - `benchmark_runner --criterion`
     - `cargo bench --bench sql`
   
     It also makes the environment-based harness configuration testable without putting configuration logic in the thin Criterion entry point.
   
     ### Add regression coverage
   
     The tests cover:
   
     - Preserving the row count from a `SELECT` followed by cleanup DDL.
     - Safe and unsafe Criterion namespace values.
     - Case-insensitive filesystem collision prevention.
     - Criterion's 64-byte directory-name limit.
     - `BENCH_NAMESPACE` handling in both entry points.
     - Rejection of `--criterion-namespace` outside Criterion mode.
     - Separate Criterion artifact directories for different namespaces.
   
     The environment-variable tests use subprocesses where needed so process-wide environment changes cannot leak into concurrently running tests.
   
     ## What is the testing strategy for this PR?
   
     The following checks were run:
   
     ```shell
     cargo test -p datafusion-benchmarks --lib --bins
     cargo check -p datafusion-benchmarks --bench sql
     ./dev/rust_lint.sh
     git diff --check
   
     The tests include an artifact-level check that runs the same SQL benchmark under two namespaces and verifies that Criterion writes separate  estimates.json files.
   
     ## Are there any user-facing changes?
   
     Yes, for benchmark users.
   
     Criterion-mode SQL benchmarks gain the optional --criterion-namespace argument and corresponding BENCH_NAMESPACE environment variable. Fixed-
     iteration SQL benchmark output now reports the last result-producing query's row count when later DDL statements execute.
   
     There are no changes to DataFusion's public library APIs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #24806:
URL: https://github.com/apache/datafusion/pull/24806#issuecomment-5641401002

   Darn, sorry. I had worked on it last night but I didn't get around to completing it and pushing the fixes. I'll create a new PR to followup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #24806:
URL: https://github.com/apache/datafusion/pull/24806#issuecomment-5471560371

   fyi @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24806:
URL: https://github.com/apache/datafusion/pull/24806#issuecomment-5471467414

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24806?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `94.69965%` with `15 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 81.53%. Comparing base ([`e4cf35c`](https://app.codecov.io/gh/apache/datafusion/commit/e4cf35cbca92e6fda855d5eac4104b47e0e600e9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2288cfc`](https://app.codecov.io/gh/apache/datafusion/commit/2288cfc3500785af8de58b72aa1fd99c3803d826?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24806?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [benchmarks/src/bin/benchmark\_runner.rs](https://app.codecov.io/gh/apache/datafusion/pull/24806?src=pr&el=tree&filepath=benchmarks%2Fsrc%2Fbin%2Fbenchmark_runner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YmVuY2htYXJrcy9zcmMvYmluL2JlbmNobWFya19ydW5uZXIucnM=) | 92.40% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24806?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [benchmarks/src/sql\_benchmark\_runner.rs](https://app.codecov.io/gh/apache/datafusion/pull/24806?src=pr&el=tree&filepath=benchmarks%2Fsrc%2Fsql_benchmark_runner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YmVuY2htYXJrcy9zcmMvc3FsX2JlbmNobWFya19ydW5uZXIucnM=) | 97.16% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24806?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@           Coverage Diff            @@
   ##             main   #24806    +/-   ##
   ========================================
     Coverage   81.52%   81.53%            
   ========================================
     Files        1123     1123            
     Lines      406148   406423   +275     
     Branches   406148   406423   +275     
   ========================================
   + Hits       331124   331378   +254     
   - Misses      55659    55677    +18     
   - Partials    19365    19368     +3     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24806?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #24806:
URL: https://github.com/apache/datafusion/pull/24806


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #24806:
URL: https://github.com/apache/datafusion/pull/24806#issuecomment-5585447323

   ping @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #24806:
URL: https://github.com/apache/datafusion/pull/24806#discussion_r3960392320


##########
benchmarks/benches/sql.rs:
##########
@@ -40,61 +38,21 @@ static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;
 #[global_allocator]
 static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;
 
-#[derive(Debug, Parser)]
-#[command(ignore_errors = true)]
-struct EnvParser {
-    #[command(flatten)]
-    options: CommonOpt,
-
-    #[arg(
-        env = "BENCH_PERSIST_RESULTS",
-        long = "persist_results",
-        default_value = "false",
-        action = clap::ArgAction::SetTrue
-    )]
-    persist_results: bool,
-
-    #[arg(
-        env = "BENCH_VALIDATE",
-        long = "validate_results",
-        default_value = "false",
-        action = clap::ArgAction::SetTrue
-    )]
-    validate: bool,
-
-    #[arg(env = "BENCH_NAME")]
-    name: Option<String>,
-
-    #[arg(env = "BENCH_SUBGROUP")]
-    subgroup: Option<String>,
-
-    #[arg(env = "BENCH_QUERY")]
-    query: Option<String>,
-}
-
 pub fn sql(c: &mut Criterion) {
     env_logger::init();
 
     let start = Instant::now();
-    let args = EnvParser::parse();
-    let config = SqlRunConfig {
-        common: args.options,
-        filter: BenchmarkFilter {
-            name: args.name,
-            subgroup: args.subgroup,
-            query: args.query,
-        },
-        replacements: default_criterion_replacements(),
-        query_filename: None,
-        persist_results: args.persist_results,
-        validate_results: args.validate,
-        output: None,
-    };
+    let (config, criterion_namespace) = criterion_harness_config_from_env();

Review Comment:
   👍 for extracting to a method



##########
benchmarks/src/bin/benchmark_runner.rs:
##########
@@ -70,10 +70,28 @@ enum CliAction {
     Criterion {
         config: SqlRunConfig,
         save_baseline: Option<String>,
+        criterion_namespace: Option<String>,
     },
     DryRun(DryRunOutput),
 }
 
+fn parse_criterion_namespace(value: &str) -> std::result::Result<String, String> {

Review Comment:
   This appears to duplicate  [`validate_criterion_namespace` in `sql_benchmark_runner.rs`](https://github.com/apache/datafusion/blob/d69046a2332ef85a95d00861bdda08f4556aacad/benchmarks/src/sql_benchmark_runner.rs#L171-L189) and could drift apart. 
   
   The clap `value_parser` could delegate to a shared public validator in `sql_benchmark_runner`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Prepare SQL runner for native benchmark migration [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24806:
URL: https://github.com/apache/datafusion/pull/24806#issuecomment-5640992221

   Let's get this one done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org