You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/08/13 14:55:03 UTC
[PR] bench: add parquet_row_filter_skip suite for the fully-matched RowFilter skip [datafusion]
zhuqi-lucas opened a new pull request, #24328:
URL: https://github.com/apache/datafusion/pull/24328
## Which issue does this PR close?
- Part of #23696 — a benchmark that mechanistically exercises the per-RG fully-matched `RowFilter` skip.
## Rationale for this change
#23696 adds a per-row-group fully-matched `RowFilter` skip, but **none of the existing benchmarks exercise it**:
- `sort_tpch` / `tpch` don't enable `pushdown_filters` by default, so there is no `RowFilter` to skip;
- ClickBench's `URL LIKE …` / equality predicates rarely make a row group's min/max fall entirely inside the satisfying range, so fully-matched RGs are rare.
As raised in review (we should verify the optimization improves something mechanistically, otherwise add a benchmark first), this adds a suite that **necessarily** triggers the skip.
## What changes are included in this PR?
A new `sql_benchmarks/parquet_row_filter_skip/` suite:
- The **load SQL** enables `pushdown_filters` and `COPY`s a clustered Parquet file — a fixed-width, zero-padded, monotonically increasing string key (`skey`) so each row group holds a disjoint, sorted range — plus 14 payload columns.
- The **query** applies a low-selectivity range filter (`skey >= '0000100000'`, `skey` not projected). The first row group straddles the threshold; every later RG is fully matched by statistics, so the per-row `RowFilter` is skipped on the fully-matched run (and `skey` isn't decoded there).
- `bench.sh` integration: `./bench.sh run parquet_row_filter_skip`, data generated inline by the load SQL. Knobs: `PRED_ROWS` (row count), `RG_SIZE` (parquet row-group size).
## Are these changes tested?
Smoke-tested locally via `cargo bench --bench sql -- --test`.
Local A/B (main vs #23696, 10M rows / 10 RGs / `skey >= '0000100000'`):
- **9 of 10 row groups fully matched** → `row_filter_skipped_fully_matched=9`;
- **~18% faster** with the optimization (main ~0.143s → branch ~0.117s); an int64-key variant is ~12%.
Once this lands, `run benchmark parquet_row_filter_skip` will compare any PR (e.g. #23696) against `main` in CI.
## Are there any user-facing changes?
No — benchmark only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] bench: add parquet_row_filter_skip suite for the fully-matched RowFilter skip [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24328:
URL: https://github.com/apache/datafusion/pull/24328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] bench: add parquet_row_filter_skip suite for the fully-matched RowFilter skip [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #24328:
URL: https://github.com/apache/datafusion/pull/24328#discussion_r3776462682
##########
benchmarks/bench.sh:
##########
@@ -107,6 +107,9 @@ wide_schema: Small-projection queries on a wide synthetic dataset (10
predicate_eval: Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an
adaptive predicate-ordering system behaves across them (see https://github.com/apache/datafusion/issues/11262)
(subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift)
+parquet_row_filter_skip: Per-RG fully-matched RowFilter skip on Parquet (apache/datafusion#23696); clustered string key + low-selectivity
+ range filter + pushdown, so most row groups are fully matched and the per-row RowFilter is skipped on them
+ (data generated inline by the suite's load SQL; knobs: PRED_ROWS, RG_SIZE)
(toggle a system under test with its native DATAFUSION_* env var; size data with PRED_ROWS, string width with PRED_FILL)
Review Comment:
This help/usage text line about toggling a system under test (and `PRED_FILL`) belongs to `predicate_eval`, but after inserting the new `parquet_row_filter_skip` entry it now appears as part of the new suite’s description. That makes the bench list misleading.
Move the toggle line back under `predicate_eval` and keep the `parquet_row_filter_skip` description limited to its own knobs (`PRED_ROWS`, `RG_SIZE`).
##########
benchmarks/sql_benchmarks/parquet_row_filter_skip/load/clustered.sql:
##########
@@ -0,0 +1,39 @@
+-- Clustered Parquet dataset for the fully-matched RowFilter-skip benchmark.
+--
+-- `skey` is a fixed-width, zero-padded, monotonically increasing string, so
+-- each row group holds a disjoint, sorted range of keys. With
+-- `pushdown_filters=true`, a low-selectivity range predicate (see the query)
+-- leaves the first row group straddling and every later row group fully
+-- matched by min/max statistics, which is exactly what the per-RG RowFilter
+-- skip targets. `skey` is not projected by the query, so the skip also avoids
+-- decoding it on the fully-matched run.
+--
+-- Knobs: PRED_ROWS (row count), RG_SIZE (parquet row-group size).
+set datafusion.execution.parquet.pushdown_filters = true;
+
+COPY (
+ SELECT
+ lpad(CAST(value AS VARCHAR), 10, '0') AS skey,
+ (value * 7) % 1000000 AS p0,
+ (value * 13) % 1000000 AS p1,
+ (value * 17) % 1000000 AS p2,
+ (value * 19) % 1000000 AS p3,
+ (value * 23) % 1000000 AS p4,
+ (value * 29) % 1000000 AS p5,
+ (value * 31) % 1000000 AS p6,
+ (value * 37) % 1000000 AS p7,
+ (value * 41) % 1000000 AS p8,
+ (value * 43) % 1000000 AS p9,
+ (value * 47) % 1000000 AS p10,
+ (value * 53) % 1000000 AS p11,
+ (value * 59) % 1000000 AS p12,
+ (value * 61) % 1000000 AS p13
+ FROM generate_series(1, ${PRED_ROWS:-10000000})
+)
+TO 'sql_benchmarks/parquet_row_filter_skip/scratch/clustered.parquet'
+STORED AS PARQUET
+OPTIONS ('max_row_group_size' '${RG_SIZE:-1000000}');
Review Comment:
The COPY writer option key here is inconsistent with other COPY/Parquet statement overrides (which use the `format.` prefix). Using `max_row_group_size` may be rejected as an unknown option or silently ignored, making `RG_SIZE` ineffective and potentially breaking the benchmark.
Use `format.max_row_group_size` for consistency with existing COPY usage (e.g. `datafusion/sqllogictest/test_files/copy.slt`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] bench: add parquet_row_filter_skip suite for the fully-matched RowFilter skip [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24328:
URL: https://github.com/apache/datafusion/pull/24328#issuecomment-5289942275
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24328?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.17%. Comparing base ([`9b3b518`](https://app.codecov.io/gh/apache/datafusion/commit/9b3b518950c0c6a564b5525d6b274000f27dccd9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`811dfb0`](https://app.codecov.io/gh/apache/datafusion/commit/811dfb0e70244c7d87f1ce2b0bce3aed0bfc25ba?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 20 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24328 +/- ##
==========================================
- Coverage 81.29% 81.17% -0.13%
==========================================
Files 1110 1109 -1
Lines 385336 388038 +2702
Branches 385336 388038 +2702
==========================================
+ Hits 313261 314974 +1713
- Misses 53594 54517 +923
- Partials 18481 18547 +66
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24328?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org