You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/23 03:37:13 UTC
(datafusion) 01/01: bench: make the spill_views memory limits overridable from the environment (#25634)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25634-ebdb657777d512b20ad40f50ce8fb85a39ee3175
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67
Author: Adrian Garcia Badaracco <17...@users.noreply.github.com>
AuthorDate: Wed Sep 23 03:36:49 2026 +0000
bench: make the spill_views memory limits overridable from the environment (#25634)
## Which issue does this PR close?
- Follow-up to https://github.com/apache/datafusion/pull/25625.
## Rationale for this change
Each `spill_views` query hardcodes its memory limit, and a benchmark-bot
trigger comment can only set environment variables. So the only way to
change a limit today is another DataFusion PR.
At the current defaults every query spills on `main` and passes (q05,
the GROUP BY, passed 30 of 30 executions at every limit from 96M to
192M). But https://github.com/apache/datafusion/pull/23565 fails q05 at
96M on every bot run (4 of 4), which aborts the whole suite run, so that
PR gets no numbers.
## What changes are included in this PR?
- `SPILL_VIEWS_LIMIT_REPEATED` (default `40M`) and
`SPILL_VIEWS_LIMIT_DISTINCT` (default `96M`) set the limit of each
subgroup, also as `--limit-repeated` / `--limit-distinct` on
`benchmark_runner`. The defaults do not change.
- The docs suggest `128M` as an override: q05 stops spilling at about
`192M`, so larger values no longer measure the spill path.
## Are these changes tested?
Ran the suite through `benchmark_runner` at the defaults and with
`SPILL_VIEWS_LIMIT_DISTINCT=128M`. The suite's asserts read back the
limit each query ran with, so a wrong override fails the run.
## Are there any user-facing changes?
No, benchmarks only.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <no...@anthropic.com>
---
benchmarks/sql_benchmarks/README.md | 4 +++-
.../spill_views/benchmarks/distinct/q04.benchmark | 2 +-
.../spill_views/benchmarks/distinct/q05.benchmark | 2 +-
.../spill_views/benchmarks/repeated/q01.benchmark | 2 +-
.../spill_views/benchmarks/repeated/q02.benchmark | 2 +-
.../spill_views/benchmarks/repeated/q03.benchmark | 2 +-
.../spill_views/spill_views.benchmark.template | 10 ++++++++++
.../sql_benchmarks/spill_views/spill_views.suite | 20 +++++++++++++++++++-
8 files changed, 37 insertions(+), 7 deletions(-)
diff --git a/benchmarks/sql_benchmarks/README.md b/benchmarks/sql_benchmarks/README.md
index e83c221ad7..340c05657e 100644
--- a/benchmarks/sql_benchmarks/README.md
+++ b/benchmarks/sql_benchmarks/README.md
@@ -47,7 +47,7 @@ in the community:
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in le [...]
| `parquet_row_filter_skip` | Micro-benchmark for the per-row-group fully-matched RowFilter skip on Parquet scans ([#23696](https://github.com/apache/datafusion/issues/23696)). Subgroups (`--subgroup`): `skip` (clustered key, most row groups fully matched by statistics so the per-row filter is skipped), `control` (scrambled key, no row group is ever fully matched). Size the data with `PRED_ROWS` and the row-group size with `RG_SIZE`. |
-| `spill_views` | Sort and `GROUP BY` queries that spill `StringView` and `BinaryView` columns read from Parquet ([#23564](https://github.com/apache/datafusion/issues/23564)). Each query sets its own memory limit and `target_partitions = 4`, so it spills in any environment. Subgroups (`--subgroup`): `repeated` (sort with a low-cardinality payload that shares one dictionary buffer, where spill compaction can remove data), `distinct` (sort and `GROUP BY` on all-distinct strings, where comp [...]
+| `spill_views` | Sort and `GROUP BY` queries that spill `StringView` and `BinaryView` columns read from Parquet ([#23564](https://github.com/apache/datafusion/issues/23564)). Each query sets its own memory limit and `target_partitions = 4`, so it spills in any environment. Subgroups (`--subgroup`): `repeated` (sort with a low-cardinality payload that shares one dictionary buffer, where spill compaction can remove data), `distinct` (sort and `GROUP BY` on all-distinct strings, where comp [...]
| `projection_subquery` | `IN`, `NOT IN` and `EXISTS` subqueries that sit in the `SELECT` list instead of a filter ([#25341](https://github.com/apache/datafusion/issues/25341)). Every query projects the boolean subquery result and aggregates it, so the measured cost is the decorrelation plan and not the size of the output. `q07` correlates on `<` instead of `=`, so it keeps the nested-loop plan and is the control that must not change. Size the two inline tables with `PSQ_ROWS` (default ` [...]
# Running Benchmarks
@@ -177,6 +177,8 @@ Some benchmarks use custom environment variables as outlined below:
| PRED_FILL | Used in the predicate_eval benchmark as the string-column width knob (filler chars per marker). | `30` |
| RG_SIZE | Used in the parquet_row_filter_skip benchmark as the Parquet `max_row_group_size` for the generated datasets. | `1000000` |
| PSQ_ROWS | Used in the projection_subquery benchmark to size the two inline tables. | `30000` |
+| SPILL_VIEWS_LIMIT_REPEATED | Used in the spill_views benchmark as the memory limit of the `repeated` subgroup. | `40M` |
+| SPILL_VIEWS_LIMIT_DISTINCT | Used in the spill_views benchmark as the memory limit of the `distinct` subgroup. | `96M` |
## How it works
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark
index f705df06ee..d3bb3c5b43 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=04
DATASET=distinct
NAME=q04_sort_string_all_distinct
-SPILL_LIMIT=96M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_DISTINCT:-96M}
VIEW_TYPE=Utf8View
DISTINCT_S=1000000
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark
index 6af904c2a7..24dad0366c 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=05
DATASET=distinct
NAME=q05_group_by_string_all_distinct
-SPILL_LIMIT=96M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_DISTINCT:-96M}
VIEW_TYPE=Utf8View
DISTINCT_S=1000000
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark
index cdbfa55f6c..21d2463ea6 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=01
DATASET=low_card_1
NAME=q01_sort_string_1_distinct
-SPILL_LIMIT=40M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_REPEATED:-40M}
VIEW_TYPE=Utf8View
DISTINCT_S=1
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark
index 7473cb3e58..365b1cb34b 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=02
DATASET=low_card_1000
NAME=q02_sort_string_1000_distinct
-SPILL_LIMIT=40M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_REPEATED:-40M}
VIEW_TYPE=Utf8View
DISTINCT_S=1000
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark
index 659d492ba8..b92406933a 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=03
DATASET=low_card_1000_binary
NAME=q03_sort_binary_1000_distinct
-SPILL_LIMIT=40M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_REPEATED:-40M}
VIEW_TYPE=BinaryView
DISTINCT_S=1000
diff --git a/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template b/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template
index a58dfc9643..e8fdd2a57d 100644
--- a/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template
+++ b/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template
@@ -7,6 +7,10 @@
# VIEW_TYPE expected Arrow type of column `s` (Utf8View, BinaryView)
# DISTINCT_S expected number of distinct values of `s`
#
+# Each qNN.benchmark takes SPILL_LIMIT from the subgroup's environment
+# variable, SPILL_VIEWS_LIMIT_REPEATED or SPILL_VIEWS_LIMIT_DISTINCT, and
+# falls back to the value the suite metadata declares as the default.
+#
# The load script COPYs a 1M-row Parquet file into scratch/ and registers it
# as table `t` (id BIGINT in shuffled order, s = a 64-byte payload). Reading
# the data back from Parquet is load-bearing: the Parquet reader makes the
@@ -17,6 +21,12 @@
# limits were sized for 4 partitions so that each query spills on main
# without running out of memory. Do not remove the pin: with more partitions
# the per-partition sort reservations can exceed the limit.
+#
+# The assert below reads the limit back from information_schema, so it also
+# shows which limit an overridden run used. The memory pool reports the
+# limit in whole K, M or G units, so give an overridden limit in the same
+# form: 128M passes the assert, 1024M and 1.5G do not, because the pool
+# reports both as 1G.
load sql_benchmarks/spill_views/load/${DATASET}.sql
diff --git a/benchmarks/sql_benchmarks/spill_views/spill_views.suite b/benchmarks/sql_benchmarks/spill_views/spill_views.suite
index ce314862fe..d027ac269c 100644
--- a/benchmarks/sql_benchmarks/spill_views/spill_views.suite
+++ b/benchmarks/sql_benchmarks/spill_views/spill_views.suite
@@ -1,7 +1,21 @@
-description = "Sort and aggregation queries that spill StringView and BinaryView columns read from Parquet (https://github.com/apache/datafusion/issues/23564). Each query sets its own memory limit and target_partitions = 4, so it spills with any environment. The repeated subgroup sorts rows whose string payload has few distinct values, so the Parquet reader makes views into one shared dictionary buffer and the spill writer can compact that buffer. The distinct subgroup sorts and groups a [...]
+description = "Sort and aggregation queries that spill StringView and BinaryView columns read from Parquet (https://github.com/apache/datafusion/issues/23564). Each query sets its own memory limit and target_partitions = 4, so it spills with any environment. The repeated subgroup sorts rows whose string payload has few distinct values, so the Parquet reader makes views into one shared dictionary buffer and the spill writer can compact that buffer. The distinct subgroup sorts and groups a [...]
query_pattern = "q{QUERY_ID_PADDED}.benchmark"
+[[options]]
+name = "limit-repeated"
+env = "SPILL_VIEWS_LIMIT_REPEATED"
+default = "40M"
+values = ["40M", "..."]
+help = "Sets the memory limit of the repeated subgroup. The queries must still spill, so keep the value well below the working set. Give a whole number of K, M or G units, for example 128M: the run asserts the limit that the memory pool reports back, and the pool reports 1024M as 1G."
+
+[[options]]
+name = "limit-distinct"
+env = "SPILL_VIEWS_LIMIT_DISTINCT"
+default = "96M"
+values = ["96M", "..."]
+help = "Sets the memory limit of the distinct subgroup. The queries must still spill, so keep the value well below the working set. Give a whole number of K, M or G units, for example 128M: the run asserts the limit that the memory pool reports back, and the pool reports 1024M as 1G."
+
[[examples]]
command = "cargo run --release --bin benchmark_runner -- spill_views"
description = "Run all spill_views queries."
@@ -9,3 +23,7 @@ description = "Run all spill_views queries."
[[examples]]
command = "cargo run --release --bin benchmark_runner -- spill_views --subgroup distinct"
description = "Run only the all-distinct (cost case) queries."
+
+[[examples]]
+command = "cargo run --release --bin benchmark_runner -- spill_views --subgroup distinct --limit-distinct 128M"
+description = "Raise the memory limit of the distinct subgroup, for example when a query runs out of memory at the default."
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org