You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/21 14:46:42 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by bd17e435c401c32af0c1dc81bcbcbe82621fa825
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6d51eeee9 Publish built docs triggered by bd17e435c401c32af0c1dc81bcbcbe82621fa825
6d51eeee9 is described below
commit 6d51eeee9289dcbe4ec22c0d20c2d719b3221a16
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jul 21 14:46:37 2026 +0000
Publish built docs triggered by bd17e435c401c32af0c1dc81bcbcbe82621fa825
---
_sources/contributors-guide/benchmarking.md.txt | 113 ++++++++++++----
contributors-guide/benchmarking.html | 164 ++++++++++++++++++++----
searchindex.js | 2 +-
3 files changed, 228 insertions(+), 51 deletions(-)
diff --git a/_sources/contributors-guide/benchmarking.md.txt b/_sources/contributors-guide/benchmarking.md.txt
index 7dde09788..88352e70d 100644
--- a/_sources/contributors-guide/benchmarking.md.txt
+++ b/_sources/contributors-guide/benchmarking.md.txt
@@ -265,14 +265,16 @@ planner.
TPC-H **SF1000**, reference cluster above, **AQE on**, 1 iteration,
`target_partitions=64`, `prefer_hash_join=false`,
-`enable_dynamic_filter_pushdown=false`. All three engines plan `SortMergeJoin`. Times
-in seconds; lower is better.
+`enable_dynamic_filter_pushdown=false`, and the sort-shuffle spill cap overridden to
+uncapped (`ballista.shuffle.sort_based.memory_limit_per_task_bytes=0`; the shipped
+default is 256 MiB). All three engines plan `SortMergeJoin`. Times in seconds; lower
+is better.
Versions under test:
| Engine | Version |
| -------- | ----------------------------------------------- |
-| Ballista | `main` @ `49d1fec8` |
+| Ballista | `#2084` @ `fac1fa22` |
| Spark | 3.5.3 (vanilla, Comet disabled) |
| Comet | `main` @ `b0165552` (1.0.0-SNAPSHOT, Spark 3.5) |
@@ -287,29 +289,29 @@ Spark each completed the full suite in one run.
| Query | Spark | Comet | Ballista (AQE on) | Rows |
| --------------------: | ---------: | ---------: | ----------------: | -----: |
-| 1 | 427.1 | 46.6 | 21.4 | 4 |
-| 2 | 75.9 | 39.2 | 64.1 | 100 |
-| 3 | 154.1 | 195.4 | 202.5 | 10 |
-| 4 | 82.0 | 42.6 | 54.6 | 5 |
-| 5 | 336.4 | 493.4 | 474.7 | 5 |
-| 6 | 28.9 | 14.6 | 27.7 | 1 |
-| 7 | 180.6 | 149.5 | 457.0 | 4 |
-| 8 | 391.8 | 642.1 | 731.0 | 2 |
-| 9 | 509.8 | 843.5 | 934.3 | 175 |
-| 10 | 151.1 | 104.8 | 136.8 | 20 |
-| 11 | 44.7 | 44.1 | 85.4 | 0 [1] |
-| 12 | 74.1 | 49.9 | 70.9 | 2 |
-| 13 | 98.7 | 58.3 | 92.5 | 30 |
-| 14 | 43.0 | 27.4 | 38.9 | 1 |
-| 15 | 121.5 | 65.6 | 84.5 | 1 [2] |
-| 16 | 24.5 | 17.5 | 24.3 | 27840 |
-| 17 | 406.7 | 285.8 | 355.3 | 1 |
+| 1 | 427.1 | 46.6 | 59.4 | 4 |
+| 2 | 75.9 | 39.2 | 57.0 | 100 |
+| 3 | 154.1 | 195.4 | 214.8 | 10 |
+| 4 | 82.0 | 42.6 | 65.2 | 5 |
+| 5 | 336.4 | 493.4 | 456.6 | 5 |
+| 6 | 28.9 | 14.6 | 29.5 | 1 |
+| 7 | 180.6 | 149.5 | 369.0 | 4 |
+| 8 | 391.8 | 642.1 | 540.3 | 2 |
+| 9 | 509.8 | 843.5 | 764.9 | 175 |
+| 10 | 151.1 | 104.8 | 168.3 | 20 |
+| 11 | 44.7 | 44.1 | 91.4 | 0 [1] |
+| 12 | 74.1 | 49.9 | 74.9 | 2 |
+| 13 | 98.7 | 58.3 | 91.9 | 30 |
+| 14 | 43.0 | 27.4 | 43.0 | 1 |
+| 15 | 121.5 | 65.6 | 116.5 | 1 [2] |
+| 16 | 24.5 | 17.5 | 29.0 | 27840 |
+| 17 | 406.7 | 285.8 | 392.8 | 1 |
| 18 | 428.8 | 370.5 | OOM | 100 |
-| 19 | 58.9 | 35.5 | 120.4 | 1 |
-| 20 | 105.9 | 67.6 | 111.2 | 110759 |
-| 21 | 562.2 | 460.1 | 694.7 | 100 |
-| 22 | 36.8 | 21.5 | 35.6 | 7 |
-| **Total (excl. Q18)** | **3914.7** | **3705.0** | **4817.8** | |
+| 19 | 58.9 | 35.5 | 183.9 | 1 |
+| 20 | 105.9 | 67.6 | 112.2 | 110759 |
+| 21 | 562.2 | 460.1 | 765.2 | 100 |
+| 22 | 36.8 | 21.5 | 35.2 | 7 |
+| **Total (excl. Q18)** | **3914.7** | **3705.0** | **4661.0** | |
The **Total** row sums the 21 queries **excluding Q18**, because Q18 does not complete
on Ballista at this sizing (below), so a 22-query total would not be comparable across
@@ -318,6 +320,10 @@ engines. Q18's own times are in its row.
Row counts agree across all three engines on every query, including Q18 (Spark and
Comet return 100; Ballista OOMs before producing a result).
+The **AQE-off** Ballista column has been removed pending a re-run at a matched core
+count; the numbers above for AQE off were taken at a different executor size and are
+not comparable to a fresh AQE-off run, so they were dropped rather than left stale.
+
[1] Q11 returns 0 rows for every engine at this scale factor: the query's threshold
constant is tuned for SF1.
@@ -344,6 +350,63 @@ regression.
did not produce an answer is recorded as `FAIL`, or `OOM` where the failure is a
known memory exhaustion.
+### Hash join with a per-partition build-size fallback (AQE on, 2×16 cores, 64 partitions)
+
+A second Ballista configuration exercises `prefer_hash_join=true` together with the
+AQE hash-join safety fallback
+([`ballista.optimizer.hash_join_max_build_partition_bytes`](https://github.com/apache/datafusion-ballista/pull/2084)).
+The fallback lowers a Partitioned hash join to `SortMergeJoin` per join when the
+build side's largest partition exceeds the configured budget, so a hash-join run does
+not abort on the queries whose non-spillable hash-join build exceeds one task slot's
+pool — notably Q18
+([#2025](https://github.com/apache/datafusion-ballista/issues/2025)). That fallback
+is what makes `prefer_hash_join=true` usable across the whole suite instead of
+failing partway.
+
+This run uses a **larger cluster than the reference above** and a **different
+`target_partitions`**, so its numbers are not comparable to the table above: TPC-H
+**SF1000**, **2 executors × 16 cores** (one per node, ~2.8 GB per task slot),
+**`target_partitions=64`**, **AQE on**, `prefer_hash_join=true`,
+`hash_join_max_build_partition_bytes=67108864` (64 MiB),
+`enable_dynamic_filter_pushdown=false`, sort-shuffle spill uncapped. Each query ran
+on a **freshly restarted cluster**. Ballista @ `afef9afc`. Times in seconds.
+
+| Query | Ballista (AQE on, hash join + 64 MiB fallback) |
+| --------: | ---------------------------------------------: |
+| 1 | 72.1 |
+| 2 | 52.9 |
+| 3 | 154.2 |
+| 4 | 72.9 |
+| 5 | 355.4 |
+| 6 | 65.9 |
+| 7 | 371.7 |
+| 8 | 452.4 |
+| 9 | 559.8 |
+| 10 | 152.3 |
+| 11 | 74.6 |
+| 12 | 88.7 |
+| 13 | 99.2 |
+| 14 | 38.6 |
+| 15 | 62.5 |
+| 16 | 29.0 |
+| 17 | 446.3 |
+| 18 | 744.6 |
+| 19 | 80.8 |
+| 20 | 117.0 |
+| 21 | 605.6 |
+| 22 | 20.5 |
+| **Total** | **4716.9** |
+
+**All 22 queries completed — no OOMs, no failures.** At the 64 MiB threshold the
+large-build joins (Q5, Q7–Q9, Q17, Q18, Q21) fall back to `SortMergeJoin` while
+smaller-build joins stay hash, so the heavy queries run at roughly sort-merge speed
+but the suite runs end to end. Q18 completes at **744.6 s** where the same
+configuration with the fallback disabled (`hash_join_max_build_partition_bytes=0`)
+exhausts the per-slot pool and fails the job
+([#2025](https://github.com/apache/datafusion-ballista/issues/2025)). The fallback is
+opt-in and off by default (a `0` budget); the 64 MiB value here is tuned to this
+cluster's ~2.8 GB task-slot pool.
+
### Recording a result
- Pin the **exact commit** the numbers came from, not a branch name.
diff --git a/contributors-guide/benchmarking.html b/contributors-guide/benchmarking.html
index b3904c7d7..634d1908d 100644
--- a/contributors-guide/benchmarking.html
+++ b/contributors-guide/benchmarking.html
@@ -629,8 +629,10 @@ planner.</p>
<h2>Results<a class="headerlink" href="#results" title="Link to this heading">#</a></h2>
<p>TPC-H <strong>SF1000</strong>, reference cluster above, <strong>AQE on</strong>, 1 iteration,
<code class="docutils literal notranslate"><span class="pre">target_partitions=64</span></code>, <code class="docutils literal notranslate"><span class="pre">prefer_hash_join=false</span></code>,
-<code class="docutils literal notranslate"><span class="pre">enable_dynamic_filter_pushdown=false</span></code>. All three engines plan <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code>. Times
-in seconds; lower is better.</p>
+<code class="docutils literal notranslate"><span class="pre">enable_dynamic_filter_pushdown=false</span></code>, and the sort-shuffle spill cap overridden to
+uncapped (<code class="docutils literal notranslate"><span class="pre">ballista.shuffle.sort_based.memory_limit_per_task_bytes=0</span></code>; the shipped
+default is 256 MiB). All three engines plan <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code>. Times in seconds; lower
+is better.</p>
<p>Versions under test:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
@@ -640,7 +642,7 @@ in seconds; lower is better.</p>
</thead>
<tbody>
<tr class="row-even"><td><p>Ballista</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">main</span></code> @ <code class="docutils literal notranslate"><span class="pre">49d1fec8</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">#2084</span></code> @ <code class="docutils literal notranslate"><span class="pre">fac1fa22</span></code></p></td>
</tr>
<tr class="row-odd"><td><p>Spark</p></td>
<td><p>3.5.3 (vanilla, Comet disabled)</p></td>
@@ -671,103 +673,103 @@ Spark each completed the full suite in one run.</p>
<tr class="row-even"><td class="text-right"><p>1</p></td>
<td class="text-right"><p>427.1</p></td>
<td class="text-right"><p>46.6</p></td>
-<td class="text-right"><p>21.4</p></td>
+<td class="text-right"><p>59.4</p></td>
<td class="text-right"><p>4</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>2</p></td>
<td class="text-right"><p>75.9</p></td>
<td class="text-right"><p>39.2</p></td>
-<td class="text-right"><p>64.1</p></td>
+<td class="text-right"><p>57.0</p></td>
<td class="text-right"><p>100</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>3</p></td>
<td class="text-right"><p>154.1</p></td>
<td class="text-right"><p>195.4</p></td>
-<td class="text-right"><p>202.5</p></td>
+<td class="text-right"><p>214.8</p></td>
<td class="text-right"><p>10</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>4</p></td>
<td class="text-right"><p>82.0</p></td>
<td class="text-right"><p>42.6</p></td>
-<td class="text-right"><p>54.6</p></td>
+<td class="text-right"><p>65.2</p></td>
<td class="text-right"><p>5</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>5</p></td>
<td class="text-right"><p>336.4</p></td>
<td class="text-right"><p>493.4</p></td>
-<td class="text-right"><p>474.7</p></td>
+<td class="text-right"><p>456.6</p></td>
<td class="text-right"><p>5</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>6</p></td>
<td class="text-right"><p>28.9</p></td>
<td class="text-right"><p>14.6</p></td>
-<td class="text-right"><p>27.7</p></td>
+<td class="text-right"><p>29.5</p></td>
<td class="text-right"><p>1</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>7</p></td>
<td class="text-right"><p>180.6</p></td>
<td class="text-right"><p>149.5</p></td>
-<td class="text-right"><p>457.0</p></td>
+<td class="text-right"><p>369.0</p></td>
<td class="text-right"><p>4</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>8</p></td>
<td class="text-right"><p>391.8</p></td>
<td class="text-right"><p>642.1</p></td>
-<td class="text-right"><p>731.0</p></td>
+<td class="text-right"><p>540.3</p></td>
<td class="text-right"><p>2</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>9</p></td>
<td class="text-right"><p>509.8</p></td>
<td class="text-right"><p>843.5</p></td>
-<td class="text-right"><p>934.3</p></td>
+<td class="text-right"><p>764.9</p></td>
<td class="text-right"><p>175</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>10</p></td>
<td class="text-right"><p>151.1</p></td>
<td class="text-right"><p>104.8</p></td>
-<td class="text-right"><p>136.8</p></td>
+<td class="text-right"><p>168.3</p></td>
<td class="text-right"><p>20</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>11</p></td>
<td class="text-right"><p>44.7</p></td>
<td class="text-right"><p>44.1</p></td>
-<td class="text-right"><p>85.4</p></td>
+<td class="text-right"><p>91.4</p></td>
<td class="text-right"><p>0 [1]</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>12</p></td>
<td class="text-right"><p>74.1</p></td>
<td class="text-right"><p>49.9</p></td>
-<td class="text-right"><p>70.9</p></td>
+<td class="text-right"><p>74.9</p></td>
<td class="text-right"><p>2</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>13</p></td>
<td class="text-right"><p>98.7</p></td>
<td class="text-right"><p>58.3</p></td>
-<td class="text-right"><p>92.5</p></td>
+<td class="text-right"><p>91.9</p></td>
<td class="text-right"><p>30</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>14</p></td>
<td class="text-right"><p>43.0</p></td>
<td class="text-right"><p>27.4</p></td>
-<td class="text-right"><p>38.9</p></td>
+<td class="text-right"><p>43.0</p></td>
<td class="text-right"><p>1</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>15</p></td>
<td class="text-right"><p>121.5</p></td>
<td class="text-right"><p>65.6</p></td>
-<td class="text-right"><p>84.5</p></td>
+<td class="text-right"><p>116.5</p></td>
<td class="text-right"><p>1 [2]</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>16</p></td>
<td class="text-right"><p>24.5</p></td>
<td class="text-right"><p>17.5</p></td>
-<td class="text-right"><p>24.3</p></td>
+<td class="text-right"><p>29.0</p></td>
<td class="text-right"><p>27840</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>17</p></td>
<td class="text-right"><p>406.7</p></td>
<td class="text-right"><p>285.8</p></td>
-<td class="text-right"><p>355.3</p></td>
+<td class="text-right"><p>392.8</p></td>
<td class="text-right"><p>1</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>18</p></td>
@@ -779,31 +781,31 @@ Spark each completed the full suite in one run.</p>
<tr class="row-even"><td class="text-right"><p>19</p></td>
<td class="text-right"><p>58.9</p></td>
<td class="text-right"><p>35.5</p></td>
-<td class="text-right"><p>120.4</p></td>
+<td class="text-right"><p>183.9</p></td>
<td class="text-right"><p>1</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>20</p></td>
<td class="text-right"><p>105.9</p></td>
<td class="text-right"><p>67.6</p></td>
-<td class="text-right"><p>111.2</p></td>
+<td class="text-right"><p>112.2</p></td>
<td class="text-right"><p>110759</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p>21</p></td>
<td class="text-right"><p>562.2</p></td>
<td class="text-right"><p>460.1</p></td>
-<td class="text-right"><p>694.7</p></td>
+<td class="text-right"><p>765.2</p></td>
<td class="text-right"><p>100</p></td>
</tr>
<tr class="row-odd"><td class="text-right"><p>22</p></td>
<td class="text-right"><p>36.8</p></td>
<td class="text-right"><p>21.5</p></td>
-<td class="text-right"><p>35.6</p></td>
+<td class="text-right"><p>35.2</p></td>
<td class="text-right"><p>7</p></td>
</tr>
<tr class="row-even"><td class="text-right"><p><strong>Total (excl. Q18)</strong></p></td>
<td class="text-right"><p><strong>3914.7</strong></p></td>
<td class="text-right"><p><strong>3705.0</strong></p></td>
-<td class="text-right"><p><strong>4817.8</strong></p></td>
+<td class="text-right"><p><strong>4661.0</strong></p></td>
<td class="text-right"><p></p></td>
</tr>
</tbody>
@@ -814,6 +816,9 @@ on Ballista at this sizing (below), so a 22-query total would not be comparable
engines. Q18’s own times are in its row.</p>
<p>Row counts agree across all three engines on every query, including Q18 (Spark and
Comet return 100; Ballista OOMs before producing a result).</p>
+<p>The <strong>AQE-off</strong> Ballista column has been removed pending a re-run at a matched core
+count; the numbers above for AQE off were taken at a different executor size and are
+not comparable to a fresh AQE-off run, so they were dropped rather than left stale.</p>
<p>[1] Q11 returns 0 rows for every engine at this scale factor: the query’s threshold
constant is tuned for SF1.</p>
<p>[2] Q15 is a multi-statement query (<code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">VIEW</span></code> / <code class="docutils literal notranslate"><span class="pre">SELECT</span></code> / <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">VIEW</span></code>); all three
@@ -834,6 +839,114 @@ regression.</p>
<p><code class="docutils literal notranslate"><span class="pre">TBD</span></code> means not yet measured on this cluster at this commit; a query that ran but
did not produce an answer is recorded as <code class="docutils literal notranslate"><span class="pre">FAIL</span></code>, or <code class="docutils literal notranslate"><span class="pre">OOM</span></code> where the failure is a
known memory exhaustion.</p>
+<section id="hash-join-with-a-per-partition-build-size-fallback-aqe-on-216-cores-64-partitions">
+<h3>Hash join with a per-partition build-size fallback (AQE on, 2×16 cores, 64 partitions)<a class="headerlink" href="#hash-join-with-a-per-partition-build-size-fallback-aqe-on-216-cores-64-partitions" title="Link to this heading">#</a></h3>
+<p>A second Ballista configuration exercises <code class="docutils literal notranslate"><span class="pre">prefer_hash_join=true</span></code> together with the
+AQE hash-join safety fallback
+(<a class="reference external" href="https://github.com/apache/datafusion-ballista/pull/2084"><code class="docutils literal notranslate"><span class="pre">ballista.optimizer.hash_join_max_build_partition_bytes</span></code></a>).
+The fallback lowers a Partitioned hash join to <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code> per join when the
+build side’s largest partition exceeds the configured budget, so a hash-join run does
+not abort on the queries whose non-spillable hash-join build exceeds one task slot’s
+pool — notably Q18
+(<a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2025">#2025</a>). That fallback
+is what makes <code class="docutils literal notranslate"><span class="pre">prefer_hash_join=true</span></code> usable across the whole suite instead of
+failing partway.</p>
+<p>This run uses a <strong>larger cluster than the reference above</strong> and a <strong>different
+<code class="docutils literal notranslate"><span class="pre">target_partitions</span></code></strong>, so its numbers are not comparable to the table above: TPC-H
+<strong>SF1000</strong>, <strong>2 executors × 16 cores</strong> (one per node, ~2.8 GB per task slot),
+<strong><code class="docutils literal notranslate"><span class="pre">target_partitions=64</span></code></strong>, <strong>AQE on</strong>, <code class="docutils literal notranslate"><span class="pre">prefer_hash_join=true</span></code>,
+<code class="docutils literal notranslate"><span class="pre">hash_join_max_build_partition_bytes=67108864</span></code> (64 MiB),
+<code class="docutils literal notranslate"><span class="pre">enable_dynamic_filter_pushdown=false</span></code>, sort-shuffle spill uncapped. Each query ran
+on a <strong>freshly restarted cluster</strong>. Ballista @ <code class="docutils literal notranslate"><span class="pre">afef9afc</span></code>. Times in seconds.</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head text-right"><p>Query</p></th>
+<th class="head text-right"><p>Ballista (AQE on, hash join + 64 MiB fallback)</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td class="text-right"><p>1</p></td>
+<td class="text-right"><p>72.1</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>2</p></td>
+<td class="text-right"><p>52.9</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>3</p></td>
+<td class="text-right"><p>154.2</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>4</p></td>
+<td class="text-right"><p>72.9</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>5</p></td>
+<td class="text-right"><p>355.4</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>6</p></td>
+<td class="text-right"><p>65.9</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>7</p></td>
+<td class="text-right"><p>371.7</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>8</p></td>
+<td class="text-right"><p>452.4</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>9</p></td>
+<td class="text-right"><p>559.8</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>10</p></td>
+<td class="text-right"><p>152.3</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>11</p></td>
+<td class="text-right"><p>74.6</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>12</p></td>
+<td class="text-right"><p>88.7</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>13</p></td>
+<td class="text-right"><p>99.2</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>14</p></td>
+<td class="text-right"><p>38.6</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>15</p></td>
+<td class="text-right"><p>62.5</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>16</p></td>
+<td class="text-right"><p>29.0</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>17</p></td>
+<td class="text-right"><p>446.3</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>18</p></td>
+<td class="text-right"><p>744.6</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>19</p></td>
+<td class="text-right"><p>80.8</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>20</p></td>
+<td class="text-right"><p>117.0</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p>21</p></td>
+<td class="text-right"><p>605.6</p></td>
+</tr>
+<tr class="row-odd"><td class="text-right"><p>22</p></td>
+<td class="text-right"><p>20.5</p></td>
+</tr>
+<tr class="row-even"><td class="text-right"><p><strong>Total</strong></p></td>
+<td class="text-right"><p><strong>4716.9</strong></p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p><strong>All 22 queries completed — no OOMs, no failures.</strong> At the 64 MiB threshold the
+large-build joins (Q5, Q7–Q9, Q17, Q18, Q21) fall back to <code class="docutils literal notranslate"><span class="pre">SortMergeJoin</span></code> while
+smaller-build joins stay hash, so the heavy queries run at roughly sort-merge speed
+but the suite runs end to end. Q18 completes at <strong>744.6 s</strong> where the same
+configuration with the fallback disabled (<code class="docutils literal notranslate"><span class="pre">hash_join_max_build_partition_bytes=0</span></code>)
+exhausts the per-slot pool and fails the job
+(<a class="reference external" href="https://github.com/apache/datafusion-ballista/issues/2025">#2025</a>). The fallback is
+opt-in and off by default (a <code class="docutils literal notranslate"><span class="pre">0</span></code> budget); the 64 MiB value here is tuned to this
+cluster’s ~2.8 GB task-slot pool.</p>
+</section>
<section id="recording-a-result">
<h3>Recording a result<a class="headerlink" href="#recording-a-result" title="Link to this heading">#</a></h3>
<ul class="simple">
@@ -937,6 +1050,7 @@ knowing about before interpreting a number:</p>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-compare-against-comet">Why compare against Comet</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#results">Results</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#hash-join-with-a-per-partition-build-size-fallback-aqe-on-216-cores-64-partitions">Hash join with a per-partition build-size fallback (AQE on, 2×16 cores, 64 partitions)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#recording-a-result">Recording a result</a></li>
</ul>
</li>
diff --git a/searchindex.js b/searchindex.js
index 945adf49f..99301185b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior (opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a persona":[[25,"adding-a-persona"]],"Aggregate Functions":[[49,"aggregate-functions"]],"Apache DataFusion Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior (opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a persona":[[25,"adding-a-persona"]],"Aggregate Functions":[[49,"aggregate-functions"]],"Apache DataFusion Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org