You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/28 02:41:38 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by bd5563ade5392ca322f06efe027cd6b8c51b8ccc
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d38c599b1 Publish built docs triggered by bd5563ade5392ca322f06efe027cd6b8c51b8ccc
d38c599b1 is described below
commit d38c599b1eea5c3057614b5f25669bd8f3149da6
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue Jul 28 02:41:33 2026 +0000
Publish built docs triggered by bd5563ade5392ca322f06efe027cd6b8c51b8ccc
---
_sources/user-guide/configs.md.txt | 2 +-
_sources/user-guide/tuning-guide.md.txt | 13 ++++++++-----
searchindex.js | 2 +-
user-guide/configs.html | 2 +-
user-guide/tuning-guide.html | 7 +++++--
5 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt b/_sources/user-guide/configs.md.txt
index c40f1a582..d174102e7 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -119,7 +119,7 @@ standard DataFusion settings.
| ballista.client.pull | Boolean | false | Should client employ pull or push job tracking. In pull mode client will make a request to server in the loop, until job finishes. Pull mode is kept for legacy clients. [...]
| ballista.client.use_tls | Boolean | false | Should connection between client, scheduler, and executors use TLS. [...]
| ballista.job.name | Utf8 | (none) | Sets the job name that will appear in the web user interface for any submitted jobs [...]
-| ballista.optimizer.broadcast_join_threshold_bytes | UInt64 | 10485760 | Byte-size threshold below which a hash join's smaller side is promoted to CollectLeft and lowered via the broadcast pattern. Governs broadcast selection under both the static distributed planner and adaptive query planning (AQE). Set to 0 to disable promotion. [...]
+| ballista.optimizer.broadcast_join_threshold_bytes | UInt64 | 10485760 | Byte-size threshold below which a hash join's smaller side is promoted to CollectLeft and lowered via the broadcast pattern. Governs broadcast selection under both the static distributed planner and adaptive query planning (AQE). It also caps null-aware anti joins with a known build size because they require single-task CollectLeft execution. Set to 0 to disable promotion and reject null- [...]
| ballista.optimizer.broadcast_join_threshold_rows | UInt64 | 1000000 | Row-count threshold below which a hash join's smaller side is promoted to CollectLeft and lowered via the broadcast pattern, used as a fallback when byte-size statistics are unavailable. Applies to adaptive query planning (AQE). Set to 0 to disable promotion via the row-count path. [...]
| ballista.optimizer.hash_join_max_build_partition_bytes | UInt64 | 67108864 | Maximum per-partition hash-join build-side bytes for a Partitioned hash join under AQE. A build partition larger than this falls back to SortMergeJoin (spillable). Defaults to 64 MiB; 0 disables the check, which makes AQE use a hash join regardless of build size. [...]
| ballista.planner.adaptive.enabled | Boolean | false | Enables Adaptive Query Planning (EXPERIMENTAL) [...]
diff --git a/_sources/user-guide/tuning-guide.md.txt b/_sources/user-guide/tuning-guide.md.txt
index 6e0ba0c51..bf1768922 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -253,11 +253,11 @@ Adaptive Query Planning is EXPERIMENTAL, should be used for testing purposes onl
### Configuration
-| key | type | default | description |
-| ------------------------------------------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| ballista.planner.adaptive.enabled | Boolean | false | Enables the adaptive planner. Experimental. |
-| ballista.optimizer.broadcast_join_threshold_bytes | UInt64 | 10485760 | Byte-size threshold below which a hash join's smaller side is broadcast (`CollectLeft`). Governs both the static planner and AQE. Set to 0 to disable broadcast joins. |
-| ballista.optimizer.broadcast_join_threshold_rows | UInt64 | 1000000 | Row-count fallback threshold used when byte-size statistics are unavailable. Applies to AQE. Set to 0 to disable promotion via the row-count path. |
+| key | type | default | description |
+| ------------------------------------------------- | ------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| ballista.planner.adaptive.enabled | Boolean | false | Enables the adaptive planner. Experimental. |
+| ballista.optimizer.broadcast_join_threshold_bytes | UInt64 | 10485760 | Byte-size threshold below which a hash join's smaller side is broadcast (`CollectLeft`). Also caps null-aware anti joins with known build sizes because they run in one task. Set to 0 to disable broadcasts and reject null-aware anti joins. |
+| ballista.optimizer.broadcast_join_threshold_rows | UInt64 | 1000000 | Row-count fallback threshold used when byte-size statistics are unavailable. Applies to AQE. Set to 0 to disable promotion via the row-count path. |
### What AQE does today
@@ -271,6 +271,9 @@ implemented:
- **Broadcast join selection.** When a join input's runtime size falls under
`ballista.optimizer.broadcast_join_threshold_bytes` (or the row-count
fallback), the smaller side is broadcast (`CollectLeft`) instead of shuffled.
+ Null-aware anti joins use `CollectLeft` with a single probe task because their
+ state cannot be coordinated across executors. A known oversized build side is
+ rejected instead of producing an incorrect distributed result.
- **Empty stage elimination.** When a completed stage produces zero rows, its
downstream exchange is replaced with an empty execution node, and emptiness
is propagated up the plan so downstream stages are skipped entirely.
diff --git a/searchindex.js b/searchindex.js
index f78afe3c8..8c9a7d421 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"API changes":[[29,"api-changes"]],"AQE join behavior (opt-in)":[[28,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[22,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[51,"adaptive-query-execution-experimental"]],"Adding a persona":[[26,"adding-a-persona"]],"Aggregate Functions":[[50,"aggregate-functions"]],"Apache DataFusion Ballista":[[27,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache Da [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"API changes":[[29,"api-changes"]],"AQE join behavior (opt-in)":[[28,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[22,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[51,"adaptive-query-execution-experimental"]],"Adding a persona":[[26,"adding-a-persona"]],"Aggregate Functions":[[50,"aggregate-functions"]],"Apache DataFusion Ballista":[[27,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache Da [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index ab02408f9..00db90d02 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -540,7 +540,7 @@ standard DataFusion settings.</p>
<tr class="row-odd"><td><p>ballista.optimizer.broadcast_join_threshold_bytes</p></td>
<td><p>UInt64</p></td>
<td><p>10485760</p></td>
-<td><p>Byte-size threshold below which a hash join’s smaller side is promoted to CollectLeft and lowered via the broadcast pattern. Governs broadcast selection under both the static distributed planner and adaptive query planning (AQE). Set to 0 to disable promotion.</p></td>
+<td><p>Byte-size threshold below which a hash join’s smaller side is promoted to CollectLeft and lowered via the broadcast pattern. Governs broadcast selection under both the static distributed planner and adaptive query planning (AQE). It also caps null-aware anti joins with a known build size because they require single-task CollectLeft execution. Set to 0 to disable promotion and reject null-aware anti joins.</p></td>
</tr>
<tr class="row-even"><td><p>ballista.optimizer.broadcast_join_threshold_rows</p></td>
<td><p>UInt64</p></td>
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 17da10cc4..25ae3c84a 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -642,7 +642,7 @@ clear that AQE was used:</p>
<tr class="row-odd"><td><p>ballista.optimizer.broadcast_join_threshold_bytes</p></td>
<td><p>UInt64</p></td>
<td><p>10485760</p></td>
-<td><p>Byte-size threshold below which a hash join’s smaller side is broadcast (<code class="docutils literal notranslate"><span class="pre">CollectLeft</span></code>). Governs both the static planner and AQE. Set to 0 to disable broadcast joins.</p></td>
+<td><p>Byte-size threshold below which a hash join’s smaller side is broadcast (<code class="docutils literal notranslate"><span class="pre">CollectLeft</span></code>). Also caps null-aware anti joins with known build sizes because they run in one task. Set to 0 to disable broadcasts and reject null-aware anti joins.</p></td>
</tr>
<tr class="row-even"><td><p>ballista.optimizer.broadcast_join_threshold_rows</p></td>
<td><p>UInt64</p></td>
@@ -664,7 +664,10 @@ implemented:</p>
smaller side drives the join.</p></li>
<li><p><strong>Broadcast join selection.</strong> When a join input’s runtime size falls under
<code class="docutils literal notranslate"><span class="pre">ballista.optimizer.broadcast_join_threshold_bytes</span></code> (or the row-count
-fallback), the smaller side is broadcast (<code class="docutils literal notranslate"><span class="pre">CollectLeft</span></code>) instead of shuffled.</p></li>
+fallback), the smaller side is broadcast (<code class="docutils literal notranslate"><span class="pre">CollectLeft</span></code>) instead of shuffled.
+Null-aware anti joins use <code class="docutils literal notranslate"><span class="pre">CollectLeft</span></code> with a single probe task because their
+state cannot be coordinated across executors. A known oversized build side is
+rejected instead of producing an incorrect distributed result.</p></li>
<li><p><strong>Empty stage elimination.</strong> When a completed stage produces zero rows, its
downstream exchange is replaced with an empty execution node, and emptiness
is propagated up the plan so downstream stages are skipped entirely.</p></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org