You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/16 20:12:18 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by 3c50281c4bf703cf1aebb10a28cb7305bda171f1
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 929eea044 Publish built docs triggered by 3c50281c4bf703cf1aebb10a28cb7305bda171f1
929eea044 is described below
commit 929eea04429eea5c511f3edb4c740c2d663f7f8d
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Aug 16 20:12:13 2026 +0000
Publish built docs triggered by 3c50281c4bf703cf1aebb10a28cb7305bda171f1
---
_sources/user-guide/scheduler.md.txt | 48 ++++++++++++++++++++++++++++++++
searchindex.js | 2 +-
user-guide/scheduler.html | 54 ++++++++++++++++++++++++++++++++++++
3 files changed, 103 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/scheduler.md.txt b/_sources/user-guide/scheduler.md.txt
index abd0cadd7..2311ea42e 100644
--- a/_sources/user-guide/scheduler.md.txt
+++ b/_sources/user-guide/scheduler.md.txt
@@ -19,6 +19,54 @@
# Ballista Scheduler
+## Fetching Query Results
+
+By default a client fetches the result partitions of a query directly from the executors that
+produced them, over Arrow Flight. This keeps the scheduler out of the data path, but it requires
+every client to have network access to every executor — which is not the case in isolated
+environments where only the scheduler is reachable.
+
+For those deployments the scheduler can advertise a different address for clients to fetch results
+from, and can optionally host an Arrow Flight proxy itself.
+
+> Note: this is plain Arrow Flight, used to move result partitions. It is not Flight SQL, which was
+> removed in Ballista 46.0.0 — the proxy only serves Ballista's `FetchPartition` action, so generic
+> Flight SQL or JDBC clients cannot use this endpoint.
+
+Two independent options control this:
+
+| Option | Description |
+| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `--enable-embedded-flight-proxy` | Runs an Arrow Flight proxy inside the scheduler process, on the scheduler's own host and port. The proxy forwards each fetch to the executor that owns the partition. |
+| `--advertise-flight-endpoint` | The `HOST:PORT` address clients are told to fetch results from, instead of the executors. Use it to point clients at a load balancer or a standalone proxy. |
+
+The first controls whether a proxy _runs_; the second controls what clients are _told_. Setting
+both is a supported combination: the embedded proxy runs, and clients are pointed at the advertised
+address, which is how you put a load balancer in front of one or more schedulers.
+
+To let clients fetch results through the scheduler itself:
+
+```bash
+ballista-scheduler --enable-embedded-flight-proxy
+```
+
+To point clients at a load balancer that fronts the schedulers:
+
+```bash
+ballista-scheduler --enable-embedded-flight-proxy \
+ --advertise-flight-endpoint ballista-flight.example.com:50050
+```
+
+Both options are disabled by default, and the embedded proxy should be enabled deliberately: it
+puts result traffic on the scheduler's process and thread pool, competing with query planning and
+task scheduling. Under load this is a known source of scheduler congestion, so prefer a separate
+proxy — advertised with `--advertise-flight-endpoint` — for clusters where result volume is
+significant.
+
+> `--advertise-flight-sql-endpoint` is accepted as a deprecated alias of
+> `--advertise-flight-endpoint`. Passing either flag with no value used to start the embedded proxy;
+> that is deprecated too and logs a warning — use `--enable-embedded-flight-proxy` instead.
+
## REST API
The scheduler also provides a REST API that allows jobs to be monitored.
diff --git a/searchindex.js b/searchindex.js
index aa1dad953..e7323aadd 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"API changes":[[31,"api-changes"]],"AQE join behavior (opt-in)":[[30,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[23,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[53,"adaptive-query-execution-experimental"]],"Adding a persona":[[28,"adding-a-persona"]],"Aggregate Functions":[[52,"aggregate-functions"]],"Apache DataFusion Ballista":[[29,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache Da [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"API changes":[[31,"api-changes"]],"AQE join behavior (opt-in)":[[30,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[23,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[53,"adaptive-query-execution-experimental"]],"Adding a persona":[[28,"adding-a-persona"]],"Aggregate Functions":[[52,"aggregate-functions"]],"Apache DataFusion Ballista":[[29,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache Da [...]
\ No newline at end of file
diff --git a/user-guide/scheduler.html b/user-guide/scheduler.html
index 3e5e9b47c..e85d33c7c 100644
--- a/user-guide/scheduler.html
+++ b/user-guide/scheduler.html
@@ -393,6 +393,59 @@
-->
<section id="ballista-scheduler">
<h1>Ballista Scheduler<a class="headerlink" href="#ballista-scheduler" title="Link to this heading">#</a></h1>
+<section id="fetching-query-results">
+<h2>Fetching Query Results<a class="headerlink" href="#fetching-query-results" title="Link to this heading">#</a></h2>
+<p>By default a client fetches the result partitions of a query directly from the executors that
+produced them, over Arrow Flight. This keeps the scheduler out of the data path, but it requires
+every client to have network access to every executor — which is not the case in isolated
+environments where only the scheduler is reachable.</p>
+<p>For those deployments the scheduler can advertise a different address for clients to fetch results
+from, and can optionally host an Arrow Flight proxy itself.</p>
+<blockquote>
+<div><p>Note: this is plain Arrow Flight, used to move result partitions. It is not Flight SQL, which was
+removed in Ballista 46.0.0 — the proxy only serves Ballista’s <code class="docutils literal notranslate"><span class="pre">FetchPartition</span></code> action, so generic
+Flight SQL or JDBC clients cannot use this endpoint.</p>
+</div></blockquote>
+<p>Two independent options control this:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Option</p></th>
+<th class="head"><p>Description</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--enable-embedded-flight-proxy</span></code></p></td>
+<td><p>Runs an Arrow Flight proxy inside the scheduler process, on the scheduler’s own host and port. The proxy forwards each fetch to the executor that owns the partition.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--advertise-flight-endpoint</span></code></p></td>
+<td><p>The <code class="docutils literal notranslate"><span class="pre">HOST:PORT</span></code> address clients are told to fetch results from, instead of the executors. Use it to point clients at a load balancer or a standalone proxy.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>The first controls whether a proxy <em>runs</em>; the second controls what clients are <em>told</em>. Setting
+both is a supported combination: the embedded proxy runs, and clients are pointed at the advertised
+address, which is how you put a load balancer in front of one or more schedulers.</p>
+<p>To let clients fetch results through the scheduler itself:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ballista-scheduler<span class="w"> </span>--enable-embedded-flight-proxy
+</pre></div>
+</div>
+<p>To point clients at a load balancer that fronts the schedulers:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ballista-scheduler<span class="w"> </span>--enable-embedded-flight-proxy<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--advertise-flight-endpoint<span class="w"> </span>ballista-flight.example.com:50050
+</pre></div>
+</div>
+<p>Both options are disabled by default, and the embedded proxy should be enabled deliberately: it
+puts result traffic on the scheduler’s process and thread pool, competing with query planning and
+task scheduling. Under load this is a known source of scheduler congestion, so prefer a separate
+proxy — advertised with <code class="docutils literal notranslate"><span class="pre">--advertise-flight-endpoint</span></code> — for clusters where result volume is
+significant.</p>
+<blockquote>
+<div><p><code class="docutils literal notranslate"><span class="pre">--advertise-flight-sql-endpoint</span></code> is accepted as a deprecated alias of
+<code class="docutils literal notranslate"><span class="pre">--advertise-flight-endpoint</span></code>. Passing either flag with no value used to start the embedded proxy;
+that is deprecated too and logs a warning — use <code class="docutils literal notranslate"><span class="pre">--enable-embedded-flight-proxy</span></code> instead.</p>
+</div></blockquote>
+</section>
<section id="rest-api">
<h2>REST API<a class="headerlink" href="#rest-api" title="Link to this heading">#</a></h2>
<p>The scheduler also provides a REST API that allows jobs to be monitored.</p>
@@ -520,6 +573,7 @@
</div>
<nav id="pst-page-toc-nav" class="page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="pst-show_toc_level nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fetching-query-results">Fetching Query Results</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rest-api">REST API</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#web-tui-configuration">Web TUI Configuration</a></li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org