You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/23 09:09:08 UTC
(datafusion-ballista) branch asf-site updated: Publish built docs triggered by 97d611d071317158578877f790eaa290bcffc30f
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6fa8e6e2c Publish built docs triggered by 97d611d071317158578877f790eaa290bcffc30f
6fa8e6e2c is described below
commit 6fa8e6e2c7046e082438d457b1dfaafc7aca7c0f
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Sep 23 09:09:04 2026 +0000
Publish built docs triggered by 97d611d071317158578877f790eaa290bcffc30f
---
_sources/upgrading/55.0.0.md.txt | 27 +++++++++++++++++++++++++++
_sources/user-guide/configs.md.txt | 15 ++++++++-------
_sources/user-guide/history-server.md.txt | 14 ++++++++++++++
_sources/user-guide/scheduler.md.txt | 17 +++++++++++++++++
searchindex.js | 2 +-
upgrading/55.0.0.html | 26 ++++++++++++++++++++++++++
user-guide/configs.html | 15 ++++++++++-----
user-guide/history-server.html | 11 +++++++++++
user-guide/scheduler.html | 16 ++++++++++++++++
9 files changed, 130 insertions(+), 13 deletions(-)
diff --git a/_sources/upgrading/55.0.0.md.txt b/_sources/upgrading/55.0.0.md.txt
index 82ff7af67..4b504dafb 100644
--- a/_sources/upgrading/55.0.0.md.txt
+++ b/_sources/upgrading/55.0.0.md.txt
@@ -139,6 +139,33 @@ image is bumped to match rather than sitting idle in a ready state.
### API changes
+#### `SchedulerConfig` now carries a scheduler identifier
+
+`SchedulerConfig` has a new `scheduler_id: String` field used as the scheduler's
+unique identity in scheduler state and history event logs. Its autogenerated
+default follows the same UUID-backed instance identity used for executor ids. It
+is stored separately from the scheduler callback endpoint built from
+`external_host` and `bind_port`. `SchedulerConfig::scheduler_name()` is renamed
+to `SchedulerConfig::scheduler_endpoint()`. Code that constructs
+`SchedulerConfig` with a struct literal must provide this field. Prefer the
+existing constructors/builders where possible; use
+`SchedulerConfig::with_scheduler_id(...)` when an embedding needs a stable
+scheduler id. The scheduler CLI accepts `--id`.
+
+`ExecutorProcessConfig` similarly has a new `executor_id: String` field used as
+the executor's unique identity when it registers with the scheduler. The
+executor CLI accepts `--id`; if unset, executor startup generates the same
+UUID-backed instance identity as before. Code that constructs
+`ExecutorProcessConfig` with a struct literal must provide this field or use the
+existing default. Active executors registered with the same scheduler must have
+unique ids; duplicate active ids are rejected and can be reused after the
+previous executor is removed from scheduler state.
+
+The generated protobuf fields
+`LaunchTaskParams.scheduler_id` and `LaunchMultiTaskParams.scheduler_id` are
+renamed to `scheduler_endpoint` because they carry scheduler callback endpoints.
+The protobuf field number remains `2`.
+
#### Shuffle fetch addresses byte ranges, not just partitions
The shuffle fetch action carried a bool saying whether the producer used the
diff --git a/_sources/user-guide/configs.md.txt b/_sources/user-guide/configs.md.txt
index 72dafc54e..55b1ed687 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -191,10 +191,11 @@ _Example: Specifying configuration options when starting the scheduler_
./ballista-scheduler --scheduler-policy push-staged --event-loop-buffer-size 1000000 --task-distribution round-robin
```
-| key | type | default | description |
-| -------------------------------------------- | ------ | ----------- | -------------------------------------------------------------------------------------------------------------------------- |
-| scheduler-policy | Utf8 | pull-staged | Sets the task scheduling policy for the scheduler, possible values: pull-staged, push-staged. |
-| event-loop-buffer-size | UInt32 | 10000 | Sets the event loop buffer size. for a system of high throughput, a larger value like 1000000 is recommended. |
-| task-distribution | Utf8 | bias | Sets the task distribution policy for the scheduler, possible values: bias, round-robin |
-| finished-job-data-clean-up-interval-seconds | UInt64 | 300 | Sets the delayed interval for cleaning up finished job data, mainly the shuffle data, 0 means the cleaning up is disabled. |
-| finished-job-state-clean-up-interval-seconds | UInt64 | 3600 | Sets the delayed interval for cleaning up finished job state stored in the backend, 0 means the cleaning up is disabled. |
+| key | type | default | description |
+| -------------------------------------------- | ------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| id | Utf8 | generated UUID | Sets the scheduler identity exposed in scheduler state and history event logs. The default matches executor UUID-backed instance ids. Executor callbacks use the scheduler endpoint. |
+| scheduler-policy | Utf8 | pull-staged | Sets the task scheduling policy for the scheduler, possible values: pull-staged, push-staged. |
+| event-loop-buffer-size | UInt32 | 10000 | Sets the event loop buffer size. for a system of high throughput, a larger value like 1000000 is recommended. |
+| task-distribution | Utf8 | bias | Sets the task distribution policy for the scheduler, possible values: bias, round-robin |
+| finished-job-data-clean-up-interval-seconds | UInt64 | 300 | Sets the delayed interval for cleaning up finished job data, mainly the shuffle data, 0 means the cleaning up is disabled. |
+| finished-job-state-clean-up-interval-seconds | UInt64 | 3600 | Sets the delayed interval for cleaning up finished job state stored in the backend, 0 means the cleaning up is disabled. |
diff --git a/_sources/user-guide/history-server.md.txt b/_sources/user-guide/history-server.md.txt
index 93ba6274e..639529c2a 100644
--- a/_sources/user-guide/history-server.md.txt
+++ b/_sources/user-guide/history-server.md.txt
@@ -39,6 +39,20 @@ to:
ballista-scheduler --event-log-dir /var/lib/ballista/history
```
+Each `JobStart` event records the unique scheduler id for the scheduler that
+created it. If you do not set an id, the scheduler generates the same
+UUID-backed instance identity used for executor ids at startup; set `--id` when
+you want a stable value for a deployment, test, or local debugging session:
+
+```shell
+ballista-scheduler \
+ --event-log-dir /var/lib/ballista/history \
+ --id scheduler-a
+```
+
+Use unique scheduler ids when collecting event logs from multiple schedulers so
+history records can be attributed to the scheduler that produced them.
+
The scheduler writes one file per job, in
[JSON Lines](https://jsonlines.org/) format. While the job runs, records are
appended to `<job_id>.eventlog.running`; when it reaches a terminal state
diff --git a/_sources/user-guide/scheduler.md.txt b/_sources/user-guide/scheduler.md.txt
index 03aa92fc3..c32da1f04 100644
--- a/_sources/user-guide/scheduler.md.txt
+++ b/_sources/user-guide/scheduler.md.txt
@@ -19,6 +19,23 @@
# Ballista Scheduler
+## Scheduler Identity
+
+Each scheduler has a unique identifier used in cluster state, `/api/state`, and
+history event logs. Set it explicitly with `--id` when you want a stable value;
+otherwise the scheduler generates the same UUID-backed instance identity used
+for executor ids at startup. Use unique scheduler ids when multiple schedulers
+write to shared history or cluster state.
+
+Executor ids follow the same model: each active executor registered with the
+same scheduler needs a unique id. Set it explicitly with `--id` when you want a
+stable value; otherwise the executor generates one at startup. A second active
+executor using an id that is already registered is rejected; the id can be
+reused after the previous executor is removed from scheduler state.
+
+Executors report task status to the scheduler callback endpoint, which is built
+from `--external-host` and `--bind-port`.
+
## Fetching Query Results
By default a client fetches the result partitions of a query directly from the executors that
diff --git a/searchindex.js b/searchindex.js
index ae2b5be68..97f8c5a80 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"API changes":[[31,"api-changes"]],"AQE join behavior (opt-in)":[[30,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[23,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[54,"adaptive-query-execution-experimental"]],"Adding a persona":[[28,"adding-a-persona"]],"Aggregate Functions":[[53,"aggregate-functions"]],"Apache DataFusion Ballista":[[29,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache Da [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"API changes":[[31,"api-changes"]],"AQE join behavior (opt-in)":[[30,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution (AQE)":[[23,"adaptive-query-execution-aqe"]],"Adaptive Query Execution (Experimental)":[[54,"adaptive-query-execution-experimental"]],"Adding a persona":[[28,"adding-a-persona"]],"Aggregate Functions":[[53,"aggregate-functions"]],"Apache DataFusion Ballista":[[29,null]],"Apache DataFusion Ballista 0.10.0 Changelog":[[0,null]],"Apache Da [...]
\ No newline at end of file
diff --git a/upgrading/55.0.0.html b/upgrading/55.0.0.html
index c5322b885..ee4e3e8c3 100644
--- a/upgrading/55.0.0.html
+++ b/upgrading/55.0.0.html
@@ -503,6 +503,31 @@ image is bumped to match rather than sitting idle in a ready state.</p>
</section>
<section id="api-changes">
<h3>API changes<a class="headerlink" href="#api-changes" title="Link to this heading">#</a></h3>
+<section id="schedulerconfig-now-carries-a-scheduler-identifier">
+<h4><code class="docutils literal notranslate"><span class="pre">SchedulerConfig</span></code> now carries a scheduler identifier<a class="headerlink" href="#schedulerconfig-now-carries-a-scheduler-identifier" title="Link to this heading">#</a></h4>
+<p><code class="docutils literal notranslate"><span class="pre">SchedulerConfig</span></code> has a new <code class="docutils literal notranslate"><span class="pre">scheduler_id:</span> <span class="pre">String</span></code> field used as the scheduler’s
+unique identity in scheduler state and history event logs. Its autogenerated
+default follows the same UUID-backed instance identity used for executor ids. It
+is stored separately from the scheduler callback endpoint built from
+<code class="docutils literal notranslate"><span class="pre">external_host</span></code> and <code class="docutils literal notranslate"><span class="pre">bind_port</span></code>. <code class="docutils literal notranslate"><span class="pre">SchedulerConfig::scheduler_name()</span></code> is renamed
+to <code class="docutils literal notranslate"><span class="pre">SchedulerConfig::scheduler_endpoint()</span></code>. Code that constructs
+<code class="docutils literal notranslate"><span class="pre">SchedulerConfig</span></code> with a struct literal must provide this field. Prefer the
+existing constructors/builders where possible; use
+<code class="docutils literal notranslate"><span class="pre">SchedulerConfig::with_scheduler_id(...)</span></code> when an embedding needs a stable
+scheduler id. The scheduler CLI accepts <code class="docutils literal notranslate"><span class="pre">--id</span></code>.</p>
+<p><code class="docutils literal notranslate"><span class="pre">ExecutorProcessConfig</span></code> similarly has a new <code class="docutils literal notranslate"><span class="pre">executor_id:</span> <span class="pre">String</span></code> field used as
+the executor’s unique identity when it registers with the scheduler. The
+executor CLI accepts <code class="docutils literal notranslate"><span class="pre">--id</span></code>; if unset, executor startup generates the same
+UUID-backed instance identity as before. Code that constructs
+<code class="docutils literal notranslate"><span class="pre">ExecutorProcessConfig</span></code> with a struct literal must provide this field or use the
+existing default. Active executors registered with the same scheduler must have
+unique ids; duplicate active ids are rejected and can be reused after the
+previous executor is removed from scheduler state.</p>
+<p>The generated protobuf fields
+<code class="docutils literal notranslate"><span class="pre">LaunchTaskParams.scheduler_id</span></code> and <code class="docutils literal notranslate"><span class="pre">LaunchMultiTaskParams.scheduler_id</span></code> are
+renamed to <code class="docutils literal notranslate"><span class="pre">scheduler_endpoint</span></code> because they carry scheduler callback endpoints.
+The protobuf field number remains <code class="docutils literal notranslate"><span class="pre">2</span></code>.</p>
+</section>
<section id="shuffle-fetch-addresses-byte-ranges-not-just-partitions">
<h4>Shuffle fetch addresses byte ranges, not just partitions<a class="headerlink" href="#shuffle-fetch-addresses-byte-ranges-not-just-partitions" title="Link to this heading">#</a></h4>
<p>The shuffle fetch action carried a bool saying whether the producer used the
@@ -666,6 +691,7 @@ struct.</p>
</ul>
</li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#api-changes">API changes</a><ul class="nav section-nav flex-column">
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#schedulerconfig-now-carries-a-scheduler-identifier"><code class="docutils literal notranslate"><span class="pre">SchedulerConfig</span></code> now carries a scheduler identifier</a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#shuffle-fetch-addresses-byte-ranges-not-just-partitions">Shuffle fetch addresses byte ranges, not just partitions</a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#ballista-executor-as-task-status-signature-reshaped"><code class="docutils literal notranslate"><span class="pre">ballista_executor::as_task_status</span></code> signature reshaped</a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#sessionconfigext-and-ballistaconfig-changes">SessionConfigExt and BallistaConfig Changes</a></li>
diff --git a/user-guide/configs.html b/user-guide/configs.html
index af6a97680..5ca14eebe 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -759,27 +759,32 @@ manage the whole cluster are also needed to be taken care of.</p>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>scheduler-policy</p></td>
+<tr class="row-even"><td><p>id</p></td>
+<td><p>Utf8</p></td>
+<td><p>generated UUID</p></td>
+<td><p>Sets the scheduler identity exposed in scheduler state and history event logs. The default matches executor UUID-backed instance ids. Executor callbacks use the scheduler endpoint.</p></td>
+</tr>
+<tr class="row-odd"><td><p>scheduler-policy</p></td>
<td><p>Utf8</p></td>
<td><p>pull-staged</p></td>
<td><p>Sets the task scheduling policy for the scheduler, possible values: pull-staged, push-staged.</p></td>
</tr>
-<tr class="row-odd"><td><p>event-loop-buffer-size</p></td>
+<tr class="row-even"><td><p>event-loop-buffer-size</p></td>
<td><p>UInt32</p></td>
<td><p>10000</p></td>
<td><p>Sets the event loop buffer size. for a system of high throughput, a larger value like 1000000 is recommended.</p></td>
</tr>
-<tr class="row-even"><td><p>task-distribution</p></td>
+<tr class="row-odd"><td><p>task-distribution</p></td>
<td><p>Utf8</p></td>
<td><p>bias</p></td>
<td><p>Sets the task distribution policy for the scheduler, possible values: bias, round-robin</p></td>
</tr>
-<tr class="row-odd"><td><p>finished-job-data-clean-up-interval-seconds</p></td>
+<tr class="row-even"><td><p>finished-job-data-clean-up-interval-seconds</p></td>
<td><p>UInt64</p></td>
<td><p>300</p></td>
<td><p>Sets the delayed interval for cleaning up finished job data, mainly the shuffle data, 0 means the cleaning up is disabled.</p></td>
</tr>
-<tr class="row-even"><td><p>finished-job-state-clean-up-interval-seconds</p></td>
+<tr class="row-odd"><td><p>finished-job-state-clean-up-interval-seconds</p></td>
<td><p>UInt64</p></td>
<td><p>3600</p></td>
<td><p>Sets the delayed interval for cleaning up finished job state stored in the backend, 0 means the cleaning up is disabled.</p></td>
diff --git a/user-guide/history-server.html b/user-guide/history-server.html
index 061a0d167..c91e8c70b 100644
--- a/user-guide/history-server.html
+++ b/user-guide/history-server.html
@@ -410,6 +410,17 @@ to:</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>ballista-scheduler<span class="w"> </span>--event-log-dir<span class="w"> </span>/var/lib/ballista/history
</pre></div>
</div>
+<p>Each <code class="docutils literal notranslate"><span class="pre">JobStart</span></code> event records the unique scheduler id for the scheduler that
+created it. If you do not set an id, the scheduler generates the same
+UUID-backed instance identity used for executor ids at startup; set <code class="docutils literal notranslate"><span class="pre">--id</span></code> when
+you want a stable value for a deployment, test, or local debugging session:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>ballista-scheduler<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--event-log-dir<span class="w"> </span>/var/lib/ballista/history<span class="w"> </span><span class="se">\</span>
+<span class="w"> </span>--id<span class="w"> </span>scheduler-a
+</pre></div>
+</div>
+<p>Use unique scheduler ids when collecting event logs from multiple schedulers so
+history records can be attributed to the scheduler that produced them.</p>
<p>The scheduler writes one file per job, in
<a class="reference external" href="https://jsonlines.org/">JSON Lines</a> format. While the job runs, records are
appended to <code class="docutils literal notranslate"><span class="pre"><job_id>.eventlog.running</span></code>; when it reaches a terminal state
diff --git a/user-guide/scheduler.html b/user-guide/scheduler.html
index 01b2ab5ea..72da734bf 100644
--- a/user-guide/scheduler.html
+++ b/user-guide/scheduler.html
@@ -394,6 +394,21 @@
-->
<section id="ballista-scheduler">
<h1>Ballista Scheduler<a class="headerlink" href="#ballista-scheduler" title="Link to this heading">#</a></h1>
+<section id="scheduler-identity">
+<h2>Scheduler Identity<a class="headerlink" href="#scheduler-identity" title="Link to this heading">#</a></h2>
+<p>Each scheduler has a unique identifier used in cluster state, <code class="docutils literal notranslate"><span class="pre">/api/state</span></code>, and
+history event logs. Set it explicitly with <code class="docutils literal notranslate"><span class="pre">--id</span></code> when you want a stable value;
+otherwise the scheduler generates the same UUID-backed instance identity used
+for executor ids at startup. Use unique scheduler ids when multiple schedulers
+write to shared history or cluster state.</p>
+<p>Executor ids follow the same model: each active executor registered with the
+same scheduler needs a unique id. Set it explicitly with <code class="docutils literal notranslate"><span class="pre">--id</span></code> when you want a
+stable value; otherwise the executor generates one at startup. A second active
+executor using an id that is already registered is rejected; the id can be
+reused after the previous executor is removed from scheduler state.</p>
+<p>Executors report task status to the scheduler callback endpoint, which is built
+from <code class="docutils literal notranslate"><span class="pre">--external-host</span></code> and <code class="docutils literal notranslate"><span class="pre">--bind-port</span></code>.</p>
+</section>
<section id="fetching-query-results">
<h2>Fetching Query Results<a class="headerlink" href="#fetching-query-results" title="Link to this heading">#</a></h2>
<p>By default a client fetches the result partitions of a query directly from the executors that
@@ -578,6 +593,7 @@ that is deprecated too and logs a warning — use <code class="docutils literal
</div>
<nav id="pst-page-toc-nav" class="page-toc" aria-labelledby="pst-page-navigation-heading-2">
<ul class="pst-show_toc_level nav section-nav flex-column">
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#scheduler-identity">Scheduler Identity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fetching-query-results">Fetching Query Results</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#rest-api">REST API</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#web-tui-configuration">Web TUI Configuration</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org