You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/04 23:31:47 UTC
(datafusion-comet) branch asf-site updated (314f4f6a5e -> 6f8601ad5b)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard 314f4f6a5e Publish built docs triggered by be9b7e30026650ec38750147c2c7df7b3cf1d28d
new 6f8601ad5b Publish built docs triggered by 33f21da181c14df7fa3e6a3104c3d7215810147e
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (314f4f6a5e)
\
N -- N -- N refs/heads/asf-site (6f8601ad5b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/user-guide/latest/iceberg-writes.md.txt | 2 +-
searchindex.js | 2 +-
user-guide/latest/iceberg-writes.html | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: Publish built docs triggered by 33f21da181c14df7fa3e6a3104c3d7215810147e
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 6f8601ad5b3b6041d92c2ecc09c8804e7c020b4f
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Oct 4 23:31:41 2026 +0000
Publish built docs triggered by 33f21da181c14df7fa3e6a3104c3d7215810147e
---
...id-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/iceberg-writes.md.txt | 32 +++++++++++++---------
_sources/user-guide/latest/iceberg-writes.md.txt | 23 ++++++++--------
contributor-guide/iceberg-writes.html | 29 +++++++++++++-------
searchindex.js | 2 +-
user-guide/latest/iceberg-writes.html | 23 ++++++++--------
6 files changed, 64 insertions(+), 47 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index d798159cb4..428b71ab9b 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/iceberg-writes.md.txt b/_sources/contributor-guide/iceberg-writes.md.txt
index 3214bfed17..8f52afc251 100644
--- a/_sources/contributor-guide/iceberg-writes.md.txt
+++ b/_sources/contributor-guide/iceberg-writes.md.txt
@@ -272,19 +272,25 @@ discards them. The price is one footer read per written file.
## Failure Handling and Cleanup Ownership
A failed attempt must leave no data files behind, as iceberg-java's `DataWriter.abort()` does, and a
-failed job must not commit anything. The files are always owned by exactly one side:
-
-| Phase | Owner | Mechanism |
-| ------------------------------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Writing, closing writers, encoding the manifest, building the batch | Native | `TrackingLocationGenerator` records every location. `AbortOnDrop` deletes them on an error, and also when the plan is dropped mid-write, from inside or outside a Tokio runtime. |
-| After the batch reaches the JVM, until the task succeeds | JVM task | `WrittenFileCleanup`, a task failure listener registered before the payload is read, takes the locations before the manifest is decoded. |
-| Job failure after some tasks completed | Driver (`IcebergCommitExec`) | Calls `BatchWrite.abort` with the completed messages, then deletes their files through the table's `FileIO`. |
-| Commit failure | Iceberg | `SparkWrite.abort` on the genuine `TaskCommit` messages, the same as the stock path. |
-
-The handoff between the first two rows is why the payload carries the locations separately from the
-manifest: a failure decoding the manifest would otherwise lose the list of files to delete. All
-deletion is best effort and logged. It must never replace the original exception, and anything it
-misses is unreferenced and reclaimed by Iceberg's `remove_orphan_files`.
+failed job must not commit anything. Cleanup must never have an ownership gap. During the handoff,
+the native guard intentionally stays armed after the JVM takes the locations and is disarmed only
+when the JVM polls the output stream to EOF:
+
+| Phase | Owner | Mechanism |
+| ------------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
+| Writing, closing writers, encoding the manifest, building the batch | Native | `TrackingLocationGenerator` records every location. `AbortOnDrop` deletes on error or plan drop and stays armed after yielding the batch. |
+| JVM has recorded locations, before its final EOF poll | Native + JVM task | `WrittenFileCleanup` owns the decoded locations while `AbortOnDrop` is still armed. A failure may make both sides attempt best-effort deletion. |
+| After the JVM's EOF poll, until the task succeeds | JVM task | The EOF poll disarms `AbortOnDrop`; `WrittenFileCleanup` remains registered for the rest of the task. |
+| Job failure after some tasks completed | Driver (`IcebergCommitExec`) | Calls `BatchWrite.abort` with the completed messages, then deletes their files through the table's `FileIO`. |
+| Commit failure | Iceberg | `SparkWrite.abort` on the genuine `TaskCommit` messages, the same as the stock path. |
+
+The handoff between the first two rows intentionally overlaps: `WrittenFileCleanup` takes the
+locations before the final EOF poll, and that poll is the acknowledgement that lets `AbortOnDrop`
+disarm. Deleting the same file from both sides during a failure in that narrow window is harmless
+because cleanup is best effort. The payload carries the locations separately from the manifest so a
+failure decoding the manifest cannot lose the list of files to delete. Cleanup is logged and must
+never replace the original exception; anything it misses is unreferenced and reclaimed by Iceberg's
+`remove_orphan_files`.
The same planning-time rule applies to failures: a failure that happens because the native side
rejected something the gate admitted is a bug in the gate, even if cleanup works.
diff --git a/_sources/user-guide/latest/iceberg-writes.md.txt b/_sources/user-guide/latest/iceberg-writes.md.txt
index 52e0eb7715..191cc5f12b 100644
--- a/_sources/user-guide/latest/iceberg-writes.md.txt
+++ b/_sources/user-guide/latest/iceberg-writes.md.txt
@@ -180,7 +180,7 @@ A write is eligible only when ALL of the following hold:
| `write.metadata.metrics.*` | any value (manifest metrics are re-derived on the JVM with Iceberg's own logic) [...]
| `write.spark.fanout.enabled` | any value (the native writer implements both clustered and fanout modes) [...]
| `write.target-file-size-bytes` | any value (the two writers can choose different roll points; see accepted divergences) [...]
-| data location URI scheme | `file`, `memory`, `s3`, `s3a`, `gs` (`gs` only when the `FileIO` opening the data location is a `GCSFileIO`; see below) [...]
+| data location URI scheme | `file`, `memory`, `s3`, `s3a`, `gs`, matched case-sensitively (`S3://` falls back). `s3`, `s3a` and `gs` need a bucket in the authority (`s3://bucket/...`), so a hostless form such as `s3:/bucket/key` falls back. `gs` only when the `FileIO` opening the data location is a `GCSFileIO`; see below [...]
| resolved `table.locationProvider()` | Iceberg's built-in `DefaultLocationProvider` [...]
| partition spec | any, except an identity partition on a `float` or `double` column (see below) [...]
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) [...]
@@ -252,16 +252,17 @@ Partial results are never committed. The commit set is exactly the commit messag
successful tasks — a failed task contributes none — and if the job fails, the driver-side
commit operator aborts without committing anything. A failed task attempt also deletes the
data files it created, as iceberg-java's writer abort does. The native writer records every
-location it hands to a file writer, and exactly one side owns deleting them at any moment: the
-native writer owns them until its output batch reaches the JVM (so it cleans up a failed write,
-a task torn down before the write completed — for example because the operator feeding it threw
-— and a failure encoding the manifest or building that batch), and the JVM owns them from then
-on through a task failure listener. The handoff does not depend on decoding the manifest: the
-native side reports the locations in the output batch alongside it, and the listener is handed
-them before the manifest is decoded, so a failure in that decode still cleans up. Both
-deletions are best-effort and never mask the original failure; anything they miss is invisible
-to every reader, since readers resolve files through committed manifests only, and is reclaimed
-by Iceberg's normal `remove_orphan_files` maintenance.
+location it hands to a file writer, and cleanup has no ownership gap. Native keeps its cleanup
+guard armed after yielding the output batch. The JVM reads those locations into a task failure
+listener first, then polls the native output to EOF; that EOF is the acknowledgement that lets
+native disarm. A failure in this narrow handoff window can therefore trigger best-effort deletion
+on both sides, which is harmless. The native guard still covers a failed write, a task torn down
+before the write completed — for example because the operator feeding it threw — and a failure
+encoding the manifest or building the output batch. The handoff does not depend on decoding the
+manifest: the locations are read before the manifest is decoded, so a failure in that decode still
+cleans up. Cleanup never masks the original failure; anything it misses is invisible to every
+reader, since readers resolve files through committed manifests only, and is reclaimed by Iceberg's
+normal `remove_orphan_files` maintenance.
When one task fails, the tasks that had already completed leave committed-nothing data files
too. The committer collects each task's commit message as that task finishes, so on a job
diff --git a/contributor-guide/iceberg-writes.html b/contributor-guide/iceberg-writes.html
index ecb82a791d..5a8c41f6fb 100644
--- a/contributor-guide/iceberg-writes.html
+++ b/contributor-guide/iceberg-writes.html
@@ -825,7 +825,9 @@ discards them. The price is one footer read per written file.</p>
<section id="failure-handling-and-cleanup-ownership">
<h2>Failure Handling and Cleanup Ownership<a class="headerlink" href="#failure-handling-and-cleanup-ownership" title="Link to this heading">#</a></h2>
<p>A failed attempt must leave no data files behind, as iceberg-java’s <code class="docutils literal notranslate"><span class="pre">DataWriter.abort()</span></code> does, and a
-failed job must not commit anything. The files are always owned by exactly one side:</p>
+failed job must not commit anything. Cleanup must never have an ownership gap. During the handoff,
+the native guard intentionally stays armed after the JVM takes the locations and is disarmed only
+when the JVM polls the output stream to EOF:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Phase</p></th>
@@ -836,27 +838,34 @@ failed job must not commit anything. The files are always owned by exactly one s
<tbody>
<tr class="row-even"><td><p>Writing, closing writers, encoding the manifest, building the batch</p></td>
<td><p>Native</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">TrackingLocationGenerator</span></code> records every location. <code class="docutils literal notranslate"><span class="pre">AbortOnDrop</span></code> deletes them on an error, and also when the plan is dropped mid-write, from inside or outside a Tokio runtime.</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">TrackingLocationGenerator</span></code> records every location. <code class="docutils literal notranslate"><span class="pre">AbortOnDrop</span></code> deletes on error or plan drop and stays armed after yielding the batch.</p></td>
</tr>
-<tr class="row-odd"><td><p>After the batch reaches the JVM, until the task succeeds</p></td>
+<tr class="row-odd"><td><p>JVM has recorded locations, before its final EOF poll</p></td>
+<td><p>Native + JVM task</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">WrittenFileCleanup</span></code> owns the decoded locations while <code class="docutils literal notranslate"><span class="pre">AbortOnDrop</span></code> is still armed. A failure may make both sides attempt best-effort deletion.</p></td>
+</tr>
+<tr class="row-even"><td><p>After the JVM’s EOF poll, until the task succeeds</p></td>
<td><p>JVM task</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">WrittenFileCleanup</span></code>, a task failure listener registered before the payload is read, takes the locations before the manifest is decoded.</p></td>
+<td><p>The EOF poll disarms <code class="docutils literal notranslate"><span class="pre">AbortOnDrop</span></code>; <code class="docutils literal notranslate"><span class="pre">WrittenFileCleanup</span></code> remains registered for the rest of the task.</p></td>
</tr>
-<tr class="row-even"><td><p>Job failure after some tasks completed</p></td>
+<tr class="row-odd"><td><p>Job failure after some tasks completed</p></td>
<td><p>Driver (<code class="docutils literal notranslate"><span class="pre">IcebergCommitExec</span></code>)</p></td>
<td><p>Calls <code class="docutils literal notranslate"><span class="pre">BatchWrite.abort</span></code> with the completed messages, then deletes their files through the table’s <code class="docutils literal notranslate"><span class="pre">FileIO</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p>Commit failure</p></td>
+<tr class="row-even"><td><p>Commit failure</p></td>
<td><p>Iceberg</p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">SparkWrite.abort</span></code> on the genuine <code class="docutils literal notranslate"><span class="pre">TaskCommit</span></code> messages, the same as the stock path.</p></td>
</tr>
</tbody>
</table>
</div>
-<p>The handoff between the first two rows is why the payload carries the locations separately from the
-manifest: a failure decoding the manifest would otherwise lose the list of files to delete. All
-deletion is best effort and logged. It must never replace the original exception, and anything it
-misses is unreferenced and reclaimed by Iceberg’s <code class="docutils literal notranslate"><span class="pre">remove_orphan_files</span></code>.</p>
+<p>The handoff between the first two rows intentionally overlaps: <code class="docutils literal notranslate"><span class="pre">WrittenFileCleanup</span></code> takes the
+locations before the final EOF poll, and that poll is the acknowledgement that lets <code class="docutils literal notranslate"><span class="pre">AbortOnDrop</span></code>
+disarm. Deleting the same file from both sides during a failure in that narrow window is harmless
+because cleanup is best effort. The payload carries the locations separately from the manifest so a
+failure decoding the manifest cannot lose the list of files to delete. Cleanup is logged and must
+never replace the original exception; anything it misses is unreferenced and reclaimed by Iceberg’s
+<code class="docutils literal notranslate"><span class="pre">remove_orphan_files</span></code>.</p>
<p>The same planning-time rule applies to failures: a failure that happens because the native side
rejected something the gate admitted is a bug in the gate, even if cleanup works.</p>
</section>
diff --git a/searchindex.js b/searchindex.js
index 17be421250..913167ee35 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[57, "id1"]], "%": [[55, "id1"]], "&": [[45, "id1"]], "*": [[55, "id2"]], "+": [[55, "id3"]], "-": [[55, "id4"]], "/": [[55, "id5"]], "1. Format Your Code": [[42, "format-your-code"]], "1. Install Comet": [[63, "install-comet"], [75, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[42, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[63, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/iceberg-writes.html b/user-guide/latest/iceberg-writes.html
index b205566965..fd0ffa73a5 100644
--- a/user-guide/latest/iceberg-writes.html
+++ b/user-guide/latest/iceberg-writes.html
@@ -982,7 +982,7 @@ iceberg-java resolves per-write options and <code class="docutils literal notran
<td><p>any value (the two writers can choose different roll points; see accepted divergences)</p></td>
</tr>
<tr class="row-even"><td><p>data location URI scheme</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">file</span></code>, <code class="docutils literal notranslate"><span class="pre">memory</span></code>, <code class="docutils literal notranslate"><span class="pre">s3</span></code>, <code class="docutils literal notranslate"><span class="pre">s3a</span></code>, <code class="docutils literal notranslate"><span class="pre">gs</span></code> (<code class="docutils literal notranslate"><span class="pre">gs</span></code> only [...]
+<td><p><code class="docutils literal notranslate"><span class="pre">file</span></code>, <code class="docutils literal notranslate"><span class="pre">memory</span></code>, <code class="docutils literal notranslate"><span class="pre">s3</span></code>, <code class="docutils literal notranslate"><span class="pre">s3a</span></code>, <code class="docutils literal notranslate"><span class="pre">gs</span></code>, matched case-sensitively (<code class="docutils literal notranslate"><span class="p [...]
</tr>
<tr class="row-odd"><td><p>resolved <code class="docutils literal notranslate"><span class="pre">table.locationProvider()</span></code></p></td>
<td><p>Iceberg’s built-in <code class="docutils literal notranslate"><span class="pre">DefaultLocationProvider</span></code></p></td>
@@ -1056,16 +1056,17 @@ attempt id is embedded in its data file names.</p>
successful tasks — a failed task contributes none — and if the job fails, the driver-side
commit operator aborts without committing anything. A failed task attempt also deletes the
data files it created, as iceberg-java’s writer abort does. The native writer records every
-location it hands to a file writer, and exactly one side owns deleting them at any moment: the
-native writer owns them until its output batch reaches the JVM (so it cleans up a failed write,
-a task torn down before the write completed — for example because the operator feeding it threw
-— and a failure encoding the manifest or building that batch), and the JVM owns them from then
-on through a task failure listener. The handoff does not depend on decoding the manifest: the
-native side reports the locations in the output batch alongside it, and the listener is handed
-them before the manifest is decoded, so a failure in that decode still cleans up. Both
-deletions are best-effort and never mask the original failure; anything they miss is invisible
-to every reader, since readers resolve files through committed manifests only, and is reclaimed
-by Iceberg’s normal <code class="docutils literal notranslate"><span class="pre">remove_orphan_files</span></code> maintenance.</p>
+location it hands to a file writer, and cleanup has no ownership gap. Native keeps its cleanup
+guard armed after yielding the output batch. The JVM reads those locations into a task failure
+listener first, then polls the native output to EOF; that EOF is the acknowledgement that lets
+native disarm. A failure in this narrow handoff window can therefore trigger best-effort deletion
+on both sides, which is harmless. The native guard still covers a failed write, a task torn down
+before the write completed — for example because the operator feeding it threw — and a failure
+encoding the manifest or building the output batch. The handoff does not depend on decoding the
+manifest: the locations are read before the manifest is decoded, so a failure in that decode still
+cleans up. Cleanup never masks the original failure; anything it misses is invisible to every
+reader, since readers resolve files through committed manifests only, and is reclaimed by Iceberg’s
+normal <code class="docutils literal notranslate"><span class="pre">remove_orphan_files</span></code> maintenance.</p>
<p>When one task fails, the tasks that had already completed leave committed-nothing data files
too. The committer collects each task’s commit message as that task finishes, so on a job
failure it aborts with the completed messages and deletes their data files through the table
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org