You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/11 13:17:26 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by 9a7b1c1b99ad9a1ddd8657b659723f6fdacf35dd
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 18c6467348 Publish built docs triggered by 9a7b1c1b99ad9a1ddd8657b659723f6fdacf35dd
18c6467348 is described below
commit 18c6467348a7ee0b60f6b75ad5e531f861ff3dc8
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Oct 11 13:17:20 2026 +0000
Publish built docs triggered by 9a7b1c1b99ad9a1ddd8657b659723f6fdacf35dd
---
_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/iceberg-writes.md.txt | 11 ++++++++---
_sources/user-guide/latest/iceberg-writes.md.txt | 9 +++++++--
contributor-guide/iceberg-writes.html | 11 ++++++++---
searchindex.js | 2 +-
user-guide/latest/iceberg-writes.html | 9 +++++++--
6 files changed, 32 insertions(+), 12 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index 6dbbe9148a..70cda79be5 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/iceberg-writes.md.txt b/_sources/contributor-guide/iceberg-writes.md.txt
index 1ef7539329..2cdfb9038b 100644
--- a/_sources/contributor-guide/iceberg-writes.md.txt
+++ b/_sources/contributor-guide/iceberg-writes.md.txt
@@ -312,9 +312,14 @@ Per-bucket `fs.s3a.bucket.<bucket>.*` keys are split into a complete bucket name
suffix. `fs.s3a.bucket.target.other.endpoint` belongs to bucket `target.other`, so it does not
make a write to `target` ineligible.
The Hadoop check excludes keys whose only source is Hadoop's built-in `core-default.xml`, while
-retaining programmatic, site-XML, and custom `*-default.xml` settings. It also ignores Spark's
-session-wide S3A vectored-read and `downgrade.syncable.exceptions` settings, which cannot affect a
-data-file write request. Keep those allowlists aligned when adding storage support. Reaching
+retaining programmatic, site-XML, and custom `*-default.xml` settings. It also ignores the keys in
+`IgnoredHadoopS3Keys`: Spark's session-wide S3A vectored-read and `downgrade.syncable.exceptions`
+settings, plus the committer, `bulk.delete.page.size` and `experimental.input.fadvise` settings that
+deployments set cluster-wide (Spark 4.1 sets two of the committer keys itself). A key belongs on
+that list only if S3A would not apply it to a data-file write request. Settings that S3A does apply,
+such as encryption, ACLs, storage class, signing and the credentials provider, must keep falling
+back. The list matches the exact key, so a per-bucket spelling is not ignored. Keep those
+allowlists aligned when adding storage support. Reaching
`FileIOBuilder.with_prop` is not proof of support: the builder accepts unknown properties and the
backend ignores them. A fallback reason must contain property names only, never values.
diff --git a/_sources/user-guide/latest/iceberg-writes.md.txt b/_sources/user-guide/latest/iceberg-writes.md.txt
index a04f6be61c..2d84caeaca 100644
--- a/_sources/user-guide/latest/iceberg-writes.md.txt
+++ b/_sources/user-guide/latest/iceberg-writes.md.txt
@@ -201,7 +201,7 @@ A write is eligible only when ALL of the following hold:
| `write.target-file-size-bytes` | any value (the two writers can choose different roll points; see accepted divergences) [...]
| data location URI scheme | `file`, `memory`, `s3`, `s3a`, `gs`, matched case-sensitively (`S3://` falls back). `s3`, `s3a` and `gs` need a bucket in the authority (`s3://bucket/...`), so a hostless form such as `s3:/bucket/key` falls back. `gs` only when the `FileIO` opening the data location is a `GCSFileIO`; see below [...]
| resolved `table.locationProvider()` | Iceberg's built-in `DefaultLocationProvider` [...]
-| Hadoop S3A settings for an `s3` / `s3a` data location | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back [...]
+| Hadoop S3A settings for an `s3` / `s3a` data location | only `fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, `endpoint.region`, and `path.style.access`, including their `fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting falls back, apart from the ignored settings listed below [...]
| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location | the S3 endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, `kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain settings parsed by the pinned iceberg-rust version, plus Comet's credential-provider class and built-in web-identity properties. When a custom provider is configured, its vendor-owned `s3.*` [...]
| partition spec | any, except an identity partition on a `float` or `double` column, or a `void` field whose source column was dropped beside a live field (see below) [...]
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) and the v3 types `variant`, `unknown`, `timestamp_ns`, `geometry` and `geography` [...]
@@ -242,7 +242,12 @@ properties; session and catalog Hadoop options are neither checked nor forwarded
`core-default.xml` values are not treated as explicit settings, but
programmatic settings and values from site or custom `*-default.xml` resources are. Spark's
session-wide S3A vectored-read and `downgrade.syncable.exceptions` compatibility settings are also
-ignored because they cannot alter an Iceberg data-file write request. Unknown explicit
+ignored, as are the S3A committer (`committer.name`, `committer.magic.enabled`,
+`committer.threads`), `bulk.delete.page.size` and `experimental.input.fadvise` settings, because
+none of them can alter an Iceberg data-file write request. Spark 4.1 sets `committer.name` and
+`committer.magic.enabled` itself when `spark-hadoop-cloud` is on the classpath. Ignored settings are
+matched by their plain `fs.s3a.<name>` keys, so a per-bucket `fs.s3a.bucket.<data-bucket>.<name>`
+form still falls back. Unknown explicit
`fs.s3a.*`, `s3.*`, or `client.*` settings therefore fall back at planning time instead of being
silently ignored by the native storage backend. The exception is a vendor-owned `s3.*` /
`client.*` property when
diff --git a/contributor-guide/iceberg-writes.html b/contributor-guide/iceberg-writes.html
index 9ebcf67bff..e964a697f9 100644
--- a/contributor-guide/iceberg-writes.html
+++ b/contributor-guide/iceberg-writes.html
@@ -862,9 +862,14 @@ Per-bucket <code class="docutils literal notranslate"><span class="pre">fs.s3a.b
suffix. <code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.target.other.endpoint</span></code> belongs to bucket <code class="docutils literal notranslate"><span class="pre">target.other</span></code>, so it does not
make a write to <code class="docutils literal notranslate"><span class="pre">target</span></code> ineligible.
The Hadoop check excludes keys whose only source is Hadoop’s built-in <code class="docutils literal notranslate"><span class="pre">core-default.xml</span></code>, while
-retaining programmatic, site-XML, and custom <code class="docutils literal notranslate"><span class="pre">*-default.xml</span></code> settings. It also ignores Spark’s
-session-wide S3A vectored-read and <code class="docutils literal notranslate"><span class="pre">downgrade.syncable.exceptions</span></code> settings, which cannot affect a
-data-file write request. Keep those allowlists aligned when adding storage support. Reaching
+retaining programmatic, site-XML, and custom <code class="docutils literal notranslate"><span class="pre">*-default.xml</span></code> settings. It also ignores the keys in
+<code class="docutils literal notranslate"><span class="pre">IgnoredHadoopS3Keys</span></code>: Spark’s session-wide S3A vectored-read and <code class="docutils literal notranslate"><span class="pre">downgrade.syncable.exceptions</span></code>
+settings, plus the committer, <code class="docutils literal notranslate"><span class="pre">bulk.delete.page.size</span></code> and <code class="docutils literal notranslate"><span class="pre">experimental.input.fadvise</span></code> settings that
+deployments set cluster-wide (Spark 4.1 sets two of the committer keys itself). A key belongs on
+that list only if S3A would not apply it to a data-file write request. Settings that S3A does apply,
+such as encryption, ACLs, storage class, signing and the credentials provider, must keep falling
+back. The list matches the exact key, so a per-bucket spelling is not ignored. Keep those
+allowlists aligned when adding storage support. Reaching
<code class="docutils literal notranslate"><span class="pre">FileIOBuilder.with_prop</span></code> is not proof of support: the builder accepts unknown properties and the
backend ignores them. A fallback reason must contain property names only, never values.</p>
</section>
diff --git a/searchindex.js b/searchindex.js
index e02f526bb4..cb16769c70 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[59, "id1"]], "%": [[57, "id1"]], "&": [[47, "id1"]], "*": [[57, "id2"]], "+": [[57, "id3"]], "-": [[57, "id4"]], "/": [[57, "id5"]], "1. Format Your Code": [[44, "format-your-code"]], "1. Install Comet": [[66, "install-comet"], [78, "install-comet"]], "1. Native Operators (nativeExecs map)": [[30, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[44, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[66, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[59, "id1"]], "%": [[57, "id1"]], "&": [[47, "id1"]], "*": [[57, "id2"]], "+": [[57, "id3"]], "-": [[57, "id4"]], "/": [[57, "id5"]], "1. Format Your Code": [[44, "format-your-code"]], "1. Install Comet": [[66, "install-comet"], [78, "install-comet"]], "1. Native Operators (nativeExecs map)": [[30, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[44, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[66, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/iceberg-writes.html b/user-guide/latest/iceberg-writes.html
index a80dfd5d01..9dff6b4114 100644
--- a/user-guide/latest/iceberg-writes.html
+++ b/user-guide/latest/iceberg-writes.html
@@ -1049,7 +1049,7 @@ iceberg-java resolves per-write options and <code class="docutils literal notran
<td><p>Iceberg’s built-in <code class="docutils literal notranslate"><span class="pre">DefaultLocationProvider</span></code></p></td>
</tr>
<tr class="row-even"><td><p>Hadoop S3A settings for an <code class="docutils literal notranslate"><span class="pre">s3</span></code> / <code class="docutils literal notranslate"><span class="pre">s3a</span></code> data location</p></td>
-<td><p>only <code class="docutils literal notranslate"><span class="pre">fs.s3a.access.key</span></code>, <code class="docutils literal notranslate"><span class="pre">secret.key</span></code>, <code class="docutils literal notranslate"><span class="pre">session.token</span></code>, <code class="docutils literal notranslate"><span class="pre">endpoint</span></code>, <code class="docutils literal notranslate"><span class="pre">endpoint.region</span></code>, and <code class="docutils litera [...]
+<td><p>only <code class="docutils literal notranslate"><span class="pre">fs.s3a.access.key</span></code>, <code class="docutils literal notranslate"><span class="pre">secret.key</span></code>, <code class="docutils literal notranslate"><span class="pre">session.token</span></code>, <code class="docutils literal notranslate"><span class="pre">endpoint</span></code>, <code class="docutils literal notranslate"><span class="pre">endpoint.region</span></code>, and <code class="docutils litera [...]
</tr>
<tr class="row-odd"><td><p>Iceberg <code class="docutils literal notranslate"><span class="pre">FileIO</span></code> S3 settings for an <code class="docutils literal notranslate"><span class="pre">s3</span></code> / <code class="docutils literal notranslate"><span class="pre">s3a</span></code> data location</p></td>
<td><p>the S3 endpoint, region, static/session credentials, path-style, SSE (<code class="docutils literal notranslate"><span class="pre">none</span></code>, <code class="docutils literal notranslate"><span class="pre">s3</span></code>, <code class="docutils literal notranslate"><span class="pre">kms</span></code>, or <code class="docutils literal notranslate"><span class="pre">custom</span></code>; not <code class="docutils literal notranslate"><span class="pre">dsse-kms</span></code>), [...]
@@ -1096,7 +1096,12 @@ properties; session and catalog Hadoop options are neither checked nor forwarded
<code class="docutils literal notranslate"><span class="pre">core-default.xml</span></code> values are not treated as explicit settings, but
programmatic settings and values from site or custom <code class="docutils literal notranslate"><span class="pre">*-default.xml</span></code> resources are. Spark’s
session-wide S3A vectored-read and <code class="docutils literal notranslate"><span class="pre">downgrade.syncable.exceptions</span></code> compatibility settings are also
-ignored because they cannot alter an Iceberg data-file write request. Unknown explicit
+ignored, as are the S3A committer (<code class="docutils literal notranslate"><span class="pre">committer.name</span></code>, <code class="docutils literal notranslate"><span class="pre">committer.magic.enabled</span></code>,
+<code class="docutils literal notranslate"><span class="pre">committer.threads</span></code>), <code class="docutils literal notranslate"><span class="pre">bulk.delete.page.size</span></code> and <code class="docutils literal notranslate"><span class="pre">experimental.input.fadvise</span></code> settings, because
+none of them can alter an Iceberg data-file write request. Spark 4.1 sets <code class="docutils literal notranslate"><span class="pre">committer.name</span></code> and
+<code class="docutils literal notranslate"><span class="pre">committer.magic.enabled</span></code> itself when <code class="docutils literal notranslate"><span class="pre">spark-hadoop-cloud</span></code> is on the classpath. Ignored settings are
+matched by their plain <code class="docutils literal notranslate"><span class="pre">fs.s3a.<name></span></code> keys, so a per-bucket <code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.<data-bucket>.<name></span></code>
+form still falls back. Unknown explicit
<code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code>, <code class="docutils literal notranslate"><span class="pre">s3.*</span></code>, or <code class="docutils literal notranslate"><span class="pre">client.*</span></code> settings therefore fall back at planning time instead of being
silently ignored by the native storage backend. The exception is a vendor-owned <code class="docutils literal notranslate"><span class="pre">s3.*</span></code> /
<code class="docutils literal notranslate"><span class="pre">client.*</span></code> property when
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org