You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/05 15:31:02 UTC
(datafusion-comet) branch asf-site updated: Publish built docs triggered by a1f8bd3ea2fbd76e0a0eeae1184d4040a67ad3f4
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c84e1ad2f4 Publish built docs triggered by a1f8bd3ea2fbd76e0a0eeae1184d4040a67ad3f4
c84e1ad2f4 is described below
commit c84e1ad2f4d7c8464ef2f18a92bde04c4fc9b215
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sat Sep 5 15:30:57 2026 +0000
Publish built docs triggered by a1f8bd3ea2fbd76e0a0eeae1184d4040a67ad3f4
---
_sources/user-guide/latest/configs.md.txt | 1 +
_sources/user-guide/latest/datasources.md.txt | 62 +++++++++++++++++++++++
_sources/user-guide/latest/iceberg.md.txt | 2 +-
searchindex.js | 2 +-
user-guide/latest/configs.html | 4 ++
user-guide/latest/datasources.html | 72 +++++++++++++++++++++++++++
user-guide/latest/iceberg.html | 2 +-
7 files changed, 142 insertions(+), 3 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt b/_sources/user-guide/latest/configs.md.txt
index 68e6533bf6..4b0d9e2ccd 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -32,6 +32,7 @@ Comet provides the following configuration settings.
| `spark.comet.scan.icebergNative.enabled` | Whether to enable native Iceberg table scan using iceberg-rust. When enabled, Iceberg tables are read directly through native execution, bypassing Spark's DataSource V2 API for better performance. | true |
| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this config is true (default), Comet falls back to Spark for ShortType columns because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may produce different results). Set to false to allow native execution of ShortType columns if you know your data does not contain unsigned UINT_8 columns from improperly encoded Parquet files. F [...]
| `spark.hadoop.fs.comet.libhdfs.schemes` | Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas. Valid only when built with hdfs-opendal feature enabled. | |
+| `spark.hadoop.fs.comet.s3Compliant.schemes` | Defines filesystem schemes (e.g., blob, minio, r2) that Comet treats as S3-compliant, separated by commas. Such schemes reuse the `fs.s3a.*` credential surface and accept vendor-style `fs.<scheme>.<authority>.*` keys. Empty by default, so no alias scheme is claimed unless opted in. Read from the Hadoop configuration, so it must be set before the SparkSession is created. | |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index d5f738d13b..84994c00c2 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -216,6 +216,68 @@ Beyond credential providers, Comet's Parquet scan supports additional S3 configu
All configuration options support bucket-specific overrides using the pattern `fs.s3a.bucket.{bucket-name}.{option}`.
+### S3-Compliant Filesystem Schemes
+
+Some environments front an S3-compatible service (MinIO, Ceph RGW, Cloudflare R2, Wasabi, and
+similar) with a vendor-branded Hadoop filesystem client that registers its own URL scheme, for
+example `blob://`, instead of `s3://` or `s3a://`. Comet can treat such schemes as aliases for
+`s3://` so the native Parquet and Iceberg scans read them directly, without the caller rewriting
+URLs.
+
+This is opt-in and disabled by default. Enable it by listing the schemes to treat as S3-compliant
+aliases in `spark.hadoop.fs.comet.s3Compliant.schemes` (Hadoop key
+`fs.comet.s3Compliant.schemes`), a comma-separated, case-insensitive list. This mirrors the
+existing `fs.comet.libhdfs.schemes` config.
+
+```shell
+--conf spark.hadoop.fs.comet.s3Compliant.schemes=blob
+```
+
+Multiple schemes can be listed together:
+
+```shell
+--conf spark.hadoop.fs.comet.s3Compliant.schemes=blob,minio,r2
+```
+
+With no configuration, Comet claims none of these aliases, so for example a `blob://` path falls
+back to Spark unchanged. The empty default is deliberate: short scheme names like `blob` are not
+unique to S3-compatible storage. Azure Blob Storage is the clearest example, so claiming `blob://`
+unconditionally would risk misrouting paths that were never meant for Comet's S3 client. Only add a
+scheme here if you intend Comet to treat it as S3-compatible.
+
+For each scheme `<s>` listed in `fs.comet.s3Compliant.schemes`, Comet also reads vendor-style,
+per-authority Hadoop keys of the form `fs.<s>.<authority>.<property>` (the authority is typically
+the bucket or account name from the URL) and translates them into the `fs.s3a.*` surface described
+above. The recognized vendor-style properties and their `fs.s3a.*` targets are:
+
+| Vendor property (`fs.<s>.<authority>.<property>`) | Translated `fs.s3a.*` suffix |
+| ------------------------------------------------- | ---------------------------- |
+| `awsAccessKeyId` | `access.key` |
+| `awsSecretAccessKey` | `secret.key` |
+| `awsSessionToken` | `session.token` |
+| `endpoint` | `endpoint` |
+| `region` | `endpoint.region` |
+| `pathStyleAccess` | `path.style.access` |
+
+Unrecognized `fs.<s>.<authority>.*` properties are ignored.
+
+A URL with no authority, such as the triple-slash `blob:///bucket/key` form, reports its authority
+as the literal string `default`. For that case, list the keys under `fs.<s>.default.<property>`.
+Comet resolves `default` to the bucket taken from the URL path (`bucket` here) and applies the
+translated settings at that bucket's scope, exactly as if they had been written under
+`fs.<s>.bucket.<property>`.
+
+When `fs.<s>.<authority>.endpoint` is set, Comet defaults path-style access to enabled for that
+bucket, since most non-AWS S3-compatible services require it. This is only a default: set
+`fs.s3a.bucket.<bucket>.path.style.access` (or the equivalent per-scheme key) explicitly to
+override it.
+
+Once translated, these values are applied at the same `fs.s3a.bucket.{bucket-name}.*` scope
+described in [Additional S3 Configuration Options](#additional-s3-configuration-options), so the
+credential providers and options documented above also apply to alias-scheme URLs. The same
+translation feeds the native Iceberg scan; see
+[Object store configuration (S3)](iceberg.md#object-store-configuration-s3) in the Iceberg guide.
+
### Examples
The following examples demonstrate how to configure S3 access using different authentication methods.
diff --git a/_sources/user-guide/latest/iceberg.md.txt b/_sources/user-guide/latest/iceberg.md.txt
index 300670a45d..f30c0fff68 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -144,7 +144,7 @@ scala> spark.sql("SELECT * FROM rest_cat.db.test_table").show()
### Object store configuration (S3)
-The native reader has its own Rust object store client and does not go through Iceberg's JVM FileIO, neither `S3FileIO` nor the older Hadoop S3A filesystem. It configures that client from the catalog's `s3.*` properties (the same keys `S3FileIO` reads) or from `spark.hadoop.fs.s3a.*` settings, so S3 configuration only reaches the native reader through one of those two channels.
+The native reader has its own Rust object store client and does not go through Iceberg's JVM FileIO, neither `S3FileIO` nor the older Hadoop S3A filesystem. It configures that client from the catalog's `s3.*` properties (the same keys `S3FileIO` reads), from `spark.hadoop.fs.s3a.*` settings, or, for a scheme opted into `spark.hadoop.fs.comet.s3Compliant.schemes`, from vendor-style `fs.<scheme>.<authority>.*` keys (see [S3-Compliant Filesystem Schemes](datasources.md#s3-compliant-filesyst [...]
For a custom S3-compatible endpoint, configure the catalog with the endpoint, path-style access, region, and credentials (Hive shown):
diff --git a/searchindex.js b/searchindex.js
index df0a90d46f..7ae1dda146 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index b77797a7f1..8ab1be17d6 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -809,6 +809,10 @@ under the License.
<td><p>Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas. Valid only when built with hdfs-opendal feature enabled.</p></td>
<td><p></p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spark.hadoop.fs.comet.s3Compliant.schemes</span></code></p></td>
+<td><p>Defines filesystem schemes (e.g., blob, minio, r2) that Comet treats as S3-compliant, separated by commas. Such schemes reuse the <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> credential surface and accept vendor-style <code class="docutils literal notranslate"><span class="pre">fs.<scheme>.<authority>.*</span></code> keys. Empty by default, so no alias scheme is claimed unless opted in. Read from the Hadoop configuration, so it mu [...]
+<td><p></p></td>
+</tr>
</tbody>
</table>
</div>
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index 73efe335cc..5da48be308 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -1010,6 +1010,77 @@ installed.</p>
</div>
<p>All configuration options support bucket-specific overrides using the pattern <code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.{bucket-name}.{option}</span></code>.</p>
</section>
+<section id="s3-compliant-filesystem-schemes">
+<h3>S3-Compliant Filesystem Schemes<a class="headerlink" href="#s3-compliant-filesystem-schemes" title="Link to this heading">#</a></h3>
+<p>Some environments front an S3-compatible service (MinIO, Ceph RGW, Cloudflare R2, Wasabi, and
+similar) with a vendor-branded Hadoop filesystem client that registers its own URL scheme, for
+example <code class="docutils literal notranslate"><span class="pre">blob://</span></code>, instead of <code class="docutils literal notranslate"><span class="pre">s3://</span></code> or <code class="docutils literal notranslate"><span class="pre">s3a://</span></code>. Comet can treat such schemes as aliases for
+<code class="docutils literal notranslate"><span class="pre">s3://</span></code> so the native Parquet and Iceberg scans read them directly, without the caller rewriting
+URLs.</p>
+<p>This is opt-in and disabled by default. Enable it by listing the schemes to treat as S3-compliant
+aliases in <code class="docutils literal notranslate"><span class="pre">spark.hadoop.fs.comet.s3Compliant.schemes</span></code> (Hadoop key
+<code class="docutils literal notranslate"><span class="pre">fs.comet.s3Compliant.schemes</span></code>), a comma-separated, case-insensitive list. This mirrors the
+existing <code class="docutils literal notranslate"><span class="pre">fs.comet.libhdfs.schemes</span></code> config.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>--conf<span class="w"> </span>spark.hadoop.fs.comet.s3Compliant.schemes<span class="o">=</span>blob
+</pre></div>
+</div>
+<p>Multiple schemes can be listed together:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>--conf<span class="w"> </span>spark.hadoop.fs.comet.s3Compliant.schemes<span class="o">=</span>blob,minio,r2
+</pre></div>
+</div>
+<p>With no configuration, Comet claims none of these aliases, so for example a <code class="docutils literal notranslate"><span class="pre">blob://</span></code> path falls
+back to Spark unchanged. The empty default is deliberate: short scheme names like <code class="docutils literal notranslate"><span class="pre">blob</span></code> are not
+unique to S3-compatible storage. Azure Blob Storage is the clearest example, so claiming <code class="docutils literal notranslate"><span class="pre">blob://</span></code>
+unconditionally would risk misrouting paths that were never meant for Comet’s S3 client. Only add a
+scheme here if you intend Comet to treat it as S3-compatible.</p>
+<p>For each scheme <code class="docutils literal notranslate"><span class="pre"><s></span></code> listed in <code class="docutils literal notranslate"><span class="pre">fs.comet.s3Compliant.schemes</span></code>, Comet also reads vendor-style,
+per-authority Hadoop keys of the form <code class="docutils literal notranslate"><span class="pre">fs.<s>.<authority>.<property></span></code> (the authority is typically
+the bucket or account name from the URL) and translates them into the <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> surface described
+above. The recognized vendor-style properties and their <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> targets are:</p>
+<div class="pst-scrollable-table-container"><table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Vendor property (<code class="docutils literal notranslate"><span class="pre">fs.<s>.<authority>.<property></span></code>)</p></th>
+<th class="head"><p>Translated <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> suffix</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">awsAccessKeyId</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">access.key</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">awsSecretAccessKey</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">secret.key</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">awsSessionToken</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">session.token</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">endpoint</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">endpoint</span></code></p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">region</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">endpoint.region</span></code></p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">pathStyleAccess</span></code></p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">path.style.access</span></code></p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<p>Unrecognized <code class="docutils literal notranslate"><span class="pre">fs.<s>.<authority>.*</span></code> properties are ignored.</p>
+<p>A URL with no authority, such as the triple-slash <code class="docutils literal notranslate"><span class="pre">blob:///bucket/key</span></code> form, reports its authority
+as the literal string <code class="docutils literal notranslate"><span class="pre">default</span></code>. For that case, list the keys under <code class="docutils literal notranslate"><span class="pre">fs.<s>.default.<property></span></code>.
+Comet resolves <code class="docutils literal notranslate"><span class="pre">default</span></code> to the bucket taken from the URL path (<code class="docutils literal notranslate"><span class="pre">bucket</span></code> here) and applies the
+translated settings at that bucket’s scope, exactly as if they had been written under
+<code class="docutils literal notranslate"><span class="pre">fs.<s>.bucket.<property></span></code>.</p>
+<p>When <code class="docutils literal notranslate"><span class="pre">fs.<s>.<authority>.endpoint</span></code> is set, Comet defaults path-style access to enabled for that
+bucket, since most non-AWS S3-compatible services require it. This is only a default: set
+<code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.<bucket>.path.style.access</span></code> (or the equivalent per-scheme key) explicitly to
+override it.</p>
+<p>Once translated, these values are applied at the same <code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.{bucket-name}.*</span></code> scope
+described in <a class="reference internal" href="#additional-s3-configuration-options">Additional S3 Configuration Options</a>, so the
+credential providers and options documented above also apply to alias-scheme URLs. The same
+translation feeds the native Iceberg scan; see
+<a class="reference internal" href="iceberg.html#object-store-configuration-s3"><span class="std std-ref">Object store configuration (S3)</span></a> in the Iceberg guide.</p>
+</section>
<section id="examples">
<h3>Examples<a class="headerlink" href="#examples" title="Link to this heading">#</a></h3>
<p>The following examples demonstrate how to configure S3 access using different authentication methods.</p>
@@ -1262,6 +1333,7 @@ As with S3, standard Hadoop ABFS configurations (the <code class="docutils liter
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#root-ca-certificates">Root CA Certificates</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#supported-credential-providers">Supported Credential Providers</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#additional-s3-configuration-options">Additional S3 Configuration Options</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#s3-compliant-filesystem-schemes">S3-Compliant Filesystem Schemes</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#examples">Examples</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#limitations">Limitations</a></li>
</ul>
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index f47f08727a..09cd6ef2e5 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -893,7 +893,7 @@ configure Spark to use a REST catalog with Comet’s native Iceberg scan:</p>
</section>
<section id="object-store-configuration-s3">
<h3>Object store configuration (S3)<a class="headerlink" href="#object-store-configuration-s3" title="Link to this heading">#</a></h3>
-<p>The native reader has its own Rust object store client and does not go through Iceberg’s JVM FileIO, neither <code class="docutils literal notranslate"><span class="pre">S3FileIO</span></code> nor the older Hadoop S3A filesystem. It configures that client from the catalog’s <code class="docutils literal notranslate"><span class="pre">s3.*</span></code> properties (the same keys <code class="docutils literal notranslate"><span class="pre">S3FileIO</span></code> reads) or from <code cla [...]
+<p>The native reader has its own Rust object store client and does not go through Iceberg’s JVM FileIO, neither <code class="docutils literal notranslate"><span class="pre">S3FileIO</span></code> nor the older Hadoop S3A filesystem. It configures that client from the catalog’s <code class="docutils literal notranslate"><span class="pre">s3.*</span></code> properties (the same keys <code class="docutils literal notranslate"><span class="pre">S3FileIO</span></code> reads), from <code class [...]
<p>For a custom S3-compatible endpoint, configure the catalog with the endpoint, path-style access, region, and credentials (Hive shown):</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span><span class="w"> </span>--conf<span class="w"> </span>spark.sql.catalog.s3_cat<span class="o">=</span>org.apache.iceberg.spark.SparkCatalog<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w"> </span>spark.sql.catalog.s3_cat.type<span class="o">=</span>hive<span class="w"> </span><span class="se">\</span>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org