You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/16 01:01:32 UTC
(datafusion-comet) 01/01: Publish built docs triggered by 5e7d918ef5ec64ca13f996f42868758474632798
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 315b4a4080195c8313e9257483852ba5bc5761db
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Fri Oct 16 01:01:25 2026 +0000
Publish built docs triggered by 5e7d918ef5ec64ca13f996f42868758474632798
---
...id-09341aca03757d9d3637a2eaff73472d472d0673.svg | 2 +-
.../s3-credential-provider-design.md.txt | 2 +-
_sources/user-guide/1.1/datasources.md.txt | 4 ++-
_sources/user-guide/1.1/iceberg.md.txt | 1 +
_sources/user-guide/latest/datasources.md.txt | 40 +++++++++++++++++-----
.../s3-credential-provider-design.html | 2 +-
searchindex.js | 2 +-
user-guide/1.1/datasources.html | 4 ++-
user-guide/1.1/iceberg.html | 1 +
user-guide/latest/datasources.html | 40 +++++++++++++++++-----
10 files changed, 76 insertions(+), 22 deletions(-)
diff --git a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
index e28ad8f87b..104593de2e 100644
--- a/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
+++ b/_images/mermaid-09341aca03757d9d3637a2eaff73472d472d0673.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 1014" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-sv [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/s3-credential-provider-design.md.txt b/_sources/contributor-guide/s3-credential-provider-design.md.txt
index d4fa3e1baa..a30548566b 100644
--- a/_sources/contributor-guide/s3-credential-provider-design.md.txt
+++ b/_sources/contributor-guide/s3-credential-provider-design.md.txt
@@ -154,7 +154,7 @@ The cache holds the `FileIO` and its bridge. The bridge reuses its credential un
## Property-bag handling on the Parquet path
-The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so strip [...]
+The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so strip [...]
## Built-in adapters
diff --git a/_sources/user-guide/1.1/datasources.md.txt b/_sources/user-guide/1.1/datasources.md.txt
index faf63102ec..d52dac9f30 100644
--- a/_sources/user-guide/1.1/datasources.md.txt
+++ b/_sources/user-guide/1.1/datasources.md.txt
@@ -39,7 +39,9 @@ See the [Iceberg Guide] and [Iceberg Writes](iceberg-writes.md) for more informa
Comet provides experimental Rust-based CSV scan support. When `spark.comet.scan.csv.v2.enabled` is enabled, CSV files
are read in Rust for improved performance. This feature is experimental and performance benefits are
workload-dependent. Only Spark's DataSource V2 CSV scan is accelerated, and Spark reads CSV through the V1 API by
-default, so also remove `csv` from `spark.sql.sources.useV1SourceList`.
+default, so also remove `csv` from `spark.sql.sources.useV1SourceList`. Queries that use
+`input_file_name()`, `input_file_block_start()` or `input_file_block_length()` fall back to Spark's CSV
+reader, which sets the values these functions read.
Alternatively, when `spark.comet.convert.csv.enabled` is enabled, data from Spark's CSV reader is immediately
converted into Arrow format, allowing the Comet pipeline to take over after that.
diff --git a/_sources/user-guide/1.1/iceberg.md.txt b/_sources/user-guide/1.1/iceberg.md.txt
index 3e1ed36335..5f2130424e 100644
--- a/_sources/user-guide/1.1/iceberg.md.txt
+++ b/_sources/user-guide/1.1/iceberg.md.txt
@@ -186,6 +186,7 @@ The following scenarios will fall back to the JVM Iceberg reader:
- Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto)
- Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes)
- Tables backed by Avro or ORC data files (only Parquet is accelerated)
+- Queries that use `input_file_name()`, `input_file_block_start()` or `input_file_block_length()`
- Scans whose data or delete files span more than one S3 bucket (the native reader uses one
object-store configuration per scan)
- Tables partitioned on `BINARY` or `DECIMAL` (with precision >28) columns
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index 47e2eb7837..43ef755c15 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -115,6 +115,29 @@ See the dedicated [Comet and Iceberg Guide](iceberg.md).
Comet supports most standard storage systems, such as local file system and object storage.
+### Scans over more than one object store
+
+A native Parquet or CSV scan reads each of its partitions through one object store, such as one S3
+bucket, one GCS bucket, or one HDFS name node. When a scan's files live in more than one store,
+Comet puts the files of each store into separate partitions, so such a scan can run with a different
+number of partitions than Spark would use. A native Parquet scan packs each store's files on their
+own, so `spark.sql.files.maxPartitionNum` applies to each store's files separately. A native CSV
+scan that would split a partition falls back to Spark when Spark planned the operators above it for
+the scan's own partitioning, as Spark 3.4 does for a scan of one partition.
+
+Such a scan forwards the object store settings of every scheme it reads, so its files can mix
+schemes such as `s3a://`, `gs://` and `hdfs://`. It falls back to Spark when it reads one bucket
+through an S3-compliant alias and through another scheme (for example `blob://bucket` and
+`s3a://bucket`), because the alias settings would then apply to both. It also falls back when an
+alias path names no bucket (`blob:///`) next to other S3 paths, because that path's alias settings
+would apply to every bucket. An alias scheme that is also listed in `fs.comet.libhdfs.schemes` is
+read through libhdfs, so Comet translates none of its settings and such a path does not cause
+either fallback. ABFS is the exception to mixing stores: Comet resolves the credentials of one
+container and account per scan, so a scan whose files span more than one ABFS container or account,
+or that mixes `abfs`/`abfss` paths with other schemes, falls back to Spark (see [Azure](#azure)). A
+native Parquet scan of a bucketed table also falls back when its files span more than one store,
+because each table bucket is read as one partition.
+
### HDFS
The Apache DataFusion Comet Rust-based reader seamlessly scans files from remote HDFS for [supported formats](#supported-spark-data-sources)
@@ -344,11 +367,12 @@ above. The recognized vendor-style properties and their `fs.s3a.*` targets are:
Unrecognized `fs.<s>.<authority>.*` properties are ignored.
-A URL with no authority, such as the triple-slash `blob:///bucket/key` form, reports its authority
-as the literal string `default`. For that case, list the keys under `fs.<s>.default.<property>`.
-Comet resolves `default` to the bucket taken from the URL path (`bucket` here) and applies the
-translated settings at that bucket's scope, exactly as if they had been written under
-`fs.<s>.bucket.<property>`.
+Keys under the literal authority `default`, `fs.<s>.default.<property>`, apply to every bucket of
+that scheme that has no `fs.<s>.<bucket>.<property>` key of its own, whether the URL names the
+bucket as its authority (`blob://bucket/key`) or in its path. A URL with no authority, such as the
+triple-slash `blob:///bucket/key` form, reports its authority as `default`, so these keys are the
+way to configure it. Comet applies them at the scope of the bucket the URL names (`bucket` here),
+exactly as if they had been written under `fs.<s>.bucket.<property>`.
When `fs.<s>.<authority>.endpoint` is set, Comet defaults path-style access to enabled for that
bucket, since most non-AWS S3-compatible services require it. This is only a default: set
@@ -361,9 +385,9 @@ credential providers and options documented above also apply to alias-scheme URL
translation feeds the native Iceberg scan; see
[Object store configuration (S3)](iceberg.md#object-store-configuration-s3) in the Iceberg guide.
-A native Parquet scan whose alias-scheme paths span more than one bucket falls back to Spark. Alias
-schemes apply to native scans only: a native Iceberg write to an alias-scheme location falls back to
-iceberg-java.
+A native Parquet or CSV scan whose alias-scheme paths span more than one bucket resolves the alias
+settings for each bucket. Alias schemes apply to native scans only: a native Iceberg write to an
+alias-scheme location falls back to iceberg-java.
### Examples
diff --git a/contributor-guide/s3-credential-provider-design.html b/contributor-guide/s3-credential-provider-design.html
index 5551b382de..4fb3669567 100644
--- a/contributor-guide/s3-credential-provider-design.html
+++ b/contributor-guide/s3-credential-provider-design.html
@@ -700,7 +700,7 @@ under the License.
</section>
<section id="property-bag-handling-on-the-parquet-path">
<h2>Property-bag handling on the Parquet path<a class="headerlink" href="#property-bag-handling-on-the-parquet-path" title="Link to this heading">#</a></h2>
-<p>The Parquet path forwards the full <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> config subset as <code class="docutils literal notranslate"><span class="pre">catalog_properties</span></code>, so an SPI provider sees the same <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> config Spark would. <code class="docutils literal notranslate"><span class="pre">forward_catalog_properties</span></code> in <code class="doc [...]
+<p>The Parquet path forwards the full <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> config subset as <code class="docutils literal notranslate"><span class="pre">catalog_properties</span></code>, so an SPI provider sees the same <code class="docutils literal notranslate"><span class="pre">fs.s3a.*</span></code> config Spark would. <code class="docutils literal notranslate"><span class="pre">forward_catalog_properties</span></code> in <code class="doc [...]
</section>
<section id="built-in-adapters">
<h2>Built-in adapters<a class="headerlink" href="#built-in-adapters" title="Link to this heading">#</a></h2>
diff --git a/searchindex.js b/searchindex.js
index f51db2f752..e5079c444b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[60, "id1"]], "%": [[58, "id1"]], "&": [[48, "id1"]], "*": [[58, "id2"]], "+": [[58, "id3"]], "-": [[58, "id4"]], "/": [[58, "id5"]], "1. Format Your Code": [[45, "format-your-code"]], "1. Install Comet": [[67, "install-comet"], [79, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[45, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[67, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[60, "id1"]], "%": [[58, "id1"]], "&": [[48, "id1"]], "*": [[58, "id2"]], "+": [[58, "id3"]], "-": [[58, "id4"]], "/": [[58, "id5"]], "1. Format Your Code": [[45, "format-your-code"]], "1. Install Comet": [[67, "install-comet"], [79, "install-comet"]], "1. Native Operators (nativeExecs map)": [[31, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[45, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[67, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/1.1/datasources.html b/user-guide/1.1/datasources.html
index 4e06a60373..0c0185545f 100644
--- a/user-guide/1.1/datasources.html
+++ b/user-guide/1.1/datasources.html
@@ -864,7 +864,9 @@ See the <a class="reference internal" href="iceberg.html"><span class="std std-d
<p>Comet provides experimental Rust-based CSV scan support. When <code class="docutils literal notranslate"><span class="pre">spark.comet.scan.csv.v2.enabled</span></code> is enabled, CSV files
are read in Rust for improved performance. This feature is experimental and performance benefits are
workload-dependent. Only Spark’s DataSource V2 CSV scan is accelerated, and Spark reads CSV through the V1 API by
-default, so also remove <code class="docutils literal notranslate"><span class="pre">csv</span></code> from <code class="docutils literal notranslate"><span class="pre">spark.sql.sources.useV1SourceList</span></code>.</p>
+default, so also remove <code class="docutils literal notranslate"><span class="pre">csv</span></code> from <code class="docutils literal notranslate"><span class="pre">spark.sql.sources.useV1SourceList</span></code>. Queries that use
+<code class="docutils literal notranslate"><span class="pre">input_file_name()</span></code>, <code class="docutils literal notranslate"><span class="pre">input_file_block_start()</span></code> or <code class="docutils literal notranslate"><span class="pre">input_file_block_length()</span></code> fall back to Spark’s CSV
+reader, which sets the values these functions read.</p>
<p>Alternatively, when <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.csv.enabled</span></code> is enabled, data from Spark’s CSV reader is immediately
converted into Arrow format, allowing the Comet pipeline to take over after that.</p>
</section>
diff --git a/user-guide/1.1/iceberg.html b/user-guide/1.1/iceberg.html
index 954222d874..26e18d3268 100644
--- a/user-guide/1.1/iceberg.html
+++ b/user-guide/1.1/iceberg.html
@@ -999,6 +999,7 @@ projected is read natively)</p></li>
<li><p>Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto)</p></li>
<li><p>Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes)</p></li>
<li><p>Tables backed by Avro or ORC data files (only Parquet is accelerated)</p></li>
+<li><p>Queries that use <code class="docutils literal notranslate"><span class="pre">input_file_name()</span></code>, <code class="docutils literal notranslate"><span class="pre">input_file_block_start()</span></code> or <code class="docutils literal notranslate"><span class="pre">input_file_block_length()</span></code></p></li>
<li><p>Scans whose data or delete files span more than one S3 bucket (the native reader uses one
object-store configuration per scan)</p></li>
<li><p>Tables partitioned on <code class="docutils literal notranslate"><span class="pre">BINARY</span></code> or <code class="docutils literal notranslate"><span class="pre">DECIMAL</span></code> (with precision >28) columns</p></li>
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index 54e0778f7f..b0d8b05056 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -936,6 +936,28 @@ them, and the conversion and the Comet operators above it would read ahead of th
<section id="supported-storages">
<h2>Supported Storages<a class="headerlink" href="#supported-storages" title="Link to this heading">#</a></h2>
<p>Comet supports most standard storage systems, such as local file system and object storage.</p>
+<section id="scans-over-more-than-one-object-store">
+<h3>Scans over more than one object store<a class="headerlink" href="#scans-over-more-than-one-object-store" title="Link to this heading">#</a></h3>
+<p>A native Parquet or CSV scan reads each of its partitions through one object store, such as one S3
+bucket, one GCS bucket, or one HDFS name node. When a scan’s files live in more than one store,
+Comet puts the files of each store into separate partitions, so such a scan can run with a different
+number of partitions than Spark would use. A native Parquet scan packs each store’s files on their
+own, so <code class="docutils literal notranslate"><span class="pre">spark.sql.files.maxPartitionNum</span></code> applies to each store’s files separately. A native CSV
+scan that would split a partition falls back to Spark when Spark planned the operators above it for
+the scan’s own partitioning, as Spark 3.4 does for a scan of one partition.</p>
+<p>Such a scan forwards the object store settings of every scheme it reads, so its files can mix
+schemes such as <code class="docutils literal notranslate"><span class="pre">s3a://</span></code>, <code class="docutils literal notranslate"><span class="pre">gs://</span></code> and <code class="docutils literal notranslate"><span class="pre">hdfs://</span></code>. It falls back to Spark when it reads one bucket
+through an S3-compliant alias and through another scheme (for example <code class="docutils literal notranslate"><span class="pre">blob://bucket</span></code> and
+<code class="docutils literal notranslate"><span class="pre">s3a://bucket</span></code>), because the alias settings would then apply to both. It also falls back when an
+alias path names no bucket (<code class="docutils literal notranslate"><span class="pre">blob:///</span></code>) next to other S3 paths, because that path’s alias settings
+would apply to every bucket. An alias scheme that is also listed in <code class="docutils literal notranslate"><span class="pre">fs.comet.libhdfs.schemes</span></code> is
+read through libhdfs, so Comet translates none of its settings and such a path does not cause
+either fallback. ABFS is the exception to mixing stores: Comet resolves the credentials of one
+container and account per scan, so a scan whose files span more than one ABFS container or account,
+or that mixes <code class="docutils literal notranslate"><span class="pre">abfs</span></code>/<code class="docutils literal notranslate"><span class="pre">abfss</span></code> paths with other schemes, falls back to Spark (see <a class="reference internal" href="#azure">Azure</a>). A
+native Parquet scan of a bucketed table also falls back when its files span more than one store,
+because each table bucket is read as one partition.</p>
+</section>
<section id="hdfs">
<h3>HDFS<a class="headerlink" href="#hdfs" title="Link to this heading">#</a></h3>
<p>The Apache DataFusion Comet Rust-based reader seamlessly scans files from remote HDFS for <a class="reference internal" href="#supported-spark-data-sources">supported formats</a></p>
@@ -1212,11 +1234,12 @@ above. The recognized vendor-style properties and their <code class="docutils li
</table>
</div>
<p>Unrecognized <code class="docutils literal notranslate"><span class="pre">fs.<s>.<authority>.*</span></code> properties are ignored.</p>
-<p>A URL with no authority, such as the triple-slash <code class="docutils literal notranslate"><span class="pre">blob:///bucket/key</span></code> form, reports its authority
-as the literal string <code class="docutils literal notranslate"><span class="pre">default</span></code>. For that case, list the keys under <code class="docutils literal notranslate"><span class="pre">fs.<s>.default.<property></span></code>.
-Comet resolves <code class="docutils literal notranslate"><span class="pre">default</span></code> to the bucket taken from the URL path (<code class="docutils literal notranslate"><span class="pre">bucket</span></code> here) and applies the
-translated settings at that bucket’s scope, exactly as if they had been written under
-<code class="docutils literal notranslate"><span class="pre">fs.<s>.bucket.<property></span></code>.</p>
+<p>Keys under the literal authority <code class="docutils literal notranslate"><span class="pre">default</span></code>, <code class="docutils literal notranslate"><span class="pre">fs.<s>.default.<property></span></code>, apply to every bucket of
+that scheme that has no <code class="docutils literal notranslate"><span class="pre">fs.<s>.<bucket>.<property></span></code> key of its own, whether the URL names the
+bucket as its authority (<code class="docutils literal notranslate"><span class="pre">blob://bucket/key</span></code>) or in its path. A URL with no authority, such as the
+triple-slash <code class="docutils literal notranslate"><span class="pre">blob:///bucket/key</span></code> form, reports its authority as <code class="docutils literal notranslate"><span class="pre">default</span></code>, so these keys are the
+way to configure it. Comet applies them at the scope of the bucket the URL names (<code class="docutils literal notranslate"><span class="pre">bucket</span></code> here),
+exactly as if they had been written under <code class="docutils literal notranslate"><span class="pre">fs.<s>.bucket.<property></span></code>.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">fs.<s>.<authority>.endpoint</span></code> is set, Comet defaults path-style access to enabled for that
bucket, since most non-AWS S3-compatible services require it. This is only a default: set
<code class="docutils literal notranslate"><span class="pre">fs.s3a.bucket.<bucket>.path.style.access</span></code> (or the equivalent per-scheme key) explicitly to
@@ -1226,9 +1249,9 @@ described in <a class="reference internal" href="#additional-s3-configuration-op
credential providers and options documented above also apply to alias-scheme URLs. The same
translation feeds the native Iceberg scan; see
<a class="reference internal" href="iceberg.html#object-store-configuration-s3"><span class="std std-ref">Object store configuration (S3)</span></a> in the Iceberg guide.</p>
-<p>A native Parquet scan whose alias-scheme paths span more than one bucket falls back to Spark. Alias
-schemes apply to native scans only: a native Iceberg write to an alias-scheme location falls back to
-iceberg-java.</p>
+<p>A native Parquet or CSV scan whose alias-scheme paths span more than one bucket resolves the alias
+settings for each bucket. Alias schemes apply to native scans only: a native Iceberg write to an
+alias-scheme location falls back to iceberg-java.</p>
</section>
<section id="examples">
<h3>Examples<a class="headerlink" href="#examples" title="Link to this heading">#</a></h3>
@@ -1472,6 +1495,7 @@ The Hadoop ABFS configuration you already have (the <code class="docutils litera
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#supported-storages">Supported Storages</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#scans-over-more-than-one-object-store">Scans over more than one object store</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#hdfs">HDFS</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#building-comet-with-hdfs-support">Building Comet with HDFS support</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#local-hdfs-development">Local HDFS development</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org