You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/07 21:54:38 UTC
(datafusion-comet) 01/01: Publish built docs triggered by e76c35f1e782b78525d447ba09d6f580f9bf6dec
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 11a606b2438f94cd7b690a5d8cfe0b842e4118dd
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Oct 7 21:54:30 2026 +0000
Publish built docs triggered by e76c35f1e782b78525d447ba09d6f580f9bf6dec
---
_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg | 2 +-
_sources/contributor-guide/adding_a_new_operator.md.txt | 5 ++++-
_sources/user-guide/latest/datasources.md.txt | 10 +++++++++-
contributor-guide/adding_a_new_operator.html | 5 ++++-
searchindex.js | 2 +-
user-guide/latest/datasources.html | 9 ++++++++-
6 files changed, 27 insertions(+), 6 deletions(-)
diff --git a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
index b4db196818..f1b72d43ea 100644
--- a/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
+++ b/_images/mermaid-f5b40459e5fe9b892b845c9198fe1c9420e05d14.svg
@@ -1 +1 @@
-<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
+<svg id="my-svg" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" class="flowchart" style="max-width: 511.73px; background-color: transparent;" viewBox="0 0 511.7299499511719 966" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#my-svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#my-svg [...]
\ No newline at end of file
diff --git a/_sources/contributor-guide/adding_a_new_operator.md.txt b/_sources/contributor-guide/adding_a_new_operator.md.txt
index 09cac3a1c7..85946f44fd 100644
--- a/_sources/contributor-guide/adding_a_new_operator.md.txt
+++ b/_sources/contributor-guide/adding_a_new_operator.md.txt
@@ -148,7 +148,10 @@ own operators itself. Spark computes a typed operation's rows one at a time, as
the conversion fills a whole Arrow batch first. So the rule leaves the output unconverted where a
limit, a `mapPartitions` function, or code reading `Dataset.rdd` could stop reading it early, unless
an operator that reads all of its input first, such as an exchange, a sort, or a hash aggregate,
-sits in between. Fusing the deserializer, the `Invoke` that calls the user function, and the
+sits in between. For the same reason, it leaves the output unconverted when the plan uses
+`input_file_name()`, `input_file_block_start()` or `input_file_block_length()`: filling the batch
+moves the scan's reader past the file that these report when Spark evaluates them above the
+conversion. Fusing the deserializer, the `Invoke` that calls the user function, and the
serializer of `Dataset.map` into one projection in the JVM codegen dispatcher was tried in
[#5714](https://github.com/apache/datafusion-comet/pull/5714) and dropped. The dispatcher only calls
into Spark's own classes, and the conversion gets nearly the same speedup for `map` while also
diff --git a/_sources/user-guide/latest/datasources.md.txt b/_sources/user-guide/latest/datasources.md.txt
index 2aed4e6acb..558e03b20d 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -87,6 +87,11 @@ The same types apply to the output of typed `Dataset` operations, such as `map`,
converts when `spark.comet.convert.typedDataset.enabled=true`. A column of any other type keeps
the operators above the typed operation on Spark.
+Comet does not convert a source, or the output of a typed `Dataset` operation, when the query uses
+`input_file_name()`, `input_file_block_start()` or `input_file_block_length()`, so the operators
+above it run in Spark. These functions report the file that Spark's reader is on when Spark evaluates
+them, and the conversion and the Comet operators above it would read ahead of that.
+
## Data Catalogs
### Apache Iceberg
@@ -289,7 +294,10 @@ URLs.
This is opt-in and disabled by default. Enable it by listing the schemes to treat as S3-compliant
aliases in `spark.hadoop.fs.comet.s3Compliant.schemes` (Hadoop key
`fs.comet.s3Compliant.schemes`), a comma-separated, case-insensitive list. This mirrors the
-existing `fs.comet.libhdfs.schemes` config.
+existing `fs.comet.libhdfs.schemes` config. The list entries are case-insensitive, but an Iceberg
+table location must write the alias in lowercase (`blob://`, not `BLOB://`): the native Iceberg
+reader opens each recorded location as written, and its S3 backend accepts only a lowercase
+scheme prefix, so Comet declines such a location up front and leaves that scan to Spark.
```shell
--conf spark.hadoop.fs.comet.s3Compliant.schemes=blob
diff --git a/contributor-guide/adding_a_new_operator.html b/contributor-guide/adding_a_new_operator.html
index 03229108fc..e89edcdc07 100644
--- a/contributor-guide/adding_a_new_operator.html
+++ b/contributor-guide/adding_a_new_operator.html
@@ -670,7 +670,10 @@ own operators itself. Spark computes a typed operation’s rows one at a time, a
the conversion fills a whole Arrow batch first. So the rule leaves the output unconverted where a
limit, a <code class="docutils literal notranslate"><span class="pre">mapPartitions</span></code> function, or code reading <code class="docutils literal notranslate"><span class="pre">Dataset.rdd</span></code> could stop reading it early, unless
an operator that reads all of its input first, such as an exchange, a sort, or a hash aggregate,
-sits in between. Fusing the deserializer, the <code class="docutils literal notranslate"><span class="pre">Invoke</span></code> that calls the user function, and the
+sits in between. For the same reason, it leaves the output unconverted when the plan uses
+<code class="docutils literal notranslate"><span class="pre">input_file_name()</span></code>, <code class="docutils literal notranslate"><span class="pre">input_file_block_start()</span></code> or <code class="docutils literal notranslate"><span class="pre">input_file_block_length()</span></code>: filling the batch
+moves the scan’s reader past the file that these report when Spark evaluates them above the
+conversion. Fusing the deserializer, the <code class="docutils literal notranslate"><span class="pre">Invoke</span></code> that calls the user function, and the
serializer of <code class="docutils literal notranslate"><span class="pre">Dataset.map</span></code> into one projection in the JVM codegen dispatcher was tried in
<a class="reference external" href="https://github.com/apache/datafusion-comet/pull/5714">#5714</a> and dropped. The dispatcher only calls
into Spark’s own classes, and the conversion gets nearly the same speedup for <code class="docutils literal notranslate"><span class="pre">map</span></code> while also
diff --git a/searchindex.js b/searchindex.js
index 3b8ba207f3..2540d88f7a 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[58, "id1"]], "%": [[56, "id1"]], "&": [[46, "id1"]], "*": [[56, "id2"]], "+": [[56, "id3"]], "-": [[56, "id4"]], "/": [[56, "id5"]], "1. Format Your Code": [[43, "format-your-code"]], "1. Install Comet": [[65, "install-comet"], [77, "install-comet"]], "1. Native Operators (nativeExecs map)": [[29, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[43, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[65, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[58, "id1"]], "%": [[56, "id1"]], "&": [[46, "id1"]], "*": [[56, "id2"]], "+": [[56, "id3"]], "-": [[56, "id4"]], "/": [[56, "id5"]], "1. Format Your Code": [[43, "format-your-code"]], "1. Install Comet": [[65, "install-comet"], [77, "install-comet"]], "1. Native Operators (nativeExecs map)": [[29, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[43, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[65, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/datasources.html b/user-guide/latest/datasources.html
index db5aea7b00..6cb32bb8b4 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -907,6 +907,10 @@ still produces the RDD rows; conversion lets eligible downstream operators execu
<p>The same types apply to the output of typed <code class="docutils literal notranslate"><span class="pre">Dataset</span></code> operations, such as <code class="docutils literal notranslate"><span class="pre">map</span></code>, which Comet
converts when <code class="docutils literal notranslate"><span class="pre">spark.comet.convert.typedDataset.enabled=true</span></code>. A column of any other type keeps
the operators above the typed operation on Spark.</p>
+<p>Comet does not convert a source, or the output of a typed <code class="docutils literal notranslate"><span class="pre">Dataset</span></code> operation, when the query uses
+<code class="docutils literal notranslate"><span class="pre">input_file_name()</span></code>, <code class="docutils literal notranslate"><span class="pre">input_file_block_start()</span></code> or <code class="docutils literal notranslate"><span class="pre">input_file_block_length()</span></code>, so the operators
+above it run in Spark. These functions report the file that Spark’s reader is on when Spark evaluates
+them, and the conversion and the Comet operators above it would read ahead of that.</p>
</section>
</section>
<section id="data-catalogs">
@@ -1146,7 +1150,10 @@ URLs.</p>
<p>This is opt-in and disabled by default. Enable it by listing the schemes to treat as S3-compliant
aliases in <code class="docutils literal notranslate"><span class="pre">spark.hadoop.fs.comet.s3Compliant.schemes</span></code> (Hadoop key
<code class="docutils literal notranslate"><span class="pre">fs.comet.s3Compliant.schemes</span></code>), a comma-separated, case-insensitive list. This mirrors the
-existing <code class="docutils literal notranslate"><span class="pre">fs.comet.libhdfs.schemes</span></code> config.</p>
+existing <code class="docutils literal notranslate"><span class="pre">fs.comet.libhdfs.schemes</span></code> config. The list entries are case-insensitive, but an Iceberg
+table location must write the alias in lowercase (<code class="docutils literal notranslate"><span class="pre">blob://</span></code>, not <code class="docutils literal notranslate"><span class="pre">BLOB://</span></code>): the native Iceberg
+reader opens each recorded location as written, and its S3 backend accepts only a lowercase
+scheme prefix, so Comet declines such a location up front and leaves that scan to Spark.</p>
<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>--conf<span class="w"> </span>spark.hadoop.fs.comet.s3Compliant.schemes<span class="o">=</span>blob
</pre></div>
</div>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org