You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/04 22:55:58 UTC

(datafusion-comet) branch asf-site updated: Publish built docs triggered by d1e14804f48535b30f99114b2531d057ed133ae6

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 09f439754 Publish built docs triggered by d1e14804f48535b30f99114b2531d057ed133ae6
09f439754 is described below

commit 09f439754e6c376682a580184754027f75f35fc1
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon May 4 22:55:53 2026 +0000

    Publish built docs triggered by d1e14804f48535b30f99114b2531d057ed133ae6
---
 _sources/user-guide/latest/metrics.md.txt | 23 +++++++++++++++++++++++
 searchindex.js                            |  2 +-
 user-guide/latest/metrics.html            | 20 ++++++++++++++++++++
 3 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/_sources/user-guide/latest/metrics.md.txt b/_sources/user-guide/latest/metrics.md.txt
index 509d0ae8c..0579b3081 100644
--- a/_sources/user-guide/latest/metrics.md.txt
+++ b/_sources/user-guide/latest/metrics.md.txt
@@ -64,3 +64,26 @@ Here is a guide to some of the native metrics.
 | `ipc_time`        | Time to encode batches in IPC format and compress using ZSTD. |
 | `mempool_time`    | Time interacting with memory pool.                            |
 | `write_time`      | Time spent writing bytes to disk.                             |
+
+## Task-Level Input Metrics on Spark 4.1+
+
+Comet's native scans set `inputMetrics.bytesRead` to the actual file IO performed by the
+DataFusion parquet reader (`bytes_scanned`). This is the truthful number you would see at the
+filesystem layer.
+
+Spark 4.1 changed its own parquet reader to pre-open the `SeekableInputStream` and read the file
+footer outside the `FileScanRDD.compute()` thread. Spark's `inputMetrics.bytesRead` is updated
+from a Hadoop FileSystem thread-local byte counter that only captures reads on the
+`compute()` thread, so reads serviced by the pre-opened stream's internal buffer go uncounted.
+The under-count is largest when the file fits in the pre-fetched buffer (tiny files, unit test
+sizes) and shrinks as files grow large enough that subsequent row-group reads cross the buffer
+and trigger fresh FS reads on the `compute()` thread.
+
+This is purely an observability difference: `inputMetrics.bytesRead` is reported to listeners
+and the Spark UI but is not consumed by the planner, the optimizer, or AQE, so the discrepancy
+does not affect query plans, partitioning, or correctness. Records read (`recordsRead`) is
+unaffected and remains exactly equal between Comet and Spark.
+
+If you compare Comet's `bytesRead` against vanilla Spark's on Spark 4.1+ (via the Spark UI or
+the REST API), expect Comet's number to be substantially larger for small files, and closer to
+Spark's for large files. Comet's value reflects what the storage layer actually delivered.
diff --git a/searchindex.js b/searchindex.js
index 857c11524..687282f2f 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[14, "format-your-code"]], "1. Install Comet": [[16, "install-comet"], [24, "install-comet"]], "1. Native Operators (nativeExecs map)": [[4, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[14, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[16, "clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[24, "clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[4, "sink-operators-sinks-m [...]
\ No newline at end of file
diff --git a/user-guide/latest/metrics.html b/user-guide/latest/metrics.html
index 7c1e5c30c..aec58d676 100644
--- a/user-guide/latest/metrics.html
+++ b/user-guide/latest/metrics.html
@@ -563,6 +563,26 @@ logged for each native plan (and there is one plan per task, so this is very ver
 </div>
 </section>
 </section>
+<section id="task-level-input-metrics-on-spark-4-1">
+<h2>Task-Level Input Metrics on Spark 4.1+<a class="headerlink" href="#task-level-input-metrics-on-spark-4-1" title="Link to this heading">#</a></h2>
+<p>Comet’s native scans set <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> to the actual file IO performed by the
+DataFusion parquet reader (<code class="docutils literal notranslate"><span class="pre">bytes_scanned</span></code>). This is the truthful number you would see at the
+filesystem layer.</p>
+<p>Spark 4.1 changed its own parquet reader to pre-open the <code class="docutils literal notranslate"><span class="pre">SeekableInputStream</span></code> and read the file
+footer outside the <code class="docutils literal notranslate"><span class="pre">FileScanRDD.compute()</span></code> thread. Spark’s <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> is updated
+from a Hadoop FileSystem thread-local byte counter that only captures reads on the
+<code class="docutils literal notranslate"><span class="pre">compute()</span></code> thread, so reads serviced by the pre-opened stream’s internal buffer go uncounted.
+The under-count is largest when the file fits in the pre-fetched buffer (tiny files, unit test
+sizes) and shrinks as files grow large enough that subsequent row-group reads cross the buffer
+and trigger fresh FS reads on the <code class="docutils literal notranslate"><span class="pre">compute()</span></code> thread.</p>
+<p>This is purely an observability difference: <code class="docutils literal notranslate"><span class="pre">inputMetrics.bytesRead</span></code> is reported to listeners
+and the Spark UI but is not consumed by the planner, the optimizer, or AQE, so the discrepancy
+does not affect query plans, partitioning, or correctness. Records read (<code class="docutils literal notranslate"><span class="pre">recordsRead</span></code>) is
+unaffected and remains exactly equal between Comet and Spark.</p>
+<p>If you compare Comet’s <code class="docutils literal notranslate"><span class="pre">bytesRead</span></code> against vanilla Spark’s on Spark 4.1+ (via the Spark UI or
+the REST API), expect Comet’s number to be substantially larger for small files, and closer to
+Spark’s for large files. Comet’s value reflects what the storage layer actually delivered.</p>
+</section>
 </section>
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org