You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/20 17:37:46 UTC
(datafusion-comet) branch asf-site updated (8aeeff73db -> 9805346d51)
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
discard 8aeeff73db Publish built docs triggered by 2b35fb8eec0fccff56f705073de572b34f0fe3dc
new 9805346d51 Publish built docs triggered by 05a3083bd489e213e0119c4a46ac41d73d613d3b
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (8aeeff73db)
\
N -- N -- N refs/heads/asf-site (9805346d51)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
_sources/user-guide/latest/metrics.md.txt | 40 +++++++++++++++++++++----------
searchindex.js | 2 +-
user-guide/latest/metrics.html | 37 ++++++++++++++++++++++++----
3 files changed, 60 insertions(+), 19 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: Publish built docs triggered by 05a3083bd489e213e0119c4a46ac41d73d613d3b
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 9805346d51424f2d35a4fa353d4d791c82d86d5c
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Thu Aug 20 17:37:37 2026 +0000
Publish built docs triggered by 05a3083bd489e213e0119c4a46ac41d73d613d3b
---
.../optimizing_expressions.md.txt | 2 +-
_sources/user-guide/latest/metrics.md.txt | 40 +++++++++++++++-------
contributor-guide/optimizing_expressions.html | 2 +-
searchindex.js | 2 +-
user-guide/latest/metrics.html | 37 +++++++++++++++++---
5 files changed, 62 insertions(+), 21 deletions(-)
diff --git a/_sources/contributor-guide/optimizing_expressions.md.txt b/_sources/contributor-guide/optimizing_expressions.md.txt
index ecdb8a7263..4b11dd0487 100644
--- a/_sources/contributor-guide/optimizing_expressions.md.txt
+++ b/_sources/contributor-guide/optimizing_expressions.md.txt
@@ -149,7 +149,7 @@ for the lightest one that fits.
| **Preallocate builders to known size** | Repeated buffer growth/reallocation | `spark_unhex`: preallocate `BinaryBuilder` to the known output length |
| **Compile-time lookup tables** | Per-element range matches / branching | `spark_unhex`: 256-entry hex table instead of per-digit range match |
| **Cache compiled regex** (thread-local, keyed by the constant arg) | `Regex::new()` per row | `parse_url` QUERY-with-key (50x): the key is constant across a batch |
-| **Read from the offset buffer directly** | `list_array.value(i)` allocating a sliced `ArrayRef` per row | `spark_size` Map path and scalar path: `value_length` / offsets instead of materializing a sliced array |
+| **Read from the offset buffer directly** | `list_array.value(i)` allocating a sliced `ArrayRef` per row | `spark_size` scalar path: `value_length` instead of materializing a sliced array; Map array path (~6-13x): `MapArray::offsets().windows(2)` diff, `length` kernel doesn't support Map |
| **Reuse Arrow `length` kernel + cheap null rewrite** | Per-row `value_length` / builder loop, or `zip` via `MutableArrayData` | `spark_size` (~12x on no-null List): `length` for List/LargeList/FixedSizeList, then `into_parts` + `set_indices` to patch null slots to `-1` |
| **Typed scans over flat values buffers + hash probe** | A per-element Arrow `eq`/compute kernel that allocates per call | `spark_arrays_overlap` (up to 18x): scan buffers directly, hash probe for large lists |
| **ASCII / byte-offset fast path** | `chars().count()` and per-char UTF-8 decoding | `substring` (up to 10x), `spark_lpad` (2x): slice by byte offset when input is ASCII |
diff --git a/_sources/user-guide/latest/metrics.md.txt b/_sources/user-guide/latest/metrics.md.txt
index 2e36aba291..a8b2d6bdb8 100644
--- a/_sources/user-guide/latest/metrics.md.txt
+++ b/_sources/user-guide/latest/metrics.md.txt
@@ -33,12 +33,22 @@ Comet operators report the following metrics in the Spark SQL UI.
Comet adds some additional metrics:
-| Metric | Description |
-| ------------------------------- | ------------------------------------------------------------- |
-| `native shuffle time` | Total time in native code excluding any child operators. |
-| `repartition time` | Time to repartition batches. |
-| `memory pool time` | Time interacting with memory pool. |
-| `encoding and compression time` | Time to encode batches in IPC format and compress using ZSTD. |
+| Metric | Description |
+| ------------------------------- | --------------------------------------------------------------------------- |
+| `native shuffle time` | Total time in native code excluding any child operators. |
+| `repartition time` | Time to repartition batches. |
+| `partition interleaving time` | Time to interleave partitioned batches before writing them. |
+| `memory pool time` | Time interacting with memory pool. |
+| `encoding and compression time` | Time to encode batches in IPC format and compress using ZSTD. |
+| `disk spilled bytes` | Actual bytes written to native shuffle spill files on disk. |
+| `memory spilled bytes` | Uncompressed Arrow backing-buffer and partition-index data before spilling. |
+
+Disk and memory spilled bytes measure different representations of the same native shuffle spill.
+Disk spill bytes count the actual bytes written to disk: compressed when shuffle compression is
+enabled and uncompressed when `spark.shuffle.compress=false`. Memory spill bytes follow Spark's
+`memoryBytesSpilled` semantics and include uncompressed in-memory Arrow backing-buffer and
+partition-index data rather than their on-disk size. These values also appear in Spark's task
+metrics and Spark UI as `diskBytesSpilled` and `memoryBytesSpilled`, respectively.
## Native Metrics
@@ -57,13 +67,17 @@ Here is a guide to some of the native metrics.
### ShuffleWriterExec
-| Metric | Description |
-| ----------------- | ------------------------------------------------------------- |
-| `elapsed_compute` | Total time excluding any child operators. |
-| `repart_time` | Time to repartition batches. |
-| `ipc_time` | Time to encode batches in IPC format and compress using ZSTD. |
-| `mempool_time` | Time interacting with memory pool. |
-| `write_time` | Time spent writing bytes to disk. |
+| Metric | Description |
+| ---------------------- | --------------------------------------------------------------------- |
+| `elapsed_compute` | Total time excluding any child operators. |
+| `repart_time` | Time to repartition batches. |
+| `interleave_time` | Time to interleave partitioned batches before writing them. |
+| `ipc_time` | Time to encode batches in IPC format and compress using ZSTD. |
+| `mempool_time` | Time interacting with memory pool. |
+| `write_time` | Time spent writing bytes to disk. |
+| `spill_count` | Number of native shuffle spills. |
+| `spilled_bytes` | Actual bytes written to native shuffle spill files on disk. |
+| `memory_spilled_bytes` | Uncompressed Arrow backing-buffer and partition-index memory spilled. |
## Task-Level Input Metrics on Spark 4.1+
diff --git a/contributor-guide/optimizing_expressions.html b/contributor-guide/optimizing_expressions.html
index 3bca1b97d7..70836e10d5 100644
--- a/contributor-guide/optimizing_expressions.html
+++ b/contributor-guide/optimizing_expressions.html
@@ -701,7 +701,7 @@ for the lightest one that fits.</p>
</tr>
<tr class="row-odd"><td><p><strong>Read from the offset buffer directly</strong></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">list_array.value(i)</span></code> allocating a sliced <code class="docutils literal notranslate"><span class="pre">ArrayRef</span></code> per row</p></td>
-<td><p><code class="docutils literal notranslate"><span class="pre">spark_size</span></code> Map path and scalar path: <code class="docutils literal notranslate"><span class="pre">value_length</span></code> / offsets instead of materializing a sliced array</p></td>
+<td><p><code class="docutils literal notranslate"><span class="pre">spark_size</span></code> scalar path: <code class="docutils literal notranslate"><span class="pre">value_length</span></code> instead of materializing a sliced array; Map array path (~6-13x): <code class="docutils literal notranslate"><span class="pre">MapArray::offsets().windows(2)</span></code> diff, <code class="docutils literal notranslate"><span class="pre">length</span></code> kernel doesn’t support Map</p></td>
</tr>
<tr class="row-even"><td><p><strong>Reuse Arrow <code class="docutils literal notranslate"><span class="pre">length</span></code> kernel + cheap null rewrite</strong></p></td>
<td><p>Per-row <code class="docutils literal notranslate"><span class="pre">value_length</span></code> / builder loop, or <code class="docutils literal notranslate"><span class="pre">zip</span></code> via <code class="docutils literal notranslate"><span class="pre">MutableArrayData</span></code></p></td>
diff --git a/searchindex.js b/searchindex.js
index 76442c101b..6d67fb9297 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[55, "id1"]], "%": [[53, "id1"]], "&": [[43, "id1"]], "*": [[53, "id2"]], "+": [[53, "id3"]], "-": [[53, "id4"]], "/": [[53, "id5"]], "1. Format Your Code": [[40, "format-your-code"]], "1. Install Comet": [[61, "install-comet"], [71, "install-comet"]], "1. Native Operators (nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and Verify": [[40, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[61, "clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/metrics.html b/user-guide/latest/metrics.html
index dd320b7216..9bc9654550 100644
--- a/user-guide/latest/metrics.html
+++ b/user-guide/latest/metrics.html
@@ -811,15 +811,30 @@ under the License.
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">repartition</span> <span class="pre">time</span></code></p></td>
<td><p>Time to repartition batches.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">memory</span> <span class="pre">pool</span> <span class="pre">time</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">partition</span> <span class="pre">interleaving</span> <span class="pre">time</span></code></p></td>
+<td><p>Time to interleave partitioned batches before writing them.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">memory</span> <span class="pre">pool</span> <span class="pre">time</span></code></p></td>
<td><p>Time interacting with memory pool.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">encoding</span> <span class="pre">and</span> <span class="pre">compression</span> <span class="pre">time</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">encoding</span> <span class="pre">and</span> <span class="pre">compression</span> <span class="pre">time</span></code></p></td>
<td><p>Time to encode batches in IPC format and compress using ZSTD.</p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">disk</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
+<td><p>Actual bytes written to native shuffle spill files on disk.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">memory</span> <span class="pre">spilled</span> <span class="pre">bytes</span></code></p></td>
+<td><p>Uncompressed Arrow backing-buffer and partition-index data before spilling.</p></td>
+</tr>
</tbody>
</table>
</div>
+<p>Disk and memory spilled bytes measure different representations of the same native shuffle spill.
+Disk spill bytes count the actual bytes written to disk: compressed when shuffle compression is
+enabled and uncompressed when <code class="docutils literal notranslate"><span class="pre">spark.shuffle.compress=false</span></code>. Memory spill bytes follow Spark’s
+<code class="docutils literal notranslate"><span class="pre">memoryBytesSpilled</span></code> semantics and include uncompressed in-memory Arrow backing-buffer and
+partition-index data rather than their on-disk size. These values also appear in Spark’s task
+metrics and Spark UI as <code class="docutils literal notranslate"><span class="pre">diskBytesSpilled</span></code> and <code class="docutils literal notranslate"><span class="pre">memoryBytesSpilled</span></code>, respectively.</p>
</section>
</section>
<section id="native-metrics">
@@ -864,15 +879,27 @@ logged for each native plan (and there is one plan per task, so this is very ver
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">repart_time</span></code></p></td>
<td><p>Time to repartition batches.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">ipc_time</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">interleave_time</span></code></p></td>
+<td><p>Time to interleave partitioned batches before writing them.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">ipc_time</span></code></p></td>
<td><p>Time to encode batches in IPC format and compress using ZSTD.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">mempool_time</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">mempool_time</span></code></p></td>
<td><p>Time interacting with memory pool.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">write_time</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">write_time</span></code></p></td>
<td><p>Time spent writing bytes to disk.</p></td>
</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">spill_count</span></code></p></td>
+<td><p>Number of native shuffle spills.</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">spilled_bytes</span></code></p></td>
+<td><p>Actual bytes written to native shuffle spill files on disk.</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">memory_spilled_bytes</span></code></p></td>
+<td><p>Uncompressed Arrow backing-buffer and partition-index memory spilled.</p></td>
+</tr>
</tbody>
</table>
</div>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org