You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/02 22:33:44 UTC
(datafusion-site) branch asf-staging updated: Commit build products
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new c17e900 Commit build products
c17e900 is described below
commit c17e9001708eb53a64bd9bd32876bba810f02d8a
Author: Build Pelican (action) <pr...@infra.apache.org>
AuthorDate: Fri Oct 2 22:33:39 2026 +0000
Commit build products
---
blog/2026/10/01/datafusion-comet-1.1.0/index.html | 7 +++++--
blog/feeds/all-en.atom.xml | 7 +++++--
blog/feeds/blog.atom.xml | 7 +++++--
blog/feeds/pmc.atom.xml | 7 +++++--
4 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/blog/2026/10/01/datafusion-comet-1.1.0/index.html b/blog/2026/10/01/datafusion-comet-1.1.0/index.html
index 53301ff..28fbc2c 100644
--- a/blog/2026/10/01/datafusion-comet-1.1.0/index.html
+++ b/blog/2026/10/01/datafusion-comet-1.1.0/index.html
@@ -252,7 +252,10 @@ grants an operator less memory than it asked for (a partial grant), the operator
<li><strong>Less log noise when spilling.</strong> 1.0.0 logged a warning and a memory dump for every partial grant, so a
spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
deadlock the task.</li>
-<li><strong>Metrics.</strong> Native memory usage, spill, and aggregate memory metrics now appear in Spark's task metrics.</li>
+<li><strong>Metrics.</strong> Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+ task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+ 1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+ in the SQL tab.</li>
</ul>
<p><code>spark.comet.exec.memoryPool.fraction</code> is deprecated. It was meant to leave room for untracked memory, but Spark
hands out the whole pool anyway, so it never did. Size <code>spark.executor.memoryOverhead</code> instead.</p>
@@ -265,7 +268,7 @@ disables itself with a warning otherwise.</p>
<p>Contributors can find more detail in the new <a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html">memory management guide</a>.</p>
<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> for driving this work, <a href="https://github.com/peterxcli">@peterxcli</a> for the memory pool lifecycle and shuffle spill
accounting fixes, <a href="https://github.com/ywskycn">@ywskycn</a> for reporting native memory usage to Spark, <a href="https://github.com/1fanwang">@1fanwang</a> for the Arrow import leak
-fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill and memory metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
+fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
<a href="https://github.com/comphead">@comphead</a>, and <a href="https://github.com/mbutrovich">@mbutrovich</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5934">#5934</a>, <a href="https://github.com/apache/datafusion-comet/pull/6162">#6162</a>, <a href="https://github.com/apache/datafusion-comet/pull/6048">#6048</a>, <a href="https://github.com/apache/datafusion-comet/pull/6128">#6128</a>, <a href="https://github.com/apache/datafusion-comet/pull/6205">#6205< [...]
<h2 id="more-iceberg-improvements">More Iceberg Improvements<a class="headerlink" href="#more-iceberg-improvements" title="Permanent link">¶</a></h2>
<ul>
diff --git a/blog/feeds/all-en.atom.xml b/blog/feeds/all-en.atom.xml
index 8701273..8861b8b 100644
--- a/blog/feeds/all-en.atom.xml
+++ b/blog/feeds/all-en.atom.xml
@@ -171,7 +171,10 @@ grants an operator less memory than it asked for (a partial grant), the operator
<li><strong>Less log noise when spilling.</strong> 1.0.0 logged a warning and a memory dump for every partial grant, so a
spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
deadlock the task.</li>
-<li><strong>Metrics.</strong> Native memory usage, spill, and aggregate memory metrics now appear in Spark's task metrics.</li>
+<li><strong>Metrics.</strong> Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+ task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+ 1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+ in the SQL tab.</li>
</ul>
<p><code>spark.comet.exec.memoryPool.fraction</code> is deprecated. It was meant to leave room for untracked memory, but Spark
hands out the whole pool anyway, so it never did. Size <code>spark.executor.memoryOverhead</code> instead.</p>
@@ -184,7 +187,7 @@ disables itself with a warning otherwise.</p>
<p>Contributors can find more detail in the new <a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html">memory management guide</a>.</p>
<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> for driving this work, <a href="https://github.com/peterxcli">@peterxcli</a> for the memory pool lifecycle and shuffle spill
accounting fixes, <a href="https://github.com/ywskycn">@ywskycn</a> for reporting native memory usage to Spark, <a href="https://github.com/1fanwang">@1fanwang</a> for the Arrow import leak
-fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill and memory metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
+fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
<a href="https://github.com/comphead">@comphead</a>, and <a href="https://github.com/mbutrovich">@mbutrovich</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5934">#5934</a>, <a href="https://github.com/apache/datafusion-comet/pull/6162">#6162</a>, <a href="https://github.com/apache/datafusion-comet/pull/6048">#6048</a>, <a href="https://github.com/apache/datafusion-comet/pull/6128">#6128</a> [...]
<h2 id="more-iceberg-improvements">More Iceberg Improvements<a class="headerlink" href="#more-iceberg-improvements" title="Permanent link">¶</a></h2>
<ul>
diff --git a/blog/feeds/blog.atom.xml b/blog/feeds/blog.atom.xml
index 7f542b4..604a2e0 100644
--- a/blog/feeds/blog.atom.xml
+++ b/blog/feeds/blog.atom.xml
@@ -171,7 +171,10 @@ grants an operator less memory than it asked for (a partial grant), the operator
<li><strong>Less log noise when spilling.</strong> 1.0.0 logged a warning and a memory dump for every partial grant, so a
spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
deadlock the task.</li>
-<li><strong>Metrics.</strong> Native memory usage, spill, and aggregate memory metrics now appear in Spark's task metrics.</li>
+<li><strong>Metrics.</strong> Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+ task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+ 1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+ in the SQL tab.</li>
</ul>
<p><code>spark.comet.exec.memoryPool.fraction</code> is deprecated. It was meant to leave room for untracked memory, but Spark
hands out the whole pool anyway, so it never did. Size <code>spark.executor.memoryOverhead</code> instead.</p>
@@ -184,7 +187,7 @@ disables itself with a warning otherwise.</p>
<p>Contributors can find more detail in the new <a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html">memory management guide</a>.</p>
<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> for driving this work, <a href="https://github.com/peterxcli">@peterxcli</a> for the memory pool lifecycle and shuffle spill
accounting fixes, <a href="https://github.com/ywskycn">@ywskycn</a> for reporting native memory usage to Spark, <a href="https://github.com/1fanwang">@1fanwang</a> for the Arrow import leak
-fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill and memory metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
+fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
<a href="https://github.com/comphead">@comphead</a>, and <a href="https://github.com/mbutrovich">@mbutrovich</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5934">#5934</a>, <a href="https://github.com/apache/datafusion-comet/pull/6162">#6162</a>, <a href="https://github.com/apache/datafusion-comet/pull/6048">#6048</a>, <a href="https://github.com/apache/datafusion-comet/pull/6128">#6128</a> [...]
<h2 id="more-iceberg-improvements">More Iceberg Improvements<a class="headerlink" href="#more-iceberg-improvements" title="Permanent link">¶</a></h2>
<ul>
diff --git a/blog/feeds/pmc.atom.xml b/blog/feeds/pmc.atom.xml
index 9be02e2..47f868e 100644
--- a/blog/feeds/pmc.atom.xml
+++ b/blog/feeds/pmc.atom.xml
@@ -171,7 +171,10 @@ grants an operator less memory than it asked for (a partial grant), the operator
<li><strong>Less log noise when spilling.</strong> 1.0.0 logged a warning and a memory dump for every partial grant, so a
spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
deadlock the task.</li>
-<li><strong>Metrics.</strong> Native memory usage, spill, and aggregate memory metrics now appear in Spark's task metrics.</li>
+<li><strong>Metrics.</strong> Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+ task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+ 1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+ in the SQL tab.</li>
</ul>
<p><code>spark.comet.exec.memoryPool.fraction</code> is deprecated. It was meant to leave room for untracked memory, but Spark
hands out the whole pool anyway, so it never did. Size <code>spark.executor.memoryOverhead</code> instead.</p>
@@ -184,7 +187,7 @@ disables itself with a warning otherwise.</p>
<p>Contributors can find more detail in the new <a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html">memory management guide</a>.</p>
<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> for driving this work, <a href="https://github.com/peterxcli">@peterxcli</a> for the memory pool lifecycle and shuffle spill
accounting fixes, <a href="https://github.com/ywskycn">@ywskycn</a> for reporting native memory usage to Spark, <a href="https://github.com/1fanwang">@1fanwang</a> for the Arrow import leak
-fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill and memory metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
+fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
<a href="https://github.com/comphead">@comphead</a>, and <a href="https://github.com/mbutrovich">@mbutrovich</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5934">#5934</a>, <a href="https://github.com/apache/datafusion-comet/pull/6162">#6162</a>, <a href="https://github.com/apache/datafusion-comet/pull/6048">#6048</a>, <a href="https://github.com/apache/datafusion-comet/pull/6128">#6128</a> [...]
<h2 id="more-iceberg-improvements">More Iceberg Improvements<a class="headerlink" href="#more-iceberg-improvements" title="Permanent link">¶</a></h2>
<ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org