You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/06 18:51:00 UTC

(datafusion-site) branch asf-site updated: Commit build products

This is an automated email from the ASF dual-hosted git repository.

github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 6ee1376  Commit build products
6ee1376 is described below

commit 6ee13766aa0e3963190e1eab25394154853f67fa
Author: Build Pelican (action) <pr...@infra.apache.org>
AuthorDate: Tue Oct 6 18:50:55 2026 +0000

    Commit build products
---
 .../2026/10/01/datafusion-comet-1.1.0/index.html   | 612 +++++++++++++++++++++
 output/author/pmc.html                             |  32 ++
 output/category/blog.html                          |  32 ++
 output/feed.xml                                    |  24 +-
 output/feeds/all-en.atom.xml                       | 427 +++++++++++++-
 output/feeds/blog.atom.xml                         | 427 +++++++++++++-
 output/feeds/pmc.atom.xml                          | 427 +++++++++++++-
 output/feeds/pmc.rss.xml                           |  24 +-
 .../images/comet-1.1.0/comet-executor-memory.svg   |  88 +++
 output/index.html                                  |  41 ++
 10 files changed, 2129 insertions(+), 5 deletions(-)

diff --git a/output/2026/10/01/datafusion-comet-1.1.0/index.html b/output/2026/10/01/datafusion-comet-1.1.0/index.html
new file mode 100644
index 0000000..73fa983
--- /dev/null
+++ b/output/2026/10/01/datafusion-comet-1.1.0/index.html
@@ -0,0 +1,612 @@
+<!doctype html>
+<html class="no-js" lang="en" dir="ltr">
+  <head>
+    <meta charset="utf-8">
+    <meta http-equiv="x-ua-compatible" content="ie=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Apache DataFusion Comet 1.1.0 Release - Apache DataFusion Blog</title>
+<link href="/blog/favicon.ico" rel="icon" type="image/x-icon">
+<link href="/blog/favicon.ico" rel="shortcut icon" type="image/x-icon">
+<link href="/blog/css/bootstrap.min.css" rel="stylesheet">
+<link href="/blog/css/fontawesome.all.min.css" rel="stylesheet">
+<link href="/blog/css/headerlink.css" rel="stylesheet">
+<link href="/blog/highlight/default.min.css" rel="stylesheet">
+<link href="/blog/css/app.css" rel="stylesheet">
+<link href="/blog/css/dark-mode.css" rel="stylesheet">
+<script src="/blog/js/dark-mode.js"></script>
+<script src="/blog/highlight/highlight.js"></script>
+<script>hljs.highlightAll();</script>  </head>
+  <body class="d-flex flex-column h-100">
+  <main class="flex-shrink-0">
+<!-- nav bar -->
+<nav class="navbar navbar-expand-lg navbar-dark bg-dark" aria-label="Fifth navbar example">
+    <div class="container-fluid">
+        <a class="navbar-brand" href="/blog/"><img src="/blog/images/logo_original4x.png" style="height: 32px;"/> Apache DataFusion Blog</a>
+        <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarADP" aria-controls="navbarADP" aria-expanded="false" aria-label="Toggle navigation">
+            <span class="navbar-toggler-icon"></span>
+        </button>
+
+        <div class="collapse navbar-collapse" id="navbarADP">
+            <ul class="navbar-nav me-auto mb-2 mb-lg-0">
+                <li class="nav-item">
+                    <a class="nav-link" href="/blog/about.html">About</a>
+                </li>
+                <li class="nav-item">
+                    <a class="nav-link" href="/blog/feed.xml">RSS</a>
+                </li>
+            </ul>
+            <button id="dark-mode-toggle" type="button" class="dark-mode-toggle" aria-label="Toggle dark mode" aria-pressed="false" title="Toggle dark mode">
+                <span class="sun-icon" aria-hidden="true">☀</span>
+                <span class="moon-icon" aria-hidden="true">☾</span>
+            </button>
+        </div>
+    </div>
+</nav>    
+<!-- article contents -->
+<div id="contents">
+  <div class="bg-white p-4 p-md-5 rounded">
+    <div class="row justify-content-center">
+      <div class="col-12 col-md-8 main-content">
+        <h1>
+          Apache DataFusion Comet 1.1.0 Release
+        </h1>
+        <p>Posted on: Thu 01 October 2026 by pmc</p>
+
+        <aside class="toc-container d-md-none mb-2">
+          <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#native-iceberg-writes-experimental">Native Iceberg Writes (Experimental)</a><ul>
+<li><a href="#splitting-the-write-operator">Splitting the write operator</a></li>
+<li><a href="#writing-parquet-natively">Writing Parquet natively</a></li>
+<li><a href="#matching-iceberg-java">Matching iceberg-java</a></li>
+<li><a href="#failure-handling">Failure handling</a></li>
+<li><a href="#known-differences">Known differences</a></li>
+</ul>
+</li>
+<li><a href="#memory-management">Memory Management</a><ul>
+<li><a href="#where-the-memory-goes">Where the memory goes</a></li>
+<li><a href="#measuring-used-memory">Measuring used memory</a></li>
+<li><a href="#memory-pool-fixes">Memory pool fixes</a></li>
+<li><a href="#on-heap-mode">On-heap mode</a></li>
+</ul>
+</li>
+<li><a href="#more-iceberg-improvements">More Iceberg Improvements</a></li>
+<li><a href="#remote-shuffle-with-celeborn">Remote Shuffle with Celeborn</a></li>
+<li><a href="#native-parquet-writes-on-spark-40-experimental">Native Parquet Writes on Spark 4.0+ (Experimental)</a></li>
+<li><a href="#s3-credentials">S3 Credentials</a></li>
+<li><a href="#performance">Performance</a><ul>
+<li><a href="#shuffle">Shuffle</a></li>
+<li><a href="#planning-and-execution">Planning and Execution</a></li>
+<li><a href="#expressions">Expressions</a></li>
+</ul>
+</li>
+<li><a href="#expanded-coverage">Expanded Coverage</a></li>
+<li><a href="#previewing-comet-plans">Previewing Comet Plans</a></li>
+<li><a href="#correctness-fixes">Correctness Fixes</a><ul>
+<li><a href="#wrong-results">Wrong results</a></li>
+<li><a href="#errors-spark-raises-that-comet-did-not">Errors Spark raises that Comet did not</a></li>
+<li><a href="#query-failures-and-crashes">Query failures and crashes</a></li>
+<li><a href="#hangs-and-resource-use">Hangs and resource use</a></li>
+</ul>
+</li>
+<li><a href="#upgrading-to-110">Upgrading to 1.1.0</a><ul>
+<li><a href="#platform">Platform</a></li>
+<li><a href="#size-settings-now-read-correctly">Size settings now read correctly</a></li>
+<li><a href="#more-memory-before-spilling">More memory before spilling</a></li>
+<li><a href="#conditions-for-enabling-comet">Conditions for enabling Comet</a></li>
+<li><a href="#deprecated-and-removed-settings">Deprecated and removed settings</a></li>
+</ul>
+</li>
+<li><a href="#compatibility">Compatibility</a></li>
+<li><a href="#get-started-with-comet-110">Get Started with Comet 1.1.0</a></li>
+</ul>
+</div>
+        </aside>
+
+        <!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 1.1.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers roughly seven weeks of development since 1.0.0: 398 commits from 40 contributors. See the
+<a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md">change log</a> for the full list of changes.</p>
+<p>The highlights:</p>
+<ul>
+<li><strong>Native Iceberg writes</strong> (experimental): Comet can write Iceberg data files with iceberg-rust instead of
+  iceberg-java.</li>
+<li><strong>Memory management</strong>: Comet now measures the native memory its pools don't track, logs it on every executor,
+  and fixes several long-standing pool accounting bugs.</li>
+<li><strong>Native shuffle over Apache Celeborn</strong>: 1.1.0 includes the native writer and reader, but released Celeborn
+  clients currently fall back to ordinary Spark/Celeborn shuffle.</li>
+<li><strong>Native Parquet writes on Spark 4.0+</strong> (experimental), now built on Spark's own write path.</li>
+</ul>
+<h2 id="native-iceberg-writes-experimental">Native Iceberg Writes (Experimental)<a class="headerlink" href="#native-iceberg-writes-experimental" title="Permanent link">¶</a></h2>
+<p>Until now, Comet accelerated Iceberg reads but left writes to the JVM. In 1.1.0, Comet can write Iceberg data
+files natively using <a href="https://github.com/apache/iceberg-rust">iceberg-rust</a>. The feature is <strong>experimental and
+disabled by default</strong>, and it only engages under fairly strict conditions.</p>
+<h3 id="splitting-the-write-operator">Splitting the write operator<a class="headerlink" href="#splitting-the-write-operator" title="Permanent link">¶</a></h3>
+<p>Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
+and validates against the catalog. Because file writing is bundled with the metadata and commit steps, there was
+no separate piece for Comet to replace.</p>
+<p>Setting <code>spark.comet.write.iceberg.splitOperator.enabled=true</code> splits eligible Iceberg writes into two operators:</p>
+<ol>
+<li><strong><code>IcebergWrite</code></strong> writes data files on the executors and returns each task's commit message.</li>
+<li><strong><code>IcebergCommit</code></strong> collects the commit messages on the driver and performs the normal Iceberg commit, once.</li>
+</ol>
+<p>With only the split enabled, iceberg-java still writes the files; the native writer, described next, replaces
+that step. The split covers <code>INSERT INTO</code> and DataFrame <code>append</code>, static and dynamic <code>INSERT OVERWRITE</code>, and
+copy-on-write <code>DELETE</code>, <code>UPDATE</code>, and <code>MERGE</code>, on every supported Spark version. Merge-on-read writes are left
+alone. When Comet can't split a write (an unrecognized write class, CTAS on Spark 3.4, or a write that needs
+Spark's commit coordinator), it plans the write exactly as if Comet weren't there.</p>
+<h3 id="writing-parquet-natively">Writing Parquet natively<a class="headerlink" href="#writing-parquet-natively" title="Permanent link">¶</a></h3>
+<p>The native writer requires the split. Also setting <code>spark.comet.iceberg.write.enabled=true</code> hands each task's
+Parquet writing to iceberg-rust. The driver passes the native writer everything it needs: schema, partition spec,
+data location, Parquet settings, writer mode, and object-store configuration. Each task writes its files and
+returns their metadata as an in-memory Iceberg manifest.</p>
+<p>From there, iceberg-java takes over again. The JVM reads the manifest, recomputes each file's metrics from its
+Parquet footer, and builds the same commit message the JVM writer would have produced. Snapshots, manifest
+lists, commit validation, and retries all work as before.</p>
+<p>The native writer consumes Arrow batches from a Comet operator, so the query feeding the write must run in Comet
+too. Writes from a local relation, such as <code>INSERT ... VALUES</code>, also need
+<code>spark.comet.exec.localTableScan.enabled=true</code>.</p>
+<p>To see which path a write took, check the physical plan: a native write shows <code>CometIcebergWrite</code> under
+<code>IcebergCommit</code>, and a write that fell back shows <code>IcebergWrite</code>.</p>
+<h3 id="matching-iceberg-java">Matching iceberg-java<a class="headerlink" href="#matching-iceberg-java" title="Permanent link">¶</a></h3>
+<p>The native writer aims to write the same table iceberg-java would, down to the metadata. Manifest metrics drive
+pruning for every future reader, so rather than trust the native writer's numbers, Comet recomputes them with
+iceberg-java's own code. Parity tests write the same rows through both writers and compare the committed counts
+and bounds. The cost is one small footer read per file.</p>
+<p>Eligibility is an allowlist. A write goes native only if its whole configuration matches the documented set of
+supported settings. Anything else, including properties added by future Iceberg versions, falls back to
+iceberg-java and reports why in Comet's extended <code>EXPLAIN</code> output. Encryption, object-storage layout, custom
+location providers, bloom filters, and unrecognized <code>parquet.*</code> properties all fall back. For a partitioned
+table, Iceberg asks Spark to cluster and sort rows by partition value, such as <code>bucket(16, id)</code> or <code>days(ts)</code>,
+before writing. That step must also run natively, which it now can because Iceberg's system functions have
+native implementations.</p>
+<p>Comet decides all of this at planning time, including checking that every iceberg-java class it relies on is
+where it expects. An Iceberg release that moves one falls back instead of failing tasks mid-write. CI also runs
+Apache Iceberg's own Spark test suites, for Iceberg 1.8.1 through 1.11.0, with the native writer enabled.</p>
+<h3 id="failure-handling">Failure handling<a class="headerlink" href="#failure-handling" title="Permanent link">¶</a></h3>
+<p>Only successful tasks' commit messages are committed. A failed task deletes the files it wrote, as iceberg-java
+does, and a failed job commits nothing and deletes the completed tasks' files. Retries can't collide, because
+each attempt's ID is part of its file names. Anything cleanup misses is invisible to readers and is removed by
+Iceberg's normal <code>remove_orphan_files</code> maintenance.</p>
+<h3 id="known-differences">Known differences<a class="headerlink" href="#known-differences" title="Permanent link">¶</a></h3>
+<p>Files written by parquet-rs, the Rust Parquet library iceberg-rust uses, differ from those written by
+parquet-java (formerly parquet-mr), which iceberg-java uses. Enabling the feature accepts these differences.
+Most are cosmetic, such as footer metadata, <code>created_by</code>, and page encoding labels. Two are worth knowing about:</p>
+<ul>
+<li><strong>Files won't split at the same points.</strong> Both writers check file size every 1000 rows, but they estimate it
+  differently, so they can roll over to a new file at different rows.</li>
+<li><strong>High-cardinality columns keep a dictionary page.</strong> parquet-java drops dictionary encoding early when it isn't
+  saving space, while parquet-rs keeps it up to <code>write.parquet.dict-size-bytes</code>. Results are the same, but
+  selective reads of native-written files fetch more bytes
+  (<a href="https://github.com/apache/datafusion-comet/issues/6114">#6114</a>).</li>
+</ul>
+<p>The <a href="https://datafusion.apache.org/comet/user-guide/latest/iceberg-writes.html">Iceberg Writes guide</a> lists every supported setting and every difference. Please try it on a non-production
+table and tell us how it goes. Feedback from real workloads is what this feature needs before it loses the
+experimental label.</p>
+<p>Thanks to <a href="https://github.com/jordepic">@jordepic</a> for designing and implementing the split-operator plan, write detection, and the native
+writer, and to <a href="https://github.com/andygrove">@andygrove</a> for the fidelity and failure-handling work, with contributions from
+<a href="https://github.com/zhangfengcdt">@zhangfengcdt</a>, <a href="https://github.com/snmvaughan">@snmvaughan</a>, <a href="https://github.com/liupoyi-1031">@liupoyi-1031</a>, and <a href="https://github.com/0lai0">@0lai0</a>, and reviews from <a href="https://github.com/sunchao">@sunchao</a>, <a href="https://github.com/comphead">@comphead</a>,
+<a href="https://github.com/unikdahal">@unikdahal</a>, and <a href="https://github.com/mbutrovich">@mbutrovich</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/4658">#4658</a>, <a href="https://github.com/apache/datafusion-comet/pull/5298">#5298</a>, <a href="https://github.com/apache/datafusion-comet/pull/5361">#5361</a>, <a href="https://github.com/apache/datafusion-comet/pull/5663">#5663</a>, <a href="https://github.com/apache/datafusion-comet/pull/5780">#578 [...]
+<h2 id="memory-management">Memory Management<a class="headerlink" href="#memory-management" title="Permanent link">¶</a></h2>
+<p>A recurring problem for Comet users has been executors killed by the cluster manager (on Kubernetes,
+<code>ExecutorLostFailure</code> with exit code 137) even though Comet stayed within its memory pool. 1.1.0 explains why
+this happens, lets you measure it, and fixes pool bugs that made it worse.</p>
+<h3 id="where-the-memory-goes">Where the memory goes<a class="headerlink" href="#where-the-memory-goes" title="Permanent link">¶</a></h3>
+<p>Comet's native operators allocate from the Rust heap but charge their reservations against Spark's off-heap
+pool, sized by <code>spark.memory.offHeap.size</code>. Operators only reserve memory for data they deliberately hold onto:
+sort buffers, hash join build sides, aggregation state, and buffered shuffle partitions.</p>
+<p>Plenty of memory is never reserved: working memory in expression kernels, decompression buffers, Parquet reader
+state, object store buffers, JVM-side Arrow buffers, and allocator overhead. All of it has to fit in
+<code>spark.executor.memoryOverhead</code>, and until now there was no way to see how much of it there was.</p>
+<p><img alt="The executor container holds the JVM heap, the off-heap memory pool, and the memory overhead. Spark and Comet share the off-heap pool, where Comet's sorts, joins, aggregations, and shuffles reserve memory. The memory overhead holds the JVM's own overhead plus Comet's native memory that the pool does not track." class="img-fluid" src="/blog/images/comet-1.1.0/comet-executor-memory.svg" width="100%"/></p>
+<h3 id="measuring-used-memory">Measuring used memory<a class="headerlink" href="#measuring-used-memory" title="Permanent link">¶</a></h3>
+<p>Comet now wraps its native allocator in a counter that tracks every byte allocated and not yet freed. It only
+observes and never rejects an allocation. Arrow buffers the JVM imports from native code are now tracked
+separately, so they aren't counted twice.</p>
+<p>Each executor uses the counter to log its native memory usage every 10 seconds while Comet runs:</p>
+<pre><code>Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native plans, 8 memory pools)
+</code></pre>
+<p><code>reserved</code> is what the pools track, and the container already has room for it. <code>allocated</code> is everything
+Comet's native code holds. The difference is what has to fit in <code>spark.executor.memoryOverhead</code>, next to the
+JVM's own non-heap memory. To size the overhead, run a representative workload, take the largest difference from
+the log, add it to the overhead you had before Comet, and leave a margin. Setting
+<code>spark.comet.memory.logInterval=1s</code> for that run makes it less likely to miss a short peak.</p>
+<p>The executor also warns when its native memory looks larger than the container allows, and the
+<a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/memory.html">tuning guide</a> walks through sizing with examples. One gotcha: setting <code>spark.executor.memoryOverhead</code>
+<em>replaces</em> the value Spark derives from <code>spark.executor.memoryOverheadFactor</code> instead of adding to it, so on a
+large executor it can shrink the container. For large executors, raise the factor instead.</p>
+<h3 id="memory-pool-fixes">Memory pool fixes<a class="headerlink" href="#memory-pool-fixes" title="Permanent link">¶</a></h3>
+<p>Comet's off-heap memory comes from one of two pools. <code>fair_unified</code>, the default, caps each operator at an even
+share of the task's memory. <code>greedy_unified</code> gives memory to operators first come, first served. When Spark
+grants an operator less memory than it asked for (a partial grant), the operator spills.</p>
+<ul>
+<li><strong><code>fair_unified</code> limited a whole task to one operator's share.</strong> Since 0.15.0, the pool compared all of a
+  task's reservations against a single operator's share, and each new operator shrank the limit for the rest.
+  Each operator now gets its own share, so tasks with several operators can use more memory before spilling.</li>
+<li><strong>A partial grant from Spark no longer panics the task.</strong> DataFusion sometimes has to record memory that
+  already exists, such as a spilled batch being read back, and both pools panicked when Spark granted less than
+  they asked for. They now track the shortfall and repay it before returning memory to Spark.</li>
+<li><strong>Leaks on failure paths.</strong> A plan that failed during setup or teardown could leak its task's memory pool, and
+  a failed Arrow import leaked the vectors it had already imported.</li>
+<li><strong>Configuration units.</strong> 1.0.0 misread three size settings, including <code>spark.memory.offHeap.size</code> written as
+  a bare number. See <a href="#upgrading-to-110">Upgrading to 1.1.0</a> for what changes.</li>
+<li><strong>Less log noise when spilling.</strong> 1.0.0 logged a warning and a memory dump for every partial grant, so a
+  spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
+  deadlock the task.</li>
+<li><strong>Metrics.</strong> Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+  task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+  1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+  in the SQL tab.</li>
+</ul>
+<p><code>spark.comet.exec.memoryPool.fraction</code> is deprecated. It was meant to leave room for untracked memory, but Spark
+hands out the whole pool anyway, so it never did. Size <code>spark.executor.memoryOverhead</code> instead.</p>
+<h3 id="on-heap-mode">On-heap mode<a class="headerlink" href="#on-heap-mode" title="Permanent link">¶</a></h3>
+<p>On-heap mode exists so that Spark's and Iceberg's test suites can run against Comet; production runs off-heap.
+Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes that
+accounting, along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
+off-heap memory however it's enabled, including when <code>CometSparkSessionExtensions</code> is registered directly, and
+disables itself with a warning otherwise.</p>
+<p>Contributors can find more detail in the new <a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html">memory management guide</a>.</p>
+<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> for driving this work, <a href="https://github.com/peterxcli">@peterxcli</a> for the memory pool lifecycle and shuffle spill
+accounting fixes, <a href="https://github.com/ywskycn">@ywskycn</a> for reporting native memory usage to Spark, <a href="https://github.com/1fanwang">@1fanwang</a> for the Arrow import leak
+fix, and <a href="https://github.com/sunchao">@sunchao</a> for the native aggregate spill metrics, with reviews from <a href="https://github.com/sunchao">@sunchao</a>,
+<a href="https://github.com/comphead">@comphead</a>, and <a href="https://github.com/mbutrovich">@mbutrovich</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5934">#5934</a>, <a href="https://github.com/apache/datafusion-comet/pull/6162">#6162</a>, <a href="https://github.com/apache/datafusion-comet/pull/6048">#6048</a>, <a href="https://github.com/apache/datafusion-comet/pull/6128">#6128</a>, <a href="https://github.com/apache/datafusion-comet/pull/6205">#6205< [...]
+<h2 id="more-iceberg-improvements">More Iceberg Improvements<a class="headerlink" href="#more-iceberg-improvements" title="Permanent link">¶</a></h2>
+<ul>
+<li><strong>V3 deletion vectors</strong> are applied on native scans.</li>
+<li><strong>Iceberg system functions</strong> (<code>bucket</code>, <code>truncate</code>, <code>years</code>, <code>months</code>, <code>days</code>, and <code>hours</code>) run natively.</li>
+<li><strong>Scan planning metrics and scan time</strong> appear in the Spark UI for native Iceberg scans.</li>
+<li><strong>A wrong-results fix.</strong> A filter such as <code>bucket(4, id) = 2</code> combined with another predicate was pushed to
+  the native scan as <code>id = 2</code>, returning too few rows.</li>
+<li>Tables partitioned by an unknown transform can be read natively, and <code>IS NULL</code> / <code>IS NOT NULL</code> checks on list
+  and map columns no longer force a fallback.</li>
+</ul>
+<p>Thanks to <a href="https://github.com/mbutrovich">@mbutrovich</a> for deletion vector support, <a href="https://github.com/parthchandra">@parthchandra</a> for the scan metrics, <a href="https://github.com/ErikBPF">@ErikBPF</a> for the
+null-check fix, and <a href="https://github.com/andygrove">@andygrove</a> for the native system functions and residual fix, with reviews from
+<a href="https://github.com/sunchao">@sunchao</a>, <a href="https://github.com/rich7420">@rich7420</a>, <a href="https://github.com/unikdahal">@unikdahal</a>, and <a href="https://github.com/jordepic">@jordepic</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5853">#5853</a>, <a href="https://github.com/apache/datafusion-comet/pull/5638">#5638</a>, <a href="https://github.com/apache/datafusion-comet/pull/6027">#6027</a>, <a href="https://github.com/apache/datafu [...]
+<h2 id="remote-shuffle-with-celeborn">Remote Shuffle with Celeborn<a class="headerlink" href="#remote-shuffle-with-celeborn" title="Permanent link">¶</a></h2>
+<p>1.1.0 adds Comet's side of native shuffle over Apache Celeborn: map tasks push Comet's Arrow data straight to
+Celeborn, and reducers read it back natively. In 1.0.0, Celeborn users always got ordinary Spark shuffle.</p>
+<p>Native Celeborn shuffle is unavailable with released Celeborn 0.6.x and 0.7.x clients in Comet 1.1.0 because
+these clients do not pass Comet's push-completion compatibility checks. Set <code>spark.shuffle.manager</code> to
+<code>org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager</code> to use Comet alongside
+ordinary Spark/Celeborn shuffle; Comet can still accelerate other operators. These clients retain
+ordinary Spark/Celeborn shuffle even when <code>spark.comet.shuffle.mode=native</code> is requested, and the
+default <code>auto</code> mode also retains ordinary shuffle. Follow-up work to enable native shuffle with
+released clients is tracked in <a href="https://github.com/apache/datafusion-comet/issues/6523">#6523</a>.
+Native shuffle over Celeborn does not support <code>spark.io.encryption.enabled=true</code>, and Celeborn
+is not bundled with Comet. The <a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/celeborn.html">Celeborn tuning guide</a> has the full requirements.</p>
+<p>Thanks to <a href="https://github.com/pingzh">@pingzh</a> for this work, with reviews from <a href="https://github.com/sunchao">@sunchao</a>, <a href="https://github.com/ziting-openai">@ziting-openai</a>, and <a href="https://github.com/andygrove">@andygrove</a>. Related
+PRs: <a href="https://github.com/apache/datafusion-comet/pull/5473">#5473</a>, <a href="https://github.com/apache/datafusion-comet/pull/5481">#5481</a>, <a href="https://github.com/apache/datafusion-comet/pull/5513">#5513</a>, <a href="https://github.com/apache/datafusion-comet/pull/5531">#5531</a>, <a href="https://github.com/apache/datafusion-comet/pull/5537">#5537</a>.</p>
+<h2 id="native-parquet-writes-on-spark-40-experimental">Native Parquet Writes on Spark 4.0+ (Experimental)<a class="headerlink" href="#native-parquet-writes-on-spark-40-experimental" title="Permanent link">¶</a></h2>
+<p>Native Parquet writes remain <strong>experimental and disabled by default</strong>, but on Spark 4.0+ they now build on
+Spark's own write path.</p>
+<p>In 1.0.0, a native write replaced Spark's entire write command, so Comet had to reimplement the commit protocol,
+save modes, and job commit. On Spark 4.0+, Comet replaces only <code>WriteFilesExec</code>, the per-task write step that
+Spark 4.0 made pluggable, and Spark handles the rest. That means:</p>
+<ul>
+<li>File names come from Spark's commit protocol, so committers that track individual files, like the S3A magic
+  committer, work.</li>
+<li>Column names, nullability, and field IDs come from the target table, not the query.</li>
+<li>Bytes-written and rows-written metrics are correct on HDFS.</li>
+</ul>
+<p>The writer handles non-partitioned, non-bucketed writes to local and HDFS paths, and skips writes that set
+<code>spark.sql.files.maxRecordsPerFile</code>. To enable it, set <code>spark.comet.parquet.write.enabled=true</code> and
+<code>spark.comet.operator.WriteFilesExec.allowIncompatible=true</code>. Spark 3.4 and 3.5 keep the old write path.</p>
+<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> and <a href="https://github.com/sunchao">@sunchao</a> for this work, with reviews from <a href="https://github.com/comphead">@comphead</a>, <a href="https://github.com/peterxcli">@peterxcli</a>,
+<a href="https://github.com/rich7420">@rich7420</a>, and <a href="https://github.com/parthchandra">@parthchandra</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5763">#5763</a>, <a href="https://github.com/apache/datafusion-comet/pull/5369">#5369</a>.</p>
+<h2 id="s3-credentials">S3 Credentials<a class="headerlink" href="#s3-credentials" title="Permanent link">¶</a></h2>
+<p>Comet's native S3 access now works with more of the credential setups Spark supports:</p>
+<ul>
+<li><strong>Built-in credential provider adapters.</strong> In 1.0.0, a native Parquet scan failed with
+  <code>Unsupported credential provider</code> for classes Spark accepts but Comet didn't reimplement, such as
+  <code>DefaultAWSCredentialsProviderChain</code>. Two new adapters fix this by setting
+  <code>spark.hadoop.fs.s3a.comet.credential.provider.class</code>.
+  <code>HadoopS3ACredentialProviderAdapter</code> (recommended) uses Hadoop S3A's own provider chain, so it supports
+  everything S3A does, including web identity, assumed roles, and per-bucket settings.
+  <code>AwsSdkCredentialProviderAdapter</code> wraps a specific AWS SDK provider class.</li>
+<li><strong>STS throttling protection on EKS with IRSA.</strong> When many executors start at once, STS can throttle their
+  credential requests. In 1.0.0, the native Iceberg path then fell back to the EKS node role, which usually can't
+  read the bucket, and the job failed with <code>403</code> errors. Native Iceberg reads and writes now fetch web-identity
+  credentials themselves, retry throttled requests with backoff, never fall back to the node role, and share one
+  credential per executor. This happens automatically.</li>
+<li><strong>Per-location credentials.</strong> A provider implementing <code>CometS3LocationScopedCredentialProvider</code> can return
+  different credentials for different prefixes in the same bucket, such as <code>warehouse/sales</code> and
+  <code>warehouse/finance</code>.</li>
+<li><strong>Custom S3-compatible schemes.</strong> Vendor filesystems that front an S3-compatible store with their own URL
+  scheme, such as <code>blob://</code>, can be read natively by listing the scheme in
+  <code>spark.hadoop.fs.comet.s3Compliant.schemes</code>.</li>
+</ul>
+<p>See the <a href="https://datafusion.apache.org/comet/user-guide/latest/s3-credential-providers.html">S3 credential providers guide</a> for details.</p>
+<p>Thanks to <a href="https://github.com/parthchandra">@parthchandra</a> for the credential adapters and STS throttling protection, <a href="https://github.com/snmvaughan">@snmvaughan</a> for
+per-location credentials, and <a href="https://github.com/comphead">@comphead</a> for custom S3-compatible schemes, with reviews from <a href="https://github.com/sunchao">@sunchao</a> and
+<a href="https://github.com/andygrove">@andygrove</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/6023">#6023</a>, <a href="https://github.com/apache/datafusion-comet/pull/6025">#6025</a>, <a href="https://github.com/apache/datafusion-comet/pull/6031">#6031</a>, <a href="https://github.com/apache/datafusion-comet/pull/5314">#5314</a>.</p>
+<h2 id="performance">Performance<a class="headerlink" href="#performance" title="Permanent link">¶</a></h2>
+<h3 id="shuffle">Shuffle<a class="headerlink" href="#shuffle" title="Permanent link">¶</a></h3>
+<ul>
+<li>In 1.0.0, the native shuffle writer ran with a <strong>1-byte write buffer</strong> by default: its 1 MiB default was
+  declared in MiB but passed to native code as bytes. It now gets the intended 1 MiB.</li>
+<li><strong>Round-robin repartitioning</strong> assigns rows by position, as Spark does, instead of hashing every column of
+  every row. That's much cheaper on wide nested schemas, and it spreads repeated values across reducers the way
+  Spark does.</li>
+<li>A task <strong>spills all of its shuffle partitions into one file</strong> instead of one file each.</li>
+<li><strong>Zstd and Arrow IPC compression contexts are reused</strong> across shuffle blocks.</li>
+<li>Shuffle blocks are <strong>decoded against a cached schema</strong> instead of re-parsing it for every block.</li>
+<li>Per-partition scratch buffers are reused, and <code>ArrowWriter</code> bulk-copies fixed-width columns.</li>
+</ul>
+<p>Thanks to the contributors who drove this work, especially <a href="https://github.com/peterxcli">@peterxcli</a>, <a href="https://github.com/dwsmith1983">@dwsmith1983</a>, <a href="https://github.com/pingzh">@pingzh</a>, and
+<a href="https://github.com/andygrove">@andygrove</a>, with reviews from <a href="https://github.com/sunchao">@sunchao</a> and <a href="https://github.com/mbutrovich">@mbutrovich</a>.</p>
+<h3 id="planning-and-execution">Planning and Execution<a class="headerlink" href="#planning-and-execution" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>Native dynamic filter pushdown</strong>: a hash join's build-side keys filter the probe-side Parquet scan at
+  runtime, so the scan can skip data that can't match.</li>
+<li><strong>Adaptive partial aggregation</strong> for eligible native shuffle plans: when grouping keys are nearly unique, the
+  partial aggregate passes rows through instead of building a hash table that doesn't reduce them.</li>
+<li><strong>Parsed plan data is cached</strong> across a stage's tasks.</li>
+<li>New <strong>native Parquet scan I/O and read-amplification metrics</strong>.</li>
+</ul>
+<h3 id="expressions">Expressions<a class="headerlink" href="#expressions" title="Permanent link">¶</a></h3>
+<p>Many expression kernels got faster. In kernel microbenchmarks, <code>map_sort</code> is up to 3x faster on multi-entry string
+maps and 18x faster on single-entry maps. The map lookup behind <code>element_at</code> and <code>GetMapValue</code> is vectorized,
+<code>collect_list</code> and <code>collect_set</code> have a native <code>GroupsAccumulator</code>, and regex patterns are compiled once per
+planned expression. Smaller gains cover date and time field extraction, <code>posexplode</code>, unnesting, nested hashing,
+decimal overflow checks, <code>list_extract</code>, and approximate percentile merges.</p>
+<h2 id="expanded-coverage">Expanded Coverage<a class="headerlink" href="#expanded-coverage" title="Permanent link">¶</a></h2>
+<p>New native support includes the <code>mode</code>, <code>max_by</code> / <code>min_by</code>, <code>listagg</code> / <code>string_agg</code> (Spark 4.0+), and
+<code>regr_*</code> regression aggregates; <code>WindowGroupLimitExec</code>; <code>explode_outer</code>; <code>make_interval</code>; native
+<code>sequence</code> for integral types and <code>unbase64</code>; <code>_metadata</code> constant columns in the native Parquet
+scan; nested types as shuffle hash partitioning keys; <code>BinaryType</code> in sort-merge joins; and Spark 4's
+<code>EmptyRelationExec</code>.</p>
+<p>More expressions now use <a href="https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/">codegen dispatch</a> by default, where Comet runs Spark's own generated code for an
+expression inside the native pipeline, so the result matches Spark without falling back. They include
+<code>translate</code>, <code>to_csv</code>, <code>encode</code>, <code>lpad</code> / <code>rpad</code>, <code>round</code> on floats, <code>abs</code> on intervals, <code>timestampadd</code> /
+<code>timestampdiff</code>, <code>next_day</code> and <code>levenshtein</code> on collated input, and Spark expressions that call a Java or
+Scala method directly. <code>rlike</code> runs natively by
+default for literal patterns that behave the same as in Java.</p>
+<p>Spark 4 <strong>Variant</strong> support also improved: native Parquet scans can project Variant columns directly. Thanks to
+<a href="https://github.com/peterxcli">@peterxcli</a> for driving Variant support, with reviews from <a href="https://github.com/sunchao">@sunchao</a>. Related PRs: <a href="https://github.com/apache/datafusion-comet/pull/5868">#5868</a>, <a href="https://github.com/apache/datafusion-comet/pull/5794">#5794</a>.</p>
+<p>There's also <strong>experimental native support for the in-memory cache</strong> (<code>spark.comet.exec.inMemoryCache.enabled</code>,
+off by default).</p>
+<h2 id="previewing-comet-plans">Previewing Comet Plans<a class="headerlink" href="#previewing-comet-plans" title="Permanent link">¶</a></h2>
+<p>The new <code>spark.comet.explain.planOnly.enabled</code> setting logs the plan Comet would have run to the driver log,
+then lets Spark run the query as normal. It shows how much of a production workload Comet would accelerate, and
+why anything falls back, without running any of it through Comet.</p>
+<p>Thanks to <a href="https://github.com/andygrove">@andygrove</a> for this feature, with reviews from <a href="https://github.com/coderfender">@coderfender</a> and <a href="https://github.com/sunchao">@sunchao</a>. Related PRs:
+<a href="https://github.com/apache/datafusion-comet/pull/5394">#5394</a>.</p>
+<h2 id="correctness-fixes">Correctness Fixes<a class="headerlink" href="#correctness-fixes" title="Permanent link">¶</a></h2>
+<p>1.1.0 fixes many cases where Comet returned different results from Spark, failed where Spark succeeds, or
+accepted input that Spark rejects. Besides the Iceberg and memory fixes above, these are the ones most likely to
+affect 1.0.0 users. The <a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md">change log</a> has the rest.</p>
+<h3 id="wrong-results">Wrong results<a class="headerlink" href="#wrong-results" title="Permanent link">¶</a></h3>
+<ul>
+<li>Decimal <code>SUM</code> returned NULL, or raised an overflow error under ANSI, when an intermediate sum overflowed
+  but the final result fit (<a href="https://github.com/apache/datafusion-comet/pull/6041">#6041</a>, <a href="https://github.com/dwsmith1983">@dwsmith1983</a>).</li>
+<li>After a late shuffle fallback, <code>avg</code> could return NULL and <code>collect_list</code> / <code>collect_set</code> could produce
+  mismatched buffers (<a href="https://github.com/apache/datafusion-comet/pull/5421">#5421</a>, <a href="https://github.com/sunchao">@sunchao</a>).</li>
+<li>Exchange reuse could share one shuffle between plans that differ, such as <code>COUNT(*) + 1</code> and <code>COUNT(*) - 1</code>,
+  semi and anti joins, or <code>explode</code> and <code>explode_outer</code> (<a href="https://github.com/apache/datafusion-comet/pull/5470">#5470</a>, <a href="https://github.com/sunchao">@sunchao</a> and <a href="https://github.com/apache/datafusion-comet/pull/5828">#5828</a>, <a href="https://github.com/ErikBPF">@ErikBPF</a>).</li>
+<li>Two ABFS containers in the same storage account shared a cached object store, so a read could return the
+  other container's data (<a href="https://github.com/apache/datafusion-comet/pull/5053">#5053</a>, <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+<li>Dictionary-encoded values hashed differently from the same values decoded, and a null struct hashed its
+  fields, which affected joins, aggregates and shuffle partitioning (<a href="https://github.com/apache/datafusion-comet/pull/5757">#5757</a> and <a href="https://github.com/apache/datafusion-comet/pull/5754">#5754</a>, <a href="https://github.com/viirya">@viirya</a>).</li>
+<li>Parquet field names containing non-ASCII characters that differ only in case read as NULL (<a href="https://github.com/apache/datafusion-comet/pull/5602">#5602</a>,
+  <a href="https://github.com/comphead">@comphead</a>).</li>
+<li><code>IN</code>, <code>InSet</code>, nested <code>=</code>, <code>arrays_overlap</code> and <code>array_position</code> now treat <code>-0.0</code> and <code>0.0</code>, and every NaN
+  encoding, as Spark does (<a href="https://github.com/apache/datafusion-comet/pull/6073">#6073</a>, <a href="https://github.com/mizulun">@mizulun</a>, <a href="https://github.com/apache/datafusion-comet/pull/5235">#5235</a>, <a href="https://github.com/divyankshah">@divyankshah</a> and <a href="https://github.com/apache/datafusion-comet/pull/5472">#5472</a>, <a href="https://github.com/sunchao">@sunchao</a>).</li>
+<li>Decimal to double and float casts were off by one unit in the last place for most <code>DECIMAL(38,18)</code> values
+  (<a href="https://github.com/apache/datafusion-comet/pull/5684">#5684</a>, <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+<li>String to timestamp casts now follow Spark's parsing rules for short fields, time zones and signed years
+  (<a href="https://github.com/apache/datafusion-comet/pull/5682">#5682</a> and <a href="https://github.com/apache/datafusion-comet/pull/5858">#5858</a>, <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+</ul>
+<h3 id="errors-spark-raises-that-comet-did-not">Errors Spark raises that Comet did not<a class="headerlink" href="#errors-spark-raises-that-comet-did-not" title="Permanent link">¶</a></h3>
+<ul>
+<li>Casts and expressions routed through codegen dispatch could skip ANSI errors raised inside a constant
+  subexpression (<a href="https://github.com/apache/datafusion-comet/pull/5623">#5623</a>, <a href="https://github.com/andygrove">@andygrove</a>).</li>
+<li>Rejected <code>TIMESTAMP_NTZ</code> casts returned NULL under ANSI instead of raising <code>CAST_INVALID_INPUT</code> (<a href="https://github.com/apache/datafusion-comet/pull/5752">#5752</a>,
+  <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+<li>Out-of-range Parquet <code>TIMESTAMP_MILLIS</code> values, top-level or nested, silently wrapped (<a href="https://github.com/apache/datafusion-comet/pull/5177">#5177</a> and <a href="https://github.com/apache/datafusion-comet/pull/5740">#5740</a>,
+  <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+<li>Nested Parquet struct, list and map fields now follow Spark's conversion rules instead of returning NULL on
+  overflow or accepting values Spark rejects (<a href="https://github.com/apache/datafusion-comet/pull/5681">#5681</a>, <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+</ul>
+<h3 id="query-failures-and-crashes">Query failures and crashes<a class="headerlink" href="#query-failures-and-crashes" title="Permanent link">¶</a></h3>
+<ul>
+<li><code>collect_list</code> and <code>collect_set</code> over nested arguments failed with "column types must match schema types"
+  (<a href="https://github.com/apache/datafusion-comet/pull/5159">#5159</a>, <a href="https://github.com/andygrove">@andygrove</a>).</li>
+<li>Native shuffle failed with a 2 GB task serialization error on jobs with very many partitions (<a href="https://github.com/apache/datafusion-comet/pull/5392">#5392</a>,
+  <a href="https://github.com/parthchandra">@parthchandra</a>).</li>
+<li>A Scala UDF from a user jar failed with a <code>ClassCastException</code> (<a href="https://github.com/apache/datafusion-comet/pull/5282">#5282</a>, <a href="https://github.com/andygrove">@andygrove</a>).</li>
+<li><code>rpad</code> and <code>lpad</code> panicked on a NULL length (<a href="https://github.com/apache/datafusion-comet/pull/5680">#5680</a>, <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+<li>Structs with duplicate field names failed the task in native shuffle, and panicked in the native Parquet
+  scan (<a href="https://github.com/apache/datafusion-comet/pull/5866">#5866</a>, <a href="https://github.com/dwsmith1983">@dwsmith1983</a> and <a href="https://github.com/apache/datafusion-comet/pull/5786">#5786</a>, <a href="https://github.com/ErikBPF">@ErikBPF</a>).</li>
+</ul>
+<h3 id="hangs-and-resource-use">Hangs and resource use<a class="headerlink" href="#hangs-and-resource-use" title="Permanent link">¶</a></h3>
+<ul>
+<li>The JVM hung on exit when an application returned from <code>main</code> without calling <code>spark.stop()</code> (<a href="https://github.com/apache/datafusion-comet/pull/5748">#5748</a>,
+  <a href="https://github.com/zhangfengcdt">@zhangfengcdt</a>).</li>
+<li>The native scan busy-polled while waiting on S3 or HDFS reads, keeping one core per task at 100% (<a href="https://github.com/apache/datafusion-comet/pull/6219">#6219</a>,
+  <a href="https://github.com/mixermt">@mixermt</a> and <a href="https://github.com/andygrove">@andygrove</a>).</li>
+<li>One task could force-spill another task's shuffle buffers, and a failed shuffle write leaked its memory
+  reservation (<a href="https://github.com/apache/datafusion-comet/pull/5493">#5493</a> and <a href="https://github.com/apache/datafusion-comet/pull/5461">#5461</a>, <a href="https://github.com/peterxcli">@peterxcli</a>).</li>
+</ul>
+<h2 id="upgrading-to-110">Upgrading to 1.1.0<a class="headerlink" href="#upgrading-to-110" title="Permanent link">¶</a></h2>
+<p>A few changes in 1.1.0 can affect a deployment. The <a href="https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html">Comet Upgrade Guide</a> has the details.</p>
+<h3 id="platform">Platform<a class="headerlink" href="#platform" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong>JDK 17 or later is required.</strong> JDK 11 support was removed, as announced in 1.0.0.</li>
+<li><strong>Spark 3.4 is still deprecated.</strong> Comet still publishes Spark 3.4 binaries, but Spark's SQL test suite runs
+  against 3.4 only on demand, so 3.4-specific regressions are more likely. We recommend Spark 3.5 or later.</li>
+</ul>
+<h3 id="size-settings-now-read-correctly">Size settings now read correctly<a class="headerlink" href="#size-settings-now-read-correctly" title="Permanent link">¶</a></h3>
+<p>1.0.0 misread three size settings, so jobs that set them may behave differently:</p>
+<ul>
+<li><strong><code>spark.memory.offHeap.size</code> as a bare number of bytes</strong> was read as MiB, which made <code>fair_unified</code>'s
+  per-operator shares about a million times too large. With correct shares, operators can spill sooner, and an
+  operator that can't spill can fail when it exceeds its share. Sizes with a unit, like <code>16g</code>, aren't affected.
+  <code>spark.comet.exec.memoryPool=greedy_unified</code> restores the old behavior.</li>
+<li><strong><code>spark.comet.maxTempDirectorySize</code> with a unit</strong>, like <code>10g</code>, was ignored in favor of the 100 GB default.
+  It's now enforced, so a query that spills past it fails.</li>
+<li><strong><code>spark.comet.shuffle.native.writeBufferSize</code> with a unit</strong> was treated as bytes, so <code>64m</code> meant 64 bytes. It
+  now means what it says. These buffers use untracked native memory, so check large values against
+  <code>spark.executor.memoryOverhead</code>.</li>
+</ul>
+<p>Malformed values for <code>spark.comet.maxTempDirectorySize</code> and <code>spark.comet.explain.native.enabled</code> now fail the
+query instead of falling back to the default.</p>
+<h3 id="more-memory-before-spilling">More memory before spilling<a class="headerlink" href="#more-memory-before-spilling" title="Permanent link">¶</a></h3>
+<p>With the <code>fair_unified</code> fix, tasks with several operators can reserve more memory before spilling than in
+0.15.0 through 1.0.0, especially on executors running few tasks at once. If you sized executors for one of those
+releases, check the memory usage log for headroom.</p>
+<h3 id="conditions-for-enabling-comet">Conditions for enabling Comet<a class="headerlink" href="#conditions-for-enabling-comet" title="Permanent link">¶</a></h3>
+<p>Comet requires Spark's off-heap memory. <code>CometPlugin</code> already enforced this, and 1.1.0 applies the same check
+when <code>CometSparkSessionExtensions</code> is registered directly, disabling Comet with a warning. Both checks read
+<code>spark.memory.offHeap.enabled</code> from the SparkContext, so set it when the application starts, not on a
+<code>SparkSession.builder</code> after the context exists.</p>
+<p>Comet also checks the shuffle manager that's actually running, not the session's <code>spark.shuffle.manager</code>. A
+session that names <code>CometShuffleManager</code> after the context started with a different one now runs without Comet,
+with a warning, instead of failing with a <code>ClassCastException</code>.</p>
+<h3 id="deprecated-and-removed-settings">Deprecated and removed settings<a class="headerlink" href="#deprecated-and-removed-settings" title="Permanent link">¶</a></h3>
+<ul>
+<li><strong><code>spark.comet.exec.memoryPool.fraction</code></strong> is deprecated and will be removed in a future major release. It
+  still works, and the driver warns when it's set.</li>
+<li><strong>Testing-only on-heap settings</strong> were removed: <code>spark.comet.memoryOverhead</code>,
+  <code>spark.comet.exec.onHeap.memoryPool</code>, and <code>spark.comet.shuffle.jvm.memoryFactor</code> (and its old name,
+  <code>spark.comet.columnar.shuffle.memory.factor</code>). The <a href="https://datafusion.apache.org/comet/about/versioning_policy.html#testing-and-internal-configurations-are-exempt">versioning policy</a> exempts testing settings, and Comet
+  ignores them if they're still set.</li>
+</ul>
+<h2 id="compatibility">Compatibility<a class="headerlink" href="#compatibility" title="Permanent link">¶</a></h2>
+<p>Supported platforms:</p>
+<ul>
+<li><strong>Spark 3.4.3</strong> with Java 17 and Scala 2.12/2.13 (deprecated)</li>
+<li><strong>Spark 3.5.9</strong> with Java 17 and Scala 2.12/2.13</li>
+<li><strong>Spark 4.0.4</strong> with Java 17/21 and Scala 2.13</li>
+<li><strong>Spark 4.1.3</strong> with Java 17/21 and Scala 2.13</li>
+<li><strong>Spark 4.2.0</strong> with Java 17 and Scala 2.13 (experimental, for early evaluation only)</li>
+</ul>
+<p>See the <a href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html">Spark Version Compatibility</a> page for known limitations specific to each version.</p>
+<p>This release builds on <strong>DataFusion 55.1</strong> and <strong>Arrow 59.2</strong>.</p>
+<h2 id="get-started-with-comet-110">Get Started with Comet 1.1.0<a class="headerlink" href="#get-started-with-comet-110" title="Permanent link">¶</a></h2>
+<p>Follow the <a href="https://datafusion.apache.org/comet/user-guide/1.1/installation.html">Comet 1.1.0 Installation Guide</a>
+to get up and running, then point Comet at your existing Spark workloads.</p>
+
+<!--
+  Comments Section
+  Loaded only after explicit visitor consent to comply with ASF policy.
+-->
+
+<div>
+  <hr>
+  <h3 id="comments">Comments<a class="headerlink" href="#comments" title="Permanent link">&para;</a></h3>
+
+  <!-- Local loader script -->
+  <script src="/blog/js/giscus-consent.js" defer></script>
+
+  <!-- Consent UI -->
+  <div id="giscus-consent">
+    <p>
+        We use <a href="https://giscus.app/">Giscus</a> for comments, powered by GitHub Discussions.
+        To respect your privacy, Giscus and comments will load only if you click "Show Comments"
+    </p>
+
+    <div class="consent-actions">
+      <button id="giscus-load" type="button">Show Comments</button>
+      <button id="giscus-revoke" type="button" hidden>Hide Comments</button>
+    </div>
+
+    <noscript>JavaScript is required to load comments from Giscus.</noscript>
+  </div>
+
+  <!-- Container where Giscus will render -->
+  <div id="comment-thread"></div>
+</div>      </div>
+      <aside class="toc-container d-none d-md-block col-md-4 col-xl-3 ms-xl-2">
+        <div class="toc"><span class="toctitle">Contents</span><ul>
+<li><a href="#native-iceberg-writes-experimental">Native Iceberg Writes (Experimental)</a><ul>
+<li><a href="#splitting-the-write-operator">Splitting the write operator</a></li>
+<li><a href="#writing-parquet-natively">Writing Parquet natively</a></li>
+<li><a href="#matching-iceberg-java">Matching iceberg-java</a></li>
+<li><a href="#failure-handling">Failure handling</a></li>
+<li><a href="#known-differences">Known differences</a></li>
+</ul>
+</li>
+<li><a href="#memory-management">Memory Management</a><ul>
+<li><a href="#where-the-memory-goes">Where the memory goes</a></li>
+<li><a href="#measuring-used-memory">Measuring used memory</a></li>
+<li><a href="#memory-pool-fixes">Memory pool fixes</a></li>
+<li><a href="#on-heap-mode">On-heap mode</a></li>
+</ul>
+</li>
+<li><a href="#more-iceberg-improvements">More Iceberg Improvements</a></li>
+<li><a href="#remote-shuffle-with-celeborn">Remote Shuffle with Celeborn</a></li>
+<li><a href="#native-parquet-writes-on-spark-40-experimental">Native Parquet Writes on Spark 4.0+ (Experimental)</a></li>
+<li><a href="#s3-credentials">S3 Credentials</a></li>
+<li><a href="#performance">Performance</a><ul>
+<li><a href="#shuffle">Shuffle</a></li>
+<li><a href="#planning-and-execution">Planning and Execution</a></li>
+<li><a href="#expressions">Expressions</a></li>
+</ul>
+</li>
+<li><a href="#expanded-coverage">Expanded Coverage</a></li>
+<li><a href="#previewing-comet-plans">Previewing Comet Plans</a></li>
+<li><a href="#correctness-fixes">Correctness Fixes</a><ul>
+<li><a href="#wrong-results">Wrong results</a></li>
+<li><a href="#errors-spark-raises-that-comet-did-not">Errors Spark raises that Comet did not</a></li>
+<li><a href="#query-failures-and-crashes">Query failures and crashes</a></li>
+<li><a href="#hangs-and-resource-use">Hangs and resource use</a></li>
+</ul>
+</li>
+<li><a href="#upgrading-to-110">Upgrading to 1.1.0</a><ul>
+<li><a href="#platform">Platform</a></li>
+<li><a href="#size-settings-now-read-correctly">Size settings now read correctly</a></li>
+<li><a href="#more-memory-before-spilling">More memory before spilling</a></li>
+<li><a href="#conditions-for-enabling-comet">Conditions for enabling Comet</a></li>
+<li><a href="#deprecated-and-removed-settings">Deprecated and removed settings</a></li>
+</ul>
+</li>
+<li><a href="#compatibility">Compatibility</a></li>
+<li><a href="#get-started-with-comet-110">Get Started with Comet 1.1.0</a></li>
+</ul>
+</div>
+      </aside>
+    </div>
+  </div>
+</div>    
+    <!-- footer -->
+    <div class="row g-0">
+      <div class="col-12">
+        <p style="font-style: italic; font-size: 0.8rem; text-align: center;">
+          Copyright 2026, <a href="https://www.apache.org/">The Apache Software Foundation</a>, Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br/>
+          Apache&reg; and the Apache logo are trademarks of The Apache Software Foundation.
+        </p>
+        <p style="font-size: 0.8rem; text-align: center;">
+          <a href="https://www.apache.org/">Foundation</a> |
+          <a href="https://www.apache.org/events/current-event.html">Events</a> |
+          <a href="https://www.apache.org/licenses/">License</a> |
+          <a href="https://www.apache.org/security/">Security</a> |
+          <a href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a> |
+          <a href="https://www.apache.org/foundation/thanks.html">Thanks</a> |
+          <a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy</a>
+        </p>
+      </div>
+    </div>
+    <script src="/blog/js/bootstrap.bundle.min.js"></script>  </main>
+  </body>
+</html>
diff --git a/output/author/pmc.html b/output/author/pmc.html
index 5c9cdac..328d500 100644
--- a/output/author/pmc.html
+++ b/output/author/pmc.html
@@ -21,6 +21,38 @@
     <h2>Articles by pmc</h2>
 
 
+        <article>
+            <header> <h2><a href="https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 1.1.0 Release">Apache DataFusion Comet 1.1.0 Release</a></h2> </header>
+            <section><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 1.1.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers roughly seven weeks of development …</p></section>
+            <footer>
+                <p>Published: <time datetime="2026-10-01T00:00:00+00:00"> Thu 01 October 2026 </time></p>
+                <address>By
+                        <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+                </address>
+            </footer>
+        </article>
         <article>
             <header> <h2><a href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="bookmark" title="Permalink to Apache DataFusion 55.0.0 Released">Apache DataFusion 55.0.0 Released</a></h2> </header>
             <section><!--
diff --git a/output/category/blog.html b/output/category/blog.html
index 035c5d5..c199836 100644
--- a/output/category/blog.html
+++ b/output/category/blog.html
@@ -22,6 +22,38 @@
     <h2>Articles in the blog category</h2>
 
 
+        <article>
+            <header> <h2><a href="https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0" rel="bookmark" title="Permalink to Apache DataFusion Comet 1.1.0 Release">Apache DataFusion Comet 1.1.0 Release</a></h2> </header>
+            <section><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 1.1.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers roughly seven weeks of development …</p></section>
+            <footer>
+                <p>Published: <time datetime="2026-10-01T00:00:00+00:00"> Thu 01 October 2026 </time></p>
+                <address>By
+                        <a href="https://datafusion.apache.org/blog/author/pmc.html">pmc</a>
+                </address>
+            </footer>
+        </article>
         <article>
             <header> <h2><a href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="bookmark" title="Permalink to Apache DataFusion 55.0.0 Released">Apache DataFusion 55.0.0 Released</a></h2> </header>
             <section><!--
diff --git a/output/feed.xml b/output/feed.xml
index b5b098e..adba3b4 100644
--- a/output/feed.xml
+++ b/output/feed.xml
@@ -1,5 +1,27 @@
 <?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feed.xml" rel="self"/><lastBuildDate>Tue, 25 Aug 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion 55.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0</link><description>&lt;!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feed.xml" rel="self"/><lastBuildDate>Thu, 01 Oct 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 1.1.0 Release</title><link>https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0</link><description>&lt;!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development …&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 01 Oct 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-10-01:/blog/2026/10/01/datafusion-comet-1.1.0</guid><category>blog</category></item><item><title>Apache DataFusion 55.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0</link><description>&lt;!--
 {% comment %}
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
diff --git a/output/feeds/all-en.atom.xml b/output/feeds/all-en.atom.xml
index 0ec1882..e232c30 100644
--- a/output/feeds/all-en.atom.xml
+++ b/output/feeds/all-en.atom.xml
@@ -1,5 +1,430 @@
 <?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-25T00:00:00+00:00</updated><entry><title>Apache DataFusion 55.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="alternate"/><published>2026-08-25T00:00: [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/all-en.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-10-01T00:00:00+00:00</updated><entry><title>Apache DataFusion Comet 1.1.0 Release</title><link href="https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0" rel="alternate"/><published>2026-10- [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development …&lt;/p&gt;</summary><content type="html">&lt;!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development since 1.0.0: 398 commits from 40 contributors. See the
+&lt;a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md"&gt;change log&lt;/a&gt; for the full list of changes.&lt;/p&gt;
+&lt;p&gt;The highlights:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Native Iceberg writes&lt;/strong&gt; (experimental): Comet can write Iceberg data files with iceberg-rust instead of
+  iceberg-java.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Memory management&lt;/strong&gt;: Comet now measures the native memory its pools don't track, logs it on every executor,
+  and fixes several long-standing pool accounting bugs.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Native shuffle over Apache Celeborn&lt;/strong&gt;: 1.1.0 includes the native writer and reader, but released Celeborn
+  clients currently fall back to ordinary Spark/Celeborn shuffle.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Native Parquet writes on Spark 4.0+&lt;/strong&gt; (experimental), now built on Spark's own write path.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="native-iceberg-writes-experimental"&gt;Native Iceberg Writes (Experimental)&lt;a class="headerlink" href="#native-iceberg-writes-experimental" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Until now, Comet accelerated Iceberg reads but left writes to the JVM. In 1.1.0, Comet can write Iceberg data
+files natively using &lt;a href="https://github.com/apache/iceberg-rust"&gt;iceberg-rust&lt;/a&gt;. The feature is &lt;strong&gt;experimental and
+disabled by default&lt;/strong&gt;, and it only engages under fairly strict conditions.&lt;/p&gt;
+&lt;h3 id="splitting-the-write-operator"&gt;Splitting the write operator&lt;a class="headerlink" href="#splitting-the-write-operator" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
+and validates against the catalog. Because file writing is bundled with the metadata and commit steps, there was
+no separate piece for Comet to replace.&lt;/p&gt;
+&lt;p&gt;Setting &lt;code&gt;spark.comet.write.iceberg.splitOperator.enabled=true&lt;/code&gt; splits eligible Iceberg writes into two operators:&lt;/p&gt;
+&lt;ol&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;IcebergWrite&lt;/code&gt;&lt;/strong&gt; writes data files on the executors and returns each task's commit message.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;IcebergCommit&lt;/code&gt;&lt;/strong&gt; collects the commit messages on the driver and performs the normal Iceberg commit, once.&lt;/li&gt;
+&lt;/ol&gt;
+&lt;p&gt;With only the split enabled, iceberg-java still writes the files; the native writer, described next, replaces
+that step. The split covers &lt;code&gt;INSERT INTO&lt;/code&gt; and DataFrame &lt;code&gt;append&lt;/code&gt;, static and dynamic &lt;code&gt;INSERT OVERWRITE&lt;/code&gt;, and
+copy-on-write &lt;code&gt;DELETE&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, and &lt;code&gt;MERGE&lt;/code&gt;, on every supported Spark version. Merge-on-read writes are left
+alone. When Comet can't split a write (an unrecognized write class, CTAS on Spark 3.4, or a write that needs
+Spark's commit coordinator), it plans the write exactly as if Comet weren't there.&lt;/p&gt;
+&lt;h3 id="writing-parquet-natively"&gt;Writing Parquet natively&lt;a class="headerlink" href="#writing-parquet-natively" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;The native writer requires the split. Also setting &lt;code&gt;spark.comet.iceberg.write.enabled=true&lt;/code&gt; hands each task's
+Parquet writing to iceberg-rust. The driver passes the native writer everything it needs: schema, partition spec,
+data location, Parquet settings, writer mode, and object-store configuration. Each task writes its files and
+returns their metadata as an in-memory Iceberg manifest.&lt;/p&gt;
+&lt;p&gt;From there, iceberg-java takes over again. The JVM reads the manifest, recomputes each file's metrics from its
+Parquet footer, and builds the same commit message the JVM writer would have produced. Snapshots, manifest
+lists, commit validation, and retries all work as before.&lt;/p&gt;
+&lt;p&gt;The native writer consumes Arrow batches from a Comet operator, so the query feeding the write must run in Comet
+too. Writes from a local relation, such as &lt;code&gt;INSERT ... VALUES&lt;/code&gt;, also need
+&lt;code&gt;spark.comet.exec.localTableScan.enabled=true&lt;/code&gt;.&lt;/p&gt;
+&lt;p&gt;To see which path a write took, check the physical plan: a native write shows &lt;code&gt;CometIcebergWrite&lt;/code&gt; under
+&lt;code&gt;IcebergCommit&lt;/code&gt;, and a write that fell back shows &lt;code&gt;IcebergWrite&lt;/code&gt;.&lt;/p&gt;
+&lt;h3 id="matching-iceberg-java"&gt;Matching iceberg-java&lt;a class="headerlink" href="#matching-iceberg-java" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;The native writer aims to write the same table iceberg-java would, down to the metadata. Manifest metrics drive
+pruning for every future reader, so rather than trust the native writer's numbers, Comet recomputes them with
+iceberg-java's own code. Parity tests write the same rows through both writers and compare the committed counts
+and bounds. The cost is one small footer read per file.&lt;/p&gt;
+&lt;p&gt;Eligibility is an allowlist. A write goes native only if its whole configuration matches the documented set of
+supported settings. Anything else, including properties added by future Iceberg versions, falls back to
+iceberg-java and reports why in Comet's extended &lt;code&gt;EXPLAIN&lt;/code&gt; output. Encryption, object-storage layout, custom
+location providers, bloom filters, and unrecognized &lt;code&gt;parquet.*&lt;/code&gt; properties all fall back. For a partitioned
+table, Iceberg asks Spark to cluster and sort rows by partition value, such as &lt;code&gt;bucket(16, id)&lt;/code&gt; or &lt;code&gt;days(ts)&lt;/code&gt;,
+before writing. That step must also run natively, which it now can because Iceberg's system functions have
+native implementations.&lt;/p&gt;
+&lt;p&gt;Comet decides all of this at planning time, including checking that every iceberg-java class it relies on is
+where it expects. An Iceberg release that moves one falls back instead of failing tasks mid-write. CI also runs
+Apache Iceberg's own Spark test suites, for Iceberg 1.8.1 through 1.11.0, with the native writer enabled.&lt;/p&gt;
+&lt;h3 id="failure-handling"&gt;Failure handling&lt;a class="headerlink" href="#failure-handling" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Only successful tasks' commit messages are committed. A failed task deletes the files it wrote, as iceberg-java
+does, and a failed job commits nothing and deletes the completed tasks' files. Retries can't collide, because
+each attempt's ID is part of its file names. Anything cleanup misses is invisible to readers and is removed by
+Iceberg's normal &lt;code&gt;remove_orphan_files&lt;/code&gt; maintenance.&lt;/p&gt;
+&lt;h3 id="known-differences"&gt;Known differences&lt;a class="headerlink" href="#known-differences" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Files written by parquet-rs, the Rust Parquet library iceberg-rust uses, differ from those written by
+parquet-java (formerly parquet-mr), which iceberg-java uses. Enabling the feature accepts these differences.
+Most are cosmetic, such as footer metadata, &lt;code&gt;created_by&lt;/code&gt;, and page encoding labels. Two are worth knowing about:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Files won't split at the same points.&lt;/strong&gt; Both writers check file size every 1000 rows, but they estimate it
+  differently, so they can roll over to a new file at different rows.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;High-cardinality columns keep a dictionary page.&lt;/strong&gt; parquet-java drops dictionary encoding early when it isn't
+  saving space, while parquet-rs keeps it up to &lt;code&gt;write.parquet.dict-size-bytes&lt;/code&gt;. Results are the same, but
+  selective reads of native-written files fetch more bytes
+  (&lt;a href="https://github.com/apache/datafusion-comet/issues/6114"&gt;#6114&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/iceberg-writes.html"&gt;Iceberg Writes guide&lt;/a&gt; lists every supported setting and every difference. Please try it on a non-production
+table and tell us how it goes. Feedback from real workloads is what this feature needs before it loses the
+experimental label.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/jordepic"&gt;@jordepic&lt;/a&gt; for designing and implementing the split-operator plan, write detection, and the native
+writer, and to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for the fidelity and failure-handling work, with contributions from
+&lt;a href="https://github.com/zhangfengcdt"&gt;@zhangfengcdt&lt;/a&gt;, &lt;a href="https://github.com/snmvaughan"&gt;@snmvaughan&lt;/a&gt;, &lt;a href="https://github.com/liupoyi-1031"&gt;@liupoyi-1031&lt;/a&gt;, and &lt;a href="https://github.com/0lai0"&gt;@0lai0&lt;/a&gt;, and reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;,
+&lt;a href="https://github.com/unikdahal"&gt;@unikdahal&lt;/a&gt;, and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/4658"&gt;#4658&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5298"&gt;#5298&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5361"&gt;#5361&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5663"&gt;#5663&lt;/a& [...]
+&lt;h2 id="memory-management"&gt;Memory Management&lt;a class="headerlink" href="#memory-management" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;A recurring problem for Comet users has been executors killed by the cluster manager (on Kubernetes,
+&lt;code&gt;ExecutorLostFailure&lt;/code&gt; with exit code 137) even though Comet stayed within its memory pool. 1.1.0 explains why
+this happens, lets you measure it, and fixes pool bugs that made it worse.&lt;/p&gt;
+&lt;h3 id="where-the-memory-goes"&gt;Where the memory goes&lt;a class="headerlink" href="#where-the-memory-goes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet's native operators allocate from the Rust heap but charge their reservations against Spark's off-heap
+pool, sized by &lt;code&gt;spark.memory.offHeap.size&lt;/code&gt;. Operators only reserve memory for data they deliberately hold onto:
+sort buffers, hash join build sides, aggregation state, and buffered shuffle partitions.&lt;/p&gt;
+&lt;p&gt;Plenty of memory is never reserved: working memory in expression kernels, decompression buffers, Parquet reader
+state, object store buffers, JVM-side Arrow buffers, and allocator overhead. All of it has to fit in
+&lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;, and until now there was no way to see how much of it there was.&lt;/p&gt;
+&lt;p&gt;&lt;img alt="The executor container holds the JVM heap, the off-heap memory pool, and the memory overhead. Spark and Comet share the off-heap pool, where Comet's sorts, joins, aggregations, and shuffles reserve memory. The memory overhead holds the JVM's own overhead plus Comet's native memory that the pool does not track." class="img-fluid" src="/blog/images/comet-1.1.0/comet-executor-memory.svg" width="100%"/&gt;&lt;/p&gt;
+&lt;h3 id="measuring-used-memory"&gt;Measuring used memory&lt;a class="headerlink" href="#measuring-used-memory" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet now wraps its native allocator in a counter that tracks every byte allocated and not yet freed. It only
+observes and never rejects an allocation. Arrow buffers the JVM imports from native code are now tracked
+separately, so they aren't counted twice.&lt;/p&gt;
+&lt;p&gt;Each executor uses the counter to log its native memory usage every 10 seconds while Comet runs:&lt;/p&gt;
+&lt;pre&gt;&lt;code&gt;Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native plans, 8 memory pools)
+&lt;/code&gt;&lt;/pre&gt;
+&lt;p&gt;&lt;code&gt;reserved&lt;/code&gt; is what the pools track, and the container already has room for it. &lt;code&gt;allocated&lt;/code&gt; is everything
+Comet's native code holds. The difference is what has to fit in &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;, next to the
+JVM's own non-heap memory. To size the overhead, run a representative workload, take the largest difference from
+the log, add it to the overhead you had before Comet, and leave a margin. Setting
+&lt;code&gt;spark.comet.memory.logInterval=1s&lt;/code&gt; for that run makes it less likely to miss a short peak.&lt;/p&gt;
+&lt;p&gt;The executor also warns when its native memory looks larger than the container allows, and the
+&lt;a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/memory.html"&gt;tuning guide&lt;/a&gt; walks through sizing with examples. One gotcha: setting &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;
+&lt;em&gt;replaces&lt;/em&gt; the value Spark derives from &lt;code&gt;spark.executor.memoryOverheadFactor&lt;/code&gt; instead of adding to it, so on a
+large executor it can shrink the container. For large executors, raise the factor instead.&lt;/p&gt;
+&lt;h3 id="memory-pool-fixes"&gt;Memory pool fixes&lt;a class="headerlink" href="#memory-pool-fixes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet's off-heap memory comes from one of two pools. &lt;code&gt;fair_unified&lt;/code&gt;, the default, caps each operator at an even
+share of the task's memory. &lt;code&gt;greedy_unified&lt;/code&gt; gives memory to operators first come, first served. When Spark
+grants an operator less memory than it asked for (a partial grant), the operator spills.&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;fair_unified&lt;/code&gt; limited a whole task to one operator's share.&lt;/strong&gt; Since 0.15.0, the pool compared all of a
+  task's reservations against a single operator's share, and each new operator shrank the limit for the rest.
+  Each operator now gets its own share, so tasks with several operators can use more memory before spilling.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;A partial grant from Spark no longer panics the task.&lt;/strong&gt; DataFusion sometimes has to record memory that
+  already exists, such as a spilled batch being read back, and both pools panicked when Spark granted less than
+  they asked for. They now track the shortfall and repay it before returning memory to Spark.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Leaks on failure paths.&lt;/strong&gt; A plan that failed during setup or teardown could leak its task's memory pool, and
+  a failed Arrow import leaked the vectors it had already imported.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Configuration units.&lt;/strong&gt; 1.0.0 misread three size settings, including &lt;code&gt;spark.memory.offHeap.size&lt;/code&gt; written as
+  a bare number. See &lt;a href="#upgrading-to-110"&gt;Upgrading to 1.1.0&lt;/a&gt; for what changes.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Less log noise when spilling.&lt;/strong&gt; 1.0.0 logged a warning and a memory dump for every partial grant, so a
+  spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
+  deadlock the task.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Metrics.&lt;/strong&gt; Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+  task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+  1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+  in the SQL tab.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;&lt;code&gt;spark.comet.exec.memoryPool.fraction&lt;/code&gt; is deprecated. It was meant to leave room for untracked memory, but Spark
+hands out the whole pool anyway, so it never did. Size &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt; instead.&lt;/p&gt;
+&lt;h3 id="on-heap-mode"&gt;On-heap mode&lt;a class="headerlink" href="#on-heap-mode" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;On-heap mode exists so that Spark's and Iceberg's test suites can run against Comet; production runs off-heap.
+Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes that
+accounting, along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
+off-heap memory however it's enabled, including when &lt;code&gt;CometSparkSessionExtensions&lt;/code&gt; is registered directly, and
+disables itself with a warning otherwise.&lt;/p&gt;
+&lt;p&gt;Contributors can find more detail in the new &lt;a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html"&gt;memory management guide&lt;/a&gt;.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for driving this work, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt; for the memory pool lifecycle and shuffle spill
+accounting fixes, &lt;a href="https://github.com/ywskycn"&gt;@ywskycn&lt;/a&gt; for reporting native memory usage to Spark, &lt;a href="https://github.com/1fanwang"&gt;@1fanwang&lt;/a&gt; for the Arrow import leak
+fix, and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; for the native aggregate spill metrics, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;,
+&lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;, and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5934"&gt;#5934&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6162"&gt;#6162&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6048"&gt;#6048&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6128"&gt;#6128&lt;/a&gt [...]
+&lt;h2 id="more-iceberg-improvements"&gt;More Iceberg Improvements&lt;a class="headerlink" href="#more-iceberg-improvements" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;V3 deletion vectors&lt;/strong&gt; are applied on native scans.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Iceberg system functions&lt;/strong&gt; (&lt;code&gt;bucket&lt;/code&gt;, &lt;code&gt;truncate&lt;/code&gt;, &lt;code&gt;years&lt;/code&gt;, &lt;code&gt;months&lt;/code&gt;, &lt;code&gt;days&lt;/code&gt;, and &lt;code&gt;hours&lt;/code&gt;) run natively.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Scan planning metrics and scan time&lt;/strong&gt; appear in the Spark UI for native Iceberg scans.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;A wrong-results fix.&lt;/strong&gt; A filter such as &lt;code&gt;bucket(4, id) = 2&lt;/code&gt; combined with another predicate was pushed to
+  the native scan as &lt;code&gt;id = 2&lt;/code&gt;, returning too few rows.&lt;/li&gt;
+&lt;li&gt;Tables partitioned by an unknown transform can be read natively, and &lt;code&gt;IS NULL&lt;/code&gt; / &lt;code&gt;IS NOT NULL&lt;/code&gt; checks on list
+  and map columns no longer force a fallback.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt; for deletion vector support, &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt; for the scan metrics, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt; for the
+null-check fix, and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for the native system functions and residual fix, with reviews from
+&lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/rich7420"&gt;@rich7420&lt;/a&gt;, &lt;a href="https://github.com/unikdahal"&gt;@unikdahal&lt;/a&gt;, and &lt;a href="https://github.com/jordepic"&gt;@jordepic&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5853"&gt;#5853&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5638"&gt;#5638&lt;/a&gt;, &lt;a href="https://github.com/apache/dataf [...]
+&lt;h2 id="remote-shuffle-with-celeborn"&gt;Remote Shuffle with Celeborn&lt;a class="headerlink" href="#remote-shuffle-with-celeborn" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;1.1.0 adds Comet's side of native shuffle over Apache Celeborn: map tasks push Comet's Arrow data straight to
+Celeborn, and reducers read it back natively. In 1.0.0, Celeborn users always got ordinary Spark shuffle.&lt;/p&gt;
+&lt;p&gt;Native Celeborn shuffle is unavailable with released Celeborn 0.6.x and 0.7.x clients in Comet 1.1.0 because
+these clients do not pass Comet's push-completion compatibility checks. Set &lt;code&gt;spark.shuffle.manager&lt;/code&gt; to
+&lt;code&gt;org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager&lt;/code&gt; to use Comet alongside
+ordinary Spark/Celeborn shuffle; Comet can still accelerate other operators. These clients retain
+ordinary Spark/Celeborn shuffle even when &lt;code&gt;spark.comet.shuffle.mode=native&lt;/code&gt; is requested, and the
+default &lt;code&gt;auto&lt;/code&gt; mode also retains ordinary shuffle. Follow-up work to enable native shuffle with
+released clients is tracked in &lt;a href="https://github.com/apache/datafusion-comet/issues/6523"&gt;#6523&lt;/a&gt;.
+Native shuffle over Celeborn does not support &lt;code&gt;spark.io.encryption.enabled=true&lt;/code&gt;, and Celeborn
+is not bundled with Comet. The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/celeborn.html"&gt;Celeborn tuning guide&lt;/a&gt; has the full requirements.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/pingzh"&gt;@pingzh&lt;/a&gt; for this work, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/ziting-openai"&gt;@ziting-openai&lt;/a&gt;, and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;. Related
+PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5473"&gt;#5473&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5481"&gt;#5481&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5513"&gt;#5513&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5531"&gt;#5531&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5537"&gt;#5537&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="native-parquet-writes-on-spark-40-experimental"&gt;Native Parquet Writes on Spark 4.0+ (Experimental)&lt;a class="headerlink" href="#native-parquet-writes-on-spark-40-experimental" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Native Parquet writes remain &lt;strong&gt;experimental and disabled by default&lt;/strong&gt;, but on Spark 4.0+ they now build on
+Spark's own write path.&lt;/p&gt;
+&lt;p&gt;In 1.0.0, a native write replaced Spark's entire write command, so Comet had to reimplement the commit protocol,
+save modes, and job commit. On Spark 4.0+, Comet replaces only &lt;code&gt;WriteFilesExec&lt;/code&gt;, the per-task write step that
+Spark 4.0 made pluggable, and Spark handles the rest. That means:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;File names come from Spark's commit protocol, so committers that track individual files, like the S3A magic
+  committer, work.&lt;/li&gt;
+&lt;li&gt;Column names, nullability, and field IDs come from the target table, not the query.&lt;/li&gt;
+&lt;li&gt;Bytes-written and rows-written metrics are correct on HDFS.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;The writer handles non-partitioned, non-bucketed writes to local and HDFS paths, and skips writes that set
+&lt;code&gt;spark.sql.files.maxRecordsPerFile&lt;/code&gt;. To enable it, set &lt;code&gt;spark.comet.parquet.write.enabled=true&lt;/code&gt; and
+&lt;code&gt;spark.comet.operator.WriteFilesExec.allowIncompatible=true&lt;/code&gt;. Spark 3.4 and 3.5 keep the old write path.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; for this work, with reviews from &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;,
+&lt;a href="https://github.com/rich7420"&gt;@rich7420&lt;/a&gt;, and &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5763"&gt;#5763&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5369"&gt;#5369&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="s3-credentials"&gt;S3 Credentials&lt;a class="headerlink" href="#s3-credentials" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Comet's native S3 access now works with more of the credential setups Spark supports:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Built-in credential provider adapters.&lt;/strong&gt; In 1.0.0, a native Parquet scan failed with
+  &lt;code&gt;Unsupported credential provider&lt;/code&gt; for classes Spark accepts but Comet didn't reimplement, such as
+  &lt;code&gt;DefaultAWSCredentialsProviderChain&lt;/code&gt;. Two new adapters fix this by setting
+  &lt;code&gt;spark.hadoop.fs.s3a.comet.credential.provider.class&lt;/code&gt;.
+  &lt;code&gt;HadoopS3ACredentialProviderAdapter&lt;/code&gt; (recommended) uses Hadoop S3A's own provider chain, so it supports
+  everything S3A does, including web identity, assumed roles, and per-bucket settings.
+  &lt;code&gt;AwsSdkCredentialProviderAdapter&lt;/code&gt; wraps a specific AWS SDK provider class.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;STS throttling protection on EKS with IRSA.&lt;/strong&gt; When many executors start at once, STS can throttle their
+  credential requests. In 1.0.0, the native Iceberg path then fell back to the EKS node role, which usually can't
+  read the bucket, and the job failed with &lt;code&gt;403&lt;/code&gt; errors. Native Iceberg reads and writes now fetch web-identity
+  credentials themselves, retry throttled requests with backoff, never fall back to the node role, and share one
+  credential per executor. This happens automatically.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Per-location credentials.&lt;/strong&gt; A provider implementing &lt;code&gt;CometS3LocationScopedCredentialProvider&lt;/code&gt; can return
+  different credentials for different prefixes in the same bucket, such as &lt;code&gt;warehouse/sales&lt;/code&gt; and
+  &lt;code&gt;warehouse/finance&lt;/code&gt;.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Custom S3-compatible schemes.&lt;/strong&gt; Vendor filesystems that front an S3-compatible store with their own URL
+  scheme, such as &lt;code&gt;blob://&lt;/code&gt;, can be read natively by listing the scheme in
+  &lt;code&gt;spark.hadoop.fs.comet.s3Compliant.schemes&lt;/code&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;See the &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/s3-credential-providers.html"&gt;S3 credential providers guide&lt;/a&gt; for details.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt; for the credential adapters and STS throttling protection, &lt;a href="https://github.com/snmvaughan"&gt;@snmvaughan&lt;/a&gt; for
+per-location credentials, and &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt; for custom S3-compatible schemes, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and
+&lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/6023"&gt;#6023&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6025"&gt;#6025&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6031"&gt;#6031&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5314"&gt;#5314&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="performance"&gt;Performance&lt;a class="headerlink" href="#performance" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;h3 id="shuffle"&gt;Shuffle&lt;a class="headerlink" href="#shuffle" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;In 1.0.0, the native shuffle writer ran with a &lt;strong&gt;1-byte write buffer&lt;/strong&gt; by default: its 1 MiB default was
+  declared in MiB but passed to native code as bytes. It now gets the intended 1 MiB.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Round-robin repartitioning&lt;/strong&gt; assigns rows by position, as Spark does, instead of hashing every column of
+  every row. That's much cheaper on wide nested schemas, and it spreads repeated values across reducers the way
+  Spark does.&lt;/li&gt;
+&lt;li&gt;A task &lt;strong&gt;spills all of its shuffle partitions into one file&lt;/strong&gt; instead of one file each.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Zstd and Arrow IPC compression contexts are reused&lt;/strong&gt; across shuffle blocks.&lt;/li&gt;
+&lt;li&gt;Shuffle blocks are &lt;strong&gt;decoded against a cached schema&lt;/strong&gt; instead of re-parsing it for every block.&lt;/li&gt;
+&lt;li&gt;Per-partition scratch buffers are reused, and &lt;code&gt;ArrowWriter&lt;/code&gt; bulk-copies fixed-width columns.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Thanks to the contributors who drove this work, especially &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt;, &lt;a href="https://github.com/pingzh"&gt;@pingzh&lt;/a&gt;, and
+&lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;.&lt;/p&gt;
+&lt;h3 id="planning-and-execution"&gt;Planning and Execution&lt;a class="headerlink" href="#planning-and-execution" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Native dynamic filter pushdown&lt;/strong&gt;: a hash join's build-side keys filter the probe-side Parquet scan at
+  runtime, so the scan can skip data that can't match.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Adaptive partial aggregation&lt;/strong&gt; for eligible native shuffle plans: when grouping keys are nearly unique, the
+  partial aggregate passes rows through instead of building a hash table that doesn't reduce them.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Parsed plan data is cached&lt;/strong&gt; across a stage's tasks.&lt;/li&gt;
+&lt;li&gt;New &lt;strong&gt;native Parquet scan I/O and read-amplification metrics&lt;/strong&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="expressions"&gt;Expressions&lt;a class="headerlink" href="#expressions" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Many expression kernels got faster. In kernel microbenchmarks, &lt;code&gt;map_sort&lt;/code&gt; is up to 3x faster on multi-entry string
+maps and 18x faster on single-entry maps. The map lookup behind &lt;code&gt;element_at&lt;/code&gt; and &lt;code&gt;GetMapValue&lt;/code&gt; is vectorized,
+&lt;code&gt;collect_list&lt;/code&gt; and &lt;code&gt;collect_set&lt;/code&gt; have a native &lt;code&gt;GroupsAccumulator&lt;/code&gt;, and regex patterns are compiled once per
+planned expression. Smaller gains cover date and time field extraction, &lt;code&gt;posexplode&lt;/code&gt;, unnesting, nested hashing,
+decimal overflow checks, &lt;code&gt;list_extract&lt;/code&gt;, and approximate percentile merges.&lt;/p&gt;
+&lt;h2 id="expanded-coverage"&gt;Expanded Coverage&lt;a class="headerlink" href="#expanded-coverage" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;New native support includes the &lt;code&gt;mode&lt;/code&gt;, &lt;code&gt;max_by&lt;/code&gt; / &lt;code&gt;min_by&lt;/code&gt;, &lt;code&gt;listagg&lt;/code&gt; / &lt;code&gt;string_agg&lt;/code&gt; (Spark 4.0+), and
+&lt;code&gt;regr_*&lt;/code&gt; regression aggregates; &lt;code&gt;WindowGroupLimitExec&lt;/code&gt;; &lt;code&gt;explode_outer&lt;/code&gt;; &lt;code&gt;make_interval&lt;/code&gt;; native
+&lt;code&gt;sequence&lt;/code&gt; for integral types and &lt;code&gt;unbase64&lt;/code&gt;; &lt;code&gt;_metadata&lt;/code&gt; constant columns in the native Parquet
+scan; nested types as shuffle hash partitioning keys; &lt;code&gt;BinaryType&lt;/code&gt; in sort-merge joins; and Spark 4's
+&lt;code&gt;EmptyRelationExec&lt;/code&gt;.&lt;/p&gt;
+&lt;p&gt;More expressions now use &lt;a href="https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/"&gt;codegen dispatch&lt;/a&gt; by default, where Comet runs Spark's own generated code for an
+expression inside the native pipeline, so the result matches Spark without falling back. They include
+&lt;code&gt;translate&lt;/code&gt;, &lt;code&gt;to_csv&lt;/code&gt;, &lt;code&gt;encode&lt;/code&gt;, &lt;code&gt;lpad&lt;/code&gt; / &lt;code&gt;rpad&lt;/code&gt;, &lt;code&gt;round&lt;/code&gt; on floats, &lt;code&gt;abs&lt;/code&gt; on intervals, &lt;code&gt;timestampadd&lt;/code&gt; /
+&lt;code&gt;timestampdiff&lt;/code&gt;, &lt;code&gt;next_day&lt;/code&gt; and &lt;code&gt;levenshtein&lt;/code&gt; on collated input, and Spark expressions that call a Java or
+Scala method directly. &lt;code&gt;rlike&lt;/code&gt; runs natively by
+default for literal patterns that behave the same as in Java.&lt;/p&gt;
+&lt;p&gt;Spark 4 &lt;strong&gt;Variant&lt;/strong&gt; support also improved: native Parquet scans can project Variant columns directly. Thanks to
+&lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt; for driving Variant support, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5868"&gt;#5868&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5794"&gt;#5794&lt;/a&gt;.&lt;/p&gt;
+&lt;p&gt;There's also &lt;strong&gt;experimental native support for the in-memory cache&lt;/strong&gt; (&lt;code&gt;spark.comet.exec.inMemoryCache.enabled&lt;/code&gt;,
+off by default).&lt;/p&gt;
+&lt;h2 id="previewing-comet-plans"&gt;Previewing Comet Plans&lt;a class="headerlink" href="#previewing-comet-plans" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;The new &lt;code&gt;spark.comet.explain.planOnly.enabled&lt;/code&gt; setting logs the plan Comet would have run to the driver log,
+then lets Spark run the query as normal. It shows how much of a production workload Comet would accelerate, and
+why anything falls back, without running any of it through Comet.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for this feature, with reviews from &lt;a href="https://github.com/coderfender"&gt;@coderfender&lt;/a&gt; and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;. Related PRs:
+&lt;a href="https://github.com/apache/datafusion-comet/pull/5394"&gt;#5394&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="correctness-fixes"&gt;Correctness Fixes&lt;a class="headerlink" href="#correctness-fixes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;1.1.0 fixes many cases where Comet returned different results from Spark, failed where Spark succeeds, or
+accepted input that Spark rejects. Besides the Iceberg and memory fixes above, these are the ones most likely to
+affect 1.0.0 users. The &lt;a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md"&gt;change log&lt;/a&gt; has the rest.&lt;/p&gt;
+&lt;h3 id="wrong-results"&gt;Wrong results&lt;a class="headerlink" href="#wrong-results" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;Decimal &lt;code&gt;SUM&lt;/code&gt; returned NULL, or raised an overflow error under ANSI, when an intermediate sum overflowed
+  but the final result fit (&lt;a href="https://github.com/apache/datafusion-comet/pull/6041"&gt;#6041&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;After a late shuffle fallback, &lt;code&gt;avg&lt;/code&gt; could return NULL and &lt;code&gt;collect_list&lt;/code&gt; / &lt;code&gt;collect_set&lt;/code&gt; could produce
+  mismatched buffers (&lt;a href="https://github.com/apache/datafusion-comet/pull/5421"&gt;#5421&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Exchange reuse could share one shuffle between plans that differ, such as &lt;code&gt;COUNT(*) + 1&lt;/code&gt; and &lt;code&gt;COUNT(*) - 1&lt;/code&gt;,
+  semi and anti joins, or &lt;code&gt;explode&lt;/code&gt; and &lt;code&gt;explode_outer&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5470"&gt;#5470&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5828"&gt;#5828&lt;/a&gt;, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Two ABFS containers in the same storage account shared a cached object store, so a read could return the
+  other container's data (&lt;a href="https://github.com/apache/datafusion-comet/pull/5053"&gt;#5053&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Dictionary-encoded values hashed differently from the same values decoded, and a null struct hashed its
+  fields, which affected joins, aggregates and shuffle partitioning (&lt;a href="https://github.com/apache/datafusion-comet/pull/5757"&gt;#5757&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5754"&gt;#5754&lt;/a&gt;, &lt;a href="https://github.com/viirya"&gt;@viirya&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Parquet field names containing non-ASCII characters that differ only in case read as NULL (&lt;a href="https://github.com/apache/datafusion-comet/pull/5602"&gt;#5602&lt;/a&gt;,
+  &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;&lt;code&gt;IN&lt;/code&gt;, &lt;code&gt;InSet&lt;/code&gt;, nested &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;arrays_overlap&lt;/code&gt; and &lt;code&gt;array_position&lt;/code&gt; now treat &lt;code&gt;-0.0&lt;/code&gt; and &lt;code&gt;0.0&lt;/code&gt;, and every NaN
+  encoding, as Spark does (&lt;a href="https://github.com/apache/datafusion-comet/pull/6073"&gt;#6073&lt;/a&gt;, &lt;a href="https://github.com/mizulun"&gt;@mizulun&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5235"&gt;#5235&lt;/a&gt;, &lt;a href="https://github.com/divyankshah"&gt;@divyankshah&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5472"&gt;#5472&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Decimal to double and float casts were off by one unit in the last place for most &lt;code&gt;DECIMAL(38,18)&lt;/code&gt; values
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5684"&gt;#5684&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;String to timestamp casts now follow Spark's parsing rules for short fields, time zones and signed years
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5682"&gt;#5682&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5858"&gt;#5858&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="errors-spark-raises-that-comet-did-not"&gt;Errors Spark raises that Comet did not&lt;a class="headerlink" href="#errors-spark-raises-that-comet-did-not" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;Casts and expressions routed through codegen dispatch could skip ANSI errors raised inside a constant
+  subexpression (&lt;a href="https://github.com/apache/datafusion-comet/pull/5623"&gt;#5623&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Rejected &lt;code&gt;TIMESTAMP_NTZ&lt;/code&gt; casts returned NULL under ANSI instead of raising &lt;code&gt;CAST_INVALID_INPUT&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5752"&gt;#5752&lt;/a&gt;,
+  &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Out-of-range Parquet &lt;code&gt;TIMESTAMP_MILLIS&lt;/code&gt; values, top-level or nested, silently wrapped (&lt;a href="https://github.com/apache/datafusion-comet/pull/5177"&gt;#5177&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5740"&gt;#5740&lt;/a&gt;,
+  &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Nested Parquet struct, list and map fields now follow Spark's conversion rules instead of returning NULL on
+  overflow or accepting values Spark rejects (&lt;a href="https://github.com/apache/datafusion-comet/pull/5681"&gt;#5681&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="query-failures-and-crashes"&gt;Query failures and crashes&lt;a class="headerlink" href="#query-failures-and-crashes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;code&gt;collect_list&lt;/code&gt; and &lt;code&gt;collect_set&lt;/code&gt; over nested arguments failed with "column types must match schema types"
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5159"&gt;#5159&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Native shuffle failed with a 2 GB task serialization error on jobs with very many partitions (&lt;a href="https://github.com/apache/datafusion-comet/pull/5392"&gt;#5392&lt;/a&gt;,
+  &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;A Scala UDF from a user jar failed with a &lt;code&gt;ClassCastException&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5282"&gt;#5282&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;&lt;code&gt;rpad&lt;/code&gt; and &lt;code&gt;lpad&lt;/code&gt; panicked on a NULL length (&lt;a href="https://github.com/apache/datafusion-comet/pull/5680"&gt;#5680&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Structs with duplicate field names failed the task in native shuffle, and panicked in the native Parquet
+  scan (&lt;a href="https://github.com/apache/datafusion-comet/pull/5866"&gt;#5866&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5786"&gt;#5786&lt;/a&gt;, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="hangs-and-resource-use"&gt;Hangs and resource use&lt;a class="headerlink" href="#hangs-and-resource-use" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;The JVM hung on exit when an application returned from &lt;code&gt;main&lt;/code&gt; without calling &lt;code&gt;spark.stop()&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5748"&gt;#5748&lt;/a&gt;,
+  &lt;a href="https://github.com/zhangfengcdt"&gt;@zhangfengcdt&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;The native scan busy-polled while waiting on S3 or HDFS reads, keeping one core per task at 100% (&lt;a href="https://github.com/apache/datafusion-comet/pull/6219"&gt;#6219&lt;/a&gt;,
+  &lt;a href="https://github.com/mixermt"&gt;@mixermt&lt;/a&gt; and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;One task could force-spill another task's shuffle buffers, and a failed shuffle write leaked its memory
+  reservation (&lt;a href="https://github.com/apache/datafusion-comet/pull/5493"&gt;#5493&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5461"&gt;#5461&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="upgrading-to-110"&gt;Upgrading to 1.1.0&lt;a class="headerlink" href="#upgrading-to-110" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;A few changes in 1.1.0 can affect a deployment. The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html"&gt;Comet Upgrade Guide&lt;/a&gt; has the details.&lt;/p&gt;
+&lt;h3 id="platform"&gt;Platform&lt;a class="headerlink" href="#platform" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;JDK 17 or later is required.&lt;/strong&gt; JDK 11 support was removed, as announced in 1.0.0.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.4 is still deprecated.&lt;/strong&gt; Comet still publishes Spark 3.4 binaries, but Spark's SQL test suite runs
+  against 3.4 only on demand, so 3.4-specific regressions are more likely. We recommend Spark 3.5 or later.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="size-settings-now-read-correctly"&gt;Size settings now read correctly&lt;a class="headerlink" href="#size-settings-now-read-correctly" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;1.0.0 misread three size settings, so jobs that set them may behave differently:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.memory.offHeap.size&lt;/code&gt; as a bare number of bytes&lt;/strong&gt; was read as MiB, which made &lt;code&gt;fair_unified&lt;/code&gt;'s
+  per-operator shares about a million times too large. With correct shares, operators can spill sooner, and an
+  operator that can't spill can fail when it exceeds its share. Sizes with a unit, like &lt;code&gt;16g&lt;/code&gt;, aren't affected.
+  &lt;code&gt;spark.comet.exec.memoryPool=greedy_unified&lt;/code&gt; restores the old behavior.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.maxTempDirectorySize&lt;/code&gt; with a unit&lt;/strong&gt;, like &lt;code&gt;10g&lt;/code&gt;, was ignored in favor of the 100 GB default.
+  It's now enforced, so a query that spills past it fails.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.shuffle.native.writeBufferSize&lt;/code&gt; with a unit&lt;/strong&gt; was treated as bytes, so &lt;code&gt;64m&lt;/code&gt; meant 64 bytes. It
+  now means what it says. These buffers use untracked native memory, so check large values against
+  &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Malformed values for &lt;code&gt;spark.comet.maxTempDirectorySize&lt;/code&gt; and &lt;code&gt;spark.comet.explain.native.enabled&lt;/code&gt; now fail the
+query instead of falling back to the default.&lt;/p&gt;
+&lt;h3 id="more-memory-before-spilling"&gt;More memory before spilling&lt;a class="headerlink" href="#more-memory-before-spilling" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;With the &lt;code&gt;fair_unified&lt;/code&gt; fix, tasks with several operators can reserve more memory before spilling than in
+0.15.0 through 1.0.0, especially on executors running few tasks at once. If you sized executors for one of those
+releases, check the memory usage log for headroom.&lt;/p&gt;
+&lt;h3 id="conditions-for-enabling-comet"&gt;Conditions for enabling Comet&lt;a class="headerlink" href="#conditions-for-enabling-comet" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet requires Spark's off-heap memory. &lt;code&gt;CometPlugin&lt;/code&gt; already enforced this, and 1.1.0 applies the same check
+when &lt;code&gt;CometSparkSessionExtensions&lt;/code&gt; is registered directly, disabling Comet with a warning. Both checks read
+&lt;code&gt;spark.memory.offHeap.enabled&lt;/code&gt; from the SparkContext, so set it when the application starts, not on a
+&lt;code&gt;SparkSession.builder&lt;/code&gt; after the context exists.&lt;/p&gt;
+&lt;p&gt;Comet also checks the shuffle manager that's actually running, not the session's &lt;code&gt;spark.shuffle.manager&lt;/code&gt;. A
+session that names &lt;code&gt;CometShuffleManager&lt;/code&gt; after the context started with a different one now runs without Comet,
+with a warning, instead of failing with a &lt;code&gt;ClassCastException&lt;/code&gt;.&lt;/p&gt;
+&lt;h3 id="deprecated-and-removed-settings"&gt;Deprecated and removed settings&lt;a class="headerlink" href="#deprecated-and-removed-settings" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.exec.memoryPool.fraction&lt;/code&gt;&lt;/strong&gt; is deprecated and will be removed in a future major release. It
+  still works, and the driver warns when it's set.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Testing-only on-heap settings&lt;/strong&gt; were removed: &lt;code&gt;spark.comet.memoryOverhead&lt;/code&gt;,
+  &lt;code&gt;spark.comet.exec.onHeap.memoryPool&lt;/code&gt;, and &lt;code&gt;spark.comet.shuffle.jvm.memoryFactor&lt;/code&gt; (and its old name,
+  &lt;code&gt;spark.comet.columnar.shuffle.memory.factor&lt;/code&gt;). The &lt;a href="https://datafusion.apache.org/comet/about/versioning_policy.html#testing-and-internal-configurations-are-exempt"&gt;versioning policy&lt;/a&gt; exempts testing settings, and Comet
+  ignores them if they're still set.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="compatibility"&gt;Compatibility&lt;a class="headerlink" href="#compatibility" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Supported platforms:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.4.3&lt;/strong&gt; with Java 17 and Scala 2.12/2.13 (deprecated)&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.5.9&lt;/strong&gt; with Java 17 and Scala 2.12/2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.0.4&lt;/strong&gt; with Java 17/21 and Scala 2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.1.3&lt;/strong&gt; with Java 17/21 and Scala 2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.2.0&lt;/strong&gt; with Java 17 and Scala 2.13 (experimental, for early evaluation only)&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;See the &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html"&gt;Spark Version Compatibility&lt;/a&gt; page for known limitations specific to each version.&lt;/p&gt;
+&lt;p&gt;This release builds on &lt;strong&gt;DataFusion 55.1&lt;/strong&gt; and &lt;strong&gt;Arrow 59.2&lt;/strong&gt;.&lt;/p&gt;
+&lt;h2 id="get-started-with-comet-110"&gt;Get Started with Comet 1.1.0&lt;a class="headerlink" href="#get-started-with-comet-110" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Follow the &lt;a href="https://datafusion.apache.org/comet/user-guide/1.1/installation.html"&gt;Comet 1.1.0 Installation Guide&lt;/a&gt;
+to get up and running, then point Comet at your existing Spark workloads.&lt;/p&gt;</content><category term="blog"/></entry><entry><title>Apache DataFusion 55.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="alternate"/><published>2026-08-25T00:00:00+00:00</published><updated>2026-08-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-08-25:/blog/2026/08/25/datafusion-55.0.0</id><summary type= [...]
 {% comment %}
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
diff --git a/output/feeds/blog.atom.xml b/output/feeds/blog.atom.xml
index 8e2b3d0..68c3a52 100644
--- a/output/feeds/blog.atom.xml
+++ b/output/feeds/blog.atom.xml
@@ -1,5 +1,430 @@
 <?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-25T00:00:00+00:00</updated><entry><title>Apache DataFusion 55.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="alternate"/><published>2026-08-25T0 [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - blog</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/blog.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-10-01T00:00:00+00:00</updated><entry><title>Apache DataFusion Comet 1.1.0 Release</title><link href="https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0" rel="alternate"/><published>202 [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development …&lt;/p&gt;</summary><content type="html">&lt;!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development since 1.0.0: 398 commits from 40 contributors. See the
+&lt;a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md"&gt;change log&lt;/a&gt; for the full list of changes.&lt;/p&gt;
+&lt;p&gt;The highlights:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Native Iceberg writes&lt;/strong&gt; (experimental): Comet can write Iceberg data files with iceberg-rust instead of
+  iceberg-java.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Memory management&lt;/strong&gt;: Comet now measures the native memory its pools don't track, logs it on every executor,
+  and fixes several long-standing pool accounting bugs.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Native shuffle over Apache Celeborn&lt;/strong&gt;: 1.1.0 includes the native writer and reader, but released Celeborn
+  clients currently fall back to ordinary Spark/Celeborn shuffle.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Native Parquet writes on Spark 4.0+&lt;/strong&gt; (experimental), now built on Spark's own write path.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="native-iceberg-writes-experimental"&gt;Native Iceberg Writes (Experimental)&lt;a class="headerlink" href="#native-iceberg-writes-experimental" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Until now, Comet accelerated Iceberg reads but left writes to the JVM. In 1.1.0, Comet can write Iceberg data
+files natively using &lt;a href="https://github.com/apache/iceberg-rust"&gt;iceberg-rust&lt;/a&gt;. The feature is &lt;strong&gt;experimental and
+disabled by default&lt;/strong&gt;, and it only engages under fairly strict conditions.&lt;/p&gt;
+&lt;h3 id="splitting-the-write-operator"&gt;Splitting the write operator&lt;a class="headerlink" href="#splitting-the-write-operator" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
+and validates against the catalog. Because file writing is bundled with the metadata and commit steps, there was
+no separate piece for Comet to replace.&lt;/p&gt;
+&lt;p&gt;Setting &lt;code&gt;spark.comet.write.iceberg.splitOperator.enabled=true&lt;/code&gt; splits eligible Iceberg writes into two operators:&lt;/p&gt;
+&lt;ol&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;IcebergWrite&lt;/code&gt;&lt;/strong&gt; writes data files on the executors and returns each task's commit message.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;IcebergCommit&lt;/code&gt;&lt;/strong&gt; collects the commit messages on the driver and performs the normal Iceberg commit, once.&lt;/li&gt;
+&lt;/ol&gt;
+&lt;p&gt;With only the split enabled, iceberg-java still writes the files; the native writer, described next, replaces
+that step. The split covers &lt;code&gt;INSERT INTO&lt;/code&gt; and DataFrame &lt;code&gt;append&lt;/code&gt;, static and dynamic &lt;code&gt;INSERT OVERWRITE&lt;/code&gt;, and
+copy-on-write &lt;code&gt;DELETE&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, and &lt;code&gt;MERGE&lt;/code&gt;, on every supported Spark version. Merge-on-read writes are left
+alone. When Comet can't split a write (an unrecognized write class, CTAS on Spark 3.4, or a write that needs
+Spark's commit coordinator), it plans the write exactly as if Comet weren't there.&lt;/p&gt;
+&lt;h3 id="writing-parquet-natively"&gt;Writing Parquet natively&lt;a class="headerlink" href="#writing-parquet-natively" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;The native writer requires the split. Also setting &lt;code&gt;spark.comet.iceberg.write.enabled=true&lt;/code&gt; hands each task's
+Parquet writing to iceberg-rust. The driver passes the native writer everything it needs: schema, partition spec,
+data location, Parquet settings, writer mode, and object-store configuration. Each task writes its files and
+returns their metadata as an in-memory Iceberg manifest.&lt;/p&gt;
+&lt;p&gt;From there, iceberg-java takes over again. The JVM reads the manifest, recomputes each file's metrics from its
+Parquet footer, and builds the same commit message the JVM writer would have produced. Snapshots, manifest
+lists, commit validation, and retries all work as before.&lt;/p&gt;
+&lt;p&gt;The native writer consumes Arrow batches from a Comet operator, so the query feeding the write must run in Comet
+too. Writes from a local relation, such as &lt;code&gt;INSERT ... VALUES&lt;/code&gt;, also need
+&lt;code&gt;spark.comet.exec.localTableScan.enabled=true&lt;/code&gt;.&lt;/p&gt;
+&lt;p&gt;To see which path a write took, check the physical plan: a native write shows &lt;code&gt;CometIcebergWrite&lt;/code&gt; under
+&lt;code&gt;IcebergCommit&lt;/code&gt;, and a write that fell back shows &lt;code&gt;IcebergWrite&lt;/code&gt;.&lt;/p&gt;
+&lt;h3 id="matching-iceberg-java"&gt;Matching iceberg-java&lt;a class="headerlink" href="#matching-iceberg-java" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;The native writer aims to write the same table iceberg-java would, down to the metadata. Manifest metrics drive
+pruning for every future reader, so rather than trust the native writer's numbers, Comet recomputes them with
+iceberg-java's own code. Parity tests write the same rows through both writers and compare the committed counts
+and bounds. The cost is one small footer read per file.&lt;/p&gt;
+&lt;p&gt;Eligibility is an allowlist. A write goes native only if its whole configuration matches the documented set of
+supported settings. Anything else, including properties added by future Iceberg versions, falls back to
+iceberg-java and reports why in Comet's extended &lt;code&gt;EXPLAIN&lt;/code&gt; output. Encryption, object-storage layout, custom
+location providers, bloom filters, and unrecognized &lt;code&gt;parquet.*&lt;/code&gt; properties all fall back. For a partitioned
+table, Iceberg asks Spark to cluster and sort rows by partition value, such as &lt;code&gt;bucket(16, id)&lt;/code&gt; or &lt;code&gt;days(ts)&lt;/code&gt;,
+before writing. That step must also run natively, which it now can because Iceberg's system functions have
+native implementations.&lt;/p&gt;
+&lt;p&gt;Comet decides all of this at planning time, including checking that every iceberg-java class it relies on is
+where it expects. An Iceberg release that moves one falls back instead of failing tasks mid-write. CI also runs
+Apache Iceberg's own Spark test suites, for Iceberg 1.8.1 through 1.11.0, with the native writer enabled.&lt;/p&gt;
+&lt;h3 id="failure-handling"&gt;Failure handling&lt;a class="headerlink" href="#failure-handling" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Only successful tasks' commit messages are committed. A failed task deletes the files it wrote, as iceberg-java
+does, and a failed job commits nothing and deletes the completed tasks' files. Retries can't collide, because
+each attempt's ID is part of its file names. Anything cleanup misses is invisible to readers and is removed by
+Iceberg's normal &lt;code&gt;remove_orphan_files&lt;/code&gt; maintenance.&lt;/p&gt;
+&lt;h3 id="known-differences"&gt;Known differences&lt;a class="headerlink" href="#known-differences" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Files written by parquet-rs, the Rust Parquet library iceberg-rust uses, differ from those written by
+parquet-java (formerly parquet-mr), which iceberg-java uses. Enabling the feature accepts these differences.
+Most are cosmetic, such as footer metadata, &lt;code&gt;created_by&lt;/code&gt;, and page encoding labels. Two are worth knowing about:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Files won't split at the same points.&lt;/strong&gt; Both writers check file size every 1000 rows, but they estimate it
+  differently, so they can roll over to a new file at different rows.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;High-cardinality columns keep a dictionary page.&lt;/strong&gt; parquet-java drops dictionary encoding early when it isn't
+  saving space, while parquet-rs keeps it up to &lt;code&gt;write.parquet.dict-size-bytes&lt;/code&gt;. Results are the same, but
+  selective reads of native-written files fetch more bytes
+  (&lt;a href="https://github.com/apache/datafusion-comet/issues/6114"&gt;#6114&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/iceberg-writes.html"&gt;Iceberg Writes guide&lt;/a&gt; lists every supported setting and every difference. Please try it on a non-production
+table and tell us how it goes. Feedback from real workloads is what this feature needs before it loses the
+experimental label.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/jordepic"&gt;@jordepic&lt;/a&gt; for designing and implementing the split-operator plan, write detection, and the native
+writer, and to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for the fidelity and failure-handling work, with contributions from
+&lt;a href="https://github.com/zhangfengcdt"&gt;@zhangfengcdt&lt;/a&gt;, &lt;a href="https://github.com/snmvaughan"&gt;@snmvaughan&lt;/a&gt;, &lt;a href="https://github.com/liupoyi-1031"&gt;@liupoyi-1031&lt;/a&gt;, and &lt;a href="https://github.com/0lai0"&gt;@0lai0&lt;/a&gt;, and reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;,
+&lt;a href="https://github.com/unikdahal"&gt;@unikdahal&lt;/a&gt;, and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/4658"&gt;#4658&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5298"&gt;#5298&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5361"&gt;#5361&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5663"&gt;#5663&lt;/a& [...]
+&lt;h2 id="memory-management"&gt;Memory Management&lt;a class="headerlink" href="#memory-management" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;A recurring problem for Comet users has been executors killed by the cluster manager (on Kubernetes,
+&lt;code&gt;ExecutorLostFailure&lt;/code&gt; with exit code 137) even though Comet stayed within its memory pool. 1.1.0 explains why
+this happens, lets you measure it, and fixes pool bugs that made it worse.&lt;/p&gt;
+&lt;h3 id="where-the-memory-goes"&gt;Where the memory goes&lt;a class="headerlink" href="#where-the-memory-goes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet's native operators allocate from the Rust heap but charge their reservations against Spark's off-heap
+pool, sized by &lt;code&gt;spark.memory.offHeap.size&lt;/code&gt;. Operators only reserve memory for data they deliberately hold onto:
+sort buffers, hash join build sides, aggregation state, and buffered shuffle partitions.&lt;/p&gt;
+&lt;p&gt;Plenty of memory is never reserved: working memory in expression kernels, decompression buffers, Parquet reader
+state, object store buffers, JVM-side Arrow buffers, and allocator overhead. All of it has to fit in
+&lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;, and until now there was no way to see how much of it there was.&lt;/p&gt;
+&lt;p&gt;&lt;img alt="The executor container holds the JVM heap, the off-heap memory pool, and the memory overhead. Spark and Comet share the off-heap pool, where Comet's sorts, joins, aggregations, and shuffles reserve memory. The memory overhead holds the JVM's own overhead plus Comet's native memory that the pool does not track." class="img-fluid" src="/blog/images/comet-1.1.0/comet-executor-memory.svg" width="100%"/&gt;&lt;/p&gt;
+&lt;h3 id="measuring-used-memory"&gt;Measuring used memory&lt;a class="headerlink" href="#measuring-used-memory" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet now wraps its native allocator in a counter that tracks every byte allocated and not yet freed. It only
+observes and never rejects an allocation. Arrow buffers the JVM imports from native code are now tracked
+separately, so they aren't counted twice.&lt;/p&gt;
+&lt;p&gt;Each executor uses the counter to log its native memory usage every 10 seconds while Comet runs:&lt;/p&gt;
+&lt;pre&gt;&lt;code&gt;Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native plans, 8 memory pools)
+&lt;/code&gt;&lt;/pre&gt;
+&lt;p&gt;&lt;code&gt;reserved&lt;/code&gt; is what the pools track, and the container already has room for it. &lt;code&gt;allocated&lt;/code&gt; is everything
+Comet's native code holds. The difference is what has to fit in &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;, next to the
+JVM's own non-heap memory. To size the overhead, run a representative workload, take the largest difference from
+the log, add it to the overhead you had before Comet, and leave a margin. Setting
+&lt;code&gt;spark.comet.memory.logInterval=1s&lt;/code&gt; for that run makes it less likely to miss a short peak.&lt;/p&gt;
+&lt;p&gt;The executor also warns when its native memory looks larger than the container allows, and the
+&lt;a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/memory.html"&gt;tuning guide&lt;/a&gt; walks through sizing with examples. One gotcha: setting &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;
+&lt;em&gt;replaces&lt;/em&gt; the value Spark derives from &lt;code&gt;spark.executor.memoryOverheadFactor&lt;/code&gt; instead of adding to it, so on a
+large executor it can shrink the container. For large executors, raise the factor instead.&lt;/p&gt;
+&lt;h3 id="memory-pool-fixes"&gt;Memory pool fixes&lt;a class="headerlink" href="#memory-pool-fixes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet's off-heap memory comes from one of two pools. &lt;code&gt;fair_unified&lt;/code&gt;, the default, caps each operator at an even
+share of the task's memory. &lt;code&gt;greedy_unified&lt;/code&gt; gives memory to operators first come, first served. When Spark
+grants an operator less memory than it asked for (a partial grant), the operator spills.&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;fair_unified&lt;/code&gt; limited a whole task to one operator's share.&lt;/strong&gt; Since 0.15.0, the pool compared all of a
+  task's reservations against a single operator's share, and each new operator shrank the limit for the rest.
+  Each operator now gets its own share, so tasks with several operators can use more memory before spilling.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;A partial grant from Spark no longer panics the task.&lt;/strong&gt; DataFusion sometimes has to record memory that
+  already exists, such as a spilled batch being read back, and both pools panicked when Spark granted less than
+  they asked for. They now track the shortfall and repay it before returning memory to Spark.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Leaks on failure paths.&lt;/strong&gt; A plan that failed during setup or teardown could leak its task's memory pool, and
+  a failed Arrow import leaked the vectors it had already imported.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Configuration units.&lt;/strong&gt; 1.0.0 misread three size settings, including &lt;code&gt;spark.memory.offHeap.size&lt;/code&gt; written as
+  a bare number. See &lt;a href="#upgrading-to-110"&gt;Upgrading to 1.1.0&lt;/a&gt; for what changes.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Less log noise when spilling.&lt;/strong&gt; 1.0.0 logged a warning and a memory dump for every partial grant, so a
+  spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
+  deadlock the task.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Metrics.&lt;/strong&gt; Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+  task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+  1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+  in the SQL tab.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;&lt;code&gt;spark.comet.exec.memoryPool.fraction&lt;/code&gt; is deprecated. It was meant to leave room for untracked memory, but Spark
+hands out the whole pool anyway, so it never did. Size &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt; instead.&lt;/p&gt;
+&lt;h3 id="on-heap-mode"&gt;On-heap mode&lt;a class="headerlink" href="#on-heap-mode" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;On-heap mode exists so that Spark's and Iceberg's test suites can run against Comet; production runs off-heap.
+Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes that
+accounting, along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
+off-heap memory however it's enabled, including when &lt;code&gt;CometSparkSessionExtensions&lt;/code&gt; is registered directly, and
+disables itself with a warning otherwise.&lt;/p&gt;
+&lt;p&gt;Contributors can find more detail in the new &lt;a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html"&gt;memory management guide&lt;/a&gt;.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for driving this work, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt; for the memory pool lifecycle and shuffle spill
+accounting fixes, &lt;a href="https://github.com/ywskycn"&gt;@ywskycn&lt;/a&gt; for reporting native memory usage to Spark, &lt;a href="https://github.com/1fanwang"&gt;@1fanwang&lt;/a&gt; for the Arrow import leak
+fix, and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; for the native aggregate spill metrics, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;,
+&lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;, and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5934"&gt;#5934&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6162"&gt;#6162&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6048"&gt;#6048&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6128"&gt;#6128&lt;/a&gt [...]
+&lt;h2 id="more-iceberg-improvements"&gt;More Iceberg Improvements&lt;a class="headerlink" href="#more-iceberg-improvements" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;V3 deletion vectors&lt;/strong&gt; are applied on native scans.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Iceberg system functions&lt;/strong&gt; (&lt;code&gt;bucket&lt;/code&gt;, &lt;code&gt;truncate&lt;/code&gt;, &lt;code&gt;years&lt;/code&gt;, &lt;code&gt;months&lt;/code&gt;, &lt;code&gt;days&lt;/code&gt;, and &lt;code&gt;hours&lt;/code&gt;) run natively.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Scan planning metrics and scan time&lt;/strong&gt; appear in the Spark UI for native Iceberg scans.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;A wrong-results fix.&lt;/strong&gt; A filter such as &lt;code&gt;bucket(4, id) = 2&lt;/code&gt; combined with another predicate was pushed to
+  the native scan as &lt;code&gt;id = 2&lt;/code&gt;, returning too few rows.&lt;/li&gt;
+&lt;li&gt;Tables partitioned by an unknown transform can be read natively, and &lt;code&gt;IS NULL&lt;/code&gt; / &lt;code&gt;IS NOT NULL&lt;/code&gt; checks on list
+  and map columns no longer force a fallback.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt; for deletion vector support, &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt; for the scan metrics, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt; for the
+null-check fix, and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for the native system functions and residual fix, with reviews from
+&lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/rich7420"&gt;@rich7420&lt;/a&gt;, &lt;a href="https://github.com/unikdahal"&gt;@unikdahal&lt;/a&gt;, and &lt;a href="https://github.com/jordepic"&gt;@jordepic&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5853"&gt;#5853&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5638"&gt;#5638&lt;/a&gt;, &lt;a href="https://github.com/apache/dataf [...]
+&lt;h2 id="remote-shuffle-with-celeborn"&gt;Remote Shuffle with Celeborn&lt;a class="headerlink" href="#remote-shuffle-with-celeborn" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;1.1.0 adds Comet's side of native shuffle over Apache Celeborn: map tasks push Comet's Arrow data straight to
+Celeborn, and reducers read it back natively. In 1.0.0, Celeborn users always got ordinary Spark shuffle.&lt;/p&gt;
+&lt;p&gt;Native Celeborn shuffle is unavailable with released Celeborn 0.6.x and 0.7.x clients in Comet 1.1.0 because
+these clients do not pass Comet's push-completion compatibility checks. Set &lt;code&gt;spark.shuffle.manager&lt;/code&gt; to
+&lt;code&gt;org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager&lt;/code&gt; to use Comet alongside
+ordinary Spark/Celeborn shuffle; Comet can still accelerate other operators. These clients retain
+ordinary Spark/Celeborn shuffle even when &lt;code&gt;spark.comet.shuffle.mode=native&lt;/code&gt; is requested, and the
+default &lt;code&gt;auto&lt;/code&gt; mode also retains ordinary shuffle. Follow-up work to enable native shuffle with
+released clients is tracked in &lt;a href="https://github.com/apache/datafusion-comet/issues/6523"&gt;#6523&lt;/a&gt;.
+Native shuffle over Celeborn does not support &lt;code&gt;spark.io.encryption.enabled=true&lt;/code&gt;, and Celeborn
+is not bundled with Comet. The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/celeborn.html"&gt;Celeborn tuning guide&lt;/a&gt; has the full requirements.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/pingzh"&gt;@pingzh&lt;/a&gt; for this work, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/ziting-openai"&gt;@ziting-openai&lt;/a&gt;, and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;. Related
+PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5473"&gt;#5473&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5481"&gt;#5481&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5513"&gt;#5513&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5531"&gt;#5531&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5537"&gt;#5537&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="native-parquet-writes-on-spark-40-experimental"&gt;Native Parquet Writes on Spark 4.0+ (Experimental)&lt;a class="headerlink" href="#native-parquet-writes-on-spark-40-experimental" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Native Parquet writes remain &lt;strong&gt;experimental and disabled by default&lt;/strong&gt;, but on Spark 4.0+ they now build on
+Spark's own write path.&lt;/p&gt;
+&lt;p&gt;In 1.0.0, a native write replaced Spark's entire write command, so Comet had to reimplement the commit protocol,
+save modes, and job commit. On Spark 4.0+, Comet replaces only &lt;code&gt;WriteFilesExec&lt;/code&gt;, the per-task write step that
+Spark 4.0 made pluggable, and Spark handles the rest. That means:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;File names come from Spark's commit protocol, so committers that track individual files, like the S3A magic
+  committer, work.&lt;/li&gt;
+&lt;li&gt;Column names, nullability, and field IDs come from the target table, not the query.&lt;/li&gt;
+&lt;li&gt;Bytes-written and rows-written metrics are correct on HDFS.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;The writer handles non-partitioned, non-bucketed writes to local and HDFS paths, and skips writes that set
+&lt;code&gt;spark.sql.files.maxRecordsPerFile&lt;/code&gt;. To enable it, set &lt;code&gt;spark.comet.parquet.write.enabled=true&lt;/code&gt; and
+&lt;code&gt;spark.comet.operator.WriteFilesExec.allowIncompatible=true&lt;/code&gt;. Spark 3.4 and 3.5 keep the old write path.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; for this work, with reviews from &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;,
+&lt;a href="https://github.com/rich7420"&gt;@rich7420&lt;/a&gt;, and &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5763"&gt;#5763&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5369"&gt;#5369&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="s3-credentials"&gt;S3 Credentials&lt;a class="headerlink" href="#s3-credentials" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Comet's native S3 access now works with more of the credential setups Spark supports:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Built-in credential provider adapters.&lt;/strong&gt; In 1.0.0, a native Parquet scan failed with
+  &lt;code&gt;Unsupported credential provider&lt;/code&gt; for classes Spark accepts but Comet didn't reimplement, such as
+  &lt;code&gt;DefaultAWSCredentialsProviderChain&lt;/code&gt;. Two new adapters fix this by setting
+  &lt;code&gt;spark.hadoop.fs.s3a.comet.credential.provider.class&lt;/code&gt;.
+  &lt;code&gt;HadoopS3ACredentialProviderAdapter&lt;/code&gt; (recommended) uses Hadoop S3A's own provider chain, so it supports
+  everything S3A does, including web identity, assumed roles, and per-bucket settings.
+  &lt;code&gt;AwsSdkCredentialProviderAdapter&lt;/code&gt; wraps a specific AWS SDK provider class.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;STS throttling protection on EKS with IRSA.&lt;/strong&gt; When many executors start at once, STS can throttle their
+  credential requests. In 1.0.0, the native Iceberg path then fell back to the EKS node role, which usually can't
+  read the bucket, and the job failed with &lt;code&gt;403&lt;/code&gt; errors. Native Iceberg reads and writes now fetch web-identity
+  credentials themselves, retry throttled requests with backoff, never fall back to the node role, and share one
+  credential per executor. This happens automatically.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Per-location credentials.&lt;/strong&gt; A provider implementing &lt;code&gt;CometS3LocationScopedCredentialProvider&lt;/code&gt; can return
+  different credentials for different prefixes in the same bucket, such as &lt;code&gt;warehouse/sales&lt;/code&gt; and
+  &lt;code&gt;warehouse/finance&lt;/code&gt;.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Custom S3-compatible schemes.&lt;/strong&gt; Vendor filesystems that front an S3-compatible store with their own URL
+  scheme, such as &lt;code&gt;blob://&lt;/code&gt;, can be read natively by listing the scheme in
+  &lt;code&gt;spark.hadoop.fs.comet.s3Compliant.schemes&lt;/code&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;See the &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/s3-credential-providers.html"&gt;S3 credential providers guide&lt;/a&gt; for details.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt; for the credential adapters and STS throttling protection, &lt;a href="https://github.com/snmvaughan"&gt;@snmvaughan&lt;/a&gt; for
+per-location credentials, and &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt; for custom S3-compatible schemes, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and
+&lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/6023"&gt;#6023&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6025"&gt;#6025&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6031"&gt;#6031&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5314"&gt;#5314&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="performance"&gt;Performance&lt;a class="headerlink" href="#performance" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;h3 id="shuffle"&gt;Shuffle&lt;a class="headerlink" href="#shuffle" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;In 1.0.0, the native shuffle writer ran with a &lt;strong&gt;1-byte write buffer&lt;/strong&gt; by default: its 1 MiB default was
+  declared in MiB but passed to native code as bytes. It now gets the intended 1 MiB.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Round-robin repartitioning&lt;/strong&gt; assigns rows by position, as Spark does, instead of hashing every column of
+  every row. That's much cheaper on wide nested schemas, and it spreads repeated values across reducers the way
+  Spark does.&lt;/li&gt;
+&lt;li&gt;A task &lt;strong&gt;spills all of its shuffle partitions into one file&lt;/strong&gt; instead of one file each.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Zstd and Arrow IPC compression contexts are reused&lt;/strong&gt; across shuffle blocks.&lt;/li&gt;
+&lt;li&gt;Shuffle blocks are &lt;strong&gt;decoded against a cached schema&lt;/strong&gt; instead of re-parsing it for every block.&lt;/li&gt;
+&lt;li&gt;Per-partition scratch buffers are reused, and &lt;code&gt;ArrowWriter&lt;/code&gt; bulk-copies fixed-width columns.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Thanks to the contributors who drove this work, especially &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt;, &lt;a href="https://github.com/pingzh"&gt;@pingzh&lt;/a&gt;, and
+&lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;.&lt;/p&gt;
+&lt;h3 id="planning-and-execution"&gt;Planning and Execution&lt;a class="headerlink" href="#planning-and-execution" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Native dynamic filter pushdown&lt;/strong&gt;: a hash join's build-side keys filter the probe-side Parquet scan at
+  runtime, so the scan can skip data that can't match.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Adaptive partial aggregation&lt;/strong&gt; for eligible native shuffle plans: when grouping keys are nearly unique, the
+  partial aggregate passes rows through instead of building a hash table that doesn't reduce them.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Parsed plan data is cached&lt;/strong&gt; across a stage's tasks.&lt;/li&gt;
+&lt;li&gt;New &lt;strong&gt;native Parquet scan I/O and read-amplification metrics&lt;/strong&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="expressions"&gt;Expressions&lt;a class="headerlink" href="#expressions" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Many expression kernels got faster. In kernel microbenchmarks, &lt;code&gt;map_sort&lt;/code&gt; is up to 3x faster on multi-entry string
+maps and 18x faster on single-entry maps. The map lookup behind &lt;code&gt;element_at&lt;/code&gt; and &lt;code&gt;GetMapValue&lt;/code&gt; is vectorized,
+&lt;code&gt;collect_list&lt;/code&gt; and &lt;code&gt;collect_set&lt;/code&gt; have a native &lt;code&gt;GroupsAccumulator&lt;/code&gt;, and regex patterns are compiled once per
+planned expression. Smaller gains cover date and time field extraction, &lt;code&gt;posexplode&lt;/code&gt;, unnesting, nested hashing,
+decimal overflow checks, &lt;code&gt;list_extract&lt;/code&gt;, and approximate percentile merges.&lt;/p&gt;
+&lt;h2 id="expanded-coverage"&gt;Expanded Coverage&lt;a class="headerlink" href="#expanded-coverage" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;New native support includes the &lt;code&gt;mode&lt;/code&gt;, &lt;code&gt;max_by&lt;/code&gt; / &lt;code&gt;min_by&lt;/code&gt;, &lt;code&gt;listagg&lt;/code&gt; / &lt;code&gt;string_agg&lt;/code&gt; (Spark 4.0+), and
+&lt;code&gt;regr_*&lt;/code&gt; regression aggregates; &lt;code&gt;WindowGroupLimitExec&lt;/code&gt;; &lt;code&gt;explode_outer&lt;/code&gt;; &lt;code&gt;make_interval&lt;/code&gt;; native
+&lt;code&gt;sequence&lt;/code&gt; for integral types and &lt;code&gt;unbase64&lt;/code&gt;; &lt;code&gt;_metadata&lt;/code&gt; constant columns in the native Parquet
+scan; nested types as shuffle hash partitioning keys; &lt;code&gt;BinaryType&lt;/code&gt; in sort-merge joins; and Spark 4's
+&lt;code&gt;EmptyRelationExec&lt;/code&gt;.&lt;/p&gt;
+&lt;p&gt;More expressions now use &lt;a href="https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/"&gt;codegen dispatch&lt;/a&gt; by default, where Comet runs Spark's own generated code for an
+expression inside the native pipeline, so the result matches Spark without falling back. They include
+&lt;code&gt;translate&lt;/code&gt;, &lt;code&gt;to_csv&lt;/code&gt;, &lt;code&gt;encode&lt;/code&gt;, &lt;code&gt;lpad&lt;/code&gt; / &lt;code&gt;rpad&lt;/code&gt;, &lt;code&gt;round&lt;/code&gt; on floats, &lt;code&gt;abs&lt;/code&gt; on intervals, &lt;code&gt;timestampadd&lt;/code&gt; /
+&lt;code&gt;timestampdiff&lt;/code&gt;, &lt;code&gt;next_day&lt;/code&gt; and &lt;code&gt;levenshtein&lt;/code&gt; on collated input, and Spark expressions that call a Java or
+Scala method directly. &lt;code&gt;rlike&lt;/code&gt; runs natively by
+default for literal patterns that behave the same as in Java.&lt;/p&gt;
+&lt;p&gt;Spark 4 &lt;strong&gt;Variant&lt;/strong&gt; support also improved: native Parquet scans can project Variant columns directly. Thanks to
+&lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt; for driving Variant support, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5868"&gt;#5868&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5794"&gt;#5794&lt;/a&gt;.&lt;/p&gt;
+&lt;p&gt;There's also &lt;strong&gt;experimental native support for the in-memory cache&lt;/strong&gt; (&lt;code&gt;spark.comet.exec.inMemoryCache.enabled&lt;/code&gt;,
+off by default).&lt;/p&gt;
+&lt;h2 id="previewing-comet-plans"&gt;Previewing Comet Plans&lt;a class="headerlink" href="#previewing-comet-plans" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;The new &lt;code&gt;spark.comet.explain.planOnly.enabled&lt;/code&gt; setting logs the plan Comet would have run to the driver log,
+then lets Spark run the query as normal. It shows how much of a production workload Comet would accelerate, and
+why anything falls back, without running any of it through Comet.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for this feature, with reviews from &lt;a href="https://github.com/coderfender"&gt;@coderfender&lt;/a&gt; and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;. Related PRs:
+&lt;a href="https://github.com/apache/datafusion-comet/pull/5394"&gt;#5394&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="correctness-fixes"&gt;Correctness Fixes&lt;a class="headerlink" href="#correctness-fixes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;1.1.0 fixes many cases where Comet returned different results from Spark, failed where Spark succeeds, or
+accepted input that Spark rejects. Besides the Iceberg and memory fixes above, these are the ones most likely to
+affect 1.0.0 users. The &lt;a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md"&gt;change log&lt;/a&gt; has the rest.&lt;/p&gt;
+&lt;h3 id="wrong-results"&gt;Wrong results&lt;a class="headerlink" href="#wrong-results" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;Decimal &lt;code&gt;SUM&lt;/code&gt; returned NULL, or raised an overflow error under ANSI, when an intermediate sum overflowed
+  but the final result fit (&lt;a href="https://github.com/apache/datafusion-comet/pull/6041"&gt;#6041&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;After a late shuffle fallback, &lt;code&gt;avg&lt;/code&gt; could return NULL and &lt;code&gt;collect_list&lt;/code&gt; / &lt;code&gt;collect_set&lt;/code&gt; could produce
+  mismatched buffers (&lt;a href="https://github.com/apache/datafusion-comet/pull/5421"&gt;#5421&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Exchange reuse could share one shuffle between plans that differ, such as &lt;code&gt;COUNT(*) + 1&lt;/code&gt; and &lt;code&gt;COUNT(*) - 1&lt;/code&gt;,
+  semi and anti joins, or &lt;code&gt;explode&lt;/code&gt; and &lt;code&gt;explode_outer&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5470"&gt;#5470&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5828"&gt;#5828&lt;/a&gt;, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Two ABFS containers in the same storage account shared a cached object store, so a read could return the
+  other container's data (&lt;a href="https://github.com/apache/datafusion-comet/pull/5053"&gt;#5053&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Dictionary-encoded values hashed differently from the same values decoded, and a null struct hashed its
+  fields, which affected joins, aggregates and shuffle partitioning (&lt;a href="https://github.com/apache/datafusion-comet/pull/5757"&gt;#5757&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5754"&gt;#5754&lt;/a&gt;, &lt;a href="https://github.com/viirya"&gt;@viirya&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Parquet field names containing non-ASCII characters that differ only in case read as NULL (&lt;a href="https://github.com/apache/datafusion-comet/pull/5602"&gt;#5602&lt;/a&gt;,
+  &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;&lt;code&gt;IN&lt;/code&gt;, &lt;code&gt;InSet&lt;/code&gt;, nested &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;arrays_overlap&lt;/code&gt; and &lt;code&gt;array_position&lt;/code&gt; now treat &lt;code&gt;-0.0&lt;/code&gt; and &lt;code&gt;0.0&lt;/code&gt;, and every NaN
+  encoding, as Spark does (&lt;a href="https://github.com/apache/datafusion-comet/pull/6073"&gt;#6073&lt;/a&gt;, &lt;a href="https://github.com/mizulun"&gt;@mizulun&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5235"&gt;#5235&lt;/a&gt;, &lt;a href="https://github.com/divyankshah"&gt;@divyankshah&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5472"&gt;#5472&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Decimal to double and float casts were off by one unit in the last place for most &lt;code&gt;DECIMAL(38,18)&lt;/code&gt; values
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5684"&gt;#5684&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;String to timestamp casts now follow Spark's parsing rules for short fields, time zones and signed years
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5682"&gt;#5682&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5858"&gt;#5858&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="errors-spark-raises-that-comet-did-not"&gt;Errors Spark raises that Comet did not&lt;a class="headerlink" href="#errors-spark-raises-that-comet-did-not" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;Casts and expressions routed through codegen dispatch could skip ANSI errors raised inside a constant
+  subexpression (&lt;a href="https://github.com/apache/datafusion-comet/pull/5623"&gt;#5623&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Rejected &lt;code&gt;TIMESTAMP_NTZ&lt;/code&gt; casts returned NULL under ANSI instead of raising &lt;code&gt;CAST_INVALID_INPUT&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5752"&gt;#5752&lt;/a&gt;,
+  &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Out-of-range Parquet &lt;code&gt;TIMESTAMP_MILLIS&lt;/code&gt; values, top-level or nested, silently wrapped (&lt;a href="https://github.com/apache/datafusion-comet/pull/5177"&gt;#5177&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5740"&gt;#5740&lt;/a&gt;,
+  &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Nested Parquet struct, list and map fields now follow Spark's conversion rules instead of returning NULL on
+  overflow or accepting values Spark rejects (&lt;a href="https://github.com/apache/datafusion-comet/pull/5681"&gt;#5681&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="query-failures-and-crashes"&gt;Query failures and crashes&lt;a class="headerlink" href="#query-failures-and-crashes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;code&gt;collect_list&lt;/code&gt; and &lt;code&gt;collect_set&lt;/code&gt; over nested arguments failed with "column types must match schema types"
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5159"&gt;#5159&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Native shuffle failed with a 2 GB task serialization error on jobs with very many partitions (&lt;a href="https://github.com/apache/datafusion-comet/pull/5392"&gt;#5392&lt;/a&gt;,
+  &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;A Scala UDF from a user jar failed with a &lt;code&gt;ClassCastException&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5282"&gt;#5282&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;&lt;code&gt;rpad&lt;/code&gt; and &lt;code&gt;lpad&lt;/code&gt; panicked on a NULL length (&lt;a href="https://github.com/apache/datafusion-comet/pull/5680"&gt;#5680&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Structs with duplicate field names failed the task in native shuffle, and panicked in the native Parquet
+  scan (&lt;a href="https://github.com/apache/datafusion-comet/pull/5866"&gt;#5866&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5786"&gt;#5786&lt;/a&gt;, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="hangs-and-resource-use"&gt;Hangs and resource use&lt;a class="headerlink" href="#hangs-and-resource-use" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;The JVM hung on exit when an application returned from &lt;code&gt;main&lt;/code&gt; without calling &lt;code&gt;spark.stop()&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5748"&gt;#5748&lt;/a&gt;,
+  &lt;a href="https://github.com/zhangfengcdt"&gt;@zhangfengcdt&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;The native scan busy-polled while waiting on S3 or HDFS reads, keeping one core per task at 100% (&lt;a href="https://github.com/apache/datafusion-comet/pull/6219"&gt;#6219&lt;/a&gt;,
+  &lt;a href="https://github.com/mixermt"&gt;@mixermt&lt;/a&gt; and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;One task could force-spill another task's shuffle buffers, and a failed shuffle write leaked its memory
+  reservation (&lt;a href="https://github.com/apache/datafusion-comet/pull/5493"&gt;#5493&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5461"&gt;#5461&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="upgrading-to-110"&gt;Upgrading to 1.1.0&lt;a class="headerlink" href="#upgrading-to-110" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;A few changes in 1.1.0 can affect a deployment. The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html"&gt;Comet Upgrade Guide&lt;/a&gt; has the details.&lt;/p&gt;
+&lt;h3 id="platform"&gt;Platform&lt;a class="headerlink" href="#platform" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;JDK 17 or later is required.&lt;/strong&gt; JDK 11 support was removed, as announced in 1.0.0.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.4 is still deprecated.&lt;/strong&gt; Comet still publishes Spark 3.4 binaries, but Spark's SQL test suite runs
+  against 3.4 only on demand, so 3.4-specific regressions are more likely. We recommend Spark 3.5 or later.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="size-settings-now-read-correctly"&gt;Size settings now read correctly&lt;a class="headerlink" href="#size-settings-now-read-correctly" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;1.0.0 misread three size settings, so jobs that set them may behave differently:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.memory.offHeap.size&lt;/code&gt; as a bare number of bytes&lt;/strong&gt; was read as MiB, which made &lt;code&gt;fair_unified&lt;/code&gt;'s
+  per-operator shares about a million times too large. With correct shares, operators can spill sooner, and an
+  operator that can't spill can fail when it exceeds its share. Sizes with a unit, like &lt;code&gt;16g&lt;/code&gt;, aren't affected.
+  &lt;code&gt;spark.comet.exec.memoryPool=greedy_unified&lt;/code&gt; restores the old behavior.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.maxTempDirectorySize&lt;/code&gt; with a unit&lt;/strong&gt;, like &lt;code&gt;10g&lt;/code&gt;, was ignored in favor of the 100 GB default.
+  It's now enforced, so a query that spills past it fails.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.shuffle.native.writeBufferSize&lt;/code&gt; with a unit&lt;/strong&gt; was treated as bytes, so &lt;code&gt;64m&lt;/code&gt; meant 64 bytes. It
+  now means what it says. These buffers use untracked native memory, so check large values against
+  &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Malformed values for &lt;code&gt;spark.comet.maxTempDirectorySize&lt;/code&gt; and &lt;code&gt;spark.comet.explain.native.enabled&lt;/code&gt; now fail the
+query instead of falling back to the default.&lt;/p&gt;
+&lt;h3 id="more-memory-before-spilling"&gt;More memory before spilling&lt;a class="headerlink" href="#more-memory-before-spilling" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;With the &lt;code&gt;fair_unified&lt;/code&gt; fix, tasks with several operators can reserve more memory before spilling than in
+0.15.0 through 1.0.0, especially on executors running few tasks at once. If you sized executors for one of those
+releases, check the memory usage log for headroom.&lt;/p&gt;
+&lt;h3 id="conditions-for-enabling-comet"&gt;Conditions for enabling Comet&lt;a class="headerlink" href="#conditions-for-enabling-comet" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet requires Spark's off-heap memory. &lt;code&gt;CometPlugin&lt;/code&gt; already enforced this, and 1.1.0 applies the same check
+when &lt;code&gt;CometSparkSessionExtensions&lt;/code&gt; is registered directly, disabling Comet with a warning. Both checks read
+&lt;code&gt;spark.memory.offHeap.enabled&lt;/code&gt; from the SparkContext, so set it when the application starts, not on a
+&lt;code&gt;SparkSession.builder&lt;/code&gt; after the context exists.&lt;/p&gt;
+&lt;p&gt;Comet also checks the shuffle manager that's actually running, not the session's &lt;code&gt;spark.shuffle.manager&lt;/code&gt;. A
+session that names &lt;code&gt;CometShuffleManager&lt;/code&gt; after the context started with a different one now runs without Comet,
+with a warning, instead of failing with a &lt;code&gt;ClassCastException&lt;/code&gt;.&lt;/p&gt;
+&lt;h3 id="deprecated-and-removed-settings"&gt;Deprecated and removed settings&lt;a class="headerlink" href="#deprecated-and-removed-settings" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.exec.memoryPool.fraction&lt;/code&gt;&lt;/strong&gt; is deprecated and will be removed in a future major release. It
+  still works, and the driver warns when it's set.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Testing-only on-heap settings&lt;/strong&gt; were removed: &lt;code&gt;spark.comet.memoryOverhead&lt;/code&gt;,
+  &lt;code&gt;spark.comet.exec.onHeap.memoryPool&lt;/code&gt;, and &lt;code&gt;spark.comet.shuffle.jvm.memoryFactor&lt;/code&gt; (and its old name,
+  &lt;code&gt;spark.comet.columnar.shuffle.memory.factor&lt;/code&gt;). The &lt;a href="https://datafusion.apache.org/comet/about/versioning_policy.html#testing-and-internal-configurations-are-exempt"&gt;versioning policy&lt;/a&gt; exempts testing settings, and Comet
+  ignores them if they're still set.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="compatibility"&gt;Compatibility&lt;a class="headerlink" href="#compatibility" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Supported platforms:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.4.3&lt;/strong&gt; with Java 17 and Scala 2.12/2.13 (deprecated)&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.5.9&lt;/strong&gt; with Java 17 and Scala 2.12/2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.0.4&lt;/strong&gt; with Java 17/21 and Scala 2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.1.3&lt;/strong&gt; with Java 17/21 and Scala 2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.2.0&lt;/strong&gt; with Java 17 and Scala 2.13 (experimental, for early evaluation only)&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;See the &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html"&gt;Spark Version Compatibility&lt;/a&gt; page for known limitations specific to each version.&lt;/p&gt;
+&lt;p&gt;This release builds on &lt;strong&gt;DataFusion 55.1&lt;/strong&gt; and &lt;strong&gt;Arrow 59.2&lt;/strong&gt;.&lt;/p&gt;
+&lt;h2 id="get-started-with-comet-110"&gt;Get Started with Comet 1.1.0&lt;a class="headerlink" href="#get-started-with-comet-110" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Follow the &lt;a href="https://datafusion.apache.org/comet/user-guide/1.1/installation.html"&gt;Comet 1.1.0 Installation Guide&lt;/a&gt;
+to get up and running, then point Comet at your existing Spark workloads.&lt;/p&gt;</content><category term="blog"/></entry><entry><title>Apache DataFusion 55.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="alternate"/><published>2026-08-25T00:00:00+00:00</published><updated>2026-08-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-08-25:/blog/2026/08/25/datafusion-55.0.0</id><summary type= [...]
 {% comment %}
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
diff --git a/output/feeds/pmc.atom.xml b/output/feeds/pmc.atom.xml
index 1f3f46d..9ac8f4e 100644
--- a/output/feeds/pmc.atom.xml
+++ b/output/feeds/pmc.atom.xml
@@ -1,5 +1,430 @@
 <?xml version="1.0" encoding="utf-8"?>
-<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - pmc</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-08-25T00:00:00+00:00</updated><entry><title>Apache DataFusion 55.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="alternate"/><published>2026-08-25T00: [...]
+<feed xmlns="http://www.w3.org/2005/Atom"><title>Apache DataFusion Blog - pmc</title><link href="https://datafusion.apache.org/blog/" rel="alternate"/><link href="https://datafusion.apache.org/blog/feeds/pmc.atom.xml" rel="self"/><id>https://datafusion.apache.org/blog/</id><updated>2026-10-01T00:00:00+00:00</updated><entry><title>Apache DataFusion Comet 1.1.0 Release</title><link href="https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0" rel="alternate"/><published>2026- [...]
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development …&lt;/p&gt;</summary><content type="html">&lt;!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development since 1.0.0: 398 commits from 40 contributors. See the
+&lt;a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md"&gt;change log&lt;/a&gt; for the full list of changes.&lt;/p&gt;
+&lt;p&gt;The highlights:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Native Iceberg writes&lt;/strong&gt; (experimental): Comet can write Iceberg data files with iceberg-rust instead of
+  iceberg-java.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Memory management&lt;/strong&gt;: Comet now measures the native memory its pools don't track, logs it on every executor,
+  and fixes several long-standing pool accounting bugs.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Native shuffle over Apache Celeborn&lt;/strong&gt;: 1.1.0 includes the native writer and reader, but released Celeborn
+  clients currently fall back to ordinary Spark/Celeborn shuffle.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Native Parquet writes on Spark 4.0+&lt;/strong&gt; (experimental), now built on Spark's own write path.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="native-iceberg-writes-experimental"&gt;Native Iceberg Writes (Experimental)&lt;a class="headerlink" href="#native-iceberg-writes-experimental" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Until now, Comet accelerated Iceberg reads but left writes to the JVM. In 1.1.0, Comet can write Iceberg data
+files natively using &lt;a href="https://github.com/apache/iceberg-rust"&gt;iceberg-rust&lt;/a&gt;. The feature is &lt;strong&gt;experimental and
+disabled by default&lt;/strong&gt;, and it only engages under fairly strict conditions.&lt;/p&gt;
+&lt;h3 id="splitting-the-write-operator"&gt;Splitting the write operator&lt;a class="headerlink" href="#splitting-the-write-operator" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
+and validates against the catalog. Because file writing is bundled with the metadata and commit steps, there was
+no separate piece for Comet to replace.&lt;/p&gt;
+&lt;p&gt;Setting &lt;code&gt;spark.comet.write.iceberg.splitOperator.enabled=true&lt;/code&gt; splits eligible Iceberg writes into two operators:&lt;/p&gt;
+&lt;ol&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;IcebergWrite&lt;/code&gt;&lt;/strong&gt; writes data files on the executors and returns each task's commit message.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;IcebergCommit&lt;/code&gt;&lt;/strong&gt; collects the commit messages on the driver and performs the normal Iceberg commit, once.&lt;/li&gt;
+&lt;/ol&gt;
+&lt;p&gt;With only the split enabled, iceberg-java still writes the files; the native writer, described next, replaces
+that step. The split covers &lt;code&gt;INSERT INTO&lt;/code&gt; and DataFrame &lt;code&gt;append&lt;/code&gt;, static and dynamic &lt;code&gt;INSERT OVERWRITE&lt;/code&gt;, and
+copy-on-write &lt;code&gt;DELETE&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, and &lt;code&gt;MERGE&lt;/code&gt;, on every supported Spark version. Merge-on-read writes are left
+alone. When Comet can't split a write (an unrecognized write class, CTAS on Spark 3.4, or a write that needs
+Spark's commit coordinator), it plans the write exactly as if Comet weren't there.&lt;/p&gt;
+&lt;h3 id="writing-parquet-natively"&gt;Writing Parquet natively&lt;a class="headerlink" href="#writing-parquet-natively" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;The native writer requires the split. Also setting &lt;code&gt;spark.comet.iceberg.write.enabled=true&lt;/code&gt; hands each task's
+Parquet writing to iceberg-rust. The driver passes the native writer everything it needs: schema, partition spec,
+data location, Parquet settings, writer mode, and object-store configuration. Each task writes its files and
+returns their metadata as an in-memory Iceberg manifest.&lt;/p&gt;
+&lt;p&gt;From there, iceberg-java takes over again. The JVM reads the manifest, recomputes each file's metrics from its
+Parquet footer, and builds the same commit message the JVM writer would have produced. Snapshots, manifest
+lists, commit validation, and retries all work as before.&lt;/p&gt;
+&lt;p&gt;The native writer consumes Arrow batches from a Comet operator, so the query feeding the write must run in Comet
+too. Writes from a local relation, such as &lt;code&gt;INSERT ... VALUES&lt;/code&gt;, also need
+&lt;code&gt;spark.comet.exec.localTableScan.enabled=true&lt;/code&gt;.&lt;/p&gt;
+&lt;p&gt;To see which path a write took, check the physical plan: a native write shows &lt;code&gt;CometIcebergWrite&lt;/code&gt; under
+&lt;code&gt;IcebergCommit&lt;/code&gt;, and a write that fell back shows &lt;code&gt;IcebergWrite&lt;/code&gt;.&lt;/p&gt;
+&lt;h3 id="matching-iceberg-java"&gt;Matching iceberg-java&lt;a class="headerlink" href="#matching-iceberg-java" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;The native writer aims to write the same table iceberg-java would, down to the metadata. Manifest metrics drive
+pruning for every future reader, so rather than trust the native writer's numbers, Comet recomputes them with
+iceberg-java's own code. Parity tests write the same rows through both writers and compare the committed counts
+and bounds. The cost is one small footer read per file.&lt;/p&gt;
+&lt;p&gt;Eligibility is an allowlist. A write goes native only if its whole configuration matches the documented set of
+supported settings. Anything else, including properties added by future Iceberg versions, falls back to
+iceberg-java and reports why in Comet's extended &lt;code&gt;EXPLAIN&lt;/code&gt; output. Encryption, object-storage layout, custom
+location providers, bloom filters, and unrecognized &lt;code&gt;parquet.*&lt;/code&gt; properties all fall back. For a partitioned
+table, Iceberg asks Spark to cluster and sort rows by partition value, such as &lt;code&gt;bucket(16, id)&lt;/code&gt; or &lt;code&gt;days(ts)&lt;/code&gt;,
+before writing. That step must also run natively, which it now can because Iceberg's system functions have
+native implementations.&lt;/p&gt;
+&lt;p&gt;Comet decides all of this at planning time, including checking that every iceberg-java class it relies on is
+where it expects. An Iceberg release that moves one falls back instead of failing tasks mid-write. CI also runs
+Apache Iceberg's own Spark test suites, for Iceberg 1.8.1 through 1.11.0, with the native writer enabled.&lt;/p&gt;
+&lt;h3 id="failure-handling"&gt;Failure handling&lt;a class="headerlink" href="#failure-handling" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Only successful tasks' commit messages are committed. A failed task deletes the files it wrote, as iceberg-java
+does, and a failed job commits nothing and deletes the completed tasks' files. Retries can't collide, because
+each attempt's ID is part of its file names. Anything cleanup misses is invisible to readers and is removed by
+Iceberg's normal &lt;code&gt;remove_orphan_files&lt;/code&gt; maintenance.&lt;/p&gt;
+&lt;h3 id="known-differences"&gt;Known differences&lt;a class="headerlink" href="#known-differences" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Files written by parquet-rs, the Rust Parquet library iceberg-rust uses, differ from those written by
+parquet-java (formerly parquet-mr), which iceberg-java uses. Enabling the feature accepts these differences.
+Most are cosmetic, such as footer metadata, &lt;code&gt;created_by&lt;/code&gt;, and page encoding labels. Two are worth knowing about:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Files won't split at the same points.&lt;/strong&gt; Both writers check file size every 1000 rows, but they estimate it
+  differently, so they can roll over to a new file at different rows.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;High-cardinality columns keep a dictionary page.&lt;/strong&gt; parquet-java drops dictionary encoding early when it isn't
+  saving space, while parquet-rs keeps it up to &lt;code&gt;write.parquet.dict-size-bytes&lt;/code&gt;. Results are the same, but
+  selective reads of native-written files fetch more bytes
+  (&lt;a href="https://github.com/apache/datafusion-comet/issues/6114"&gt;#6114&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/iceberg-writes.html"&gt;Iceberg Writes guide&lt;/a&gt; lists every supported setting and every difference. Please try it on a non-production
+table and tell us how it goes. Feedback from real workloads is what this feature needs before it loses the
+experimental label.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/jordepic"&gt;@jordepic&lt;/a&gt; for designing and implementing the split-operator plan, write detection, and the native
+writer, and to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for the fidelity and failure-handling work, with contributions from
+&lt;a href="https://github.com/zhangfengcdt"&gt;@zhangfengcdt&lt;/a&gt;, &lt;a href="https://github.com/snmvaughan"&gt;@snmvaughan&lt;/a&gt;, &lt;a href="https://github.com/liupoyi-1031"&gt;@liupoyi-1031&lt;/a&gt;, and &lt;a href="https://github.com/0lai0"&gt;@0lai0&lt;/a&gt;, and reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;,
+&lt;a href="https://github.com/unikdahal"&gt;@unikdahal&lt;/a&gt;, and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/4658"&gt;#4658&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5298"&gt;#5298&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5361"&gt;#5361&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5663"&gt;#5663&lt;/a& [...]
+&lt;h2 id="memory-management"&gt;Memory Management&lt;a class="headerlink" href="#memory-management" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;A recurring problem for Comet users has been executors killed by the cluster manager (on Kubernetes,
+&lt;code&gt;ExecutorLostFailure&lt;/code&gt; with exit code 137) even though Comet stayed within its memory pool. 1.1.0 explains why
+this happens, lets you measure it, and fixes pool bugs that made it worse.&lt;/p&gt;
+&lt;h3 id="where-the-memory-goes"&gt;Where the memory goes&lt;a class="headerlink" href="#where-the-memory-goes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet's native operators allocate from the Rust heap but charge their reservations against Spark's off-heap
+pool, sized by &lt;code&gt;spark.memory.offHeap.size&lt;/code&gt;. Operators only reserve memory for data they deliberately hold onto:
+sort buffers, hash join build sides, aggregation state, and buffered shuffle partitions.&lt;/p&gt;
+&lt;p&gt;Plenty of memory is never reserved: working memory in expression kernels, decompression buffers, Parquet reader
+state, object store buffers, JVM-side Arrow buffers, and allocator overhead. All of it has to fit in
+&lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;, and until now there was no way to see how much of it there was.&lt;/p&gt;
+&lt;p&gt;&lt;img alt="The executor container holds the JVM heap, the off-heap memory pool, and the memory overhead. Spark and Comet share the off-heap pool, where Comet's sorts, joins, aggregations, and shuffles reserve memory. The memory overhead holds the JVM's own overhead plus Comet's native memory that the pool does not track." class="img-fluid" src="/blog/images/comet-1.1.0/comet-executor-memory.svg" width="100%"/&gt;&lt;/p&gt;
+&lt;h3 id="measuring-used-memory"&gt;Measuring used memory&lt;a class="headerlink" href="#measuring-used-memory" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet now wraps its native allocator in a counter that tracks every byte allocated and not yet freed. It only
+observes and never rejects an allocation. Arrow buffers the JVM imports from native code are now tracked
+separately, so they aren't counted twice.&lt;/p&gt;
+&lt;p&gt;Each executor uses the counter to log its native memory usage every 10 seconds while Comet runs:&lt;/p&gt;
+&lt;pre&gt;&lt;code&gt;Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native plans, 8 memory pools)
+&lt;/code&gt;&lt;/pre&gt;
+&lt;p&gt;&lt;code&gt;reserved&lt;/code&gt; is what the pools track, and the container already has room for it. &lt;code&gt;allocated&lt;/code&gt; is everything
+Comet's native code holds. The difference is what has to fit in &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;, next to the
+JVM's own non-heap memory. To size the overhead, run a representative workload, take the largest difference from
+the log, add it to the overhead you had before Comet, and leave a margin. Setting
+&lt;code&gt;spark.comet.memory.logInterval=1s&lt;/code&gt; for that run makes it less likely to miss a short peak.&lt;/p&gt;
+&lt;p&gt;The executor also warns when its native memory looks larger than the container allows, and the
+&lt;a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/memory.html"&gt;tuning guide&lt;/a&gt; walks through sizing with examples. One gotcha: setting &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;
+&lt;em&gt;replaces&lt;/em&gt; the value Spark derives from &lt;code&gt;spark.executor.memoryOverheadFactor&lt;/code&gt; instead of adding to it, so on a
+large executor it can shrink the container. For large executors, raise the factor instead.&lt;/p&gt;
+&lt;h3 id="memory-pool-fixes"&gt;Memory pool fixes&lt;a class="headerlink" href="#memory-pool-fixes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet's off-heap memory comes from one of two pools. &lt;code&gt;fair_unified&lt;/code&gt;, the default, caps each operator at an even
+share of the task's memory. &lt;code&gt;greedy_unified&lt;/code&gt; gives memory to operators first come, first served. When Spark
+grants an operator less memory than it asked for (a partial grant), the operator spills.&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;fair_unified&lt;/code&gt; limited a whole task to one operator's share.&lt;/strong&gt; Since 0.15.0, the pool compared all of a
+  task's reservations against a single operator's share, and each new operator shrank the limit for the rest.
+  Each operator now gets its own share, so tasks with several operators can use more memory before spilling.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;A partial grant from Spark no longer panics the task.&lt;/strong&gt; DataFusion sometimes has to record memory that
+  already exists, such as a spilled batch being read back, and both pools panicked when Spark granted less than
+  they asked for. They now track the shortfall and repay it before returning memory to Spark.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Leaks on failure paths.&lt;/strong&gt; A plan that failed during setup or teardown could leak its task's memory pool, and
+  a failed Arrow import leaked the vectors it had already imported.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Configuration units.&lt;/strong&gt; 1.0.0 misread three size settings, including &lt;code&gt;spark.memory.offHeap.size&lt;/code&gt; written as
+  a bare number. See &lt;a href="#upgrading-to-110"&gt;Upgrading to 1.1.0&lt;/a&gt; for what changes.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Less log noise when spilling.&lt;/strong&gt; 1.0.0 logged a warning and a memory dump for every partial grant, so a
+  spilling query could log hundreds. Partial grants now log at DEBUG, and the dump is gone because it could
+  deadlock the task.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Metrics.&lt;/strong&gt; Spills from native sorts, sort-merge joins, and aggregates now count toward Spark's Spill (Disk)
+  task metric in every stage. The native shuffle writer reports Spill (Disk) and Spill (Memory) separately, where
+  1.0.0 put the same number in both. Native aggregates with grouping keys also show spill counts, bytes, and rows
+  in the SQL tab.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;&lt;code&gt;spark.comet.exec.memoryPool.fraction&lt;/code&gt; is deprecated. It was meant to leave room for untracked memory, but Spark
+hands out the whole pool anyway, so it never did. Size &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt; instead.&lt;/p&gt;
+&lt;h3 id="on-heap-mode"&gt;On-heap mode&lt;a class="headerlink" href="#on-heap-mode" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;On-heap mode exists so that Spark's and Iceberg's test suites can run against Comet; production runs off-heap.
+Its memory accounting didn't protect anything, because native memory isn't on the JVM heap, so 1.1.0 removes that
+accounting, along with six of the nine pool types and several testing-only settings. Outside tests, Comet now requires
+off-heap memory however it's enabled, including when &lt;code&gt;CometSparkSessionExtensions&lt;/code&gt; is registered directly, and
+disables itself with a warning otherwise.&lt;/p&gt;
+&lt;p&gt;Contributors can find more detail in the new &lt;a href="https://datafusion.apache.org/comet/contributor-guide/memory_management.html"&gt;memory management guide&lt;/a&gt;.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for driving this work, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt; for the memory pool lifecycle and shuffle spill
+accounting fixes, &lt;a href="https://github.com/ywskycn"&gt;@ywskycn&lt;/a&gt; for reporting native memory usage to Spark, &lt;a href="https://github.com/1fanwang"&gt;@1fanwang&lt;/a&gt; for the Arrow import leak
+fix, and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; for the native aggregate spill metrics, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;,
+&lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;, and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5934"&gt;#5934&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6162"&gt;#6162&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6048"&gt;#6048&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6128"&gt;#6128&lt;/a&gt [...]
+&lt;h2 id="more-iceberg-improvements"&gt;More Iceberg Improvements&lt;a class="headerlink" href="#more-iceberg-improvements" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;V3 deletion vectors&lt;/strong&gt; are applied on native scans.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Iceberg system functions&lt;/strong&gt; (&lt;code&gt;bucket&lt;/code&gt;, &lt;code&gt;truncate&lt;/code&gt;, &lt;code&gt;years&lt;/code&gt;, &lt;code&gt;months&lt;/code&gt;, &lt;code&gt;days&lt;/code&gt;, and &lt;code&gt;hours&lt;/code&gt;) run natively.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Scan planning metrics and scan time&lt;/strong&gt; appear in the Spark UI for native Iceberg scans.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;A wrong-results fix.&lt;/strong&gt; A filter such as &lt;code&gt;bucket(4, id) = 2&lt;/code&gt; combined with another predicate was pushed to
+  the native scan as &lt;code&gt;id = 2&lt;/code&gt;, returning too few rows.&lt;/li&gt;
+&lt;li&gt;Tables partitioned by an unknown transform can be read natively, and &lt;code&gt;IS NULL&lt;/code&gt; / &lt;code&gt;IS NOT NULL&lt;/code&gt; checks on list
+  and map columns no longer force a fallback.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt; for deletion vector support, &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt; for the scan metrics, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt; for the
+null-check fix, and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for the native system functions and residual fix, with reviews from
+&lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/rich7420"&gt;@rich7420&lt;/a&gt;, &lt;a href="https://github.com/unikdahal"&gt;@unikdahal&lt;/a&gt;, and &lt;a href="https://github.com/jordepic"&gt;@jordepic&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5853"&gt;#5853&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5638"&gt;#5638&lt;/a&gt;, &lt;a href="https://github.com/apache/dataf [...]
+&lt;h2 id="remote-shuffle-with-celeborn"&gt;Remote Shuffle with Celeborn&lt;a class="headerlink" href="#remote-shuffle-with-celeborn" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;1.1.0 adds Comet's side of native shuffle over Apache Celeborn: map tasks push Comet's Arrow data straight to
+Celeborn, and reducers read it back natively. In 1.0.0, Celeborn users always got ordinary Spark shuffle.&lt;/p&gt;
+&lt;p&gt;Native Celeborn shuffle is unavailable with released Celeborn 0.6.x and 0.7.x clients in Comet 1.1.0 because
+these clients do not pass Comet's push-completion compatibility checks. Set &lt;code&gt;spark.shuffle.manager&lt;/code&gt; to
+&lt;code&gt;org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager&lt;/code&gt; to use Comet alongside
+ordinary Spark/Celeborn shuffle; Comet can still accelerate other operators. These clients retain
+ordinary Spark/Celeborn shuffle even when &lt;code&gt;spark.comet.shuffle.mode=native&lt;/code&gt; is requested, and the
+default &lt;code&gt;auto&lt;/code&gt; mode also retains ordinary shuffle. Follow-up work to enable native shuffle with
+released clients is tracked in &lt;a href="https://github.com/apache/datafusion-comet/issues/6523"&gt;#6523&lt;/a&gt;.
+Native shuffle over Celeborn does not support &lt;code&gt;spark.io.encryption.enabled=true&lt;/code&gt;, and Celeborn
+is not bundled with Comet. The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/tuning/celeborn.html"&gt;Celeborn tuning guide&lt;/a&gt; has the full requirements.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/pingzh"&gt;@pingzh&lt;/a&gt; for this work, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;, &lt;a href="https://github.com/ziting-openai"&gt;@ziting-openai&lt;/a&gt;, and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;. Related
+PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5473"&gt;#5473&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5481"&gt;#5481&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5513"&gt;#5513&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5531"&gt;#5531&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5537"&gt;#5537&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="native-parquet-writes-on-spark-40-experimental"&gt;Native Parquet Writes on Spark 4.0+ (Experimental)&lt;a class="headerlink" href="#native-parquet-writes-on-spark-40-experimental" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Native Parquet writes remain &lt;strong&gt;experimental and disabled by default&lt;/strong&gt;, but on Spark 4.0+ they now build on
+Spark's own write path.&lt;/p&gt;
+&lt;p&gt;In 1.0.0, a native write replaced Spark's entire write command, so Comet had to reimplement the commit protocol,
+save modes, and job commit. On Spark 4.0+, Comet replaces only &lt;code&gt;WriteFilesExec&lt;/code&gt;, the per-task write step that
+Spark 4.0 made pluggable, and Spark handles the rest. That means:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;File names come from Spark's commit protocol, so committers that track individual files, like the S3A magic
+  committer, work.&lt;/li&gt;
+&lt;li&gt;Column names, nullability, and field IDs come from the target table, not the query.&lt;/li&gt;
+&lt;li&gt;Bytes-written and rows-written metrics are correct on HDFS.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;The writer handles non-partitioned, non-bucketed writes to local and HDFS paths, and skips writes that set
+&lt;code&gt;spark.sql.files.maxRecordsPerFile&lt;/code&gt;. To enable it, set &lt;code&gt;spark.comet.parquet.write.enabled=true&lt;/code&gt; and
+&lt;code&gt;spark.comet.operator.WriteFilesExec.allowIncompatible=true&lt;/code&gt;. Spark 3.4 and 3.5 keep the old write path.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; for this work, with reviews from &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;,
+&lt;a href="https://github.com/rich7420"&gt;@rich7420&lt;/a&gt;, and &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5763"&gt;#5763&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5369"&gt;#5369&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="s3-credentials"&gt;S3 Credentials&lt;a class="headerlink" href="#s3-credentials" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Comet's native S3 access now works with more of the credential setups Spark supports:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Built-in credential provider adapters.&lt;/strong&gt; In 1.0.0, a native Parquet scan failed with
+  &lt;code&gt;Unsupported credential provider&lt;/code&gt; for classes Spark accepts but Comet didn't reimplement, such as
+  &lt;code&gt;DefaultAWSCredentialsProviderChain&lt;/code&gt;. Two new adapters fix this by setting
+  &lt;code&gt;spark.hadoop.fs.s3a.comet.credential.provider.class&lt;/code&gt;.
+  &lt;code&gt;HadoopS3ACredentialProviderAdapter&lt;/code&gt; (recommended) uses Hadoop S3A's own provider chain, so it supports
+  everything S3A does, including web identity, assumed roles, and per-bucket settings.
+  &lt;code&gt;AwsSdkCredentialProviderAdapter&lt;/code&gt; wraps a specific AWS SDK provider class.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;STS throttling protection on EKS with IRSA.&lt;/strong&gt; When many executors start at once, STS can throttle their
+  credential requests. In 1.0.0, the native Iceberg path then fell back to the EKS node role, which usually can't
+  read the bucket, and the job failed with &lt;code&gt;403&lt;/code&gt; errors. Native Iceberg reads and writes now fetch web-identity
+  credentials themselves, retry throttled requests with backoff, never fall back to the node role, and share one
+  credential per executor. This happens automatically.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Per-location credentials.&lt;/strong&gt; A provider implementing &lt;code&gt;CometS3LocationScopedCredentialProvider&lt;/code&gt; can return
+  different credentials for different prefixes in the same bucket, such as &lt;code&gt;warehouse/sales&lt;/code&gt; and
+  &lt;code&gt;warehouse/finance&lt;/code&gt;.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Custom S3-compatible schemes.&lt;/strong&gt; Vendor filesystems that front an S3-compatible store with their own URL
+  scheme, such as &lt;code&gt;blob://&lt;/code&gt;, can be read natively by listing the scheme in
+  &lt;code&gt;spark.hadoop.fs.comet.s3Compliant.schemes&lt;/code&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;See the &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/s3-credential-providers.html"&gt;S3 credential providers guide&lt;/a&gt; for details.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt; for the credential adapters and STS throttling protection, &lt;a href="https://github.com/snmvaughan"&gt;@snmvaughan&lt;/a&gt; for
+per-location credentials, and &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt; for custom S3-compatible schemes, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and
+&lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/6023"&gt;#6023&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6025"&gt;#6025&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/6031"&gt;#6031&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5314"&gt;#5314&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="performance"&gt;Performance&lt;a class="headerlink" href="#performance" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;h3 id="shuffle"&gt;Shuffle&lt;a class="headerlink" href="#shuffle" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;In 1.0.0, the native shuffle writer ran with a &lt;strong&gt;1-byte write buffer&lt;/strong&gt; by default: its 1 MiB default was
+  declared in MiB but passed to native code as bytes. It now gets the intended 1 MiB.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Round-robin repartitioning&lt;/strong&gt; assigns rows by position, as Spark does, instead of hashing every column of
+  every row. That's much cheaper on wide nested schemas, and it spreads repeated values across reducers the way
+  Spark does.&lt;/li&gt;
+&lt;li&gt;A task &lt;strong&gt;spills all of its shuffle partitions into one file&lt;/strong&gt; instead of one file each.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Zstd and Arrow IPC compression contexts are reused&lt;/strong&gt; across shuffle blocks.&lt;/li&gt;
+&lt;li&gt;Shuffle blocks are &lt;strong&gt;decoded against a cached schema&lt;/strong&gt; instead of re-parsing it for every block.&lt;/li&gt;
+&lt;li&gt;Per-partition scratch buffers are reused, and &lt;code&gt;ArrowWriter&lt;/code&gt; bulk-copies fixed-width columns.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Thanks to the contributors who drove this work, especially &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt;, &lt;a href="https://github.com/pingzh"&gt;@pingzh&lt;/a&gt;, and
+&lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and &lt;a href="https://github.com/mbutrovich"&gt;@mbutrovich&lt;/a&gt;.&lt;/p&gt;
+&lt;h3 id="planning-and-execution"&gt;Planning and Execution&lt;a class="headerlink" href="#planning-and-execution" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Native dynamic filter pushdown&lt;/strong&gt;: a hash join's build-side keys filter the probe-side Parquet scan at
+  runtime, so the scan can skip data that can't match.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Adaptive partial aggregation&lt;/strong&gt; for eligible native shuffle plans: when grouping keys are nearly unique, the
+  partial aggregate passes rows through instead of building a hash table that doesn't reduce them.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Parsed plan data is cached&lt;/strong&gt; across a stage's tasks.&lt;/li&gt;
+&lt;li&gt;New &lt;strong&gt;native Parquet scan I/O and read-amplification metrics&lt;/strong&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="expressions"&gt;Expressions&lt;a class="headerlink" href="#expressions" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Many expression kernels got faster. In kernel microbenchmarks, &lt;code&gt;map_sort&lt;/code&gt; is up to 3x faster on multi-entry string
+maps and 18x faster on single-entry maps. The map lookup behind &lt;code&gt;element_at&lt;/code&gt; and &lt;code&gt;GetMapValue&lt;/code&gt; is vectorized,
+&lt;code&gt;collect_list&lt;/code&gt; and &lt;code&gt;collect_set&lt;/code&gt; have a native &lt;code&gt;GroupsAccumulator&lt;/code&gt;, and regex patterns are compiled once per
+planned expression. Smaller gains cover date and time field extraction, &lt;code&gt;posexplode&lt;/code&gt;, unnesting, nested hashing,
+decimal overflow checks, &lt;code&gt;list_extract&lt;/code&gt;, and approximate percentile merges.&lt;/p&gt;
+&lt;h2 id="expanded-coverage"&gt;Expanded Coverage&lt;a class="headerlink" href="#expanded-coverage" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;New native support includes the &lt;code&gt;mode&lt;/code&gt;, &lt;code&gt;max_by&lt;/code&gt; / &lt;code&gt;min_by&lt;/code&gt;, &lt;code&gt;listagg&lt;/code&gt; / &lt;code&gt;string_agg&lt;/code&gt; (Spark 4.0+), and
+&lt;code&gt;regr_*&lt;/code&gt; regression aggregates; &lt;code&gt;WindowGroupLimitExec&lt;/code&gt;; &lt;code&gt;explode_outer&lt;/code&gt;; &lt;code&gt;make_interval&lt;/code&gt;; native
+&lt;code&gt;sequence&lt;/code&gt; for integral types and &lt;code&gt;unbase64&lt;/code&gt;; &lt;code&gt;_metadata&lt;/code&gt; constant columns in the native Parquet
+scan; nested types as shuffle hash partitioning keys; &lt;code&gt;BinaryType&lt;/code&gt; in sort-merge joins; and Spark 4's
+&lt;code&gt;EmptyRelationExec&lt;/code&gt;.&lt;/p&gt;
+&lt;p&gt;More expressions now use &lt;a href="https://datafusion.apache.org/blog/2026/06/20/datafusion-comet-0.17.0/"&gt;codegen dispatch&lt;/a&gt; by default, where Comet runs Spark's own generated code for an
+expression inside the native pipeline, so the result matches Spark without falling back. They include
+&lt;code&gt;translate&lt;/code&gt;, &lt;code&gt;to_csv&lt;/code&gt;, &lt;code&gt;encode&lt;/code&gt;, &lt;code&gt;lpad&lt;/code&gt; / &lt;code&gt;rpad&lt;/code&gt;, &lt;code&gt;round&lt;/code&gt; on floats, &lt;code&gt;abs&lt;/code&gt; on intervals, &lt;code&gt;timestampadd&lt;/code&gt; /
+&lt;code&gt;timestampdiff&lt;/code&gt;, &lt;code&gt;next_day&lt;/code&gt; and &lt;code&gt;levenshtein&lt;/code&gt; on collated input, and Spark expressions that call a Java or
+Scala method directly. &lt;code&gt;rlike&lt;/code&gt; runs natively by
+default for literal patterns that behave the same as in Java.&lt;/p&gt;
+&lt;p&gt;Spark 4 &lt;strong&gt;Variant&lt;/strong&gt; support also improved: native Parquet scans can project Variant columns directly. Thanks to
+&lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt; for driving Variant support, with reviews from &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;. Related PRs: &lt;a href="https://github.com/apache/datafusion-comet/pull/5868"&gt;#5868&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5794"&gt;#5794&lt;/a&gt;.&lt;/p&gt;
+&lt;p&gt;There's also &lt;strong&gt;experimental native support for the in-memory cache&lt;/strong&gt; (&lt;code&gt;spark.comet.exec.inMemoryCache.enabled&lt;/code&gt;,
+off by default).&lt;/p&gt;
+&lt;h2 id="previewing-comet-plans"&gt;Previewing Comet Plans&lt;a class="headerlink" href="#previewing-comet-plans" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;The new &lt;code&gt;spark.comet.explain.planOnly.enabled&lt;/code&gt; setting logs the plan Comet would have run to the driver log,
+then lets Spark run the query as normal. It shows how much of a production workload Comet would accelerate, and
+why anything falls back, without running any of it through Comet.&lt;/p&gt;
+&lt;p&gt;Thanks to &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt; for this feature, with reviews from &lt;a href="https://github.com/coderfender"&gt;@coderfender&lt;/a&gt; and &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;. Related PRs:
+&lt;a href="https://github.com/apache/datafusion-comet/pull/5394"&gt;#5394&lt;/a&gt;.&lt;/p&gt;
+&lt;h2 id="correctness-fixes"&gt;Correctness Fixes&lt;a class="headerlink" href="#correctness-fixes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;1.1.0 fixes many cases where Comet returned different results from Spark, failed where Spark succeeds, or
+accepted input that Spark rejects. Besides the Iceberg and memory fixes above, these are the ones most likely to
+affect 1.0.0 users. The &lt;a href="https://github.com/apache/datafusion-comet/blob/branch-1.1/docs/source/changelog/1.1.0.md"&gt;change log&lt;/a&gt; has the rest.&lt;/p&gt;
+&lt;h3 id="wrong-results"&gt;Wrong results&lt;a class="headerlink" href="#wrong-results" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;Decimal &lt;code&gt;SUM&lt;/code&gt; returned NULL, or raised an overflow error under ANSI, when an intermediate sum overflowed
+  but the final result fit (&lt;a href="https://github.com/apache/datafusion-comet/pull/6041"&gt;#6041&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;After a late shuffle fallback, &lt;code&gt;avg&lt;/code&gt; could return NULL and &lt;code&gt;collect_list&lt;/code&gt; / &lt;code&gt;collect_set&lt;/code&gt; could produce
+  mismatched buffers (&lt;a href="https://github.com/apache/datafusion-comet/pull/5421"&gt;#5421&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Exchange reuse could share one shuffle between plans that differ, such as &lt;code&gt;COUNT(*) + 1&lt;/code&gt; and &lt;code&gt;COUNT(*) - 1&lt;/code&gt;,
+  semi and anti joins, or &lt;code&gt;explode&lt;/code&gt; and &lt;code&gt;explode_outer&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5470"&gt;#5470&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5828"&gt;#5828&lt;/a&gt;, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Two ABFS containers in the same storage account shared a cached object store, so a read could return the
+  other container's data (&lt;a href="https://github.com/apache/datafusion-comet/pull/5053"&gt;#5053&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Dictionary-encoded values hashed differently from the same values decoded, and a null struct hashed its
+  fields, which affected joins, aggregates and shuffle partitioning (&lt;a href="https://github.com/apache/datafusion-comet/pull/5757"&gt;#5757&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5754"&gt;#5754&lt;/a&gt;, &lt;a href="https://github.com/viirya"&gt;@viirya&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Parquet field names containing non-ASCII characters that differ only in case read as NULL (&lt;a href="https://github.com/apache/datafusion-comet/pull/5602"&gt;#5602&lt;/a&gt;,
+  &lt;a href="https://github.com/comphead"&gt;@comphead&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;&lt;code&gt;IN&lt;/code&gt;, &lt;code&gt;InSet&lt;/code&gt;, nested &lt;code&gt;=&lt;/code&gt;, &lt;code&gt;arrays_overlap&lt;/code&gt; and &lt;code&gt;array_position&lt;/code&gt; now treat &lt;code&gt;-0.0&lt;/code&gt; and &lt;code&gt;0.0&lt;/code&gt;, and every NaN
+  encoding, as Spark does (&lt;a href="https://github.com/apache/datafusion-comet/pull/6073"&gt;#6073&lt;/a&gt;, &lt;a href="https://github.com/mizulun"&gt;@mizulun&lt;/a&gt;, &lt;a href="https://github.com/apache/datafusion-comet/pull/5235"&gt;#5235&lt;/a&gt;, &lt;a href="https://github.com/divyankshah"&gt;@divyankshah&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5472"&gt;#5472&lt;/a&gt;, &lt;a href="https://github.com/sunchao"&gt;@sunchao&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Decimal to double and float casts were off by one unit in the last place for most &lt;code&gt;DECIMAL(38,18)&lt;/code&gt; values
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5684"&gt;#5684&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;String to timestamp casts now follow Spark's parsing rules for short fields, time zones and signed years
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5682"&gt;#5682&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5858"&gt;#5858&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="errors-spark-raises-that-comet-did-not"&gt;Errors Spark raises that Comet did not&lt;a class="headerlink" href="#errors-spark-raises-that-comet-did-not" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;Casts and expressions routed through codegen dispatch could skip ANSI errors raised inside a constant
+  subexpression (&lt;a href="https://github.com/apache/datafusion-comet/pull/5623"&gt;#5623&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Rejected &lt;code&gt;TIMESTAMP_NTZ&lt;/code&gt; casts returned NULL under ANSI instead of raising &lt;code&gt;CAST_INVALID_INPUT&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5752"&gt;#5752&lt;/a&gt;,
+  &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Out-of-range Parquet &lt;code&gt;TIMESTAMP_MILLIS&lt;/code&gt; values, top-level or nested, silently wrapped (&lt;a href="https://github.com/apache/datafusion-comet/pull/5177"&gt;#5177&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5740"&gt;#5740&lt;/a&gt;,
+  &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Nested Parquet struct, list and map fields now follow Spark's conversion rules instead of returning NULL on
+  overflow or accepting values Spark rejects (&lt;a href="https://github.com/apache/datafusion-comet/pull/5681"&gt;#5681&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="query-failures-and-crashes"&gt;Query failures and crashes&lt;a class="headerlink" href="#query-failures-and-crashes" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;code&gt;collect_list&lt;/code&gt; and &lt;code&gt;collect_set&lt;/code&gt; over nested arguments failed with "column types must match schema types"
+  (&lt;a href="https://github.com/apache/datafusion-comet/pull/5159"&gt;#5159&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Native shuffle failed with a 2 GB task serialization error on jobs with very many partitions (&lt;a href="https://github.com/apache/datafusion-comet/pull/5392"&gt;#5392&lt;/a&gt;,
+  &lt;a href="https://github.com/parthchandra"&gt;@parthchandra&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;A Scala UDF from a user jar failed with a &lt;code&gt;ClassCastException&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5282"&gt;#5282&lt;/a&gt;, &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;&lt;code&gt;rpad&lt;/code&gt; and &lt;code&gt;lpad&lt;/code&gt; panicked on a NULL length (&lt;a href="https://github.com/apache/datafusion-comet/pull/5680"&gt;#5680&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;Structs with duplicate field names failed the task in native shuffle, and panicked in the native Parquet
+  scan (&lt;a href="https://github.com/apache/datafusion-comet/pull/5866"&gt;#5866&lt;/a&gt;, &lt;a href="https://github.com/dwsmith1983"&gt;@dwsmith1983&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5786"&gt;#5786&lt;/a&gt;, &lt;a href="https://github.com/ErikBPF"&gt;@ErikBPF&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="hangs-and-resource-use"&gt;Hangs and resource use&lt;a class="headerlink" href="#hangs-and-resource-use" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;The JVM hung on exit when an application returned from &lt;code&gt;main&lt;/code&gt; without calling &lt;code&gt;spark.stop()&lt;/code&gt; (&lt;a href="https://github.com/apache/datafusion-comet/pull/5748"&gt;#5748&lt;/a&gt;,
+  &lt;a href="https://github.com/zhangfengcdt"&gt;@zhangfengcdt&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;The native scan busy-polled while waiting on S3 or HDFS reads, keeping one core per task at 100% (&lt;a href="https://github.com/apache/datafusion-comet/pull/6219"&gt;#6219&lt;/a&gt;,
+  &lt;a href="https://github.com/mixermt"&gt;@mixermt&lt;/a&gt; and &lt;a href="https://github.com/andygrove"&gt;@andygrove&lt;/a&gt;).&lt;/li&gt;
+&lt;li&gt;One task could force-spill another task's shuffle buffers, and a failed shuffle write leaked its memory
+  reservation (&lt;a href="https://github.com/apache/datafusion-comet/pull/5493"&gt;#5493&lt;/a&gt; and &lt;a href="https://github.com/apache/datafusion-comet/pull/5461"&gt;#5461&lt;/a&gt;, &lt;a href="https://github.com/peterxcli"&gt;@peterxcli&lt;/a&gt;).&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="upgrading-to-110"&gt;Upgrading to 1.1.0&lt;a class="headerlink" href="#upgrading-to-110" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;A few changes in 1.1.0 can affect a deployment. The &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/migration-guide.html"&gt;Comet Upgrade Guide&lt;/a&gt; has the details.&lt;/p&gt;
+&lt;h3 id="platform"&gt;Platform&lt;a class="headerlink" href="#platform" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;JDK 17 or later is required.&lt;/strong&gt; JDK 11 support was removed, as announced in 1.0.0.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.4 is still deprecated.&lt;/strong&gt; Comet still publishes Spark 3.4 binaries, but Spark's SQL test suite runs
+  against 3.4 only on demand, so 3.4-specific regressions are more likely. We recommend Spark 3.5 or later.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h3 id="size-settings-now-read-correctly"&gt;Size settings now read correctly&lt;a class="headerlink" href="#size-settings-now-read-correctly" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;1.0.0 misread three size settings, so jobs that set them may behave differently:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.memory.offHeap.size&lt;/code&gt; as a bare number of bytes&lt;/strong&gt; was read as MiB, which made &lt;code&gt;fair_unified&lt;/code&gt;'s
+  per-operator shares about a million times too large. With correct shares, operators can spill sooner, and an
+  operator that can't spill can fail when it exceeds its share. Sizes with a unit, like &lt;code&gt;16g&lt;/code&gt;, aren't affected.
+  &lt;code&gt;spark.comet.exec.memoryPool=greedy_unified&lt;/code&gt; restores the old behavior.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.maxTempDirectorySize&lt;/code&gt; with a unit&lt;/strong&gt;, like &lt;code&gt;10g&lt;/code&gt;, was ignored in favor of the 100 GB default.
+  It's now enforced, so a query that spills past it fails.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.shuffle.native.writeBufferSize&lt;/code&gt; with a unit&lt;/strong&gt; was treated as bytes, so &lt;code&gt;64m&lt;/code&gt; meant 64 bytes. It
+  now means what it says. These buffers use untracked native memory, so check large values against
+  &lt;code&gt;spark.executor.memoryOverhead&lt;/code&gt;.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Malformed values for &lt;code&gt;spark.comet.maxTempDirectorySize&lt;/code&gt; and &lt;code&gt;spark.comet.explain.native.enabled&lt;/code&gt; now fail the
+query instead of falling back to the default.&lt;/p&gt;
+&lt;h3 id="more-memory-before-spilling"&gt;More memory before spilling&lt;a class="headerlink" href="#more-memory-before-spilling" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;With the &lt;code&gt;fair_unified&lt;/code&gt; fix, tasks with several operators can reserve more memory before spilling than in
+0.15.0 through 1.0.0, especially on executors running few tasks at once. If you sized executors for one of those
+releases, check the memory usage log for headroom.&lt;/p&gt;
+&lt;h3 id="conditions-for-enabling-comet"&gt;Conditions for enabling Comet&lt;a class="headerlink" href="#conditions-for-enabling-comet" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;p&gt;Comet requires Spark's off-heap memory. &lt;code&gt;CometPlugin&lt;/code&gt; already enforced this, and 1.1.0 applies the same check
+when &lt;code&gt;CometSparkSessionExtensions&lt;/code&gt; is registered directly, disabling Comet with a warning. Both checks read
+&lt;code&gt;spark.memory.offHeap.enabled&lt;/code&gt; from the SparkContext, so set it when the application starts, not on a
+&lt;code&gt;SparkSession.builder&lt;/code&gt; after the context exists.&lt;/p&gt;
+&lt;p&gt;Comet also checks the shuffle manager that's actually running, not the session's &lt;code&gt;spark.shuffle.manager&lt;/code&gt;. A
+session that names &lt;code&gt;CometShuffleManager&lt;/code&gt; after the context started with a different one now runs without Comet,
+with a warning, instead of failing with a &lt;code&gt;ClassCastException&lt;/code&gt;.&lt;/p&gt;
+&lt;h3 id="deprecated-and-removed-settings"&gt;Deprecated and removed settings&lt;a class="headerlink" href="#deprecated-and-removed-settings" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h3&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;&lt;code&gt;spark.comet.exec.memoryPool.fraction&lt;/code&gt;&lt;/strong&gt; is deprecated and will be removed in a future major release. It
+  still works, and the driver warns when it's set.&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Testing-only on-heap settings&lt;/strong&gt; were removed: &lt;code&gt;spark.comet.memoryOverhead&lt;/code&gt;,
+  &lt;code&gt;spark.comet.exec.onHeap.memoryPool&lt;/code&gt;, and &lt;code&gt;spark.comet.shuffle.jvm.memoryFactor&lt;/code&gt; (and its old name,
+  &lt;code&gt;spark.comet.columnar.shuffle.memory.factor&lt;/code&gt;). The &lt;a href="https://datafusion.apache.org/comet/about/versioning_policy.html#testing-and-internal-configurations-are-exempt"&gt;versioning policy&lt;/a&gt; exempts testing settings, and Comet
+  ignores them if they're still set.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;h2 id="compatibility"&gt;Compatibility&lt;a class="headerlink" href="#compatibility" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Supported platforms:&lt;/p&gt;
+&lt;ul&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.4.3&lt;/strong&gt; with Java 17 and Scala 2.12/2.13 (deprecated)&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 3.5.9&lt;/strong&gt; with Java 17 and Scala 2.12/2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.0.4&lt;/strong&gt; with Java 17/21 and Scala 2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.1.3&lt;/strong&gt; with Java 17/21 and Scala 2.13&lt;/li&gt;
+&lt;li&gt;&lt;strong&gt;Spark 4.2.0&lt;/strong&gt; with Java 17 and Scala 2.13 (experimental, for early evaluation only)&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;See the &lt;a href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html"&gt;Spark Version Compatibility&lt;/a&gt; page for known limitations specific to each version.&lt;/p&gt;
+&lt;p&gt;This release builds on &lt;strong&gt;DataFusion 55.1&lt;/strong&gt; and &lt;strong&gt;Arrow 59.2&lt;/strong&gt;.&lt;/p&gt;
+&lt;h2 id="get-started-with-comet-110"&gt;Get Started with Comet 1.1.0&lt;a class="headerlink" href="#get-started-with-comet-110" title="Permanent link"&gt;¶&lt;/a&gt;&lt;/h2&gt;
+&lt;p&gt;Follow the &lt;a href="https://datafusion.apache.org/comet/user-guide/1.1/installation.html"&gt;Comet 1.1.0 Installation Guide&lt;/a&gt;
+to get up and running, then point Comet at your existing Spark workloads.&lt;/p&gt;</content><category term="blog"/></entry><entry><title>Apache DataFusion 55.0.0 Released</title><link href="https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0" rel="alternate"/><published>2026-08-25T00:00:00+00:00</published><updated>2026-08-25T00:00:00+00:00</updated><author><name>pmc</name></author><id>tag:datafusion.apache.org,2026-08-25:/blog/2026/08/25/datafusion-55.0.0</id><summary type= [...]
 {% comment %}
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
diff --git a/output/feeds/pmc.rss.xml b/output/feeds/pmc.rss.xml
index 1a1c70f..33a19ad 100644
--- a/output/feeds/pmc.rss.xml
+++ b/output/feeds/pmc.rss.xml
@@ -1,5 +1,27 @@
 <?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - pmc</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/pmc.rss.xml" rel="self"/><lastBuildDate>Tue, 25 Aug 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion 55.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0</link><description>&lt;!--
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache DataFusion Blog - pmc</title><link>https://datafusion.apache.org/blog/</link><description/><atom:link href="https://datafusion.apache.org/blog/feeds/pmc.rss.xml" rel="self"/><lastBuildDate>Thu, 01 Oct 2026 00:00:00 +0000</lastBuildDate><item><title>Apache DataFusion Comet 1.1.0 Release</title><link>https://datafusion.apache.org/blog/2026/10/01/datafusion-comet-1.1.0</link><description>&lt;!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+--&gt;
+
+&lt;p&gt;The Apache DataFusion PMC is pleased to announce version 1.1.0 of the &lt;a href="https://datafusion.apache.org/comet/"&gt;Comet&lt;/a&gt; subproject.&lt;/p&gt;
+&lt;p&gt;Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.&lt;/p&gt;
+&lt;p&gt;This release covers roughly seven weeks of development …&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">pmc</dc:creator><pubDate>Thu, 01 Oct 2026 00:00:00 +0000</pubDate><guid>tag:datafusion.apache.org,2026-10-01:/blog/2026/10/01/datafusion-comet-1.1.0</guid><category>blog</category></item><item><title>Apache DataFusion 55.0.0 Released</title><link>https://datafusion.apache.org/blog/2026/08/25/datafusion-55.0.0</link><description>&lt;!--
 {% comment %}
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
diff --git a/output/images/comet-1.1.0/comet-executor-memory.svg b/output/images/comet-1.1.0/comet-executor-memory.svg
new file mode 100644
index 0000000..6cd0763
--- /dev/null
+++ b/output/images/comet-1.1.0/comet-executor-memory.svg
@@ -0,0 +1,88 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="960" height="390" viewBox="0 0 960 390" role="img" aria-labelledby="title desc">
+  <title id="title">How Comet uses executor memory</title>
+  <desc id="desc">An executor container holds the JVM heap, the off-heap memory pool, and the memory overhead. Spark and Comet both use the JVM heap and share the off-heap memory pool. The memory overhead holds the JVM's own overhead and the rest of Comet's native memory, which the pool does not track.</desc>
+  <style>
+    .sans { font-family: Inter, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif; }
+    .mono { font-family: 'JetBrains Mono', ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
+  </style>
+  <defs>
+    <clipPath id="poolClip">
+      <rect x="322" y="134" width="286" height="96" rx="8"/>
+    </clipPath>
+    <marker id="arrowHead" viewBox="0 0 10 10" refX="5" refY="5" markerWidth="5" markerHeight="5" orient="auto-start-reverse">
+      <path d="M0,0 L10,5 L0,10 z" fill="#444"/>
+    </marker>
+  </defs>
+
+  <rect width="960" height="390" fill="#ffffff"/>
+
+  <!-- Executor container -->
+  <rect x="20" y="20" width="920" height="244" rx="16" fill="#F5F6F8" stroke="#A0A4AA" stroke-width="1.5" stroke-dasharray="7 5"/>
+  <text class="sans" x="40" y="48" font-size="15" font-weight="700" fill="#555">Executor container</text>
+
+  <!-- JVM heap -->
+  <rect x="40" y="62" width="250" height="184" rx="12" fill="#ffffff" stroke="#555" stroke-width="2"/>
+  <text class="sans" x="165" y="92" font-size="18" font-weight="700" fill="#222" text-anchor="middle">JVM heap</text>
+  <text class="mono" x="165" y="116" font-size="13.5" fill="#A84808" text-anchor="middle">spark.executor.memory</text>
+  <rect x="56" y="134" width="124" height="96" rx="8" fill="#FDE7A7" stroke="#D9AE2E" stroke-width="1.5"/>
+  <text class="sans" x="118" y="188" font-size="16" font-weight="700" fill="#333" text-anchor="middle">Spark</text>
+  <rect x="190" y="134" width="84" height="96" rx="8" fill="#C8EBB5" stroke="#6BAF4E" stroke-width="1.5"/>
+  <text class="sans" x="232" y="188" font-size="16" font-weight="700" fill="#333" text-anchor="middle">Comet</text>
+
+  <!-- Off-heap memory pool: one budget, split between Spark and Comet -->
+  <rect x="306" y="62" width="318" height="184" rx="12" fill="#ffffff" stroke="#555" stroke-width="2"/>
+  <text class="sans" x="465" y="92" font-size="18" font-weight="700" fill="#222" text-anchor="middle">Off-heap memory pool</text>
+  <text class="mono" x="465" y="116" font-size="13.5" fill="#A84808" text-anchor="middle">spark.memory.offHeap.size</text>
+  <g clip-path="url(#poolClip)">
+    <rect x="322" y="134" width="143" height="96" fill="#FDE7A7"/>
+    <rect x="465" y="134" width="143" height="96" fill="#C8EBB5"/>
+  </g>
+  <rect x="322" y="134" width="286" height="96" rx="8" fill="none" stroke="#8a8a8a" stroke-width="1.5"/>
+  <line x1="465" y1="134" x2="465" y2="230" stroke="#777" stroke-width="1.5" stroke-dasharray="4 4"/>
+  <text class="sans" x="393" y="180" font-size="16" font-weight="700" fill="#333" text-anchor="middle">Spark</text>
+  <text class="sans" x="537" y="180" font-size="16" font-weight="700" fill="#333" text-anchor="middle">Comet</text>
+  <line x1="443" y1="210" x2="487" y2="210" stroke="#444" stroke-width="2" marker-start="url(#arrowHead)" marker-end="url(#arrowHead)"/>
+  <circle cx="602" cy="84" r="13" fill="#E8650A"/>
+  <text class="sans" x="602" y="89" font-size="14" font-weight="700" fill="#fff" text-anchor="middle">1</text>
+
+  <!-- Memory overhead -->
+  <rect x="640" y="62" width="280" height="184" rx="12" fill="#ffffff" stroke="#555" stroke-width="2"/>
+  <text class="sans" x="780" y="92" font-size="18" font-weight="700" fill="#222" text-anchor="middle">Memory overhead</text>
+  <text class="mono" x="780" y="116" font-size="13.5" fill="#A84808" text-anchor="middle">spark.executor.memoryOverhead</text>
+  <rect x="656" y="134" width="104" height="96" rx="8" fill="#FDE7A7" stroke="#D9AE2E" stroke-width="1.5"/>
+  <text class="sans" x="708" y="178" font-size="16" font-weight="700" fill="#333" text-anchor="middle">JVM</text>
+  <text class="sans" x="708" y="198" font-size="13" fill="#555" text-anchor="middle">overhead</text>
+  <rect x="770" y="134" width="134" height="96" rx="8" fill="#C8EBB5" stroke="#6BAF4E" stroke-width="1.5"/>
+  <text class="sans" x="837" y="188" font-size="16" font-weight="700" fill="#333" text-anchor="middle">Comet</text>
+  <circle cx="898" cy="84" r="13" fill="#E8650A"/>
+  <text class="sans" x="898" y="89" font-size="14" font-weight="700" fill="#fff" text-anchor="middle">2</text>
+
+  <!-- Legend -->
+  <g class="sans" font-size="14" fill="#444">
+    <rect x="44" y="291" width="18" height="18" rx="4" fill="#FDE7A7" stroke="#D9AE2E" stroke-width="1.5"/>
+    <text x="72" y="305">Spark</text>
+    <rect x="44" y="321" width="18" height="18" rx="4" fill="#C8EBB5" stroke="#6BAF4E" stroke-width="1.5"/>
+    <text x="72" y="335">Comet</text>
+  </g>
+
+  <!-- Callouts -->
+  <g class="sans">
+    <circle cx="319" cy="296" r="12" fill="#E8650A"/>
+    <text x="319" y="301" font-size="13" font-weight="700" fill="#fff" text-anchor="middle">1</text>
+    <text x="338" y="301" font-size="15" font-weight="700" fill="#222">Shared with Spark</text>
+    <text font-size="14" fill="#444">
+      <tspan x="338" y="324">Comet's sorts, joins, aggregations</tspan>
+      <tspan x="338" y="344">and shuffles reserve memory from</tspan>
+      <tspan x="338" y="364">the same pool that Spark uses.</tspan>
+    </text>
+
+    <circle cx="653" cy="296" r="12" fill="#E8650A"/>
+    <text x="653" y="301" font-size="13" font-weight="700" fill="#fff" text-anchor="middle">2</text>
+    <text x="672" y="301" font-size="15" font-weight="700" fill="#222">Not tracked by the pool</text>
+    <text font-size="14" fill="#444">
+      <tspan x="672" y="324">Comet also needs native memory</tspan>
+      <tspan x="672" y="344">outside the JVM. Make sure</tspan>
+      <tspan x="672" y="364">memoryOverhead has room for it.</tspan>
+    </text>
+  </g>
+</svg>
diff --git a/output/index.html b/output/index.html
index 5551bf4..a65edee 100644
--- a/output/index.html
+++ b/output/index.html
@@ -53,6 +53,47 @@
             <p><i>Here you can find the latest updates from DataFusion and related projects.</i></p>
 
 
+    <!-- Post -->
+    <div class="row">
+        <div class="callout">
+            <article class="post">
+                <header>
+                    <div class="title">
+                        <h1><a href="/blog/2026/10/01/datafusion-comet-1.1.0">Apache DataFusion Comet 1.1.0 Release</a></h1>
+                        <p>Posted on: Thu 01 October 2026 by pmc</p>
+                        <p><!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<p>The Apache DataFusion PMC is pleased to announce version 1.1.0 of the <a href="https://datafusion.apache.org/comet/">Comet</a> subproject.</p>
+<p>Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.</p>
+<p>This release covers roughly seven weeks of development …</p></p>
+                        <footer>
+                            <ul class="actions">
+                                <div style="text-align: right"><a href="/blog/2026/10/01/datafusion-comet-1.1.0" class="button medium">Continue Reading</a></div>
+                            </ul>
+                            <ul class="stats">
+                            </ul>
+                        </footer>
+            </article>
+        </div>
+    </div>
     <!-- Post -->
     <div class="row">
         <div class="callout">


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org