You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/09/22 17:30:04 UTC
(datafusion-site) branch site/comet-1.1.0 created (now 5aa9980)
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a change to branch site/comet-1.1.0
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
at 5aa9980 Add Comet 1.1.0 release blog post draft
This branch includes the following new commits:
new 5aa9980 Add Comet 1.1.0 release blog post draft
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-site) 01/01: Add Comet 1.1.0 release blog post draft
Posted by ag...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch site/comet-1.1.0
in repository https://gitbox.apache.org/repos/asf/datafusion-site.git
commit 5aa9980e4d3c854dee08cff39fcf0924487794b5
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Sep 22 11:29:50 2026 -0600
Add Comet 1.1.0 release blog post draft
Drafted from the 333 commits on apache/main since the 1.0.0 tag, with
native Iceberg writes (experimental) as the headline feature.
---
content/blog/2026-09-25-datafusion-comet-1.1.0.md | 267 ++++++++++++++++++++++
1 file changed, 267 insertions(+)
diff --git a/content/blog/2026-09-25-datafusion-comet-1.1.0.md b/content/blog/2026-09-25-datafusion-comet-1.1.0.md
new file mode 100644
index 0000000..03c54bb
--- /dev/null
+++ b/content/blog/2026-09-25-datafusion-comet-1.1.0.md
@@ -0,0 +1,267 @@
+---
+layout: post
+title: Apache DataFusion Comet 1.1.0 Release
+date: 2026-09-25
+author: pmc
+categories: [subprojects]
+---
+
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+[TOC]
+
+The Apache DataFusion PMC is pleased to announce version 1.1.0 of the [Comet](https://datafusion.apache.org/comet/) subproject.
+
+Comet is an accelerator for Apache Spark that translates Spark physical plans to DataFusion physical plans for
+improved performance and efficiency without requiring any code changes.
+
+This release covers roughly seven weeks of development since 1.0.0 and consists of 333 commits from 37
+contributors. See the [change log] for the full list of changes.
+
+[change log]: https://github.com/apache/datafusion-comet/blob/main/docs/source/changelog/1.1.0.md
+
+## Native Iceberg Writes (Experimental)
+
+Every Comet release up to now has accelerated the read side of an Iceberg workload while leaving writes
+entirely to the JVM. 1.1.0 changes that: Comet can now write Iceberg data files natively, through
+[iceberg-rust](https://github.com/apache/iceberg-rust), inside its existing native execution pipeline. The
+feature is **experimental and disabled by default**, and the rest of this section explains both what it does
+and the fairly strict conditions under which it will engage.
+
+### The problem: one operator, no AQE
+
+Spark writes an Iceberg table through a single physical operator that combines data-file writing with
+metadata writing, committing, and catalog validation. Because that operator sits outside Spark's Adaptive
+Query Execution, the sub-query feeding the write — the scans, projects, sorts, and exchanges producing the
+rows — cannot be re-planned at runtime. It is also opaque to Comet's columnar rules, so the write itself was
+never a candidate for native execution.
+
+### Step one: split the write operator
+
+The first half of the work splits that operator in two. With
+`spark.comet.write.iceberg.splitOperator.enabled=true`, Comet rewrites eligible Iceberg writes into:
+
+1. **`IcebergWrite`** — writes the data files on the executors and returns each task's serialized commit
+ message. This operator, and the sub-query feeding it, run inside AQE.
+2. **`IcebergCommit`** — collects the commit messages on the driver and performs the normal Iceberg commit,
+ including commit-time validation, outside AQE, exactly once.
+
+On its own this toggle changes nothing about who writes the bytes — data files are still written by
+iceberg-java. What it buys is that the write's input becomes visible to AQE and to Comet's columnar rules,
+and it creates the seam that the native writer plugs into.
+
+The split plan covers `INSERT INTO` / DataFrame `append`, static and dynamic `INSERT OVERWRITE`, and
+copy-on-write `DELETE` / `UPDATE` / `MERGE`, on every Spark version Comet supports. Merge-on-read delta
+writes are not intercepted. When the rewrite is skipped — an unrecognized write class, a CTAS on Spark 3.4,
+a write requiring Spark's commit coordinator — the write is planned exactly as if Comet were absent.
+
+### Step two: write the Parquet files natively
+
+With `spark.comet.iceberg.write.enabled=true`, the `IcebergWrite` operator's per-task Parquet write is
+delegated to iceberg-rust. The JVM-side planner marshals everything the native writer needs into the
+serialized plan: the write schema and partition spec as JSON, the data location, resolved Parquet writer
+settings, the writer mode (unpartitioned, fanout, or clustered, mirroring `SparkWrite`'s own choice),
+object-store configuration, and per-task IDs. On each task, iceberg-rust writes the Parquet files and returns
+its `DataFile` metadata packed as a single in-memory Iceberg V2 data manifest.
+
+Everything after that point is deliberately left to iceberg-java. The JVM decodes the manifest bytes with
+Iceberg's own `ManifestFiles.read`, re-derives each file's manifest metrics from the written Parquet footer
+using Iceberg's `MetricsConfig` logic, and wraps the result in the same `TaskCommit` message the JVM writer
+would have produced. Snapshot assignment, manifest-list aggregation, commit validation, and retries are
+untouched.
+
+### Fidelity as a design constraint
+
+The native writer has to produce the same outcome as iceberg-java, not merely a valid Iceberg table, and the
+design leans hard in that direction.
+
+Manifest metrics drive partition- and file-level pruning for every future reader of the table, so a
+divergence there would outlive the write. Rather than trusting what the native writer reports, Comet
+re-derives metrics on the JVM from each file's Parquet footer through iceberg-java's own
+`ParquetUtil.footerMetrics` and `MetricsConfig.forTable`. Metrics modes, bound truncation, the inferred-column
+cap, and list/map bounds suppression are therefore iceberg-java's code making iceberg-java's decisions, and
+the parity suite compares committed manifests byte-for-byte against JVM-written ones. The cost is one
+footer-sized ranged read per written file.
+
+Eligibility detection is an allowlist, not a denylist. A write is eligible only when its entire effective
+configuration matches a documented table of supported settings — and anything else, including any
+write-affecting property added by a future Iceberg version, any value outside the supported set, or any
+reflection failure while inspecting the write, falls back to iceberg-java with the reason reported in Comet's
+extended `EXPLAIN` output. Encryption keys, object-storage layout, custom location providers, bloom filters,
+and unvetted `parquet.*` properties all decline the native path. The plan feeding the write must also be
+fully Comet-native, which for a partitioned table includes the hash distribution and local sort Iceberg
+requests on its partition transforms — those stay native because the Iceberg system functions now have native
+implementations (see below).
+
+Eligibility is decided entirely at plan time, including the reflection surface: every iceberg-java class and
+method the executor-side commit assembly needs is eagerly resolved on the driver, so an Iceberg release that
+moves any of them declines the native path rather than failing tasks mid-write.
+
+### Failures never commit partial results
+
+The commit set is exactly the commit messages returned by successful tasks. A failed task contributes none
+and deletes the data files it created, as iceberg-java's writer abort does; ownership of those files is
+handed from the native side to a JVM task-failure listener at a well-defined point, so a failure on either
+side of the boundary still cleans up. If the job fails, the driver-side commit operator aborts without
+committing anything and deletes the completed tasks' files through the table `FileIO`. Task retries cannot
+collide, because each attempt's task attempt id is embedded in its data file names. Anything a best-effort
+deletion misses is invisible to readers, which resolve files through committed manifests only, and is
+reclaimed by Iceberg's normal `remove_orphan_files` maintenance.
+
+### Accepted divergences
+
+A handful of differences between parquet-mr and parquet-rs are unconditional, and enabling the toggle accepts
+them. They are almost entirely cosmetic — footer key-value metadata, the root schema element name,
+`created_by`, absent page CRCs and page-header statistics, `RLE_DICTIONARY` labeling, compressed page bytes —
+and none change what a reader computes. Two are worth knowing about operationally:
+
+- **File rolling lands on the same 1000-row grid as iceberg-java, but not necessarily on the same row.** Both
+ writers re-check file size against `write.target-file-size-bytes` every 1000 rows, but they compare
+ different size estimates, so nothing bounds how far apart their roll points are. Do not rely on file-layout
+ parity between the two writers.
+- **Float and double partition directory names** are rendered with Rust's shortest representation rather than
+ `Float.toString` (`f=1` where iceberg-java writes `f=1.0`). Distinct values still get distinct directories,
+ and no reader parses these names. Iceberg deprecated float and double partitioning in 1.3.
+
+The [Iceberg Writes guide] documents the full eligibility table and every accepted divergence. Please try it
+on a non-production table and tell us what you find — feedback from real workloads is exactly what this
+feature needs before it can lose the experimental label.
+
+[Iceberg Writes guide]: https://datafusion.apache.org/comet/user-guide/latest/iceberg-writes.html
+
+## More Iceberg Improvements
+
+The read side gained several things in this release too:
+
+- **Iceberg V3 deletion vectors** are now applied on native scans.
+- **Iceberg system functions** — `bucket`, `truncate`, `years`, `months`, `days`, and `hours` — have native
+ implementations. Besides being faster, this is what keeps a partitioned table's write plan fully native and
+ therefore eligible for the native writer.
+- **Scan planning metrics and scan time** are reported in the Spark UI for the native Iceberg scan.
+- Fixes for tables partitioned by an unknown transform, complex null checks on native scans, and Iceberg
+ system functions wrapped as `ApplyFunctionExpression`.
+
+## Native Parquet Writes on Spark 4.0+
+
+Separately from Iceberg, 1.1.0 hooks native Parquet writes into Spark's `WriteFilesExec` seam on Spark 4.0 and
+later, behind `spark.comet.parquet.write.enabled`. This release also preserves Catalyst nullability and field
+IDs in native Parquet writes.
+
+## Remote Shuffle with Celeborn
+
+Applications using Apache Celeborn can now use Comet's composite shuffle manager to run Comet's **native**
+shuffle over Celeborn, rather than falling back to ordinary Spark/Celeborn shuffle for every exchange. This
+landed as a series of changes adding an RSS partition writer, task-owned JNI callbacks, destination-aware
+native shuffle execution, the map-side push lifecycle, a raw native shuffle reader, and native-only planning.
+
+Native shuffle over Celeborn requires an explicit `spark.comet.shuffle.mode=native`; the default `auto` mode
+retains ordinary Spark/Celeborn shuffle. It also requires a Celeborn client that provides a safe
+push-completion API — released 0.6.0 and 0.7.0 clients do not, and those versions retain ordinary shuffle even
+in native mode. Native RSS does not support `spark.io.encryption.enabled=true`. Celeborn is an optional
+application dependency and is not bundled with Comet. See the [tuning guide] for the full set of
+requirements and the frame-size and in-flight-bytes knobs.
+
+[tuning guide]: https://datafusion.apache.org/comet/user-guide/latest/tuning.html
+
+## Performance
+
+### Shuffle
+
+Several changes target shuffle, which dominates many TPC-DS-shaped workloads:
+
+- A task now spills **every shuffle partition into one file** instead of one file per partition.
+- **Zstd compression contexts and Arrow IPC contexts are reused** across shuffle blocks instead of being
+ rebuilt per block.
+- Shuffle blocks are **decoded against a cached schema** rather than re-parsing the schema per block, and
+ expected schemas are cached for remote shuffle decoding.
+- Per-partition scratch buffers are reused in the shuffle write path.
+- `ArrowWriter` bulk-copies fixed-width columns.
+
+### Planning and Execution
+
+- **Native dynamic filter pushdown** from hash joins into Parquet scans.
+- **Adaptive partial aggregation** is enabled for eligible native shuffle plans.
+- **Parsed plan data is cached across the tasks of a stage**, avoiding repeated deserialization work.
+- **Native Parquet scan I/O and read-amplification metrics** are exposed, alongside native aggregate spill and
+ memory metrics.
+
+### Expression Kernels
+
+Ongoing kernel-level work continued across the expression library: `map_sort` is up to 3x faster for
+multi-entry string maps and 18x faster on singleton normalization; the native map lookup behind `element_at`
+and `GetMapValue` is vectorized; `collect_list` and `collect_set` gained a native `GroupsAccumulator`; user
+regex patterns are compiled once per planned expression; `hour`/`minute`/`second` and `dayofweek`/`weekday`
+skip calendar reconstruction; `posexplode` array expressions are evaluated once per batch; unnesting slices
+the child rather than gathering it; and the nested-element list hash is batched for flat struct elements.
+
+## Memory Observability
+
+Two features make Comet's memory use easier to reason about: **native allocation accounting**, and **tracing
+of Arrow memory held on the JVM side**. Together they close a long-standing gap where memory held by the
+native engine and by Arrow buffers on the JVM was hard to attribute when diagnosing executor OOMs.
+
+## Expanded Coverage
+
+New expression and operator support in this release includes the `mode` aggregate, `max_by` / `min_by`, the
+`regr_slope` / `regr_intercept` / `regr_r2` / `regr_sxx` / `regr_syy` / `regr_sxy` regression aggregates,
+`WindowGroupLimitExec`, `explode_outer`, `make_interval`, a native `spark_sequence` kernel for integral
+element types, a native `spark_unbase64` kernel, `_metadata` constant columns in the native Parquet scan,
+nested types as native shuffle hash partitioning keys, `BinaryType` for sort-merge join, and Spark 4's
+`EmptyRelationExec` as a native input. Several more expressions — `translate`, `to_csv`, `encode`, `lpad` /
+`rpad`, `round` on float/double, `abs` on intervals, `timestampadd` / `timestampdiff`, `next_day` and
+`levenshtein` on collated input, and unrecognized `StaticInvoke` / `Invoke` — are now routed through codegen
+dispatch by default, and `rlike` runs natively by default for Java-equivalent literal patterns.
+
+Spark 4 **Variant** support advanced as well, with Parquet storage adapted for Variant projection, Variant
+arrays normalized at the native Parquet boundary, and `VariantType` identity carried through schema
+serialization.
+
+This release also adds **experimental native support for an in-memory cache**
+(`spark.comet.exec.inMemoryCache.enabled`, disabled by default), support for **S3-compliant filesystems**, and
+build gates for contrib **Delta** and **Lance** scans.
+
+## Deprecations and Removals
+
+- **JDK 11 support has been removed**, as announced in the 1.0.0 release. Comet 1.1.0 requires JDK 17 or
+ later.
+- **Apache Spark 3.4 remains deprecated.** Comet continues to build and publish Spark 3.4 binaries, but
+ Spark's own SQL test suite no longer runs against Spark 3.4 on every change, so Spark 3.4-specific
+ regressions are more likely to reach a release. We recommend moving to Spark 3.5 or later.
+
+## Compatibility
+
+Supported platforms include:
+
+- **Spark 3.4.3** with Java 17 and Scala 2.12/2.13 (deprecated)
+- **Spark 3.5.9** with Java 17 and Scala 2.12/2.13
+- **Spark 4.0.4** with Java 17 and Scala 2.13
+- **Spark 4.1.3** with Java 17/21 and Scala 2.13
+- **Spark 4.2.0** with Java 17 and Scala 2.13 (experimental, for early evaluation only)
+
+See the [Spark Version Compatibility] page for known limitations specific to each version.
+
+[Spark Version Compatibility]: https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html
+
+This release builds on **DataFusion 55.1** and **Arrow 59.2**.
+
+## Get Started with Comet 1.1.0
+
+Ready to try it out? Follow the [Comet 1.1.0 Installation Guide](https://datafusion.apache.org/comet/user-guide/1.1/installation.html)
+to get up and running, then point Comet at your existing Spark workloads and see the speedup for yourself.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org