You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/06 19:23:28 UTC
(datafusion-comet) branch main updated (62c5d44921 -> 275d09db82)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
from 62c5d44921 perf: skip Spark's encoders for boxed primitive Scala UDF values in the codegen dispatcher (#6714)
new f651144b9d add benches for remaining comet native kernels (#6648)
new 275d09db82 docs: add 1.1.0 changelog (#6728)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/source/changelog/1.1.0.md | 503 +++++++++++++++++++++
docs/source/changelog/index.md | 1 +
native/spark-expr/Cargo.toml | 16 +
.../benches/bloom_filter_might_contain.rs | 144 ++++++
native/spark-expr/benches/cast_nested.rs | 141 ++++++
.../benches/{aggregate_misc.rs => hll_agg.rs} | 159 +++----
native/spark-expr/benches/hll_scalar.rs | 74 +++
7 files changed, 941 insertions(+), 97 deletions(-)
create mode 100644 docs/source/changelog/1.1.0.md
create mode 100644 native/spark-expr/benches/bloom_filter_might_contain.rs
create mode 100644 native/spark-expr/benches/cast_nested.rs
copy native/spark-expr/benches/{aggregate_misc.rs => hll_agg.rs} (52%)
create mode 100644 native/spark-expr/benches/hll_scalar.rs
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/02: add benches for remaining comet native kernels (#6648)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit f651144b9d3a6ceb7d037907aac6cde6e8a6fa26
Author: Bhargava Vadlamani <11...@users.noreply.github.com>
AuthorDate: Tue Oct 6 17:51:01 2026 +0000
add benches for remaining comet native kernels (#6648)
---
native/spark-expr/Cargo.toml | 16 ++
.../benches/bloom_filter_might_contain.rs | 144 ++++++++++++++++
native/spark-expr/benches/cast_nested.rs | 141 ++++++++++++++++
native/spark-expr/benches/hll_agg.rs | 184 +++++++++++++++++++++
native/spark-expr/benches/hll_scalar.rs | 74 +++++++++
5 files changed, 559 insertions(+)
diff --git a/native/spark-expr/Cargo.toml b/native/spark-expr/Cargo.toml
index 7020f8aab7..965bbd7916 100644
--- a/native/spark-expr/Cargo.toml
+++ b/native/spark-expr/Cargo.toml
@@ -83,6 +83,10 @@ harness = false
name = "cast_from_string"
harness = false
+[[bench]]
+name = "cast_nested"
+harness = false
+
[[bench]]
name = "cast_string_to_date"
harness = false
@@ -183,6 +187,10 @@ harness = false
name = "bloom_filter_agg"
harness = false
+[[bench]]
+name = "bloom_filter_might_contain"
+harness = false
+
[[bench]]
name = "collect"
harness = false
@@ -455,6 +463,14 @@ harness = false
name = "aggregate_stats"
harness = false
+[[bench]]
+name = "hll_agg"
+harness = false
+
+[[bench]]
+name = "hll_scalar"
+harness = false
+
[[bench]]
name = "aggregate_misc"
harness = false
diff --git a/native/spark-expr/benches/bloom_filter_might_contain.rs b/native/spark-expr/benches/bloom_filter_might_contain.rs
new file mode 100644
index 0000000000..c0e926a40f
--- /dev/null
+++ b/native/spark-expr/benches/bloom_filter_might_contain.rs
@@ -0,0 +1,144 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmark for the per-row `might_contain` probe (`BloomFilterMightContain`). The filter it
+//! probes against is built up front by running the public `BloomFilterAgg` over a column of
+//! longs, so the bench exercises only the probe path (`might_contain_longs`) and not filter
+//! construction.
+
+use arrow::array::{Array, ArrayRef, BinaryArray, Int64Array, RecordBatch};
+use arrow::datatypes::{DataType, Field, Schema};
+use criterion::{criterion_group, criterion_main, Criterion};
+use datafusion::common::config::ConfigOptions;
+use datafusion::common::ScalarValue;
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::source::DataSourceExec;
+use datafusion::logical_expr::{AggregateUDF, ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
+use datafusion::physical_expr::aggregate::AggregateExprBuilder;
+use datafusion::physical_expr::expressions::{lit, Column};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::aggregates::{AggregateExec, AggregateMode, PhysicalGroupBy};
+use datafusion::physical_plan::{collect, ExecutionPlan};
+use datafusion::prelude::SessionContext;
+use datafusion_comet_spark_expr::{
+ BloomFilterAgg, BloomFilterMightContain, SparkBloomFilterVersion,
+};
+use std::hint::black_box;
+use std::sync::Arc;
+use tokio::runtime::Runtime;
+
+const DISTINCT_ITEMS: usize = 8192;
+const NUM_BITS: i32 = (DISTINCT_ITEMS as i32) * 16;
+
+/// Build a serialized Spark bloom filter containing `0..DISTINCT_ITEMS` by running the public
+/// `BloomFilterAgg` once. Returns the filter's Spark-serialized bytes.
+fn build_filter_bytes(rt: &Runtime) -> Vec<u8> {
+ let values: ArrayRef = Arc::new(Int64Array::from(
+ (0..DISTINCT_ITEMS as i64).collect::<Vec<_>>(),
+ ));
+ let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
+ let batch = RecordBatch::try_new(Arc::clone(&schema), vec![values]).unwrap();
+
+ let scan: Arc<dyn ExecutionPlan> = Arc::new(DataSourceExec::new(Arc::new(
+ MemorySourceConfig::try_new(&[vec![batch]], Arc::clone(&schema), None).unwrap(),
+ )));
+ let udf = AggregateUDF::new_from_impl(BloomFilterAgg::new(
+ lit(ScalarValue::Int64(Some(DISTINCT_ITEMS as i64))),
+ lit(ScalarValue::Int64(Some(NUM_BITS as i64))),
+ DataType::Binary,
+ SparkBloomFilterVersion::V1,
+ ));
+ let aggr_expr = AggregateExprBuilder::new(
+ Arc::new(udf),
+ vec![Arc::new(Column::new("v", 0)) as Arc<dyn PhysicalExpr>],
+ )
+ .schema(Arc::clone(&schema))
+ .alias("bloom_filter")
+ .build()
+ .unwrap();
+
+ let aggregate: Arc<dyn ExecutionPlan> = Arc::new(
+ AggregateExec::try_new(
+ AggregateMode::Single,
+ PhysicalGroupBy::new(vec![], vec![], vec![vec![]], false),
+ vec![aggr_expr.into()],
+ vec![None],
+ scan,
+ schema,
+ )
+ .unwrap(),
+ );
+
+ let ctx = SessionContext::new();
+ let batches = rt.block_on(collect(aggregate, ctx.task_ctx())).unwrap();
+ let col = batches[0].column(0);
+ let bytes = col.as_any().downcast_ref::<BinaryArray>().unwrap();
+ bytes.value(0).to_vec()
+}
+
+/// Build an Int64 probe column: the first half are hits (`0..rows/2`, present in the filter),
+/// the second half are misses (large values not inserted). Every `null_every`-th row is null
+/// (`null_every == 0` means no nulls).
+fn create_probe_array(rows: usize, null_every: usize) -> ArrayRef {
+ let arr: Int64Array = (0..rows)
+ .map(|i| {
+ if null_every != 0 && i % null_every == 0 {
+ None
+ } else if i < rows / 2 {
+ Some(i as i64)
+ } else {
+ Some(i as i64 + 1_000_000_000)
+ }
+ })
+ .collect();
+ Arc::new(arr)
+}
+
+fn run(udf: &BloomFilterMightContain, values: &ArrayRef, rows: usize) {
+ let args = vec![ColumnarValue::Array(Arc::clone(values))];
+ black_box(
+ udf.invoke_with_args(ScalarFunctionArgs {
+ args,
+ arg_fields: vec![],
+ number_rows: rows,
+ return_field: Arc::new(Field::new("result", DataType::Boolean, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap(),
+ );
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let rows = 8192;
+ let rt = Runtime::new().unwrap();
+ let filter_bytes = build_filter_bytes(&rt);
+ let udf =
+ BloomFilterMightContain::try_new(lit(ScalarValue::Binary(Some(filter_bytes)))).unwrap();
+
+ let no_nulls = create_probe_array(rows, 0);
+ let with_nulls = create_probe_array(rows, 10);
+
+ let mut group = c.benchmark_group("bloom_filter_might_contain");
+ group.bench_function("probe no nulls", |b| b.iter(|| run(&udf, &no_nulls, rows)));
+ group.bench_function("probe with nulls", |b| {
+ b.iter(|| run(&udf, &with_nulls, rows))
+ });
+ group.finish();
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
diff --git a/native/spark-expr/benches/cast_nested.rs b/native/spark-expr/benches/cast_nested.rs
new file mode 100644
index 0000000000..5c6efd5cb4
--- /dev/null
+++ b/native/spark-expr/benches/cast_nested.rs
@@ -0,0 +1,141 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmarks for the nested `cast` paths: struct->struct (child retyped), map->map (value
+//! retyped), and array->string. Each builds an 8192-row input and casts it through the public
+//! `Cast` physical expression, mirroring the scalar cast benches.
+
+use arrow::array::{
+ ArrayRef, Int32Array, ListArray, MapArray, RecordBatch, StringArray, StructArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{DataType, Field, Fields, Int32Type, Schema};
+use criterion::{criterion_group, criterion_main, Criterion};
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion_comet_spark_expr::{Cast, EvalMode, SparkCastOptions};
+use std::hint::black_box;
+use std::sync::Arc;
+
+const NUM_ROWS: usize = 8192;
+const ENTRIES_PER_ROW: usize = 2;
+
+/// Single-column batch named `c` wrapping `array`.
+fn batch(array: ArrayRef) -> RecordBatch {
+ let schema = Schema::new(vec![Field::new("c", array.data_type().clone(), true)]);
+ RecordBatch::try_new(Arc::new(schema), vec![array]).unwrap()
+}
+
+/// `Struct<a: Int32, b: Utf8>` with `NUM_ROWS` rows.
+fn struct_input() -> ArrayRef {
+ let a = Arc::new(Int32Array::from((0..NUM_ROWS as i32).collect::<Vec<_>>())) as ArrayRef;
+ let b = Arc::new(StringArray::from(
+ (0..NUM_ROWS).map(|i| format!("s{i}")).collect::<Vec<_>>(),
+ )) as ArrayRef;
+ let fields = Fields::from(vec![
+ Field::new("a", DataType::Int32, true),
+ Field::new("b", DataType::Utf8, true),
+ ]);
+ Arc::new(StructArray::new(fields, vec![a, b], None))
+}
+
+/// `Map<Utf8, Int32>` with `NUM_ROWS` rows of `ENTRIES_PER_ROW` entries each.
+fn map_input() -> ArrayRef {
+ let total = NUM_ROWS * ENTRIES_PER_ROW;
+ let keys = Arc::new(StringArray::from(
+ (0..total).map(|i| format!("k{i}")).collect::<Vec<_>>(),
+ )) as ArrayRef;
+ let values = Arc::new(Int32Array::from((0..total as i32).collect::<Vec<_>>())) as ArrayRef;
+ let entries_fields = Fields::from(vec![
+ Field::new("key_value_key", DataType::Utf8, false),
+ Field::new("key_value_value", DataType::Int32, true),
+ ]);
+ let entries = StructArray::new(entries_fields, vec![keys, values], None);
+ let offsets = OffsetBuffer::<i32>::new(
+ (0..=NUM_ROWS)
+ .map(|i| (i * ENTRIES_PER_ROW) as i32)
+ .collect::<Vec<_>>()
+ .into(),
+ );
+ let entries_field = Arc::new(Field::new(
+ "key_value",
+ DataType::Struct(entries.fields().clone()),
+ false,
+ ));
+ Arc::new(MapArray::new(entries_field, offsets, entries, None, false))
+}
+
+/// `List<Int32>` with `NUM_ROWS` rows whose lengths cycle through 0..=3.
+fn list_input() -> ArrayRef {
+ let data = (0..NUM_ROWS).map(|i| {
+ let len = i % 4;
+ Some(
+ (0..len)
+ .map(|j| Some((i * 4 + j) as i32))
+ .collect::<Vec<_>>(),
+ )
+ });
+ Arc::new(ListArray::from_iter_primitive::<Int32Type, _, _>(data))
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let options = SparkCastOptions::new(EvalMode::Legacy, "UTC", false);
+ let col: Arc<dyn PhysicalExpr> = Arc::new(Column::new("c", 0));
+
+ let mut group = c.benchmark_group("cast_nested");
+
+ // struct<a: Int32, b: Utf8> -> struct<a: Int64, b: Utf8>: retypes the first child.
+ let struct_batch = batch(struct_input());
+ let struct_target = DataType::Struct(Fields::from(vec![
+ Field::new("a", DataType::Int64, true),
+ Field::new("b", DataType::Utf8, true),
+ ]));
+ let cast_struct = Cast::new(col.clone(), struct_target, options.clone(), None, None);
+ group.bench_function("struct_to_struct", |b| {
+ b.iter(|| black_box(cast_struct.evaluate(black_box(&struct_batch)).unwrap()))
+ });
+
+ // map<Utf8, Int32> -> map<Utf8, Int64>: retypes the value field.
+ let map_batch = batch(map_input());
+ let map_target = DataType::Map(
+ Arc::new(Field::new(
+ "entries",
+ DataType::Struct(Fields::from(vec![
+ Field::new("key", DataType::Utf8, false),
+ Field::new("value", DataType::Int64, true),
+ ])),
+ false,
+ )),
+ false,
+ );
+ let cast_map = Cast::new(col.clone(), map_target, options.clone(), None, None);
+ group.bench_function("map_to_map", |b| {
+ b.iter(|| black_box(cast_map.evaluate(black_box(&map_batch)).unwrap()))
+ });
+
+ // list<Int32> -> Utf8: renders each list to its Spark string form.
+ let list_batch = batch(list_input());
+ let cast_list = Cast::new(col, DataType::Utf8, options, None, None);
+ group.bench_function("array_to_string", |b| {
+ b.iter(|| black_box(cast_list.evaluate(black_box(&list_batch)).unwrap()))
+ });
+
+ group.finish();
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
diff --git a/native/spark-expr/benches/hll_agg.rs b/native/spark-expr/benches/hll_agg.rs
new file mode 100644
index 0000000000..c933533f7c
--- /dev/null
+++ b/native/spark-expr/benches/hll_agg.rs
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmarks for the Comet HyperLogLog aggregates in `agg_funcs`: `hll_sketch_agg`
+//! (builds a sketch from raw values) and `hll_union_agg` (merges serialized sketches).
+//! Each runs as a grouped `AggregateMode::Partial` over 8192 rows x 10 batches, mirroring
+//! the other aggregate benches. The union input is a column of real sketch bytes minted
+//! up front with `SparkHllSketch`, since the kernel deserializes and merges each one.
+
+use arrow::array::{ArrayRef, BinaryBuilder, Int64Builder, RecordBatch, StringBuilder};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{criterion_group, criterion_main, Criterion};
+use datafusion::datasource::memory::MemorySourceConfig;
+use datafusion::datasource::source::DataSourceExec;
+use datafusion::execution::TaskContext;
+use datafusion::logical_expr::AggregateUDF;
+use datafusion::physical_expr::aggregate::AggregateExprBuilder;
+use datafusion::physical_expr::expressions::Column;
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::aggregates::{AggregateExec, AggregateMode, PhysicalGroupBy};
+use datafusion::physical_plan::ExecutionPlan;
+use datafusion_comet_spark_expr::{HllSketchAgg, HllUnionAgg, SparkHllSketch};
+use futures::StreamExt;
+use std::hint::black_box;
+use std::sync::Arc;
+use tokio::runtime::Runtime;
+
+/// Spark's default `lgConfigK` for `approx_count_distinct`/HLL sketches.
+const LG_CONFIG_K: u8 = 12;
+/// Number of distinct sketches minted for the union input; rows cycle through them.
+const SKETCH_POOL: usize = 256;
+
+async fn agg_test(
+ partitions: &[Vec<RecordBatch>],
+ group_col: Arc<dyn PhysicalExpr>,
+ input_exprs: Vec<Arc<dyn PhysicalExpr>>,
+ aggregate_udf: Arc<AggregateUDF>,
+ alias: &str,
+) {
+ let schema = &partitions[0][0].schema();
+ let scan: Arc<dyn ExecutionPlan> = Arc::new(DataSourceExec::new(Arc::new(
+ MemorySourceConfig::try_new(partitions, Arc::clone(schema), None).unwrap(),
+ )));
+ let aggregate = create_aggregate(scan, group_col, input_exprs, schema, aggregate_udf, alias);
+ let mut stream = aggregate
+ .execute(0, Arc::new(TaskContext::default()))
+ .unwrap();
+ while let Some(batch) = stream.next().await {
+ let _batch = batch.unwrap();
+ }
+}
+
+fn create_aggregate(
+ scan: Arc<dyn ExecutionPlan>,
+ group_col: Arc<dyn PhysicalExpr>,
+ input_exprs: Vec<Arc<dyn PhysicalExpr>>,
+ schema: &SchemaRef,
+ aggregate_udf: Arc<AggregateUDF>,
+ alias: &str,
+) -> Arc<AggregateExec> {
+ let aggr_expr = AggregateExprBuilder::new(aggregate_udf, input_exprs)
+ .schema(schema.clone())
+ .alias(alias)
+ .with_ignore_nulls(false)
+ .with_distinct(false)
+ .build()
+ .unwrap();
+
+ Arc::new(
+ AggregateExec::try_new(
+ AggregateMode::Partial,
+ PhysicalGroupBy::new_single(vec![(group_col, "c0".to_string())]),
+ vec![aggr_expr.into()],
+ vec![None], // no filter expressions
+ scan,
+ Arc::clone(schema),
+ )
+ .unwrap(),
+ )
+}
+
+/// A pool of `SKETCH_POOL` distinct non-empty sketches, each covering a disjoint run of i64
+/// values so a union of several estimates to a meaningful cardinality.
+fn sketch_pool() -> Vec<Vec<u8>> {
+ (0..SKETCH_POOL)
+ .map(|p| {
+ let mut sketch = SparkHllSketch::new(LG_CONFIG_K);
+ let start = (p * 100) as i64;
+ for v in start..start + 100 {
+ sketch.update_i64(v);
+ }
+ sketch.to_sketch_bytes()
+ })
+ .collect()
+}
+
+/// `c0` is the Utf8 group key (1024 groups); `c1` is a spread-out Int64 input for
+/// `hll_sketch_agg`; `c2` is a column of serialized sketches for `hll_union_agg`.
+fn create_record_batch(num_rows: usize, pool: &[Vec<u8>]) -> RecordBatch {
+ let mut group_builder = StringBuilder::with_capacity(num_rows, num_rows * 8);
+ let mut c1_builder = Int64Builder::with_capacity(num_rows);
+ let mut c2_builder = BinaryBuilder::new();
+ for i in 0..num_rows {
+ group_builder.append_value(format!("group_{}", i % 1024));
+ c1_builder.append_value(i as i64);
+ c2_builder.append_value(&pool[i % pool.len()]);
+ }
+
+ let fields = vec![
+ Field::new("c0", DataType::Utf8, false),
+ Field::new("c1", DataType::Int64, false),
+ Field::new("c2", DataType::Binary, false),
+ ];
+ let columns: Vec<ArrayRef> = vec![
+ Arc::new(group_builder.finish()),
+ Arc::new(c1_builder.finish()),
+ Arc::new(c2_builder.finish()),
+ ];
+
+ RecordBatch::try_new(Arc::new(Schema::new(fields)), columns).unwrap()
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let num_rows = 8192;
+ let pool = sketch_pool();
+ let batch = create_record_batch(num_rows, &pool);
+ let batches = vec![batch; 10];
+ let partitions = &[batches];
+
+ let c0: Arc<dyn PhysicalExpr> = Arc::new(Column::new("c0", 0));
+ let c1: Arc<dyn PhysicalExpr> = Arc::new(Column::new("c1", 1));
+ let c2: Arc<dyn PhysicalExpr> = Arc::new(Column::new("c2", 2));
+
+ let rt = Runtime::new().unwrap();
+ let mut group = c.benchmark_group("hll_agg");
+
+ let sketch_udf: Arc<AggregateUDF> = Arc::new(AggregateUDF::new_from_impl(HllSketchAgg::new(
+ LG_CONFIG_K as i32,
+ )));
+ group.bench_function("hll_sketch_agg", |b| {
+ b.to_async(&rt).iter(|| {
+ black_box(agg_test(
+ partitions,
+ c0.clone(),
+ vec![c1.clone()],
+ sketch_udf.clone(),
+ "hll_sketch_agg",
+ ))
+ })
+ });
+
+ let union_udf: Arc<AggregateUDF> =
+ Arc::new(AggregateUDF::new_from_impl(HllUnionAgg::new(false)));
+ group.bench_function("hll_union_agg", |b| {
+ b.to_async(&rt).iter(|| {
+ black_box(agg_test(
+ partitions,
+ c0.clone(),
+ vec![c2.clone()],
+ union_udf.clone(),
+ "hll_union_agg",
+ ))
+ })
+ });
+
+ group.finish();
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
diff --git a/native/spark-expr/benches/hll_scalar.rs b/native/spark-expr/benches/hll_scalar.rs
new file mode 100644
index 0000000000..d3b4c79070
--- /dev/null
+++ b/native/spark-expr/benches/hll_scalar.rs
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmarks for the scalar HLL kernels: `hll_sketch_estimate` (Binary sketch -> Long estimate)
+//! and `hll_union` (two Binary sketch columns + an `allowDifferentLgConfigK` flag -> Binary).
+//! Both deserialize a sketch per row, so inputs are real sketch bytes minted up front with
+//! `SparkHllSketch`.
+
+use arrow::array::{ArrayRef, BinaryArray, BooleanArray};
+use criterion::{criterion_group, criterion_main, Criterion};
+use datafusion::physical_plan::ColumnarValue;
+use datafusion_comet_spark_expr::{spark_hll_sketch_estimate, spark_hll_union, SparkHllSketch};
+use std::hint::black_box;
+use std::sync::Arc;
+
+const LG_CONFIG_K: u8 = 12;
+
+/// Build a Binary column of `rows` sketches, each covering a distinct run of 100 i64 values
+/// offset by `base` so the two columns in a union cover different ranges.
+fn sketch_column(rows: usize, base: i64) -> ArrayRef {
+ let sketches: Vec<Vec<u8>> = (0..rows)
+ .map(|i| {
+ let mut s = SparkHllSketch::new(LG_CONFIG_K);
+ let start = base + (i as i64) * 100;
+ for v in start..start + 100 {
+ s.update_i64(v);
+ }
+ s.to_sketch_bytes()
+ })
+ .collect();
+ Arc::new(BinaryArray::from_iter_values(sketches))
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let rows = 8192;
+ let left = sketch_column(rows, 0);
+ let right = sketch_column(rows, 1_000_000);
+ let allow: ArrayRef = Arc::new(BooleanArray::from(vec![false; rows]));
+
+ let mut group = c.benchmark_group("hll_scalar");
+
+ group.bench_function("hll_sketch_estimate", |b| {
+ let args = vec![ColumnarValue::Array(Arc::clone(&left))];
+ b.iter(|| black_box(spark_hll_sketch_estimate(black_box(&args)).unwrap()))
+ });
+
+ group.bench_function("hll_union", |b| {
+ let args = vec![
+ ColumnarValue::Array(Arc::clone(&left)),
+ ColumnarValue::Array(Arc::clone(&right)),
+ ColumnarValue::Array(Arc::clone(&allow)),
+ ];
+ b.iter(|| black_box(spark_hll_union(black_box(&args)).unwrap()))
+ });
+
+ group.finish();
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 02/02: docs: add 1.1.0 changelog (#6728)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 275d09db8231215a97f6d86154556c074ed691d6
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Oct 6 18:16:39 2026 +0000
docs: add 1.1.0 changelog (#6728)
* docs: add 1.1.0 changelog (#6282)
* docs: [branch-1.1] credit co-authors in the 1.1.0 changelog (#6359)
Regenerate the 1.1.0 changelog with the generator from #6358, for the
same range as #6282 (1.0.0..021c378fc), and format it with prettier
3.9.9. The generator used for #6282 credited each PR only to the author
of its merge commit, so it left out Michael Taranov, whose commits from
#6092 are in #6219, and credited the backports only to the person who
opened them.
The PR lines for #5310, #6219, #6223 and the four backports now name
their co-authors, and the credits count each co-authored PR, which adds
Michael Taranov and brings the contributor count to 41.
* update changelog (#6569)
---
docs/source/changelog/1.1.0.md | 503 +++++++++++++++++++++++++++++++++++++++++
docs/source/changelog/index.md | 1 +
2 files changed, 504 insertions(+)
diff --git a/docs/source/changelog/1.1.0.md b/docs/source/changelog/1.1.0.md
new file mode 100644
index 0000000000..dfbaf5abda
--- /dev/null
+++ b/docs/source/changelog/1.1.0.md
@@ -0,0 +1,503 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# DataFusion Comet 1.1.0 Changelog
+
+This release consists of 416 commits from 41 contributors. See credits at the end of this changelog for more information.
+
+**Fixed bugs:**
+
+- fix: skip null slots when checking overflow in unary negation [#5162](https://github.com/apache/datafusion-comet/pull/5162) (Smallfu666)
+- fix: surface next_day and make_date ANSI errors as Spark exceptions [#5167](https://github.com/apache/datafusion-comet/pull/5167) (peterxcli)
+- fix: normalize nested field nullability in ShuffleScanExec and ExpandExec [#5138](https://github.com/apache/datafusion-comet/pull/5138) (andygrove)
+- fix: propagate the Spark task ClassLoader to JVM UDF calls [#5282](https://github.com/apache/datafusion-comet/pull/5282) (andygrove)
+- fix: avoid duplicate CheckOverflow evaluation for decimal division [#5225](https://github.com/apache/datafusion-comet/pull/5225) (peterxcli)
+- fix: match Spark whitespace trimming in to_time and try_to_time [#5364](https://github.com/apache/datafusion-comet/pull/5364) (sunchao)
+- fix: preserve Catalyst nullability and field IDs in native Parquet writes [#5369](https://github.com/apache/datafusion-comet/pull/5369) (sunchao)
+- fix: guard against silent fail_on_error loss in scalar wiring (#5074) [#5359](https://github.com/apache/datafusion-comet/pull/5359) (sam-1112)
+- fix: preserve Spark semantics for dictionary-encoded Parquet inputs and reject dictionary targets [#5234](https://github.com/apache/datafusion-comet/pull/5234) (peterxcli)
+- fix: canonicalize NaN in flat arrays_overlap float keys [#5376](https://github.com/apache/datafusion-comet/pull/5376) (sunchao)
+- fix: report native shuffle write metrics accurately [#5370](https://github.com/apache/datafusion-comet/pull/5370) (sunchao)
+- fix: Native shuffle fails with a 2GB task serialization OOM on jobs with many partitions [#5392](https://github.com/apache/datafusion-comet/pull/5392) (parthchandra)
+- fix: format NativeMemoryConsumer id in toString [#5398](https://github.com/apache/datafusion-comet/pull/5398) (ywskycn)
+- fix: ignore reader-side parquet.hadoop.vectored.io.enabled in Iceberg native-write detection [#5410](https://github.com/apache/datafusion-comet/pull/5410) (snmvaughan)
+- fix: support map casts with NullType elements [#5045](https://github.com/apache/datafusion-comet/pull/5045) (peterxcli)
+- fix: distinguish reflection failure from absent accessor in IcebergReflection [#5412](https://github.com/apache/datafusion-comet/pull/5412) (unikdahal)
+- fix: use current shuffle config in aggregate test [#5439](https://github.com/apache/datafusion-comet/pull/5439) (peterxcli)
+- fix: support wide years in native make_date [#5443](https://github.com/apache/datafusion-comet/pull/5443) (peterxcli)
+- fix: report native child spill metrics in shuffle tasks [#5445](https://github.com/apache/datafusion-comet/pull/5445) (sunchao)
+- fix: expose native memory usage to Spark [#5408](https://github.com/apache/datafusion-comet/pull/5408) (ywskycn)
+- fix: release native shuffle reservation after spill failure [#5461](https://github.com/apache/datafusion-comet/pull/5461) (peterxcli)
+- fix: track peak memory before JVM shuffle spill [#5463](https://github.com/apache/datafusion-comet/pull/5463) (peterxcli)
+- fix: RAII for memory pool registration [#5464](https://github.com/apache/datafusion-comet/pull/5464) (peterxcli)
+- fix: tighten RSS writer visibility and JNI array limits [#5475](https://github.com/apache/datafusion-comet/pull/5475) (pingzh)
+- fix: report native operator spill metrics in Spark task metrics for non-shuffle stages [#5497](https://github.com/apache/datafusion-comet/pull/5497) (peterxcli)
+- fix: report bounded shuffle allocator memory usage [#5516](https://github.com/apache/datafusion-comet/pull/5516) (ywskycn)
+- fix: use Spark type names in ANSI abs overflow errors [#5357](https://github.com/apache/datafusion-comet/pull/5357) (Smallfu666)
+- fix: make collect_list/collect_set argument coercion a normalization barrier [#5159](https://github.com/apache/datafusion-comet/pull/5159) (andygrove)
+- fix: make task-shared memory pool as ref-counted RAII guard [#5494](https://github.com/apache/datafusion-comet/pull/5494) (peterxcli)
+- fix: accept UTC timezone aliases in Python Arrow input [#5556](https://github.com/apache/datafusion-comet/pull/5556) (sunchao)
+- fix: normalize scalar float sort and window rank keys [#5469](https://github.com/apache/datafusion-comet/pull/5469) (sunchao)
+- fix: make CometExplodeExec respect batch size [#5362](https://github.com/apache/datafusion-comet/pull/5362) (andygrove)
+- fix: align Spark 4.2 Python worker configuration [#5561](https://github.com/apache/datafusion-comet/pull/5561) (sunchao)
+- fix: prevent memory leak after failed Arrow vector import [#5539](https://github.com/apache/datafusion-comet/pull/5539) (1fanwang)
+- fix: preserve Arrow Field metadata across C Data exports [#5552](https://github.com/apache/datafusion-comet/pull/5552) (peterxcli)
+- fix: rebase map offsets in mapsort so sliced maps do not overrun entries [#5630](https://github.com/apache/datafusion-comet/pull/5630) (viirya)
+- fix: scope Celeborn bootstrap hooks to Comet clients [#5627](https://github.com/apache/datafusion-comet/pull/5627) (pingzh)
+- fix: skip codegen dispatcher null short-circuit when a foldable subtree can raise [#5623](https://github.com/apache/datafusion-comet/pull/5623) (andygrove)
+- fix: accept dictionary encodings in remote shuffle [#5650](https://github.com/apache/datafusion-comet/pull/5650) (pingzh)
+- fix: make CometDiskBlockWriter spill registry per-task instead of executor-global [#5493](https://github.com/apache/datafusion-comet/pull/5493) (peterxcli)
+- fix: Bump iceberg-rust so native Iceberg writes URL-escape partition paths [#5651](https://github.com/apache/datafusion-comet/pull/5651) (andygrove)
+- fix(celeborn): reject unsafe native push completion tracking [#5665](https://github.com/apache/datafusion-comet/pull/5665) (pingzh)
+- fix: match Spark's null short-circuiting in array_join and enable it natively [#5558](https://github.com/apache/datafusion-comet/pull/5558) (Visorgood)
+- fix: make native shuffle spill metrics independent of input batching [#5628](https://github.com/apache/datafusion-comet/pull/5628) (sunchao)
+- fix: expand object store option references, uniquify constant metadata names, drop dead parquet JNI [#5653](https://github.com/apache/datafusion-comet/pull/5653) (dwsmith1983)
+- fix: match Spark's ANSI bound check for float/double to integral casts [#5683](https://github.com/apache/datafusion-comet/pull/5683) (peterxcli)
+- fix: return NULL from rpad/lpad when the length column is NULL instead of panicking [#5680](https://github.com/apache/datafusion-comet/pull/5680) (peterxcli)
+- fix: fall back for concat_ws with array arguments instead of failing natively [#5679](https://github.com/apache/datafusion-comet/pull/5679) (peterxcli)
+- fix: prevent silent overflow when reading Parquet TIMESTAMP_MILLIS values [#5177](https://github.com/apache/datafusion-comet/pull/5177) (peterxcli)
+- fix: recover native Celeborn shuffle from oversized rows [#5668](https://github.com/apache/datafusion-comet/pull/5668) (pingzh)
+- fix: report native shuffle read metrics [#5554](https://github.com/apache/datafusion-comet/pull/5554) (peterxcli)
+- fix: correctly rounded decimal to double/float cast matching BigDecimal.doubleValue/floatValue [#5684](https://github.com/apache/datafusion-comet/pull/5684) (peterxcli)
+- fix: restore columnar transitions under the native Iceberg write [#5696](https://github.com/apache/datafusion-comet/pull/5696) (andygrove)
+- fix: make columnar-to-row benchmarks exercise Comet [#5718](https://github.com/apache/datafusion-comet/pull/5718) (rich7420)
+- fix: distinguish "nothing spilled" from a spill backend with no local path [#5726](https://github.com/apache/datafusion-comet/pull/5726) (andygrove)
+- fix: propagate Arrow array copy errors [#5747](https://github.com/apache/datafusion-comet/pull/5747) (rich7420)
+- fix: keep the dictionary hash fast path off nested and reseeded buffers [#5757](https://github.com/apache/datafusion-comet/pull/5757) (viirya)
+- fix: preserve ANSI errors for rejected TIMESTAMP_NTZ casts [#5752](https://github.com/apache/datafusion-comet/pull/5752) (peterxcli)
+- fix: apply the parent struct's null mask before hashing its fields [#5754](https://github.com/apache/datafusion-comet/pull/5754) (viirya)
+- fix: read Iceberg tables partitioned by an unknown transform [#5759](https://github.com/apache/datafusion-comet/pull/5759) (andygrove)
+- fix: native Iceberg write panics on an evolved partition spec and on a timestamptz partition path [#5729](https://github.com/apache/datafusion-comet/pull/5729) (andygrove)
+- fix: let AQE optimize queries over Comet caches [#5733](https://github.com/apache/datafusion-comet/pull/5733) (peterxcli)
+- fix: dispatch Iceberg system functions wrapped as ApplyFunctionExpression [#5773](https://github.com/apache/datafusion-comet/pull/5773) (andygrove)
+- fix: attach tokio runtime threads to the JVM as daemon threads [#5748](https://github.com/apache/datafusion-comet/pull/5748) (zhangfengcdt)
+- fix: check nested TIMESTAMP_MILLIS overflow in unfiltered scans [#5740](https://github.com/apache/datafusion-comet/pull/5740) (peterxcli)
+- fix: match iceberg-java's exception for unclustered input to a clustered Iceberg write [#5779](https://github.com/apache/datafusion-comet/pull/5779) (andygrove)
+- fix: enable FIRST/LAST partial merge [#5041](https://github.com/apache/datafusion-comet/pull/5041) (peterxcli)
+- fix: preserve aggregate result identity during exchange reuse [#5470](https://github.com/apache/datafusion-comet/pull/5470) (sunchao)
+- fix: ignore structural tags when lifting expression coverage [#5471](https://github.com/apache/datafusion-comet/pull/5471) (sunchao)
+- fix: align string to timestamp parsing with Spark's segment rules [#5682](https://github.com/apache/datafusion-comet/pull/5682) (peterxcli)
+- fix: roll native Iceberg data files on iceberg-java's 1000-row grid [#5780](https://github.com/apache/datafusion-comet/pull/5780) (andygrove)
+- fix: decide libhdfs routing from the scheme as written [#5825](https://github.com/apache/datafusion-comet/pull/5825) (comphead)
+- fix: list a fanout Iceberg write's data files in a stable order [#5810](https://github.com/apache/datafusion-comet/pull/5810) (andygrove)
+- fix: isolate object-store registration by backend and configuration [#5503](https://github.com/apache/datafusion-comet/pull/5503) (sunchao)
+- fix: Delete completed tasks' data files when an Iceberg write job fails [#5663](https://github.com/apache/datafusion-comet/pull/5663) (andygrove)
+- fix: apply Spark's Parquet conversion rules to nested struct/list/map fields [#5681](https://github.com/apache/datafusion-comet/pull/5681) (peterxcli)
+- fix: preserve nulls for Boolean/Byte/Short/Integer columns in FuzzDataGenerator [#5855](https://github.com/apache/datafusion-comet/pull/5855) (Smallfu666)
+- fix: Accept explicit positive years in timestamp casts [#5858](https://github.com/apache/datafusion-comet/pull/5858) (peterxcli)
+- fix: decline structs with duplicate field names before they reach Java Arrow [#5866](https://github.com/apache/datafusion-comet/pull/5866) (dwsmith1983)
+- fix: explain ObjectHashAggregate fallback when Comet shuffle is disabled [#5746](https://github.com/apache/datafusion-comet/pull/5746) (0lai0)
+- fix: preserve join and generator semantics in plan identity [#5828](https://github.com/apache/datafusion-comet/pull/5828) (ErikBPF)
+- fix: propagate Parquet field-name folding failures [#5845](https://github.com/apache/datafusion-comet/pull/5845) (sunchao)
+- fix: decode dictionary input for PyArrow UDFs [#5560](https://github.com/apache/datafusion-comet/pull/5560) (sunchao)
+- fix: size JVM shuffle pointer array growth from the array, not the data pages [#5907](https://github.com/apache/datafusion-comet/pull/5907) (andygrove)
+- fix: render float and double Iceberg partition values like iceberg-java [#5840](https://github.com/apache/datafusion-comet/pull/5840) (andygrove)
+- fix: align time parsing and native second extraction with Spark [#5738](https://github.com/apache/datafusion-comet/pull/5738) (peterxcli)
+- fix: normalize floating-point values in native collect_set [#5166](https://github.com/apache/datafusion-comet/pull/5166) (peterxcli)
+- fix: keep Iceberg complex null checks on native scans [#5732](https://github.com/apache/datafusion-comet/pull/5732) (ErikBPF)
+- fix: decode invalid UTF-8 at the JVM to native FFI import boundary [#5310](https://github.com/apache/datafusion-comet/pull/5310) (manuzhang, andygrove)
+- fix: fall back when a struct repeats a Parquet field id [#6004](https://github.com/apache/datafusion-comet/pull/6004) (comphead)
+- fix: normalize signed zero in nested float array comparisons [#5235](https://github.com/apache/datafusion-comet/pull/5235) (divyankshah)
+- fix: fall back to Spark for native Iceberg writes to gs:// through HadoopFileIO [#5935](https://github.com/apache/datafusion-comet/pull/5935) (zhangfengcdt)
+- fix: gate the regr_r2 degenerate-case swap on the Spark patch release [#6042](https://github.com/apache/datafusion-comet/pull/6042) (dwsmith1983)
+- fix: include ABFS container in object store cache key [#5053](https://github.com/apache/datafusion-comet/pull/5053) (peterxcli)
+- fix: preserve Spark row index read errors [#6046](https://github.com/apache/datafusion-comet/pull/6046) (liupoyi-1031)
+- fix: remove the ineffective spark.executor.memoryOverhead adjustment from the driver plugin [#6054](https://github.com/apache/datafusion-comet/pull/6054) (andygrove)
+- fix: count each memory pool once in analyze_trace [#5991](https://github.com/apache/datafusion-comet/pull/5991) (andygrove)
+- fix: revert unsafe partial aggregates after final fallback [#5421](https://github.com/apache/datafusion-comet/pull/5421) (sunchao)
+- fix: restore the site's mermaid diagrams and make a dropped one fail CI [#6064](https://github.com/apache/datafusion-comet/pull/6064) (andygrove)
+- fix: write cached batches to the schema width, not the batch width [#6090](https://github.com/apache/datafusion-comet/pull/6090) (andygrove)
+- fix(iceberg): guard native Iceberg scan driver-metric double-post, add metrics docs and tests [#6085](https://github.com/apache/datafusion-comet/pull/6085) (parthchandra)
+- fix: let decimal SUM recover from an intermediate overflow like Spark [#6041](https://github.com/apache/datafusion-comet/pull/6041) (dwsmith1983)
+- fix: Nested floating-point IN membership does not match Spark for signed zero [#6073](https://github.com/apache/datafusion-comet/pull/6073) (mizulun)
+- fix: let Comet memory pools overcommit on grow instead of panicking [#6128](https://github.com/apache/datafusion-comet/pull/6128) (andygrove)
+- fix: correct two nightly test failures on Spark 3.4 and 4.2 [#6156](https://github.com/apache/datafusion-comet/pull/6156) (andygrove)
+- fix: support null calendar interval literals [#5133](https://github.com/apache/datafusion-comet/pull/5133) (peterxcli)
+- fix: wrap Iceberg split-write failures the way Spark does when abort fails [#6153](https://github.com/apache/datafusion-comet/pull/6153) (andygrove)
+- fix: support Utf8/LargeUtf8/Utf8View in native RLike without panicking [#5215](https://github.com/apache/datafusion-comet/pull/5215) (sam-1112)
+- fix: normalize noncanonical NaN literals in comparisons [#5472](https://github.com/apache/datafusion-comet/pull/5472) (sunchao)
+- fix: preserve map field metadata and honor target sorted flag in cast_map_to_map [#5227](https://github.com/apache/datafusion-comet/pull/5227) (Smallfu666)
+- fix: reject duplicate Parquet field names before decoding [#5786](https://github.com/apache/datafusion-comet/pull/5786) (ErikBPF)
+- fix: preserve Parquet conversion errors during join filtering [#6067](https://github.com/apache/datafusion-comet/pull/6067) (pingzh)
+- fix: plan Iceberg writes with Spark's own operator when Comet is disabled [#6151](https://github.com/apache/datafusion-comet/pull/6151) (andygrove)
+- fix: read shuffle write buffer, spill limit and off-heap sizes in bytes [#6191](https://github.com/apache/datafusion-comet/pull/6191) (andygrove)
+- fix: bound shuffle schema cache retention and preserve eviction order [#6098](https://github.com/apache/datafusion-comet/pull/6098) (sunchao)
+- fix: do not run Comet in on-heap mode without `spark.comet.exec.onHeap.enabled` [#6195](https://github.com/apache/datafusion-comet/pull/6195) (andygrove)
+- fix: remove misleading native opt-in for dispatch-only datetime expressions [#6182](https://github.com/apache/datafusion-comet/pull/6182) (LinSimon-901101)
+- fix: support CalendarIntervalType hashing [#5135](https://github.com/apache/datafusion-comet/pull/5135) (peterxcli)
+- fix: report `Input` column when native Iceberg scan is enabled or native shuffle is enabled [#5265](https://github.com/apache/datafusion-comet/pull/5265) (hsiang-c)
+- fix: skip the executor memory overhead warning when a factor is set or in local mode [#6198](https://github.com/apache/datafusion-comet/pull/6198) (andygrove)
+- fix: keep operators above a cached relation native after the AQE re-plan [#6208](https://github.com/apache/datafusion-comet/pull/6208) (andygrove)
+- fix: preserve current AQE logical-stage links on Comet operators [#5483](https://github.com/apache/datafusion-comet/pull/5483) (sunchao)
+- fix: check each fair_unified reservation against its own share [#6205](https://github.com/apache/datafusion-comet/pull/6205) (andygrove)
+- fix(iceberg): don't push transform residuals as their source column, fail on residual errors [#6154](https://github.com/apache/datafusion-comet/pull/6154) (andygrove)
+- fix: match Spark's duplicate field and field id semantics in parquet field lookup [#5654](https://github.com/apache/datafusion-comet/pull/5654) (dwsmith1983)
+- fix: park the native scan loop instead of busy-polling while waiting on native I/O [#6219](https://github.com/apache/datafusion-comet/pull/6219) (andygrove, mixermt)
+- fix: refresh S3 policy locations when a location's credential fails [#6223](https://github.com/apache/datafusion-comet/pull/6223) (andygrove, snmvaughan)
+- fix: inject Comet session extension rules only once per session [#6230](https://github.com/apache/datafusion-comet/pull/6230) (andygrove)
+- fix: [branch-1.1] reject a file without field ids at any depth whether or not id matching is on (#6116) [#6266](https://github.com/apache/datafusion-comet/pull/6266) (andygrove, dwsmith1983)
+- fix: [branch-1.1] decline native Iceberg writes with a custom location provider (#6216) [#6305](https://github.com/apache/datafusion-comet/pull/6305) (andygrove, liupoyi-1031)
+- fix: [branch-1.1] Native S3 scan on EKS/IRSA turns a transient STS throttle into a hard 403 storm (#6025) [#6323](https://github.com/apache/datafusion-comet/pull/6323) (andygrove, parthchandra)
+- fix: [branch-1.1] log partial memory grants at DEBUG and drop the memory usage dump (#6269) [#6346](https://github.com/apache/datafusion-comet/pull/6346) (andygrove)
+- fix: [branch-1.1] zero sliced boolean offsets at every level before exporting to the JVM (#6339) [#6449](https://github.com/apache/datafusion-comet/pull/6449) (andygrove)
+- fix: [branch-1.1] build with Rust 1.99, which deprecates the legacy f64 constants and fetch_update (#6507) [#6526](https://github.com/apache/datafusion-comet/pull/6526) (andygrove)
+- fix: [branch-1.1] make adaptive aggregation skipping opt-in (#6474) [#6488](https://github.com/apache/datafusion-comet/pull/6488) (andygrove)
+- fix: [branch-1.1] fall back to Spark for \_metadata.file_block_start and file_block_length (#6510) [#6511](https://github.com/apache/datafusion-comet/pull/6511) (andygrove)
+- fix: [branch-1.1] fall back for rank limits over nested float keys (#6468) [#6487](https://github.com/apache/datafusion-comet/pull/6487) (andygrove)
+- fix: [branch-1.1] fall back for incompatible regression aggregates (#6451) [#6489](https://github.com/apache/datafusion-comet/pull/6489) (andygrove)
+- fix: [branch-1.1] coerce native IF branches to a common type (#6458) [#6491](https://github.com/apache/datafusion-comet/pull/6491) (andygrove)
+- fix: [branch-1.1] match pre-epoch Iceberg temporal rounding (#6456) [#6486](https://github.com/apache/datafusion-comet/pull/6486) (andygrove)
+- fix: [branch-1.1] defer Parquet conversion errors until a row group is decoded, as Spark does (#6515) [#6540](https://github.com/apache/datafusion-comet/pull/6540) (andygrove)
+- fix: [branch-1.1] dispatch StaticInvoke and Invoke only into Spark's own classes (#6542) [#6554](https://github.com/apache/datafusion-comet/pull/6554) (andygrove)
+- fix: [branch-1.1] gate array distinct and union signed-zero semantics by Spark version (#5750) [#6561](https://github.com/apache/datafusion-comet/pull/6561) (andygrove, peterxcli)
+- fix: [branch-1.1] fall back from the native Iceberg scan when a nested field was added or renamed (#6543) [#6560](https://github.com/apache/datafusion-comet/pull/6560) (andygrove)
+
+**Performance related:**
+
+- perf: optimize spark_base64 in spark-expr [#4885](https://github.com/apache/datafusion-comet/pull/4885) (andygrove)
+- perf: optimize `spark_floor` (up to 4x faster) [#4911](https://github.com/apache/datafusion-comet/pull/4911) (andygrove)
+- perf: cache Iceberg reflection lookups on the planning path [#5222](https://github.com/apache/datafusion-comet/pull/5222) (andygrove)
+- perf: vectorize integer-to-decimal cast [#4939](https://github.com/apache/datafusion-comet/pull/4939) (andygrove)
+- perf: compute spark_size list lengths with Arrow length kernel [#5233](https://github.com/apache/datafusion-comet/pull/5233) (0lai0)
+- perf: make CometShuffleBenchmark completable and fast [#5388](https://github.com/apache/datafusion-comet/pull/5388) (andygrove)
+- perf: vectorize Map in spark_size via offset buffer [#5395](https://github.com/apache/datafusion-comet/pull/5395) (0lai0)
+- perf: improve ArrowWriter performance for fixed-length vectors [#5046](https://github.com/apache/datafusion-comet/pull/5046) (peterxcli)
+- perf: reuse Arrow IPC compression context across shuffle blocks [#5038](https://github.com/apache/datafusion-comet/pull/5038) (peterxcli)
+- perf: use Arrow cast for decimal rescale check [#5440](https://github.com/apache/datafusion-comet/pull/5440) (peterxcli)
+- perf: bulk copy fixed-width columns in ArrowWriter [#5442](https://github.com/apache/datafusion-comet/pull/5442) (peterxcli)
+- perf: serialize Python input directly from Comet Arrow vectors [#5368](https://github.com/apache/datafusion-comet/pull/5368) (sunchao)
+- perf: reuse per-partition scratch in the shuffle write path [#5568](https://github.com/apache/datafusion-comet/pull/5568) (dwsmith1983)
+- perf: validate shuffle IPC context reuse savings [#5727](https://github.com/apache/datafusion-comet/pull/5727) (peterxcli)
+- perf: slice the child instead of gathering it when unnesting [#5667](https://github.com/apache/datafusion-comet/pull/5667) (andygrove)
+- perf: evaluate posexplode array expressions once per batch [#5737](https://github.com/apache/datafusion-comet/pull/5737) (rich7420)
+- perf: cache expected schemas for remote shuffle decoding [#5722](https://github.com/apache/datafusion-comet/pull/5722) (pingzh)
+- perf: use Arrow cast for date to timestamp NTZ [#5735](https://github.com/apache/datafusion-comet/pull/5735) (peterxcli)
+- perf: reduce allocations when collecting cache statistics [#5734](https://github.com/apache/datafusion-comet/pull/5734) (peterxcli)
+- perf: avoid repeated decimal promotion in expression serialization [#5736](https://github.com/apache/datafusion-comet/pull/5736) (peterxcli)
+- perf: give collect_list and collect_set a native GroupsAccumulator [#5803](https://github.com/apache/datafusion-comet/pull/5803) (andygrove)
+- perf: vectorize the native map lookup behind element_at and GetMapValue [#5806](https://github.com/apache/datafusion-comet/pull/5806) (andygrove)
+- perf: compile user regex patterns once per planned expression [#5612](https://github.com/apache/datafusion-comet/pull/5612) (dwsmith1983)
+- perf: batch the nested-element list hash for flat struct elements [#5778](https://github.com/apache/datafusion-comet/pull/5778) (viirya)
+- perf: optimize `map_sort` singleton normalization (18x faster) [#5887](https://github.com/apache/datafusion-comet/pull/5887) (viirya)
+- perf: optimize `map_sort` for multi-entry string maps (up to 3x faster) [#5901](https://github.com/apache/datafusion-comet/pull/5901) (viirya)
+- perf: skip calendar reconstruction in hour/minute/second and dayofweek/weekday [#5771](https://github.com/apache/datafusion-comet/pull/5771) (peterxcli)
+- perf: reuse zstd compression contexts across shuffle blocks [#5565](https://github.com/apache/datafusion-comet/pull/5565) (dwsmith1983)
+- perf: cache parsed plan data across a stage tasks [#5615](https://github.com/apache/datafusion-comet/pull/5615) (dwsmith1983)
+- perf: spill every shuffle partition of a task into one file [#5916](https://github.com/apache/datafusion-comet/pull/5916) (peterxcli)
+- perf: decode shuffle blocks against a cached schema instead of re-parsing per block [#5809](https://github.com/apache/datafusion-comet/pull/5809) (peterxcli)
+- perf: project cached batches by buffer selection, prune on collated strings [#5543](https://github.com/apache/datafusion-comet/pull/5543) (andygrove)
+- perf: make one thread-local access per tracked allocation [#6166](https://github.com/apache/datafusion-comet/pull/6166) (andygrove)
+- perf: reuse quantile summary buffers during merge [#4932](https://github.com/apache/datafusion-comet/pull/4932) (peterxcli)
+- perf: optimize list_extract without defaults using Arrow take [#5174](https://github.com/apache/datafusion-comet/pull/5174) (peterxcli)
+- perf: track decimal overflow without rescanning results [#5044](https://github.com/apache/datafusion-comet/pull/5044) (peterxcli)
+
+**Implemented enhancements:**
+
+- feat: support `explode_outer` [#5192](https://github.com/apache/datafusion-comet/pull/5192) (comphead)
+- feat: support timestampadd and timestampdiff via codegen dispatch [#5030](https://github.com/apache/datafusion-comet/pull/5030) (andygrove)
+- feat: support \_metadata constant columns in native Parquet scan [#5237](https://github.com/apache/datafusion-comet/pull/5237) (mbutrovich)
+- feat: add `make_interval` support (codegen dispatch + native) [#5039](https://github.com/apache/datafusion-comet/pull/5039) (peterxcli)
+- feat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations [#4658](https://github.com/apache/datafusion-comet/pull/4658) (jordepic)
+- feat: detect Iceberg V2 writes and emit fall-back reasons [#5298](https://github.com/apache/datafusion-comet/pull/5298) (jordepic)
+- feat: remove native cast from boolean to decimal [#5185](https://github.com/apache/datafusion-comet/pull/5185) (andygrove)
+- feat: add micro benchmark runner and EC2 guide [#5374](https://github.com/apache/datafusion-comet/pull/5374) (andygrove)
+- feat: build gate + inert wiring for contrib Delta scans [Delta contrib split, part 2] [#4952](https://github.com/apache/datafusion-comet/pull/4952) (schenksj)
+- feat: Support native scans with unprojected Spark 4 VARIANT columns [#5377](https://github.com/apache/datafusion-comet/pull/5377) (sunchao)
+- feat: expose native aggregate spill and memory metrics [#5423](https://github.com/apache/datafusion-comet/pull/5423) (sunchao)
+- feat: support `WindowGroupLimitExec` [#4870](https://github.com/apache/datafusion-comet/pull/4870) (comphead)
+- feat: add RSS partition writer and task-owned JNI callback (1/n) [#5473](https://github.com/apache/datafusion-comet/pull/5473) (pingzh)
+- feat: native `spark_unbase64` kernel [#5451](https://github.com/apache/datafusion-comet/pull/5451) (0lai0)
+- feat: add partition writer destinations to native shuffle plans (2/n) [#5476](https://github.com/apache/datafusion-comet/pull/5476) (pingzh)
+- feat: add destination-aware native shuffle execution (3/n) [#5481](https://github.com/apache/datafusion-comet/pull/5481) (pingzh)
+- feat: bind task-owned RSS callbacks to native shuffle plans (4/n) [#5491](https://github.com/apache/datafusion-comet/pull/5491) (pingzh)
+- feat: add Celeborn shuffle manager and partition pusher (5/n) [#5501](https://github.com/apache/datafusion-comet/pull/5501) (pingzh)
+- feat: complete Celeborn map-side shuffle push lifecycle (6/n) [#5513](https://github.com/apache/datafusion-comet/pull/5513) (pingzh)
+- feat: add experimental native support for in-memory cache, disabled by default [#5051](https://github.com/apache/datafusion-comet/pull/5051) (andygrove)
+- feat: add raw Celeborn native shuffle reader (7/n) [#5531](https://github.com/apache/datafusion-comet/pull/5531) (pingzh)
+- feat: support Map for `CreateArray` literal [#5452](https://github.com/apache/datafusion-comet/pull/5452) (comphead)
+- feat: route `round` on float/double through the codegen dispatcher [#5600](https://github.com/apache/datafusion-comet/pull/5600) (andygrove)
+- feat: enable native-only Celeborn shuffle planning (8/n) [#5537](https://github.com/apache/datafusion-comet/pull/5537) (pingzh)
+- feat: support unicode case sensitive field names for reading parquet [#5602](https://github.com/apache/datafusion-comet/pull/5602) (comphead)
+- feat: implement native Iceberg V2 writer via iceberg-rust [#5361](https://github.com/apache/datafusion-comet/pull/5361) (jordepic)
+- feat: Support Iceberg system functions (bucket, truncate, years/months/days/hours) natively [#5638](https://github.com/apache/datafusion-comet/pull/5638) (andygrove)
+- feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates [#4775](https://github.com/apache/datafusion-comet/pull/4775) (andygrove)
+- feat: carry VariantType identity through schema serialization [#5631](https://github.com/apache/datafusion-comet/pull/5631) (peterxcli)
+- feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher [#5692](https://github.com/apache/datafusion-comet/pull/5692) (andygrove)
+- feat: support S3 compliant filesystems [#5314](https://github.com/apache/datafusion-comet/pull/5314) (comphead)
+- feat: add native spark_sequence kernel for integral element types [#5614](https://github.com/apache/datafusion-comet/pull/5614) (0lai0)
+- feat: support nested types as native shuffle hash partitioning keys [#5567](https://github.com/apache/datafusion-comet/pull/5567) (viirya)
+- feat: route next_day and levenshtein collated input through the codegen dispatcher [#5720](https://github.com/apache/datafusion-comet/pull/5720) (0lai0)
+- chore: add_benches hash function aggregators [#5730](https://github.com/apache/datafusion-comet/pull/5730) (coderfender)
+- test: cover Spark-to-Arrow batch conversion edge cases [#5713](https://github.com/apache/datafusion-comet/pull/5713) (peterxcli)
+- feat: normalize marked Variant arrays at the native Parquet boundary [#5715](https://github.com/apache/datafusion-comet/pull/5715) (peterxcli)
+- feat: support native concat_ws with string arrays [#5725](https://github.com/apache/datafusion-comet/pull/5725) (peterxcli)
+- feat: native dynamic filter pushdown for hash join into Parquet scans [#5699](https://github.com/apache/datafusion-comet/pull/5699) (pingzh)
+- feat: enable codegen dispatch for `lpad` and `rpad` [#5764](https://github.com/apache/datafusion-comet/pull/5764) (Satyr09)
+- feat: address remaining issues for `CreateArray` [#5766](https://github.com/apache/datafusion-comet/pull/5766) (comphead)
+- ci: stop non-gating labels from skipping Preflight [#5784](https://github.com/apache/datafusion-comet/pull/5784) (comphead)
+- feat: route `abs` on interval types through the codegen dispatcher [#5622](https://github.com/apache/datafusion-comet/pull/5622) (kazantsev-maksim)
+- ci: shard Iceberg Spark tests across four runners [#5459](https://github.com/apache/datafusion-comet/pull/5459) (sunchao)
+- test: expand ANSI coverage for round, conv and elt [#5799](https://github.com/apache/datafusion-comet/pull/5799) (rich7420)
+- test: strengthen ANSI exception assertions [#5800](https://github.com/apache/datafusion-comet/pull/5800) (rich7420)
+- chore: Improve network retry configuration for maven and artifact upload [#5782](https://github.com/apache/datafusion-comet/pull/5782) (comphead)
+- ci: gate the Delta contrib build on symbols, not on libcomet size [#5827](https://github.com/apache/datafusion-comet/pull/5827) (andygrove)
+- test: add helpers to assert whether an expression ran natively or via codegen dispatch [#5610](https://github.com/apache/datafusion-comet/pull/5610) (andygrove)
+- feat: support Spark encode expression via codegen dispatch [#5037](https://github.com/apache/datafusion-comet/pull/5037) (andygrove)
+- feat: support native aggregate function `mode` [#4782](https://github.com/apache/datafusion-comet/pull/4782) (andygrove)
+- test: name the whole dispatched subtree in the decimal promotion assertion [#5849](https://github.com/apache/datafusion-comet/pull/5849) (andygrove)
+- feat: route translate and to_csv through codegen dispatch by default [#5032](https://github.com/apache/datafusion-comet/pull/5032) (andygrove)
+- feat: expose native Parquet scan I/O and read-amplification metrics [#5453](https://github.com/apache/datafusion-comet/pull/5453) (sunchao)
+- ci: move the job routing policy out of ci.yml expressions and into compute-changes.py [#5850](https://github.com/apache/datafusion-comet/pull/5850) (andygrove)
+- feat: support max_by and min_by aggregate expressions [#4817](https://github.com/apache/datafusion-comet/pull/4817) (andygrove)
+- ci: add a Required Checks aggregator job so main can have a required status check [#5842](https://github.com/apache/datafusion-comet/pull/5842) (andygrove)
+- feat: adapt Parquet storage for Variant projection [#5794](https://github.com/apache/datafusion-comet/pull/5794) (peterxcli)
+- ci: move the Spark 3.4/3.5/4.0, Iceberg, macOS and benchmark suites behind a merge queue [#5843](https://github.com/apache/datafusion-comet/pull/5843) (andygrove)
+- feat(iceberg): Iceberg table format V3: apply deletion vector on reads [#5853](https://github.com/apache/datafusion-comet/pull/5853) (mbutrovich)
+- test: restore ANSI array access error coverage [#5798](https://github.com/apache/datafusion-comet/pull/5798) (rich7420)
+- test: cover native memory accounting boundaries [#5856](https://github.com/apache/datafusion-comet/pull/5856) (rich7420)
+- test: cover date maps in Parquet temporal fuzz tests [#5877](https://github.com/apache/datafusion-comet/pull/5877) (rich7420)
+- feat: Enable adaptive partial aggregation for eligible native shuffle plans [#5723](https://github.com/apache/datafusion-comet/pull/5723) (peterxcli)
+- ci: move the Spark 4.1 sql_hive shards behind the merge queue [#5871](https://github.com/apache/datafusion-comet/pull/5871) (andygrove)
+- ci: retry the Maven wrapper bootstrap in every job that calls ./mvnw directly [#5852](https://github.com/apache/datafusion-comet/pull/5852) (andygrove)
+- test: cover slice over expression-produced non-null element arrays (#… [#5839](https://github.com/apache/datafusion-comet/pull/5839) (sam-1112)
+- test: run the libhdfs suite manually instead of in CI [#5892](https://github.com/apache/datafusion-comet/pull/5892) (andygrove)
+- feat: Add Lance contrib build gate [#5728](https://github.com/apache/datafusion-comet/pull/5728) (wirybeaver)
+- chore: drop support for JDK 11 [#5897](https://github.com/apache/datafusion-comet/pull/5897) (manuzhang)
+- test: cover regex expression routing configurations [#5917](https://github.com/apache/datafusion-comet/pull/5917) (rich7420)
+- test: cover round expression routing configurations [#5878](https://github.com/apache/datafusion-comet/pull/5878) (rich7420)
+- test: cover to_csv routing configurations [#5921](https://github.com/apache/datafusion-comet/pull/5921) (rich7420)
+- test: cover interval expression routing configurations [#5920](https://github.com/apache/datafusion-comet/pull/5920) (rich7420)
+- test: cover array and map expression routing configurations [#5918](https://github.com/apache/datafusion-comet/pull/5918) (rich7420)
+- test: make the cancelled Iceberg abort test yield deterministically [#5919](https://github.com/apache/datafusion-comet/pull/5919) (andygrove)
+- chore: deprecate Spark 3.4 rather than removing it in 1.1.0 [#5885](https://github.com/apache/datafusion-comet/pull/5885) (andygrove)
+- test: restore the TPC-H suite’s 2 GiB off-heap budget [#5904](https://github.com/apache/datafusion-comet/pull/5904) (ErikBPF)
+- test: cover string expression routing and native opt-in [#5915](https://github.com/apache/datafusion-comet/pull/5915) (rich7420)
+- chore: run only the cache-writing jobs on push to main [#5930](https://github.com/apache/datafusion-comet/pull/5930) (andygrove)
+- ci: publish nightly SNAPSHOT jars to repository.apache.org [#5902](https://github.com/apache/datafusion-comet/pull/5902) (andygrove)
+- ci: fold the Delta build gate and PyArrow UDF suite into the merge queue tiers [#5926](https://github.com/apache/datafusion-comet/pull/5926) (andygrove)
+- refactor: use Arrow decimal precision validation [#5160](https://github.com/apache/datafusion-comet/pull/5160) (Smallfu666)
+- ci: shrink the pull request tier to the Linux build on the default Spark profile [#5939](https://github.com/apache/datafusion-comet/pull/5939) (andygrove)
+- ci: run the non-default Spark and Iceberg suites nightly instead of in the merge queue [#5963](https://github.com/apache/datafusion-comet/pull/5963) (andygrove)
+- test: cover lpad and rpad routing configurations [#5895](https://github.com/apache/datafusion-comet/pull/5895) (rich7420)
+- test: preserve Spark SQL baselines for ordering-sensitive fixtures [#5914](https://github.com/apache/datafusion-comet/pull/5914) (rich7420)
+- test: cover json_array_length routing configurations [#5940](https://github.com/apache/datafusion-comet/pull/5940) (peterxcli)
+- test: cover SQL fixture metadata and statement parsing [#5941](https://github.com/apache/datafusion-comet/pull/5941) (peterxcli)
+- test: cover crc32 on binary inputs [#5942](https://github.com/apache/datafusion-comet/pull/5942) (peterxcli)
+- test: cover datetime and timezone routing configurations [#5950](https://github.com/apache/datafusion-comet/pull/5950) (rich7420)
+- test: cover collation and predicate routing configurations [#5951](https://github.com/apache/datafusion-comet/pull/5951) (rich7420)
+- test: cover array_intersect routing configurations [#5952](https://github.com/apache/datafusion-comet/pull/5952) (rich7420)
+- test: guard native plan equality against omitted parameters [#5953](https://github.com/apache/datafusion-comet/pull/5953) (rich7420)
+- test: expand replace compatibility regression coverage [#5409](https://github.com/apache/datafusion-comet/pull/5409) (sam-1112)
+- feat: add native allocation accounting for memory observability [#5934](https://github.com/apache/datafusion-comet/pull/5934) (andygrove)
+- feat: run rlike natively by default for Java-equivalent literal patterns [#5415](https://github.com/apache/datafusion-comet/pull/5415) (sam-1112)
+- ci: write large actions/cache entries only on push to main [#5973](https://github.com/apache/datafusion-comet/pull/5973) (andygrove)
+- refactor: extract shared runtime filter components [#5937](https://github.com/apache/datafusion-comet/pull/5937) (pingzh)
+- test: skip the Spark RocksDB state-store suites when Comet is enabled [#5987](https://github.com/apache/datafusion-comet/pull/5987) (comphead)
+- chore: bench nondeterministic and json kernels [#5989](https://github.com/apache/datafusion-comet/pull/5989) (coderfender)
+- feat: add BinaryType support for SortMergeJoin [#5928](https://github.com/apache/datafusion-comet/pull/5928) (xhumanoid)
+- refactor: replace remaining hand-rolled loops with Arrow kernels [#5367](https://github.com/apache/datafusion-comet/pull/5367) (0lai0)
+- feat: support Spark 4 EmptyRelationExec as a native input [#5821](https://github.com/apache/datafusion-comet/pull/5821) (jeffw13)
+- chore: Bench agg (welford) stats [#5988](https://github.com/apache/datafusion-comet/pull/5988) (coderfender)
+- ci: rename the umbrella workflow from CI to Comet CI [#6003](https://github.com/apache/datafusion-comet/pull/6003) (andygrove)
+- ci: bootstrap Maven from the setup actions so every mvnw job is covered [#5881](https://github.com/apache/datafusion-comet/pull/5881) (andygrove)
+- ci: add dev/local-ci.sh to run the Spark SQL and Iceberg suites locally [#5974](https://github.com/apache/datafusion-comet/pull/5974) (comphead)
+- feat: narrow strict floating-point admission for scalar sort keys [#5981](https://github.com/apache/datafusion-comet/pull/5981) (0lai0)
+- feat: route compatible xxhash64 args through SparkXxhash64 [#5960](https://github.com/apache/datafusion-comet/pull/5960) (sam-1112)
+- feat: support string arrays in Spark-to-Comet conversion [#5954](https://github.com/apache/datafusion-comet/pull/5954) (rich7420)
+- feat: run length on binary input natively [#5874](https://github.com/apache/datafusion-comet/pull/5874) (dwsmith1983)
+- chore: drop the redundant width_bucket shim registrations and guard serde uniqueness [#5873](https://github.com/apache/datafusion-comet/pull/5873) (dwsmith1983)
+- feat: hook native Parquet writes into Spark's WriteFilesExec seam on Spark 4.0+ [#5763](https://github.com/apache/datafusion-comet/pull/5763) (andygrove)
+- feat(iceberg): report native Iceberg scan planning metrics and scan time in the Spark UI [#6027](https://github.com/apache/datafusion-comet/pull/6027) (parthchandra)
+- test: guard page skipping in the native Iceberg scan [#6040](https://github.com/apache/datafusion-comet/pull/6040) (dwsmith1983)
+- test: stop claiming Spark agrees on signed-zero array literals [#6055](https://github.com/apache/datafusion-comet/pull/6055) (cestercian)
+- feat: trace Arrow memory held on the JVM side [#6048](https://github.com/apache/datafusion-comet/pull/6048) (andygrove)
+- feat: unix_timestamp codegen dispatch for strings, and fix pre-epoch fractional rounding [#5789](https://github.com/apache/datafusion-comet/pull/5789) (Satyr09)
+- chore: remove JaCoCo from the build [#6084](https://github.com/apache/datafusion-comet/pull/6084) (andygrove)
+- refactor: compose CometScanRule and CometExecRule into a single CometRule [#6082](https://github.com/apache/datafusion-comet/pull/6082) (andygrove)
+- chore: add a native Iceberg write benchmark [#6038](https://github.com/apache/datafusion-comet/pull/6038) (0lai0)
+- ci: retry the Spark test pre-compile step on resolution failures [#6079](https://github.com/apache/datafusion-comet/pull/6079) (andygrove)
+- ci: focus Miri on unsafe row and hash tests [#6072](https://github.com/apache/datafusion-comet/pull/6072) (rich7420)
+- test: run the Iceberg Spark tests with the native Iceberg writer enabled [#5677](https://github.com/apache/datafusion-comet/pull/5677) (andygrove)
+- feat: support listagg / string_agg aggregate (Spark 4.0+) [#4816](https://github.com/apache/datafusion-comet/pull/4816) (andygrove)
+- ci: run label-triggered CI as a separate workflow [#6161](https://github.com/apache/datafusion-comet/pull/6161) (andygrove)
+- feat: always count native allocations and log executor native memory usage [#6162](https://github.com/apache/datafusion-comet/pull/6162) (andygrove)
+- feat: use DataFusion unnest_outer instead of Comet ListEmptyToNullExpr [#6132](https://github.com/apache/datafusion-comet/pull/6132) (comphead)
+- chore: recommend --force-with-lease when updating PR branches [#6171](https://github.com/apache/datafusion-comet/pull/6171) (mizulun)
+- feat: positional round robin shuffle keyed on a row ordinal [#6095](https://github.com/apache/datafusion-comet/pull/6095) (andygrove)
+- chore: deprecate spark.comet.exec.memoryPool.fraction [#6163](https://github.com/apache/datafusion-comet/pull/6163) (andygrove)
+- feat: add spark.comet.explain.planOnly.enabled [#5394](https://github.com/apache/datafusion-comet/pull/5394) (andygrove)
+- feat: support direct Variant projection in native Parquet scans [#5868](https://github.com/apache/datafusion-comet/pull/5868) (peterxcli)
+- test: cover JSON and cast expression routing [#6068](https://github.com/apache/datafusion-comet/pull/6068) (rich7420)
+- feat: remove memory accounting from Comet's on-heap mode [#6066](https://github.com/apache/datafusion-comet/pull/6066) (andygrove)
+- test: accept CometHashAggregateExec in the Spark 4.0 CollationSuite hash agg check [#6220](https://github.com/apache/datafusion-comet/pull/6220) (andygrove)
+- feat: per-location credentials for the S3 credential SPI [#6031](https://github.com/apache/datafusion-comet/pull/6031) (snmvaughan)
+- refactor: centralize data type support predicates [#5025](https://github.com/apache/datafusion-comet/pull/5025) (peterxcli)
+- chore: [branch-1.1] change version from 1.1.0-SNAPSHOT to 1.1.0 [#6236](https://github.com/apache/datafusion-comet/pull/6236) (andygrove)
+- feat: [branch-1.1] built-in S3 credential provider adapters for the native Parquet scan (#6023) [#6318](https://github.com/apache/datafusion-comet/pull/6318) (andygrove, parthchandra)
+- chore: [branch-1.1] Add 1.1.0 changelog [#6282](https://github.com/apache/datafusion-comet/pull/6282) (andygrove)
+- test: [branch-1.1] fix flaky mixed field id directory test on Spark 3.4 and 3.5 (#6312) [#6527](https://github.com/apache/datafusion-comet/pull/6527) (andygrove)
+- test: [branch-1.1] cover sliced boolean arrays in explode (#6473) [#6492](https://github.com/apache/datafusion-comet/pull/6492) (andygrove)
+- test: [branch-1.1] backport the Iceberg write report and the mid-write retry test (#6155, #6111) [#6307](https://github.com/apache/datafusion-comet/pull/6307) (andygrove, sam-1112)
+- feat: [branch-1.1] reuse S3 credentials until shortly before their reported expiry (#6509) [#6545](https://github.com/apache/datafusion-comet/pull/6545) (andygrove, snmvaughan)
+
+**Documentation updates:**
+
+- docs: add note about run-iceberg-tests label in CI [#5247](https://github.com/apache/datafusion-comet/pull/5247) (mbutrovich)
+- docs: add 1.0.0 TPC-DS benchmark results, remove TPC-H [#5284](https://github.com/apache/datafusion-comet/pull/5284) (mbutrovich)
+- docs: add 1.0.0 changelog to main [#5263](https://github.com/apache/datafusion-comet/pull/5263) (andygrove)
+- docs: correct Spark 4.2 version and CI test status in installation guide [#5315](https://github.com/apache/datafusion-comet/pull/5315) (andygrove)
+- docs: add suggest-native-expression skill for assessing native expression candidates [#5348](https://github.com/apache/datafusion-comet/pull/5348) (andygrove)
+- docs: document C2R cost for wide/nested schemas in tuning guide [#5458](https://github.com/apache/datafusion-comet/pull/5458) (DebadityaHait)
+- docs: update stale interval multiplication note in expressions.md [#5518](https://github.com/apache/datafusion-comet/pull/5518) (peterxcli)
+- docs: add contributing guide link [#5521](https://github.com/apache/datafusion-comet/pull/5521) (Dharan-K)
+- doc: fix benchmark examples for MacOS [#5522](https://github.com/apache/datafusion-comet/pull/5522) (xhumanoid)
+- docs: add comet meeting link [#5621](https://github.com/apache/datafusion-comet/pull/5621) (coderfender)
+- docs: add a contributor guide page for CI and the merge queue [#5863](https://github.com/apache/datafusion-comet/pull/5863) (andygrove)
+- docs: add contributor guide page on memory management [#5933](https://github.com/apache/datafusion-comet/pull/5933) (andygrove)
+- docs: how to check the scheduled CI runs are actually running [#6000](https://github.com/apache/datafusion-comet/pull/6000) (andygrove)
+- docs: explain allocator hazards and diagram where memory is allocated [#6014](https://github.com/apache/datafusion-comet/pull/6014) (andygrove)
+- docs: pre-render mermaid diagrams to SVG at build time [#6021](https://github.com/apache/datafusion-comet/pull/6021) (andygrove)
+- docs: split the PR review skill by area and correct the shuffle contributor docs [#6018](https://github.com/apache/datafusion-comet/pull/6018) (andygrove)
+- docs: correct the stale range-partitioning strict floating-point rule [#6049](https://github.com/apache/datafusion-comet/pull/6049) (andygrove)
+- docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory [#6051](https://github.com/apache/datafusion-comet/pull/6051) (andygrove)
+- docs: exempt testing-category and internal configs from the versioning policy [#6089](https://github.com/apache/datafusion-comet/pull/6089) (andygrove)
+- docs: make doc changes weekly comet sync [#6160](https://github.com/apache/datafusion-comet/pull/6160) (coderfender)
+- docs: correct the plugin and shuffle sections of the plugin overview [#6197](https://github.com/apache/datafusion-comet/pull/6197) (andygrove)
+- docs: fix stale and missing native Iceberg write details [#6150](https://github.com/apache/datafusion-comet/pull/6150) (andygrove)
+- docs: add an Iceberg writes contributor guide and review skill [#6149](https://github.com/apache/datafusion-comet/pull/6149) (andygrove)
+- docs: correct which operators run separate native plans in a task [#6194](https://github.com/apache/datafusion-comet/pull/6194) (andygrove)
+- docs: update the user guide for the 1.1.0 release [#6168](https://github.com/apache/datafusion-comet/pull/6168) (andygrove)
+- docs: [branch-1.1] backport the 1.1.0 upgrade notes and tuning guide updates (#6237, #6244, #6248) [#6265](https://github.com/apache/datafusion-comet/pull/6265) (andygrove)
+- ci: [branch-1.1] run every tier on release-branch pull requests, and the full suite before an RC (#6218) [#6284](https://github.com/apache/datafusion-comet/pull/6284) (andygrove)
+- docs: [branch-1.1] generate release docs for 1.1.0 [#6344](https://github.com/apache/datafusion-comet/pull/6344) (andygrove)
+- docs: [branch-1.1] credit co-authors in the 1.1.0 changelog [#6359](https://github.com/apache/datafusion-comet/pull/6359) (andygrove)
+
+**Other:**
+
+- chore: start 1.1.0 development [#5242](https://github.com/apache/datafusion-comet/pull/5242) (andygrove)
+- refactor: replace hand-coded rollup of expression fallback reasons onto operators [#5236](https://github.com/apache/datafusion-comet/pull/5236) (andygrove)
+- test: rename CometCastSuite to CometNativeCastSuite [#5268](https://github.com/apache/datafusion-comet/pull/5268) (andygrove)
+- refactor: use arity helper for Int to Decimal128 reinterpretation [#5193](https://github.com/apache/datafusion-comet/pull/5193) (0lai0)
+- test: add guard for Iceberg version on Variant fallback test [#5278](https://github.com/apache/datafusion-comet/pull/5278) (mbutrovich)
+- chore: update documentation links for 1.0.0 release [#5290](https://github.com/apache/datafusion-comet/pull/5290) (andygrove)
+- chore(deps): bump reqsign-core from 3.2.0 to 3.2.1 in /native in the all-other-cargo-deps group [#5287](https://github.com/apache/datafusion-comet/pull/5287) (dependabot[bot])
+- chore(deps): bump object_store_opendal from 0.57.0 to 0.58.0 in /native [#5289](https://github.com/apache/datafusion-comet/pull/5289) (dependabot[bot])
+- chore(deps): bump the codeql-actions group with 2 updates [#5286](https://github.com/apache/datafusion-comet/pull/5286) (dependabot[bot])
+- bug: Revert "chore(deps): bump object_store_opendal from 0.57.0 to 0.58.0 in /native" [#5332](https://github.com/apache/datafusion-comet/pull/5332) (coderfender)
+- test: cover the narrowing direction of cast_and_stamp_schema [#5285](https://github.com/apache/datafusion-comet/pull/5285) (andygrove)
+- chore(deps): bump opendal from 0.57.0 to 0.58.1 in /native [#5324](https://github.com/apache/datafusion-comet/pull/5324) (manuzhang)
+- chore: respect Cargo parallelism settings for native release builds [#5344](https://github.com/apache/datafusion-comet/pull/5344) (pingzh)
+- test: make expression benchmark harness fair and reproducible [#5371](https://github.com/apache/datafusion-comet/pull/5371) (andygrove)
+- chore(deps): bump the all-other-cargo-deps group in /native with 5 updates [#5360](https://github.com/apache/datafusion-comet/pull/5360) (dependabot[bot])
+- test: fix vacuous signed-zero coverage in SQL file tests [#5393](https://github.com/apache/datafusion-comet/pull/5393) (sam-1112)
+- chore: fix clippy warnings for Rust 1.98 [#5400](https://github.com/apache/datafusion-comet/pull/5400) (ywskycn)
+- chore(deps): bump the codeql-actions group with 2 updates [#5405](https://github.com/apache/datafusion-comet/pull/5405) (dependabot[bot])
+- test: strengthen signed-zero SQL assertions [#5404](https://github.com/apache/datafusion-comet/pull/5404) (sunchao)
+- chore(deps): bump actions/checkout from 6 to 7 [#5406](https://github.com/apache/datafusion-comet/pull/5406) (dependabot[bot])
+- test: add expression fallback-invariance suite [#5329](https://github.com/apache/datafusion-comet/pull/5329) (4ktLuffy)
+- refactor: delegate ANSI integer arithmetic to arrow checked kernels [#5280](https://github.com/apache/datafusion-comet/pull/5280) (kazantsev-maksim)
+- test: docs and test-coverage hardening for native collect_list / collect_set [#5055](https://github.com/apache/datafusion-comet/pull/5055) (andygrove)
+- test: deduplicate Throwable cause-chain traversal [#5441](https://github.com/apache/datafusion-comet/pull/5441) (peterxcli)
+- refactor: use Arrow timezone type [#5129](https://github.com/apache/datafusion-comet/pull/5129) (Hashim1999164)
+- chore: add md formatting to `make format` [#5460](https://github.com/apache/datafusion-comet/pull/5460) (comphead)
+- test: fail unexpected vacuous fallback-invariance checks [#5417](https://github.com/apache/datafusion-comet/pull/5417) (sunchao)
+- ci: cache Maven distributions and retry bootstrap downloads [#5422](https://github.com/apache/datafusion-comet/pull/5422) (sunchao)
+- chore: add math benches [#5479](https://github.com/apache/datafusion-comet/pull/5479) (coderfender)
+- chore: Bench string functions [#5492](https://github.com/apache/datafusion-comet/pull/5492) (coderfender)
+- chore(deps): bump actions/setup-java from 5 to 6 [#5524](https://github.com/apache/datafusion-comet/pull/5524) (dependabot[bot])
+- chore(deps): bump the codeql-actions group with 2 updates [#5523](https://github.com/apache/datafusion-comet/pull/5523) (dependabot[bot])
+- chore: bench additional math function [#5520](https://github.com/apache/datafusion-comet/pull/5520) (coderfender)
+- chore: remove redundant condition and stray println in shuffle code [#5562](https://github.com/apache/datafusion-comet/pull/5562) (viirya)
+- test: cover struct data columns in native shuffle [#5564](https://github.com/apache/datafusion-comet/pull/5564) (viirya)
+- chore: rename .claude directory to vendor-neutral .ai [#5594](https://github.com/apache/datafusion-comet/pull/5594) (andygrove)
+- chore: bench additional scalar functions [#5598](https://github.com/apache/datafusion-comet/pull/5598) (coderfender)
+- test: verify Celeborn reflection compatibility [#5604](https://github.com/apache/datafusion-comet/pull/5604) (pingzh)
+- test: enable native path in lower/upper_enabled sql fixtures [#5619](https://github.com/apache/datafusion-comet/pull/5619) (cestercian)
+- chore: add benches datetime [#5620](https://github.com/apache/datafusion-comet/pull/5620) (coderfender)
+- chore: move dead and defensive serde guards out of convert [#5595](https://github.com/apache/datafusion-comet/pull/5595) (andygrove)
+- test: exercise the Iceberg write split-operator plan in Iceberg's own suites [#5640](https://github.com/apache/datafusion-comet/pull/5640) (andygrove)
+- chore(deps): bump the codeql-actions group with 2 updates [#5669](https://github.com/apache/datafusion-comet/pull/5669) (dependabot[bot])
+- test: add explode operator microbenchmark [#5381](https://github.com/apache/datafusion-comet/pull/5381) (andygrove)
+- deps: bump DataFusion 55.0 and Arrow/Parquet 59.2 [#5262](https://github.com/apache/datafusion-comet/pull/5262) (mbutrovich)
+- chore: add benches array functions [#5700](https://github.com/apache/datafusion-comet/pull/5700) (coderfender)
+- test: restore Comet coverage for recursive HAVING and ORDER BY [#5755](https://github.com/apache/datafusion-comet/pull/5755) (rich7420)
+- test: restore Parquet V2 writer and delta encoding coverage [#5760](https://github.com/apache/datafusion-comet/pull/5760) (rich7420)
+- chore: Add benches for datetime funcs [#5767](https://github.com/apache/datafusion-comet/pull/5767) (coderfender)
+- bench: add a benchmark for the Spark hash kernels [#5765](https://github.com/apache/datafusion-comet/pull/5765) (viirya)
+- test: restore Spark 4.1 Variant shredding suites [#5745](https://github.com/apache/datafusion-comet/pull/5745) (rich7420)
+- refactor: share one helper for pushing a struct's null mask into its children [#5769](https://github.com/apache/datafusion-comet/pull/5769) (viirya)
+- ci: label pull requests by changed paths and title prefix [#5762](https://github.com/apache/datafusion-comet/pull/5762) (dwsmith1983)
+- test: cover ambiguous exact nested Parquet field matches [#5751](https://github.com/apache/datafusion-comet/pull/5751) (peterxcli)
+- bench: measure nested types as native shuffle hash partitioning keys [#5788](https://github.com/apache/datafusion-comet/pull/5788) (viirya)
+- deps: bump to datafusion 55.1.0 [#5865](https://github.com/apache/datafusion-comet/pull/5865) (comphead)
+- bench: isolate map normalization and nested key hashing [#5822](https://github.com/apache/datafusion-comet/pull/5822) (viirya)
+- bench: add a shuffle read benchmark covering the per-block schema parse [#5805](https://github.com/apache/datafusion-comet/pull/5805) (peterxcli)
+- chore(deps): bump actions/setup-java from 4 to 6 [#6012](https://github.com/apache/datafusion-comet/pull/6012) (dependabot[bot])
+- chore(deps): bump the codeql-actions group with 2 updates [#6011](https://github.com/apache/datafusion-comet/pull/6011) (dependabot[bot])
+- deps: bump the iceberg-rust pin to bb1e4a4 and document why it is pinned [#6094](https://github.com/apache/datafusion-comet/pull/6094) (andygrove)
+
+## Credits
+
+Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. A PR with commits from more than one person counts for each of them.
+
+```
+ 147 Andy Grove
+ 62 Peter Lee
+ 26 Chao Sun
+ 26 KUAN-HAO HUANG
+ 19 Ping Zhang
+ 15 Oleks V
+ 14 dustin
+ 13 Bhargava Vadlamani
+ 13 Liang-Chi Hsieh
+ 11 dependabot[bot]
+ 10 ChenChen Lai
+ 8 sam-1112
+ 6 Matt Butrovich
+ 5 Han-Yin Chang
+ 5 Parth Chandra
+ 4 Erik Bogado
+ 4 Steve Vaughan
+ 4 Wei Yan
+ 3 Jordan Epstein
+ 3 Manu Zhang
+ 2 Alexey
+ 2 Cestercian
+ 2 Daipayan Mukherjee
+ 2 Feng Zhang
+ 2 Kazantsev Maksim
+ 2 YuLun Mao
+ 2 liupoyi-1031
+ 1 Dharan-K
+ 1 Hashim Khan
+ 1 Henos D
+ 1 Jeff Wang
+ 1 LinSimon-901101
+ 1 Michael Taranov
+ 1 Scott Schenkein
+ 1 Stefan Wang
+ 1 Unik Dahal
+ 1 Viacheslav Inozemtsev
+ 1 Xuanyi Li
+ 1 debaditya
+ 1 divyank sameer shah
+ 1 hsiang-c
+```
+
+Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.
diff --git a/docs/source/changelog/index.md b/docs/source/changelog/index.md
index e79878f7e7..792dfa2df3 100644
--- a/docs/source/changelog/index.md
+++ b/docs/source/changelog/index.md
@@ -24,6 +24,7 @@ Per-release change logs for Apache DataFusion Comet.
```{toctree}
:maxdepth: 1
+1.1.0 <1.1.0>
1.0.0 <1.0.0>
0.17.1 <0.17.1>
0.17.0 <0.17.0>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org