You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/16 22:53:56 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-4798-46b9c6afe20ccb8f43ca747d87b18f408486b364 created (now d44366230a)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-4798-46b9c6afe20ccb8f43ca747d87b18f408486b364
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at d44366230a feat: support PivotFirst aggregate for the optimized PIVOT fast path (#4798)
This branch includes the following new commits:
new d44366230a feat: support PivotFirst aggregate for the optimized PIVOT fast path (#4798)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: feat: support PivotFirst aggregate for the optimized PIVOT fast path (#4798)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-4798-46b9c6afe20ccb8f43ca747d87b18f408486b364
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit d44366230aed8aedd9e7d9fca24fdfe9a668a184
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Oct 16 22:53:34 2026 +0000
feat: support PivotFirst aggregate for the optimized PIVOT fast path (#4798)
* feat: support PivotFirst aggregate for the optimized PIVOT fast path
Wire the Spark-internal PivotFirst aggregate through Comet so that
PIVOT queries which trigger Spark's two-phase fast path (12+ pivot
values with an aggregate whose output type is in
PivotFirst.supportsDataType) run natively instead of falling back.
Scaffolded with the implement-comet-expression skill.
* simplify: address /simplify review of PivotFirst
- Delegate value-type gate to Spark's `PivotFirst.supportsDataType` instead
of maintaining a parallel list in the serde.
- Wrap the plan-time pivot map/vector in `Arc` so per-group accumulator
init bumps a refcount instead of deep-cloning.
- Drop the redundant `pivot_values` field from the accumulator (only the
map and `slots.len()` are read at runtime); the accumulator now derives
its slot count from the shared map.
- Use the shared `format_state_name` helper for state field names.
- Delegate `evaluate()` to `state()` so the two paths share their slot
materialization.
- Walk each state column backwards in `merge_batch` and stop at the first
non-null so we build at most one `ScalarValue` per slot per merge.
- Drop the "pad every IN clause to 12 values" workaround from the SQL
tests and the CometAggregateSuite test - the fast-path gate is
supported-data-type only, no minimum count.
* fix: match Spark's float pivot key semantics in PivotFirst
* fix: fall back for PivotFirst pivot columns and lists Comet cannot match
Spark finds a pivot value's slot in a Scala HashMap only for atomic pivot
columns, and even there a binary value matches by reference, so no row ever
matches before Spark 4.2. Arrays and structs (including the struct that a
pivot on several columns builds) are matched through an interpreted ordering,
and from 4.2 so are binary values and non-default collations. The native
lookup reproduces none of that, so those pivot columns now fall back.
Spark also sizes the aggregate buffer from the distinct pivot values, so a
list with duplicates (including 0.0 next to -0.0) made the native partial
aggregate emit more columns than the shuffle schema declares. Such lists fall
back too.
The SQL fixture now asserts native execution for every query that should run
natively, covers each pivot column type that does, and checks each fallback.
* perf: add a GroupsAccumulator for PivotFirst
PivotFirst only had a row Accumulator, so grouped aggregation went through
DataFusion's GroupsAccumulatorAdapter, which visits every group's state on each
batch, and the accumulator built a ScalarValue for every input row.
The grouped accumulator keeps every group's slots in one flat, group-major
buffer with a validity bitmap, and returns them as the list without copying.
Pivot values are looked up in a typed index built once per aggregate, with
Spark's float rules in a new float_semantics::ieee_equality_key helper. The
ungrouped accumulator shares the index. A pivot list with a repeated value is
rejected at construction, since the serde falls back for it.
* test: add a benchmark for the optimized PIVOT plan
CometPivotBenchmark pivots sum(v) over 4M rows on a string column with 4,
32 or 128 values, grouped into 1K, 100K or 1M groups, and compares Spark, Comet
with PivotFirst disabled (so the aggregates using it fall back while the rest
of the plan stays native) and Comet with PivotFirst native. The query selects
every pivot column into the noop sink, and before timing each Comet run is
checked against Spark's answer and for the plan its name claims.
* perf: copy only the emitted groups out of PivotFirst's state
`EmitTo::First(n)` split the cell buffer with `split_off`, which leaves the
original allocation with the first part and copies the rest. The emitted
batch therefore held the buffer of every group, and the remaining groups
were copied, nearly doubling live memory for a partial emit of a few
groups. `EmitTo::take_needed` copies out whichever part is smaller.
* refactor: run ungrouped PivotFirst through the grouped accumulator
The ungrouped accumulator kept a `ScalarValue` per slot, built one with
`ScalarValue::try_from_array` for every matching input row, and repeated
the grouped accumulator's update, merge and state logic. It now wraps a
`PivotFirstGroupsAccumulator` holding one group. `evaluate` reads that
group's slots without taking them, because DataFusion requires that an
accumulator can be evaluated more than once.
That leaves native code no row-based path for a value type the grouped
accumulator does not store. The serde therefore checks the value type
against the types the native aggregate stores rather than calling
`PivotFirst.supportsDataType`, so a type that a later Spark version adds
falls back instead of failing the query.
* test: cover ungrouped, empty, multi-batch and collated PivotFirst inputs
PivotFirst.sql gains a pivot with no grouping column, the same pivot over
no rows, and a pivot over 20000 groups whose input is repartitioned by the
pivot column, so that each group has several partial states to merge.
PivotFirst_collation.sql checks on Spark 4.0+ that a UTF8_LCASE pivot
column falls back.
Every native query now also asserts that PivotFirst itself ran natively.
That was the only thing the two CometAggregateSuite tests checked that the
fixture did not, so they are removed. The fixture's comments no longer name
a nonexistent PivotTransformer or threshold, or a cast the float query
does not make.
---
.../expression-audits/agg_funcs.md | 9 +
docs/source/user-guide/latest/expressions.md | 1 +
native/core/src/execution/planner.rs | 27 +-
native/proto/src/proto/expr.proto | 16 +
native/spark-expr/src/agg_funcs/mod.rs | 2 +
native/spark-expr/src/agg_funcs/pivot_first.rs | 1142 ++++++++++++++++++++
native/spark-expr/src/float_semantics/mod.rs | 32 +
.../org/apache/comet/serde/QueryPlanSerde.scala | 1 +
.../scala/org/apache/comet/serde/aggregates.scala | 101 +-
.../sql-tests/expressions/aggregate/PivotFirst.sql | 571 ++++++++++
.../expressions/aggregate/PivotFirst_collation.sql | 38 +
.../spark/sql/benchmark/CometPivotBenchmark.scala | 141 +++
12 files changed, 2079 insertions(+), 2 deletions(-)
diff --git a/docs/source/contributor-guide/expression-audits/agg_funcs.md b/docs/source/contributor-guide/expression-audits/agg_funcs.md
index 10445f247a..c3157c0ba6 100644
--- a/docs/source/contributor-guide/expression-audits/agg_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/agg_funcs.md
@@ -109,4 +109,13 @@
- Spark 4.1.1 (audited 2026-06-24): identical to 4.0.1.
- `CometPercentile` reports `Compatible` for the single-literal-percentage, default-frequency, numeric-input, ascending form and runs it natively by default. Every other form is `Unsupported` and falls back to Spark.
+## pivot_first
+
+- Spark 3.4.3 (audited 2026-07-02): `PivotFirst(pivotColumn, valueColumn, pivotColumnValues)` is an internal `ImperativeAggregate` emitted only by the optimized-pivot fast path in `Analyzer.ResolvePivot`. It buckets `valueColumn` into an array of length `indexSize` indexed by matching `pivotColumn`. Null value columns are ignored, unmatched pivot values are dropped. Value types are gated by `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal). The index [...]
+- Spark 3.5.8 (audited 2026-07-02): swaps `StructType.fromAttributes` for `DataTypeUtils.fromAttributes`; no behavior change.
+- Spark 4.0.1 (audited 2026-07-02): identical to 3.5.8.
+- Spark 4.1.1 (audited 2026-07-02): identical to 4.0.1.
+- Spark 4.2.0 (audited 2026-10-12): the `TreeMap` is chosen for every pivot column type without proper equals (`!TypeUtils.typeWithProperEquals`), which adds binary and strings with a non-binary-equality collation, so binary pivot values now match by content. A `findPivotIndex` wrapper returns `-1` for a null key on the `TreeMap` path instead of throwing.
+- `CometPivotFirst` (in `spark/src/main/scala/org/apache/comet/serde/aggregates.scala`) forwards the aggregate to the native `SparkPivotFirst` UDAF (`native/spark-expr/src/agg_funcs/pivot_first.rs`) when the value type is in the supported set and the pivot column is a Boolean, Byte, Short, Int, Long, Float, Double, Decimal, UTF8_BINARY String, Date, Timestamp or TimestampNTZ. Those are the types Spark keys by value equality on every version, which the native lookup reproduces, including [...]
+
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md
index bb2c80a5ce..dfe819139d 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -123,6 +123,7 @@ The tables below list every Spark built-in expression with its current status.
| `percentile_approx` | ✅ | Native | Alias of `approx_percentile`; same restrictions apply |
| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark |
| `percentile_disc` | 🔜 | — | Percentile aggregate |
+| `pivot_first` | ✅ | Native | Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark's standard filtered-aggregate path. Binary, array and struct pivot columns (including pivots on more than one column), non-default collations and pivot lists with duplicate values fall back to Spark |
| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
diff --git a/native/core/src/execution/planner.rs b/native/core/src/execution/planner.rs
index 0c9bf36e1e..7c1492c2e6 100644
--- a/native/core/src/execution/planner.rs
+++ b/native/core/src/execution/planner.rs
@@ -95,7 +95,7 @@ use datafusion_comet_spark_expr::{
create_comet_physical_fun, create_comet_physical_fun_with_eval_mode, BinaryOutputStyle,
BloomFilterAgg, BloomFilterMightContain, CheckedBinaryExpr, CometCollectList, CometCollectSet,
CsvWriteOptions, EvalMode, ListPositionsExpr, SparkArraysZipFunc, SparkBloomFilterVersion,
- SparkListAgg, SparkPercentile, Subquery, SumInteger, ToCsv,
+ SparkListAgg, SparkPercentile, SparkPivotFirst, Subquery, SumInteger, ToCsv,
};
use datafusion_datasource::TableSchema;
use iceberg::expr::Bind;
@@ -3699,6 +3699,31 @@ impl PhysicalPlanner {
AggregateUDF::new_from_impl(Mode::new(datatype, expr.normalize_neg_zero));
Self::create_aggr_func_expr("mode", schema, vec![child], func)
}
+ AggExprStruct::PivotFirst(expr) => {
+ let pivot_col =
+ self.create_expr(expr.pivot_column.as_ref().unwrap(), Arc::clone(&schema))?;
+ let value_col =
+ self.create_expr(expr.value_column.as_ref().unwrap(), Arc::clone(&schema))?;
+ let value_type = to_arrow_datatype(expr.value_datatype.as_ref().unwrap());
+ // Reconstruct pivot values as ScalarValues from the serialized Literal exprs.
+ // The Scala serde builds them as Literal(v, pivot_column.dataType), so they carry
+ // the pivot column's type, which `SparkPivotFirst` builds its lookup index from.
+ let mut pivot_values = Vec::with_capacity(expr.pivot_values.len());
+ for lit_expr in &expr.pivot_values {
+ let physical = self.create_expr(lit_expr, Arc::clone(&schema))?;
+ let literal = physical.downcast_ref::<Literal>().ok_or_else(|| {
+ GeneralError(
+ "PivotFirst pivot_values must be Literal expressions".to_string(),
+ )
+ })?;
+ pivot_values.push(literal.value().clone());
+ }
+ let func = AggregateUDF::new_from_impl(SparkPivotFirst::try_new(
+ value_type,
+ pivot_values,
+ )?);
+ Self::create_aggr_func_expr("pivot_first", schema, vec![pivot_col, value_col], func)
+ }
}
}
diff --git a/native/proto/src/proto/expr.proto b/native/proto/src/proto/expr.proto
index dac44095ce..969b3aaa9c 100644
--- a/native/proto/src/proto/expr.proto
+++ b/native/proto/src/proto/expr.proto
@@ -170,6 +170,7 @@ message AggExpr {
ListAgg listAgg = 26;
HllSketchAgg hllSketchAgg = 27;
HllUnionAgg hllUnionAgg = 28;
+ PivotFirst pivotFirst = 29;
}
// Optional filter expression for SQL FILTER (WHERE ...) clause.
@@ -416,6 +417,21 @@ message Mode {
bool normalize_neg_zero = 3;
}
+// Optimized Pivot second-phase aggregate. Given a `pivot_column` and a
+// pre-computed list of `pivot_values`, buckets the incoming `value_column`
+// into an array whose length is `pivot_values.size()`. Rows whose pivot
+// value is not in `pivot_values` are ignored; if multiple rows in the same
+// group map to the same bucket, the last non-null value wins (matches
+// Spark's `PivotFirst.update`/`merge`). Output type is
+// ArrayType(value_datatype). The pivot_values are serialized as Literal
+// Exprs so the value bytes and data type reach native code faithfully.
+message PivotFirst {
+ Expr pivot_column = 1;
+ Expr value_column = 2;
+ repeated Expr pivot_values = 3;
+ DataType value_datatype = 4;
+}
+
enum EvalMode {
LEGACY = 0;
TRY = 1;
diff --git a/native/spark-expr/src/agg_funcs/mod.rs b/native/spark-expr/src/agg_funcs/mod.rs
index ca9f4ecab2..0f9bfc955b 100644
--- a/native/spark-expr/src/agg_funcs/mod.rs
+++ b/native/spark-expr/src/agg_funcs/mod.rs
@@ -31,6 +31,7 @@ mod max_min_by;
mod min_max;
mod mode;
mod percentile;
+mod pivot_first;
mod quantile_summaries;
mod regr;
mod stddev;
@@ -54,6 +55,7 @@ pub use max_min_by::MaxMinBy;
pub use min_max::SparkMinMax;
pub use mode::Mode;
pub use percentile::SparkPercentile;
+pub use pivot_first::SparkPivotFirst;
pub use regr::{Regr, RegrType};
pub use stddev::Stddev;
pub use sum_decimal::SumDecimal;
diff --git a/native/spark-expr/src/agg_funcs/pivot_first.rs b/native/spark-expr/src/agg_funcs/pivot_first.rs
new file mode 100644
index 0000000000..43252001bd
--- /dev/null
+++ b/native/spark-expr/src/agg_funcs/pivot_first.rs
@@ -0,0 +1,1142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Spark's `PivotFirst` aggregate. Used only by the second phase of the optimized pivot plan
+//! that the analyzer rule `ResolvePivot` builds. For each group, `PivotFirst` maintains one slot
+//! per pivot value; on each input row it looks the pivot column up in the pivot list, and writes
+//! the value column into that slot when a match is found and the value is non-null. Rows with
+//! unmatched pivot values are ignored; matched rows with a null value column leave the slot
+//! unchanged (matches Spark).
+//!
+//! State layout is one column per pivot slot, matching Spark's `aggBufferAttributes` (which
+//! declares `indexSize` `AttributeReference`s, one per pivot value). This keeps the shuffle
+//! schema between Partial and Final consistent with what Spark catalyst declared; otherwise
+//! the shuffle exchange rejects the batch. Spark sizes that buffer from its index of the pivot
+//! values, so the serde falls back when the list holds duplicates, and the native slot count is
+//! always the list length. The result reassembles the slots into a `ListArray` matching
+//! `PivotFirst.dataType = ArrayType(value_type)`.
+//!
+//! The optimized pivot plan groups by every column of the pivoted relation other than the pivot
+//! and aggregated columns, so the aggregate runs grouped, through [`PivotFirstGroupsAccumulator`],
+//! unless the relation has no other column. That case uses [`PivotFirstAccumulator`], which is
+//! the same accumulator holding a single group.
+
+use crate::float_semantics::ieee_equality_key;
+use arrow::array::{
+ new_null_array, Array, ArrayRef, ArrowPrimitiveType, AsArray, BooleanArray,
+ BooleanBufferBuilder, ListArray, PrimitiveArray,
+};
+use arrow::buffer::{BooleanBuffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow::compute::cast;
+use arrow::datatypes::{
+ DataType, Date32Type, Decimal128Type, Field, FieldRef, Float32Type, Float64Type, Int16Type,
+ Int32Type, Int64Type, Int8Type, TimeUnit, TimestampMicrosecondType,
+};
+use datafusion::common::utils::SingleRowListArrayBuilder;
+use datafusion::common::{internal_err, not_impl_err, HashMap, Result, ScalarValue};
+use datafusion::logical_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion::logical_expr::Volatility::Immutable;
+use datafusion::logical_expr::{
+ Accumulator, AggregateUDFImpl, EmitTo, GroupsAccumulator, Signature,
+};
+use datafusion::physical_expr::expressions::format_state_name;
+use std::fmt::Debug;
+use std::mem::size_of_val;
+use std::sync::Arc;
+
+/// UDAF implementation of Spark's `PivotFirst`.
+///
+/// `pivot_values` is a fixed, plan-time list of the pivot column values that occupy each
+/// output slot, and `pivot_index` maps an input row's pivot column value to its slot. The index
+/// is wrapped in `Arc` so that every accumulator shares it rather than rebuilding it.
+#[derive(Debug)]
+pub struct SparkPivotFirst {
+ signature: Signature,
+ value_type: DataType,
+ // Kept for `PartialEq`/`Hash` (identity of the aggregate for plan comparison) and for the
+ // slot count `state_fields` needs.
+ pivot_values: Vec<ScalarValue>,
+ pivot_index: Arc<PivotIndex>,
+}
+
+impl PartialEq for SparkPivotFirst {
+ fn eq(&self, other: &Self) -> bool {
+ self.value_type == other.value_type && self.pivot_values == other.pivot_values
+ }
+}
+
+impl Eq for SparkPivotFirst {}
+
+impl std::hash::Hash for SparkPivotFirst {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.value_type.hash(state);
+ self.pivot_values.hash(state);
+ }
+}
+
+impl SparkPivotFirst {
+ /// `pivot_values` are the pivot list in output order. The serde only sends a non-empty list
+ /// without duplicates, of a pivot column type that [`PivotIndex`] supports.
+ pub fn try_new(value_type: DataType, pivot_values: Vec<ScalarValue>) -> Result<Self> {
+ let pivot_index = Arc::new(PivotIndex::try_new(&pivot_values)?);
+ Ok(Self {
+ signature: Signature::user_defined(Immutable),
+ value_type,
+ pivot_values,
+ pivot_index,
+ })
+ }
+}
+
+impl AggregateUDFImpl for SparkPivotFirst {
+ fn name(&self) -> &str {
+ "pivot_first"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::List(Arc::new(Field::new_list_field(
+ self.value_type.clone(),
+ true,
+ ))))
+ }
+
+ fn state_fields(&self, args: StateFieldsArgs) -> Result<Vec<FieldRef>> {
+ // One field per pivot slot, matching Spark's aggBufferAttributes so the shuffle
+ // exchange sees the same schema catalyst declared. `format_state_name` is the same
+ // helper other aggregates in this crate use (see `avg.rs`, `stddev.rs`).
+ Ok((0..self.pivot_values.len())
+ .map(|i| {
+ Arc::new(Field::new(
+ format_state_name(args.name, &i.to_string()),
+ self.value_type.clone(),
+ true,
+ ))
+ })
+ .collect())
+ }
+
+ fn accumulator(&self, _acc_args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
+ Ok(Box::new(PivotFirstAccumulator {
+ groups: self.groups_accumulator()?,
+ }))
+ }
+
+ fn groups_accumulator_supported(&self, _args: AccumulatorArgs) -> bool {
+ // The ungrouped accumulator is built on the grouped one, so it cannot stand in for a
+ // value type that the grouped one does not store.
+ true
+ }
+
+ fn create_groups_accumulator(
+ &self,
+ _args: AccumulatorArgs,
+ ) -> Result<Box<dyn GroupsAccumulator>> {
+ Ok(self.groups_accumulator()?)
+ }
+}
+
+impl SparkPivotFirst {
+ /// The grouped accumulator, for any value type in Spark's `PivotFirst.supportsDataType`.
+ fn groups_accumulator(&self) -> Result<Box<dyn PivotFirstGroups>> {
+ let index = Arc::clone(&self.pivot_index);
+ let num_slots = self.pivot_values.len();
+ let value_type = self.value_type.clone();
+ Ok(match &self.value_type {
+ // Booleans are stored as 0/1 bytes, so that one primitive implementation covers
+ // every value type.
+ DataType::Boolean | DataType::Int8 => Box::new(
+ PivotFirstGroupsAccumulator::<Int8Type>::new(value_type, index, num_slots),
+ ),
+ DataType::Int16 => Box::new(PivotFirstGroupsAccumulator::<Int16Type>::new(
+ value_type, index, num_slots,
+ )),
+ DataType::Int32 => Box::new(PivotFirstGroupsAccumulator::<Int32Type>::new(
+ value_type, index, num_slots,
+ )),
+ DataType::Int64 => Box::new(PivotFirstGroupsAccumulator::<Int64Type>::new(
+ value_type, index, num_slots,
+ )),
+ DataType::Float32 => Box::new(PivotFirstGroupsAccumulator::<Float32Type>::new(
+ value_type, index, num_slots,
+ )),
+ DataType::Float64 => Box::new(PivotFirstGroupsAccumulator::<Float64Type>::new(
+ value_type, index, num_slots,
+ )),
+ DataType::Decimal128(_, _) => Box::new(
+ PivotFirstGroupsAccumulator::<Decimal128Type>::new(value_type, index, num_slots),
+ ),
+ other => return internal_err!("PivotFirst does not support value type {other}"),
+ })
+ }
+}
+
+/// Marks a row whose pivot column value is in no slot.
+const NO_SLOT: u32 = u32::MAX;
+
+/// Maps a pivot column value to its output slot the way Spark's `PivotFirst` matches it.
+///
+/// For the pivot column types the serde admits, Spark looks the value up in a Scala
+/// `HashMap[Any, Int]` built from the pivot list. Matching therefore goes through Scala's `==`
+/// on the boxed Catalyst value, which is plain value equality except for floats, where it is
+/// IEEE 754 equality: `-0.0` and `0.0` are one key, and NaN matches nothing, not even a NaN in
+/// the pivot list. A null pivot column value matches a null in the pivot list, which is what
+/// Spark's `pivotIndex.getOrElse(null, -1)` does.
+#[derive(Debug)]
+struct PivotIndex {
+ /// The Arrow type of the pivot values. A pivot column with another layout, such as a
+ /// dictionary-encoded string column, is cast to it before the lookup.
+ key_type: DataType,
+ keys: PivotKeys,
+ null_slot: Option<u32>,
+}
+
+/// The slot of each non-null pivot value.
+#[derive(Debug)]
+enum PivotKeys {
+ /// Slots of `false` and `true`.
+ Boolean([Option<u32>; 2]),
+ /// Integers, dates, timestamps and decimals, keyed on their value as an `i128`.
+ Integer(HashMap<i128, u32>),
+ /// Floats, keyed on the bits of their [`ieee_equality_key`] as an `f64`. A NaN in the pivot
+ /// list has no key, so its slot is never written.
+ Float(HashMap<u64, u32>),
+ Utf8(HashMap<Box<str>, u32>),
+}
+
+impl PivotIndex {
+ fn try_new(pivot_values: &[ScalarValue]) -> Result<Self> {
+ let Some(first) = pivot_values.first() else {
+ return internal_err!("PivotFirst requires at least one pivot value");
+ };
+ let key_type = first.data_type();
+ let mut keys = match &key_type {
+ DataType::Boolean => PivotKeys::Boolean([None, None]),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::Date32
+ | DataType::Timestamp(TimeUnit::Microsecond, _)
+ | DataType::Decimal128(_, _) => PivotKeys::Integer(HashMap::new()),
+ DataType::Float32 | DataType::Float64 => PivotKeys::Float(HashMap::new()),
+ DataType::Utf8 => PivotKeys::Utf8(HashMap::new()),
+ other => return internal_err!("PivotFirst does not support pivot column type {other}"),
+ };
+ if pivot_values.len() >= NO_SLOT as usize {
+ return internal_err!("PivotFirst supports fewer than {NO_SLOT} pivot values");
+ }
+ let mut null_slot = None;
+ for (slot, value) in pivot_values.iter().enumerate() {
+ let slot = slot as u32;
+ // `Some(true)` for a new key, `Some(false)` for a repeated one, `None` for a NaN.
+ let inserted = match (&mut keys, value) {
+ (_, value) if value.is_null() => Some(null_slot.replace(slot).is_none()),
+ (PivotKeys::Boolean(slots), ScalarValue::Boolean(Some(b))) => {
+ Some(slots[*b as usize].replace(slot).is_none())
+ }
+ (PivotKeys::Integer(map), value) => match integer_key(value) {
+ Some(key) => Some(map.insert(key, slot).is_none()),
+ None => return mismatched_pivot_value(value, &key_type),
+ },
+ (PivotKeys::Float(map), ScalarValue::Float32(Some(v))) => {
+ float_key(*v).map(|key| map.insert(key, slot).is_none())
+ }
+ (PivotKeys::Float(map), ScalarValue::Float64(Some(v))) => {
+ float_key(*v).map(|key| map.insert(key, slot).is_none())
+ }
+ (PivotKeys::Utf8(map), ScalarValue::Utf8(Some(s))) => {
+ Some(map.insert(s.as_str().into(), slot).is_none())
+ }
+ (_, value) => return mismatched_pivot_value(value, &key_type),
+ };
+ if inserted == Some(false) {
+ // Spark keeps a repeated value's last position and sizes its buffer from the
+ // distinct values, so the slot layout here would no longer match its schema.
+ return internal_err!(
+ "PivotFirst pivot value {value} is repeated; the serde should have fallen back"
+ );
+ }
+ }
+ Ok(Self {
+ key_type,
+ keys,
+ null_slot,
+ })
+ }
+
+ /// The slot of every row of `pivots`, or [`NO_SLOT`] for a row whose value is not in the
+ /// pivot list.
+ fn slots(&self, pivots: &ArrayRef) -> Result<Vec<u32>> {
+ // A timestamp's value does not depend on its time zone label, so only the unit has to
+ // agree for the column to be read as it is.
+ let same_layout = match (pivots.data_type(), &self.key_type) {
+ (DataType::Timestamp(a, _), DataType::Timestamp(b, _)) => a == b,
+ (a, b) => a == b,
+ };
+ let pivots = if same_layout {
+ Arc::clone(pivots)
+ } else {
+ cast(pivots, &self.key_type)?
+ };
+ let null_slot = self.null_slot.unwrap_or(NO_SLOT);
+ Ok(match &self.keys {
+ PivotKeys::Boolean(slots) => {
+ lookup(pivots.as_boolean().iter(), null_slot, |b| slots[b as usize])
+ }
+ PivotKeys::Integer(map) => match pivots.data_type() {
+ DataType::Int8 => integer_slots::<Int8Type>(&pivots, null_slot, map),
+ DataType::Int16 => integer_slots::<Int16Type>(&pivots, null_slot, map),
+ DataType::Int32 => integer_slots::<Int32Type>(&pivots, null_slot, map),
+ DataType::Int64 => integer_slots::<Int64Type>(&pivots, null_slot, map),
+ DataType::Date32 => integer_slots::<Date32Type>(&pivots, null_slot, map),
+ DataType::Timestamp(_, _) => {
+ integer_slots::<TimestampMicrosecondType>(&pivots, null_slot, map)
+ }
+ DataType::Decimal128(_, _) => {
+ integer_slots::<Decimal128Type>(&pivots, null_slot, map)
+ }
+ other => return internal_err!("PivotFirst cannot read pivot column type {other}"),
+ },
+ PivotKeys::Float(map) => match pivots.data_type() {
+ DataType::Float32 => float_slots::<Float32Type>(&pivots, null_slot, map),
+ DataType::Float64 => float_slots::<Float64Type>(&pivots, null_slot, map),
+ other => return internal_err!("PivotFirst cannot read pivot column type {other}"),
+ },
+ PivotKeys::Utf8(map) => lookup(pivots.as_string::<i32>().iter(), null_slot, |s| {
+ map.get(s).copied()
+ }),
+ })
+ }
+}
+
+fn mismatched_pivot_value<T>(value: &ScalarValue, key_type: &DataType) -> Result<T> {
+ internal_err!("PivotFirst pivot value {value:?} does not have the pivot column type {key_type}")
+}
+
+fn integer_key(value: &ScalarValue) -> Option<i128> {
+ Some(match value {
+ ScalarValue::Int8(Some(v)) => i128::from(*v),
+ ScalarValue::Int16(Some(v)) => i128::from(*v),
+ ScalarValue::Int32(Some(v)) | ScalarValue::Date32(Some(v)) => i128::from(*v),
+ ScalarValue::Int64(Some(v)) | ScalarValue::TimestampMicrosecond(Some(v), _) => {
+ i128::from(*v)
+ }
+ ScalarValue::Decimal128(Some(v), _, _) => *v,
+ _ => return None,
+ })
+}
+
+fn float_key(v: impl Into<f64>) -> Option<u64> {
+ ieee_equality_key(v.into()).map(f64::to_bits)
+}
+
+fn lookup<T>(
+ values: impl Iterator<Item = Option<T>>,
+ null_slot: u32,
+ slot_of: impl Fn(T) -> Option<u32>,
+) -> Vec<u32> {
+ values
+ .map(|v| match v {
+ Some(v) => slot_of(v).unwrap_or(NO_SLOT),
+ None => null_slot,
+ })
+ .collect()
+}
+
+fn integer_slots<T>(pivots: &ArrayRef, null_slot: u32, map: &HashMap<i128, u32>) -> Vec<u32>
+where
+ T: ArrowPrimitiveType,
+ T::Native: Into<i128>,
+{
+ lookup(pivots.as_primitive::<T>().iter(), null_slot, |v| {
+ map.get(&v.into()).copied()
+ })
+}
+
+fn float_slots<T>(pivots: &ArrayRef, null_slot: u32, map: &HashMap<u64, u32>) -> Vec<u32>
+where
+ T: ArrowPrimitiveType,
+ T::Native: Into<f64>,
+{
+ lookup(pivots.as_primitive::<T>().iter(), null_slot, |v| {
+ float_key(v).and_then(|key| map.get(&key).copied())
+ })
+}
+
+/// Grouped `PivotFirst` state. Each group owns `num_slots` cells, stored group-major in one flat
+/// buffer, so that a group's cells are contiguous and `evaluate` hands the buffer to the output
+/// list as it is.
+#[derive(Debug)]
+struct PivotFirstGroupsAccumulator<T: ArrowPrimitiveType> {
+ /// The value type Spark declared. `T` stores it, as `Int8` for `Boolean`.
+ value_type: DataType,
+ pivot_index: Arc<PivotIndex>,
+ num_slots: usize,
+ /// `values[group * num_slots + slot]`. Only the cells set in `is_set` hold a value.
+ values: Vec<T::Native>,
+ /// Which cells have been written. A cell nothing has written to is null, as in Spark.
+ is_set: BooleanBufferBuilder,
+}
+
+impl<T: ArrowPrimitiveType> PivotFirstGroupsAccumulator<T> {
+ fn new(value_type: DataType, pivot_index: Arc<PivotIndex>, num_slots: usize) -> Self {
+ Self {
+ value_type,
+ pivot_index,
+ num_slots,
+ values: Vec::new(),
+ is_set: BooleanBufferBuilder::new(0),
+ }
+ }
+
+ fn resize(&mut self, total_num_groups: usize) {
+ let num_cells = total_num_groups * self.num_slots;
+ if num_cells > self.values.len() {
+ self.values.resize(num_cells, T::Native::default());
+ self.is_set.append_n(num_cells - self.is_set.len(), false);
+ }
+ }
+
+ /// Writes `values[row]` into `cell` for each `(row, cell)` pair, skipping a null value, which
+ /// Spark never writes. A later row overwrites an earlier one, as Spark's `update` and `merge`
+ /// do.
+ fn write(
+ &mut self,
+ values: &ArrayRef,
+ writes: impl Iterator<Item = (usize, usize)>,
+ ) -> Result<()> {
+ let values = if self.value_type == DataType::Boolean {
+ cast(values, &DataType::Int8)?
+ } else {
+ Arc::clone(values)
+ };
+ let Some(values) = values.as_primitive_opt::<T>() else {
+ return internal_err!(
+ "PivotFirst expected {} values, got {}",
+ self.value_type,
+ values.data_type()
+ );
+ };
+ for (row, cell) in writes {
+ if values.is_valid(row) {
+ self.values[cell] = values.value(row);
+ self.is_set.set_bit(cell, true);
+ }
+ }
+ Ok(())
+ }
+
+ /// Removes the cells of the groups that `emit_to` covers, returning their values and which of
+ /// them are set. The remaining groups move to the front.
+ fn take_cells(&mut self, emit_to: EmitTo) -> (Vec<T::Native>, BooleanBuffer) {
+ // `take_needed` copies out whichever part is smaller, so emitting a few groups does not
+ // hand the output the buffer that still holds all the others. It counts cells, and each
+ // group has `num_slots` of them.
+ let cells = match emit_to {
+ EmitTo::All => EmitTo::All,
+ EmitTo::First(n) => EmitTo::First(n * self.num_slots),
+ };
+ let values = cells.take_needed(&mut self.values);
+ let num_cells = values.len();
+ let is_set = self.is_set.finish();
+ self.is_set
+ .append_buffer(&is_set.slice(num_cells, is_set.len() - num_cells));
+ (values, is_set.slice(0, num_cells))
+ }
+
+ /// Gives stored cells the declared value type: the precision and scale of a decimal, or
+ /// booleans in place of their `Int8` storage.
+ fn to_value_type(&self, cells: PrimitiveArray<T>) -> Result<ArrayRef> {
+ if self.value_type == DataType::Boolean {
+ Ok(cast(&cells, &DataType::Boolean)?)
+ } else {
+ Ok(Arc::new(cells.with_data_type(self.value_type.clone())))
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> GroupsAccumulator for PivotFirstGroupsAccumulator<T> {
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> Result<()> {
+ let [pivots, values] = values else {
+ return internal_err!(
+ "PivotFirst expects 2 inputs (pivot, value); got {}",
+ values.len()
+ );
+ };
+ self.resize(total_num_groups);
+ let slots = self.pivot_index.slots(pivots)?;
+ let num_slots = self.num_slots;
+ let writes = group_indices
+ .iter()
+ .zip(slots)
+ .enumerate()
+ .filter(|&(row, (_, slot))| {
+ slot != NO_SLOT && opt_filter.is_none_or(|f| f.is_valid(row) && f.value(row))
+ })
+ .map(|(row, (&group, slot))| (row, group * num_slots + slot as usize));
+ self.write(values, writes)
+ }
+
+ fn merge_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ total_num_groups: usize,
+ ) -> Result<()> {
+ if values.len() != self.num_slots {
+ return internal_err!(
+ "PivotFirst merge expects {} state columns; got {}",
+ self.num_slots,
+ values.len()
+ );
+ }
+ self.resize(total_num_groups);
+ // Each column is one slot, and any non-null cell overwrites the group's slot (matches
+ // Spark's `PivotFirst.merge`). A group's state holds only the pivot values its rows had,
+ // so the columns are mostly null, and only their valid rows are visited.
+ let num_slots = self.num_slots;
+ for (slot, column) in values.iter().enumerate() {
+ let cell = |row: usize| (row, group_indices[row] * num_slots + slot);
+ match column.logical_nulls() {
+ Some(nulls) => self.write(column, nulls.valid_indices().map(cell))?,
+ None => self.write(column, (0..column.len()).map(cell))?,
+ }
+ }
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> Result<ArrayRef> {
+ let (values, is_set) = self.take_cells(emit_to);
+ let num_groups = values.len() / self.num_slots;
+ let cells = PrimitiveArray::new(values.into(), Some(NullBuffer::new(is_set)));
+ let offsets =
+ OffsetBuffer::<i32>::from_lengths(std::iter::repeat_n(self.num_slots, num_groups));
+ let field = Arc::new(Field::new_list_field(self.value_type.clone(), true));
+ Ok(Arc::new(ListArray::try_new(
+ field,
+ offsets,
+ self.to_value_type(cells)?,
+ None,
+ )?))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> Result<Vec<ArrayRef>> {
+ let (values, is_set) = self.take_cells(emit_to);
+ let num_slots = self.num_slots;
+ let num_groups = values.len() / num_slots;
+ (0..num_slots)
+ .map(|slot| {
+ let column: ScalarBuffer<T::Native> = (0..num_groups)
+ .map(|group| values[group * num_slots + slot])
+ .collect();
+ let nulls = BooleanBuffer::collect_bool(num_groups, |group| {
+ is_set.value(group * num_slots + slot)
+ });
+ self.to_value_type(PrimitiveArray::new(column, Some(NullBuffer::new(nulls))))
+ })
+ .collect()
+ }
+
+ fn convert_to_state(
+ &self,
+ _values: &[ArrayRef],
+ _opt_filter: Option<&BooleanArray>,
+ ) -> Result<Vec<ArrayRef>> {
+ not_impl_err!("Input batch conversion to state not implemented")
+ }
+
+ fn size(&self) -> usize {
+ size_of_val(self)
+ + self.values.capacity() * size_of::<T::Native>()
+ + self.is_set.capacity() / 8
+ }
+}
+
+/// A [`PivotFirstGroupsAccumulator`] of any value type, as [`PivotFirstAccumulator`] uses it.
+trait PivotFirstGroups: GroupsAccumulator + Sync + Debug {
+ /// The slots of `group` in the value type, left in place. The slots of a group that no row
+ /// has reached are null.
+ fn group_slots(&self, group: usize) -> Result<ArrayRef>;
+}
+
+impl<T: ArrowPrimitiveType + Debug> PivotFirstGroups for PivotFirstGroupsAccumulator<T> {
+ fn group_slots(&self, group: usize) -> Result<ArrayRef> {
+ let first = group * self.num_slots;
+ let cells = first..first + self.num_slots;
+ if cells.end > self.values.len() {
+ return Ok(new_null_array(&self.value_type, self.num_slots));
+ }
+ let is_set =
+ BooleanBuffer::collect_bool(self.num_slots, |slot| self.is_set.get_bit(first + slot));
+ let values = PrimitiveArray::new(
+ self.values[cells].to_vec().into(),
+ Some(NullBuffer::new(is_set)),
+ );
+ self.to_value_type(values)
+ }
+}
+
+/// Ungrouped `PivotFirst`: a [`PivotFirstGroupsAccumulator`] whose rows all belong to one group.
+#[derive(Debug)]
+struct PivotFirstAccumulator {
+ groups: Box<dyn PivotFirstGroups>,
+}
+
+impl Accumulator for PivotFirstAccumulator {
+ fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
+ let num_rows = values.first().map_or(0, |v| v.len());
+ self.groups
+ .update_batch(values, &vec![0; num_rows], None, 1)
+ }
+
+ fn evaluate(&mut self) -> Result<ScalarValue> {
+ // Reads the slots rather than emitting them, because a window aggregate can evaluate
+ // the same accumulator more than once.
+ let slots = self.groups.group_slots(0)?;
+ Ok(SingleRowListArrayBuilder::new(slots).build_list_scalar())
+ }
+
+ fn size(&self) -> usize {
+ size_of_val(self) + self.groups.size()
+ }
+
+ fn state(&mut self) -> Result<Vec<ScalarValue>> {
+ // One scalar per slot, matching `state_fields`.
+ let slots = self.groups.group_slots(0)?;
+ (0..slots.len())
+ .map(|slot| ScalarValue::try_from_array(&slots, slot))
+ .collect()
+ }
+
+ fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
+ let num_rows = states.first().map_or(0, |s| s.len());
+ self.groups.merge_batch(states, &vec![0; num_rows], 1)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow::array::{
+ Date32Array, Decimal128Array, DictionaryArray, Float32Array, Float64Array, Int32Array,
+ Int64Array, Int8Array, StringArray, StringViewArray, TimestampMicrosecondArray,
+ };
+ use arrow::datatypes::Int32Type as DictKey;
+ use arrow::datatypes::Schema;
+
+ fn s(v: &str) -> ScalarValue {
+ ScalarValue::Utf8(Some(v.to_string()))
+ }
+ fn f64s(values: &[f64]) -> Vec<ScalarValue> {
+ values
+ .iter()
+ .map(|v| ScalarValue::Float64(Some(*v)))
+ .collect()
+ }
+
+ fn index(pivot_values: &[ScalarValue]) -> Arc<PivotIndex> {
+ Arc::new(PivotIndex::try_new(pivot_values).unwrap())
+ }
+
+ fn slots(pivot_values: &[ScalarValue], pivots: ArrayRef) -> Vec<u32> {
+ index(pivot_values).slots(&pivots).unwrap()
+ }
+
+ /// `SparkPivotFirst` ignores the accumulator arguments, so any well-formed set will do.
+ fn args(schema: &Schema) -> AccumulatorArgs<'_> {
+ AccumulatorArgs {
+ return_field: Arc::new(Field::new("pivot_first", DataType::Null, true)),
+ schema,
+ expr_fields: &[],
+ ignore_nulls: false,
+ order_bys: &[],
+ is_reversed: false,
+ name: "pivot_first",
+ is_distinct: false,
+ exprs: &[],
+ }
+ }
+
+ fn grouped_acc(
+ value_type: DataType,
+ pivot_values: &[ScalarValue],
+ ) -> Box<dyn GroupsAccumulator> {
+ SparkPivotFirst::try_new(value_type, pivot_values.to_vec())
+ .unwrap()
+ .create_groups_accumulator(args(&Schema::empty()))
+ .unwrap()
+ }
+
+ fn ungrouped_acc(value_type: DataType, pivot_values: &[ScalarValue]) -> Box<dyn Accumulator> {
+ SparkPivotFirst::try_new(value_type, pivot_values.to_vec())
+ .unwrap()
+ .accumulator(args(&Schema::empty()))
+ .unwrap()
+ }
+
+ /// The lists `evaluate` returned, one `Vec` of slot values per group.
+ fn int32_lists(lists: &ArrayRef) -> Vec<Vec<Option<i32>>> {
+ let lists = lists.as_list::<i32>();
+ (0..lists.len())
+ .map(|i| {
+ let list = lists.value(i);
+ list.as_primitive::<Int32Type>().iter().collect()
+ })
+ .collect()
+ }
+
+ /// The list an ungrouped accumulator's `evaluate` returned.
+ fn int32_list(value: ScalarValue) -> Vec<Option<i32>> {
+ let ScalarValue::List(list) = value else {
+ panic!("expected a list, got {value:?}");
+ };
+ let list: ArrayRef = list;
+ int32_lists(&list).remove(0)
+ }
+
+ #[test]
+ fn index_maps_values_to_slots() {
+ let pivots: ArrayRef = Arc::new(StringArray::from(vec![
+ Some("c"),
+ Some("z"),
+ None,
+ Some("a"),
+ ]));
+ // `z` is not in the list, and nor is null, so neither row has a slot.
+ assert_eq!(
+ slots(&[s("a"), s("b"), s("c")], pivots),
+ vec![2, NO_SLOT, NO_SLOT, 0]
+ );
+ }
+
+ #[test]
+ fn null_pivot_value_matches_null_rows() {
+ let pivots: ArrayRef = Arc::new(StringArray::from(vec![None, Some("a")]));
+ let pivot_values = [s("a"), ScalarValue::Utf8(None)];
+ assert_eq!(slots(&pivot_values, pivots), vec![1, 0]);
+ }
+
+ #[test]
+ fn signed_zero_pivot_values_are_one_key() {
+ // Spark keys the pivot list on the boxed Catalyst value, where `-0.0 == 0.0`, so a
+ // `-0.0` pivot value matches rows carrying either sign of zero, and the other way round.
+ let zeros: ArrayRef = Arc::new(Float64Array::from(vec![0.0, -0.0, 1.0]));
+ assert_eq!(
+ slots(&f64s(&[-0.0, 1.0]), Arc::clone(&zeros)),
+ vec![0, 0, 1]
+ );
+ assert_eq!(slots(&f64s(&[0.0, 1.0]), zeros), vec![0, 0, 1]);
+ }
+
+ #[test]
+ fn nan_pivot_value_matches_nothing() {
+ // Scala's `==` on Double is IEEE, so `pivotIndex.getOrElse(NaN, -1)` never hits even
+ // when NaN is in the pivot list, and so does a NaN with another payload.
+ let nans: ArrayRef = Arc::new(Float64Array::from(vec![
+ f64::NAN,
+ f64::from_bits(0xfff8_0000_0000_0000),
+ 2.0,
+ ]));
+ assert_eq!(
+ slots(&f64s(&[f64::NAN, 2.0]), nans),
+ vec![NO_SLOT, NO_SLOT, 1]
+ );
+ }
+
+ #[test]
+ fn float32_signed_zero_and_nan_follow_the_same_rules() {
+ let pivot_values = [
+ ScalarValue::Float32(Some(-0.0)),
+ ScalarValue::Float32(Some(f32::NAN)),
+ ];
+ let pivots: ArrayRef = Arc::new(Float32Array::from(vec![0.0, f32::NAN]));
+ assert_eq!(slots(&pivot_values, pivots), vec![0, NO_SLOT]);
+ }
+
+ #[test]
+ fn integer_like_pivot_columns() {
+ let pivots: ArrayRef = Arc::new(Int8Array::from(vec![Some(3), Some(4), None]));
+ let pivot_values = [ScalarValue::Int8(Some(4)), ScalarValue::Int8(Some(3))];
+ assert_eq!(slots(&pivot_values, pivots), vec![1, 0, NO_SLOT]);
+
+ let pivots: ArrayRef = Arc::new(Date32Array::from(vec![19723, 19724]));
+ let pivot_values = [ScalarValue::Date32(Some(19724))];
+ assert_eq!(slots(&pivot_values, pivots), vec![NO_SLOT, 0]);
+
+ let pivots: ArrayRef = Arc::new(
+ Decimal128Array::from(vec![150, 225])
+ .with_precision_and_scale(10, 2)
+ .unwrap(),
+ );
+ let pivot_values = [ScalarValue::Decimal128(Some(225), 10, 2)];
+ assert_eq!(slots(&pivot_values, pivots), vec![NO_SLOT, 0]);
+
+ let pivots: ArrayRef = Arc::new(BooleanArray::from(vec![Some(true), None, Some(false)]));
+ let pivot_values = [
+ ScalarValue::Boolean(Some(false)),
+ ScalarValue::Boolean(None),
+ ];
+ assert_eq!(slots(&pivot_values, pivots), vec![NO_SLOT, 1, 0]);
+ }
+
+ #[test]
+ fn timestamp_time_zone_label_does_not_matter() {
+ // Comet labels TimestampType values UTC, but the value is the instant either way.
+ let pivots: ArrayRef =
+ Arc::new(TimestampMicrosecondArray::from(vec![10, 20]).with_timezone("Asia/Tokyo"));
+ let pivot_values = [ScalarValue::TimestampMicrosecond(
+ Some(20),
+ Some("UTC".into()),
+ )];
+ assert_eq!(slots(&pivot_values, pivots), vec![NO_SLOT, 0]);
+ }
+
+ #[test]
+ fn string_column_in_another_layout_is_cast() {
+ let pivot_values = [s("a"), s("b")];
+ let dictionary: DictionaryArray<DictKey> = vec!["b", "x", "a"].into_iter().collect();
+ assert_eq!(
+ slots(&pivot_values, Arc::new(dictionary)),
+ vec![1, NO_SLOT, 0]
+ );
+ let view: ArrayRef = Arc::new(StringViewArray::from(vec!["a", "b"]));
+ assert_eq!(slots(&pivot_values, view), vec![0, 1]);
+ }
+
+ #[test]
+ fn repeated_pivot_values_are_rejected() {
+ // Spark collapses these into fewer buffer slots, so the serde falls back for them. Two
+ // NaNs are distinct keys for Spark, so they are allowed.
+ assert!(PivotIndex::try_new(&[s("a"), s("b"), s("a")]).is_err());
+ assert!(PivotIndex::try_new(&f64s(&[0.0, -0.0])).is_err());
+ assert!(PivotIndex::try_new(&[ScalarValue::Utf8(None), ScalarValue::Utf8(None)]).is_err());
+ assert!(PivotIndex::try_new(&f64s(&[f64::NAN, f64::NAN])).is_ok());
+ assert!(PivotIndex::try_new(&[]).is_err());
+ }
+
+ #[test]
+ fn unmatchable_pivot_value_keeps_its_state_slot() {
+ // A NaN entry is left out of the index, but it still has a slot, or `state()` would
+ // return fewer columns than `state_fields` declares and the shuffle exchange would
+ // reject the batch.
+ let mut grouped = grouped_acc(DataType::Int32, &f64s(&[f64::NAN, 0.0]));
+ let pivots: ArrayRef = Arc::new(Float64Array::from(vec![f64::NAN, -0.0]));
+ let values: ArrayRef = Arc::new(Int32Array::from(vec![10, 20]));
+ grouped
+ .update_batch(&[pivots, values], &[0, 0], None, 1)
+ .unwrap();
+ let state = grouped.state(EmitTo::All).unwrap();
+ assert_eq!(state.len(), 2);
+ assert!(state[0].is_null(0));
+ assert_eq!(state[1].as_primitive::<Int32Type>().value(0), 20);
+
+ let mut ungrouped = ungrouped_acc(DataType::Int32, &f64s(&[f64::NAN, 0.0]));
+ assert_eq!(ungrouped.state().unwrap().len(), 2);
+ }
+
+ #[test]
+ fn grouped_update_writes_each_group_slot() {
+ let mut acc = grouped_acc(DataType::Int32, &[s("a"), s("b"), s("c")]);
+ let pivots: ArrayRef = Arc::new(StringArray::from(vec!["a", "c", "b", "z", "a", "a"]));
+ let values: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(1),
+ Some(3),
+ Some(2),
+ Some(99),
+ Some(10),
+ None,
+ ]));
+ // Group 1 gets 'a' twice: the later 10 wins, and the null after it changes nothing.
+ acc.update_batch(&[pivots, values], &[0, 0, 1, 1, 1, 1], None, 2)
+ .unwrap();
+ assert_eq!(
+ int32_lists(&acc.evaluate(EmitTo::All).unwrap()),
+ vec![vec![Some(1), None, Some(3)], vec![Some(10), Some(2), None]]
+ );
+ }
+
+ #[test]
+ fn grouped_update_honors_filter() {
+ let mut acc = grouped_acc(DataType::Int32, &[s("a"), s("b")]);
+ let pivots: ArrayRef = Arc::new(StringArray::from(vec!["a", "b"]));
+ let values: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
+ let filter = BooleanArray::from(vec![Some(false), None]);
+ acc.update_batch(&[pivots, values], &[0, 0], Some(&filter), 1)
+ .unwrap();
+ assert_eq!(
+ int32_lists(&acc.evaluate(EmitTo::All).unwrap()),
+ vec![vec![None, None]]
+ );
+ }
+
+ #[test]
+ fn grouped_state_merges_across_partials() {
+ let pivot_values = [s("a"), s("b"), s("c")];
+ let mut left = grouped_acc(DataType::Int32, &pivot_values);
+ let mut right = grouped_acc(DataType::Int32, &pivot_values);
+ left.update_batch(
+ &[
+ Arc::new(StringArray::from(vec!["a", "b"])),
+ Arc::new(Int32Array::from(vec![1, 2])),
+ ],
+ &[0, 1],
+ None,
+ 2,
+ )
+ .unwrap();
+ right
+ .update_batch(
+ &[
+ Arc::new(StringArray::from(vec!["c", "a"])),
+ Arc::new(Int32Array::from(vec![3, 4])),
+ ],
+ &[0, 0],
+ None,
+ 1,
+ )
+ .unwrap();
+
+ let left_state = left.state(EmitTo::All).unwrap();
+ assert_eq!(left_state.len(), 3);
+ assert!(left_state.iter().all(|c| c.data_type() == &DataType::Int32));
+
+ // Right's group 0 is the final group 1, and its 'a' overwrites nothing there, while
+ // left's group 0 keeps its 'a'.
+ let mut merged = grouped_acc(DataType::Int32, &pivot_values);
+ merged.merge_batch(&left_state, &[0, 1], 2).unwrap();
+ merged
+ .merge_batch(&right.state(EmitTo::All).unwrap(), &[1], 2)
+ .unwrap();
+ assert_eq!(
+ int32_lists(&merged.evaluate(EmitTo::All).unwrap()),
+ vec![vec![Some(1), None, None], vec![Some(4), Some(2), Some(3)]]
+ );
+ }
+
+ #[test]
+ fn grouped_merge_reads_state_without_a_null_buffer() {
+ // A shuffle reader can hand over a state column with no null buffer at all.
+ let mut acc = grouped_acc(DataType::Int32, &[s("a"), s("b")]);
+ let a: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
+ let b: ArrayRef = Arc::new(Int32Array::from(vec![Some(3), None]));
+ assert!(a.nulls().is_none());
+ acc.merge_batch(&[a, b], &[1, 0], 2).unwrap();
+ assert_eq!(
+ int32_lists(&acc.evaluate(EmitTo::All).unwrap()),
+ vec![vec![Some(2), None], vec![Some(1), Some(3)]]
+ );
+ }
+
+ #[test]
+ fn grouped_emit_first_keeps_the_remaining_groups() {
+ let mut acc = grouped_acc(DataType::Int32, &[s("a"), s("b")]);
+ acc.update_batch(
+ &[
+ Arc::new(StringArray::from(vec!["a", "b", "a"])),
+ Arc::new(Int32Array::from(vec![1, 2, 3])),
+ ],
+ &[0, 1, 2],
+ None,
+ 3,
+ )
+ .unwrap();
+ assert_eq!(
+ int32_lists(&acc.evaluate(EmitTo::First(2)).unwrap()),
+ vec![vec![Some(1), None], vec![None, Some(2)]]
+ );
+ // The old group 2 is now group 0.
+ acc.update_batch(
+ &[
+ Arc::new(StringArray::from(vec!["b"])),
+ Arc::new(Int32Array::from(vec![4])),
+ ],
+ &[0],
+ None,
+ 1,
+ )
+ .unwrap();
+ let state = acc.state(EmitTo::All).unwrap();
+ assert_eq!(state[0].as_primitive::<Int32Type>().value(0), 3);
+ assert_eq!(state[1].as_primitive::<Int32Type>().value(0), 4);
+ assert_eq!(acc.evaluate(EmitTo::All).unwrap().len(), 0);
+ }
+
+ #[test]
+ fn grouped_emit_first_copies_out_only_the_emitted_groups() {
+ // The output of a partial emit must not hold the buffer that keeps the other groups.
+ let num_groups = 1000;
+ let mut acc = grouped_acc(DataType::Int32, &[s("a"), s("b")]);
+ acc.update_batch(
+ &[
+ Arc::new(StringArray::from(vec!["b"; num_groups])),
+ Arc::new(Int32Array::from_iter_values(0..num_groups as i32)),
+ ],
+ &(0..num_groups).collect::<Vec<_>>(),
+ None,
+ num_groups,
+ )
+ .unwrap();
+ let lists = acc.evaluate(EmitTo::First(10)).unwrap();
+ let cells = lists.as_list::<i32>().values().as_primitive::<Int32Type>();
+ assert_eq!(cells.len(), 20);
+ assert_eq!(
+ cells.values().inner().capacity(),
+ cells.len() * size_of::<i32>()
+ );
+ assert_eq!(
+ int32_lists(&acc.evaluate(EmitTo::All).unwrap())[0],
+ vec![None, Some(10)]
+ );
+ }
+
+ #[test]
+ fn grouped_boolean_values_round_trip() {
+ let pivot_values = [s("a"), s("b"), s("c")];
+ let mut partial = grouped_acc(DataType::Boolean, &pivot_values);
+ partial
+ .update_batch(
+ &[
+ Arc::new(StringArray::from(vec!["a", "b"])),
+ Arc::new(BooleanArray::from(vec![true, false])),
+ ],
+ &[0, 0],
+ None,
+ 1,
+ )
+ .unwrap();
+ let state = partial.state(EmitTo::All).unwrap();
+ assert!(state.iter().all(|c| c.data_type() == &DataType::Boolean));
+
+ let mut fin = grouped_acc(DataType::Boolean, &pivot_values);
+ fin.merge_batch(&state, &[0], 1).unwrap();
+ let lists = fin.evaluate(EmitTo::All).unwrap();
+ assert_eq!(
+ lists.data_type(),
+ &DataType::List(Arc::new(Field::new_list_field(DataType::Boolean, true)))
+ );
+ let list = lists.as_list::<i32>().value(0);
+ assert_eq!(
+ list.as_boolean().iter().collect::<Vec<_>>(),
+ vec![Some(true), Some(false), None]
+ );
+ }
+
+ #[test]
+ fn grouped_decimal_values_keep_precision_and_scale() {
+ let value_type = DataType::Decimal128(10, 2);
+ let mut acc = grouped_acc(value_type.clone(), &[s("a")]);
+ let values: ArrayRef = Arc::new(
+ Decimal128Array::from(vec![1234])
+ .with_precision_and_scale(10, 2)
+ .unwrap(),
+ );
+ acc.update_batch(
+ &[Arc::new(StringArray::from(vec!["a"])), values],
+ &[0],
+ None,
+ 1,
+ )
+ .unwrap();
+ let lists = acc.evaluate(EmitTo::All).unwrap();
+ let list = lists.as_list::<i32>().value(0);
+ assert_eq!(list.data_type(), &value_type);
+ assert_eq!(list.as_primitive::<Decimal128Type>().value(0), 1234);
+ }
+
+ #[test]
+ fn ungrouped_accumulator_is_one_group() {
+ let pivot_values = [
+ ScalarValue::Int64(Some(1)),
+ ScalarValue::Int64(Some(2)),
+ ScalarValue::Int64(None),
+ ];
+ let mut partial = ungrouped_acc(DataType::Int32, &pivot_values);
+ partial
+ .update_batch(&[
+ Arc::new(Int64Array::from(vec![
+ Some(2),
+ None,
+ Some(7),
+ Some(1),
+ Some(2),
+ ])),
+ Arc::new(Int32Array::from(vec![
+ Some(20),
+ Some(30),
+ Some(70),
+ None,
+ Some(21),
+ ])),
+ ])
+ .unwrap();
+ // 7 is in no slot, the null value leaves slot 0 empty, the later 21 wins slot 1, and
+ // the null pivot value collects the null row. Evaluating leaves the slots in place, so
+ // a second call agrees with the first.
+ for _ in 0..2 {
+ assert_eq!(
+ int32_list(partial.evaluate().unwrap()),
+ vec![None, Some(21), Some(30)]
+ );
+ }
+ let state = partial.state().unwrap();
+ assert_eq!(
+ state,
+ [None, Some(21), Some(30)].map(ScalarValue::Int32).to_vec()
+ );
+
+ // A final accumulator merges one row per partial state, and a later non-null slot wins.
+ let other = [Some(1), None, Some(31)].map(ScalarValue::Int32);
+ let columns: Vec<ArrayRef> = state
+ .into_iter()
+ .zip(other)
+ .map(|(a, b)| ScalarValue::iter_to_array([a, b]).unwrap())
+ .collect();
+ let mut fin = ungrouped_acc(DataType::Int32, &pivot_values);
+ fin.merge_batch(&columns).unwrap();
+ assert_eq!(
+ int32_list(fin.evaluate().unwrap()),
+ vec![Some(1), Some(21), Some(31)]
+ );
+ }
+
+ #[test]
+ fn ungrouped_accumulator_without_rows_is_all_null() {
+ // A pivot with no grouping column returns one row of nulls over an empty input.
+ let value_type = DataType::Decimal128(10, 2);
+ let mut acc = ungrouped_acc(value_type.clone(), &[s("a"), s("b")]);
+ assert_eq!(
+ acc.state().unwrap(),
+ vec![ScalarValue::Decimal128(None, 10, 2); 2]
+ );
+ let ScalarValue::List(list) = acc.evaluate().unwrap() else {
+ panic!("expected a list");
+ };
+ let slots = list.value(0);
+ assert_eq!(slots.data_type(), &value_type);
+ assert_eq!((slots.len(), slots.null_count()), (2, 2));
+ }
+}
diff --git a/native/spark-expr/src/float_semantics/mod.rs b/native/spark-expr/src/float_semantics/mod.rs
index 997f8b2464..fc6cdd633f 100644
--- a/native/spark-expr/src/float_semantics/mod.rs
+++ b/native/spark-expr/src/float_semantics/mod.rs
@@ -35,6 +35,9 @@
//! [`compare_floats_java`].
//! - `Murmur3Hash` and `XxHash64`: `-0.0` hashes as `0.0`, and NaN as the canonical NaN. See
//! [`hash_input`].
+//! - Scala's `==` on boxed numbers (`BoxesRunTime.equals`), which keys of a Scala `HashMap[Any, _]`
+//! such as `PivotFirst`'s pivot value index match on: IEEE 754 equality, so `-0.0` equals `0.0`
+//! and NaN equals nothing, not even another NaN. See [`ieee_equality_key`].
//!
//! A native expression must follow the rule of the Spark function it replaces, so build it from
//! these helpers rather than a local copy. Where an Arrow kernel sorts, row-encodes, hashes or
@@ -86,6 +89,18 @@ pub fn canonicalize_nan<T: Float>(v: T) -> T {
}
}
+/// The key that a Scala `HashMap[Any, _]` matches a boxed float on. Scala's `==` on boxed numbers
+/// is IEEE 754 equality, so `-0.0` and `0.0` share a key, and NaN has none: it equals nothing, not
+/// even another NaN.
+#[inline]
+pub fn ieee_equality_key<T: Float>(v: T) -> Option<T> {
+ if v.is_nan() {
+ None
+ } else {
+ Some(normalize_float(v))
+ }
+}
+
/// Spark's SQL ordering, `SQLOrderingUtil.compareDoubles`: `-0.0` equals `0.0`, all NaNs are
/// equal, and NaN is greater than every other value, including positive infinity.
#[inline]
@@ -209,6 +224,23 @@ mod tests {
}
}
+ #[test]
+ fn ieee_equality_keys() {
+ let key = |v: f64| ieee_equality_key(v).map(f64::to_bits);
+ assert_eq!(key(-0.0), Some(0.0f64.to_bits()));
+ assert_eq!(key(0.0), Some(0.0f64.to_bits()));
+ assert_eq!(key(-1.5), Some((-1.5f64).to_bits()));
+ assert_eq!(key(f64::INFINITY), Some(f64::INFINITY.to_bits()));
+ for nan in [f64::NAN, NEGATIVE_NAN, PAYLOAD_NAN] {
+ assert_eq!(key(nan), None);
+ }
+ assert_eq!(
+ ieee_equality_key(-0.0f32).map(f32::to_bits),
+ Some(0.0f32.to_bits())
+ );
+ assert_eq!(ieee_equality_key(NEGATIVE_NAN_F32), None);
+ }
+
#[test]
fn comparisons_match_compare_doubles() {
// Ascending under `compareDoubles`, with values that compare equal grouped together.
diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
index 3f3881e6d4..82e5efbe47 100644
--- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
@@ -438,6 +438,7 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
classOf[MinBy] -> CometMinBy,
classOf[Mode] -> CometMode,
classOf[Percentile] -> CometPercentile,
+ classOf[PivotFirst] -> CometPivotFirst,
classOf[RegrIntercept] -> CometRegrIntercept,
classOf[RegrR2] -> CometRegrR2,
classOf[RegrReplacement] -> CometRegrReplacement,
diff --git a/spark/src/main/scala/org/apache/comet/serde/aggregates.scala b/spark/src/main/scala/org/apache/comet/serde/aggregates.scala
index fde4788e9d..7ea6863bc9 100644
--- a/spark/src/main/scala/org/apache/comet/serde/aggregates.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/aggregates.scala
@@ -22,7 +22,7 @@ package org.apache.comet.serde
import scala.jdk.CollectionConverters._
import org.apache.spark.sql.catalyst.expressions.{Attribute, Cast, Divide, Expression, Literal}
-import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, AggregateFunction, ApproximatePercentile, Average, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAggregate, CentralMomentAgg, CollectList, CollectSet, Complete, Corr, Count, Covariance, CovPopulation, CovSample, DeclarativeAggregate, First, HyperLogLogPlusPlus, Last, Max, MaxBy, MaxMinBy, Min, MinBy, Mode, Partial, Percentile, RegrIntercept, RegrR2, RegrReplacement, RegrSlope, RegrSXY, StddevPop, StddevSamp, Su [...]
+import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, AggregateFunction, ApproximatePercentile, Average, BitAndAgg, BitOrAgg, BitXorAgg, BloomFilterAggregate, CentralMomentAgg, CollectList, CollectSet, Complete, Corr, Count, Covariance, CovPopulation, CovSample, DeclarativeAggregate, First, HyperLogLogPlusPlus, Last, Max, MaxBy, MaxMinBy, Min, MinBy, Mode, Partial, Percentile, PivotFirst, RegrIntercept, RegrR2, RegrReplacement, RegrSlope, RegrSXY, StddevPop, St [...]
import org.apache.spark.sql.catalyst.util.ArrayData
import org.apache.spark.sql.comet.CometExecUtils
import org.apache.spark.sql.internal.SQLConf
@@ -1483,6 +1483,105 @@ object CometMode extends CometAggregateExpressionSerde[Mode] with CometTypeShim
}
}
+object CometPivotFirst extends CometAggregateExpressionSerde[PivotFirst] with CometTypeShim {
+
+ private def unsupportedValueTypeReason(dt: DataType): String =
+ s"Unsupported value data type: $dt"
+
+ private def unsupportedPivotTypeReason(dt: DataType): String =
+ s"Unsupported pivot column data type: $dt"
+
+ private val emptyPivotValuesReason = "Pivot values list is empty"
+
+ private val duplicatePivotValuesReason = "Pivot values list contains duplicates"
+
+ override def getUnsupportedReasons(): Seq[String] = Seq(
+ "Value data type other than Boolean, Byte, Short, Int, Long, Float, Double or Decimal",
+ "Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, " +
+ "Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which " +
+ "includes the struct that a pivot on more than one column builds",
+ emptyPivotValuesReason,
+ s"$duplicatePivotValuesReason, including `0.0` next to `-0.0`")
+
+ // The value types the native aggregate stores. They are the types of Spark's
+ // `PivotFirst.supportsDataType` on every version through 4.2. The check does not call that
+ // method, because a type a later version adds would then reach native code that has no storage
+ // for it, and the query would fail rather than fall back. Running a new type natively means
+ // adding it here, to `getUnsupportedReasons`, and to `SparkPivotFirst::groups_accumulator`.
+ private def isSupportedValueType(dt: DataType): Boolean = dt match {
+ case BooleanType | ByteType | ShortType | IntegerType | LongType => true
+ case FloatType | DoubleType => true
+ case _: DecimalType => true
+ case _ => false
+ }
+
+ // The pivot column types for which Spark finds a value's slot in a `HashMap` keyed by plain
+ // value equality, which the native lookup reproduces. Spark keys arrays and structs in a
+ // `TreeMap` by their interpreted ordering. It keys binary values by reference identity before
+ // 4.2 and by ordering from 4.2, which orders strings with a non-UTF8_BINARY collation too.
+ private def isSupportedPivotType(dt: DataType): Boolean = dt match {
+ case BooleanType | ByteType | ShortType | IntegerType | LongType => true
+ case FloatType | DoubleType => true
+ case _: DecimalType => true
+ case DateType | TimestampType | TimestampNTZType => true
+ case st: StringType => !isStringCollationType(st)
+ case _ => false
+ }
+
+ override def getSupportLevel(expr: PivotFirst): SupportLevel = {
+ val pivotType = expr.pivotColumn.dataType
+ if (!isSupportedValueType(expr.valueDataType)) {
+ Unsupported(Some(unsupportedValueTypeReason(expr.valueDataType)))
+ } else if (!isSupportedPivotType(pivotType)) {
+ Unsupported(Some(unsupportedPivotTypeReason(pivotType)))
+ } else if (expr.pivotColumnValues.isEmpty) {
+ Unsupported(Some(emptyPivotValuesReason))
+ } else if (expr.indexSize != expr.pivotColumnValues.length) {
+ // Spark sizes the aggregate buffer from its index of the pivot values, which holds a
+ // repeated value once, while the native aggregate keeps one slot per list entry. The
+ // native partial aggregate would emit more columns than the shuffle schema declares.
+ Unsupported(Some(duplicatePivotValuesReason))
+ } else {
+ Compatible()
+ }
+ }
+
+ override def convert(
+ aggExpr: AggregateExpression,
+ expr: PivotFirst,
+ inputs: Seq[Attribute],
+ binding: Boolean,
+ conf: SQLConf): Option[ExprOuterClass.AggExpr] = {
+ val pivotColExpr = exprToProto(expr.pivotColumn, inputs, binding)
+ val valueColExpr = exprToProto(expr.valueColumn, inputs, binding)
+ val valueDt = serializeDataType(expr.valueDataType)
+ val pivotDt = expr.pivotColumn.dataType
+
+ // Spark stores the pivot values as already-evaluated raw Scala values. Rebuild them as
+ // Literal expressions carrying the pivot column data type so the native side receives
+ // the exact bytes it will later compare against. Preserve list order - the position in
+ // this list is the output array index.
+ val pivotValueExprs =
+ expr.pivotColumnValues.map(v => exprToProto(Literal(v, pivotDt), inputs, binding))
+
+ if (pivotColExpr.isDefined && valueColExpr.isDefined && valueDt.isDefined &&
+ pivotValueExprs.forall(_.isDefined)) {
+ val builder = ExprOuterClass.PivotFirst.newBuilder()
+ builder.setPivotColumn(pivotColExpr.get)
+ builder.setValueColumn(valueColExpr.get)
+ builder.setValueDatatype(valueDt.get)
+ pivotValueExprs.foreach(v => builder.addPivotValues(v.get))
+ Some(ExprOuterClass.AggExpr.newBuilder().setPivotFirst(builder).build())
+ } else if (valueDt.isEmpty) {
+ withFallbackReason(aggExpr, unsupportedValueTypeReason(expr.valueDataType))
+ None
+ } else {
+ withFallbackReason(aggExpr, "Child expression or data type not supported")
+ None
+ }
+ }
+}
+
object AggSerde {
import org.apache.spark.sql.types._
diff --git a/spark/src/test/resources/sql-tests/expressions/aggregate/PivotFirst.sql b/spark/src/test/resources/sql-tests/expressions/aggregate/PivotFirst.sql
new file mode 100644
index 0000000000..18c2eb25bc
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/aggregate/PivotFirst.sql
@@ -0,0 +1,571 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- PivotFirst is Spark's internal aggregate for the second phase of the optimized pivot plan,
+-- which the analyzer rule ResolvePivot builds whenever every aggregate output type is in
+-- PivotFirst.supportsDataType (Boolean, Byte, Short, Int, Long, Float, Double, Decimal).
+-- These SQL tests exercise that path via SQL PIVOT clauses, one per supported value type and
+-- per supported pivot column type, plus a handful of edge cases (unmatched pivot values,
+-- all-null groups, null pivot column). Every native query asserts that PivotFirst itself ran
+-- natively, so a plan without it fails rather than passing on Spark's other pivot plan. The
+-- pivot column types and pivot lists whose matching Comet does not reproduce come last and
+-- must fall back.
+
+-- ============================================================
+-- Setup: sales-like table used for most cases
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales (year int, course string, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales VALUES
+ (2012, 'dotNET', 15000),
+ (2012, 'Java', 20000),
+ (2013, 'dotNET', 48000),
+ (2013, 'Java', 30000)
+
+-- ============================================================
+-- Int values: canonical optimized-pivot query.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Long values: sum of a bigint aggregates to bigint.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_bi (year int, course string, earnings bigint) USING parquet
+
+statement
+INSERT INTO pf_sales_bi VALUES
+ (2012, 'dotNET', 15000000000),
+ (2012, 'Java', 20000000000),
+ (2013, 'dotNET', 48000000000),
+ (2013, 'Java', 30000000000)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_bi
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Double values: avg(earnings) is double.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales
+ PIVOT (avg(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Float values: the column is already float, and min keeps the
+-- type where sum would widen it to double.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_f (year int, course string, earnings float) USING parquet
+
+statement
+INSERT INTO pf_sales_f VALUES
+ (2012, 'dotNET', CAST(15000 AS FLOAT)),
+ (2012, 'Java', CAST(20000 AS FLOAT)),
+ (2013, 'dotNET', CAST(48000 AS FLOAT)),
+ (2013, 'Java', CAST(30000 AS FLOAT))
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_f
+ PIVOT (min(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Short and Byte value types (sum promotes them to bigint, so use min
+-- to keep the aggregate output type equal to the input type).
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_small (
+ year int, course string, s smallint, b tinyint) USING parquet
+
+statement
+INSERT INTO pf_sales_small VALUES
+ (2012, 'dotNET', 150, 10),
+ (2012, 'Java', 200, 20),
+ (2013, 'dotNET', 480, 30),
+ (2013, 'Java', 300, 40)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_small
+ PIVOT (min(s)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_small
+ PIVOT (min(b)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Boolean values: bool_or's output type is boolean.
+-- ============================================================
+
+statement
+CREATE TABLE pf_flags (year int, course string, flag boolean) USING parquet
+
+statement
+INSERT INTO pf_flags VALUES
+ (2012, 'dotNET', true),
+ (2012, 'Java', false),
+ (2013, 'dotNET', false),
+ (2013, 'Java', true)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_flags
+ PIVOT (bool_or(flag)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Decimal values.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_dec (year int, course string, earnings decimal(10,2)) USING parquet
+
+statement
+INSERT INTO pf_sales_dec VALUES
+ (2012, 'dotNET', 15000.00),
+ (2012, 'Java', 20000.00),
+ (2013, 'dotNET', 48000.00),
+ (2013, 'Java', 30000.00)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_dec
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Multiple aggregates in a single pivot.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales
+ PIVOT (sum(earnings) AS s, avg(earnings) AS a
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Pivot values that do not appear in the data: the corresponding
+-- output slots must be NULL.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java', 'MissingA', 'MissingB'))
+ ORDER BY year
+
+-- ============================================================
+-- Rows whose pivot column is not in the pivot list must be ignored:
+-- 'Python' below should not contribute to any output slot.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_extra (year int, course string, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_extra VALUES
+ (2012, 'dotNET', 15000),
+ (2012, 'Java', 20000),
+ (2012, 'Python', 12345),
+ (2013, 'dotNET', 48000),
+ (2013, 'Java', 30000),
+ (2013, 'Python', 67890)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_extra
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Group with no matching rows: every output slot is NULL for that
+-- pivot combination. Ensures NULL initialization matches Spark.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_partial (year int, course string, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_partial VALUES
+ (2012, 'dotNET', 15000),
+ (2012, 'Java', 20000),
+ (2013, 'Python', 42000)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_partial
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- NULL value in an aggregated column: the null must not clobber a
+-- previously written slot (Spark PivotFirst.update ignores nulls).
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_nullable (year int, course string, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_nullable VALUES
+ (2012, 'dotNET', 15000),
+ (2012, 'dotNET', NULL),
+ (2012, 'Java', NULL),
+ (2013, 'dotNET', 48000),
+ (2013, 'Java', 30000)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_nullable
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- NULL in the pivot COLUMN. Spark tracks this with a dedicated
+-- test ("pivot with null should not throw NPE") because early
+-- versions of PivotFirst NPE'd on null keys with a TreeMap. A null
+-- row matches no slot unless the pivot list holds NULL too.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_nullpivot (year int, course string, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_nullpivot VALUES
+ (2012, 'dotNET', 15000),
+ (2012, NULL, 9999),
+ (2013, 'Java', 30000),
+ (2013, NULL, 42)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_nullpivot
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', 'Java'))
+ ORDER BY year
+
+-- ============================================================
+-- Non-string pivot column: integer. Exercises the pivot value
+-- serialization path with a numeric literal.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_intpivot (region string, year int, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_intpivot VALUES
+ ('NA', 2012, 15000),
+ ('NA', 2013, 48000),
+ ('EU', 2012, 20000),
+ ('EU', 2013, 30000)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_intpivot
+ PIVOT (sum(earnings)
+ FOR year IN (2012, 2013))
+ ORDER BY region
+
+-- ============================================================
+-- Non-string pivot column: date. Exercises the pivot value
+-- serialization path with a DateType literal.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_datepivot (region string, d date, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_datepivot VALUES
+ ('NA', DATE '2024-01-01', 15000),
+ ('NA', DATE '2024-06-15', 48000),
+ ('EU', DATE '2024-01-01', 20000),
+ ('EU', DATE '2024-06-15', 30000)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_datepivot
+ PIVOT (sum(earnings)
+ FOR d IN (DATE '2024-01-01', DATE '2024-06-15'))
+ ORDER BY region
+
+-- ============================================================
+-- Non-string pivot column: double, including the two values whose
+-- key matching differs between Spark and a plain hash of the native
+-- scalar. Spark keys the pivot list on the boxed Catalyst value, so
+-- -0.0 and 0.0 are the same key while NaN matches nothing at all.
+--
+-- The zero rows are written as -0.0D / 0.0D rather than
+-- CAST(-0.0 AS DOUBLE), because the cast constant-folds to +0.0 and
+-- would make the case vacuous.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_dblpivot (region string, k double, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_dblpivot VALUES
+ ('NA', 0.0D, 15000),
+ ('NA', -0.0D, 48000),
+ ('NA', double('NaN'), 11000),
+ ('NA', 1.5D, 20000),
+ ('EU', 0.0D, 30000),
+ ('EU', double('NaN'), 12000),
+ ('EU', 1.5D, 40000)
+
+-- -0.0 in the pivot list has to collect both signs of zero (63000 / 30000),
+-- and the NaN column has to stay NULL for every group.
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_dblpivot
+ PIVOT (sum(earnings)
+ FOR k IN (-0.0D, double('NaN'), 1.5D))
+ ORDER BY region
+
+-- The mirror: a 0.0 entry in the pivot list must also catch the -0.0 rows.
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_dblpivot
+ PIVOT (sum(earnings)
+ FOR k IN (0.0D, 1.5D))
+ ORDER BY region
+
+-- ============================================================
+-- Non-string pivot column: float. Same two rules.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_fltpivot (region string, k float, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_fltpivot VALUES
+ ('NA', 0.0F, 15000),
+ ('NA', -0.0F, 48000),
+ ('NA', float('NaN'), 11000),
+ ('EU', -0.0F, 30000)
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_fltpivot
+ PIVOT (sum(earnings)
+ FOR k IN (-0.0F, float('NaN')))
+ ORDER BY region
+
+-- ============================================================
+-- NULL in the pivot LIST collects the rows whose pivot column is
+-- NULL, as Spark's HashMap lookup of a null key does.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_nullpivot
+ PIVOT (sum(earnings)
+ FOR course IN ('dotNET', NULL))
+ ORDER BY year
+
+-- ============================================================
+-- Two NaN entries are two distinct keys for Spark (NaN equals
+-- nothing), so the list has no duplicates and stays native, and
+-- both columns stay NULL.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM pf_sales_dblpivot
+ PIVOT (sum(earnings)
+ FOR k IN (double('NaN'), 1.5D, double('NaN')))
+ ORDER BY region
+
+-- ============================================================
+-- The other pivot column types that run natively: boolean,
+-- tinyint, smallint, bigint, decimal, timestamp and timestamp_ntz.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_types (
+ region string, flag boolean, b tinyint, s smallint, l bigint, d decimal(10,2),
+ ts timestamp, ntz timestamp_ntz, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_types VALUES
+ ('NA', true, 1, 100, 10000000000, 1.50,
+ TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP_NTZ '2024-01-01 00:00:00', 15000),
+ ('NA', false, 2, 200, 20000000000, 2.25,
+ TIMESTAMP '2024-06-15 12:30:00', TIMESTAMP_NTZ '2024-06-15 12:30:00', 48000),
+ ('EU', true, 2, 200, 20000000000, 2.25,
+ TIMESTAMP '2024-06-15 12:30:00', TIMESTAMP_NTZ '2024-06-15 12:30:00', 20000),
+ ('EU', NULL, 3, 300, 30000000000, 3.00,
+ TIMESTAMP '2025-01-01 00:00:00', TIMESTAMP_NTZ '2025-01-01 00:00:00', 30000)
+
+query expect_native(pivotfirst)
+SELECT region, `true`, `false` FROM (
+ SELECT region, flag, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings) FOR flag IN (true, false))
+ ORDER BY region
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT region, b, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings) FOR b IN (CAST(1 AS TINYINT), CAST(2 AS TINYINT)))
+ ORDER BY region
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT region, s, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings) FOR s IN (CAST(100 AS SMALLINT), CAST(300 AS SMALLINT)))
+ ORDER BY region
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT region, l, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings) FOR l IN (10000000000, 20000000000))
+ ORDER BY region
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT region, d, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings) FOR d IN (2.25, 3.00))
+ ORDER BY region
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT region, ts, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings)
+ FOR ts IN (TIMESTAMP '2024-01-01 00:00:00', TIMESTAMP '2024-06-15 12:30:00'))
+ ORDER BY region
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT region, ntz, earnings FROM pf_sales_types)
+ PIVOT (sum(earnings)
+ FOR ntz IN (TIMESTAMP_NTZ '2024-06-15 12:30:00', TIMESTAMP_NTZ '2025-01-01 00:00:00'))
+ ORDER BY region
+
+-- ============================================================
+-- No grouping column: the pivoted relation has only the pivot and
+-- value columns, so PivotFirst runs ungrouped.
+-- ============================================================
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT course, earnings FROM pf_sales)
+ PIVOT (sum(earnings) FOR course IN ('dotNET', 'Java'))
+
+-- Over no rows the result is still one row, of nulls. A literal
+-- WHERE false lets the optimizer replace the scan with an empty
+-- local relation, which runs in Spark. These fixtures run without
+-- constant folding, so 1 = 0 keeps the scan and the filter native.
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT course, earnings FROM pf_sales WHERE 1 = 0)
+ PIVOT (sum(earnings) FOR course IN ('dotNET', 'Java'))
+
+-- ============================================================
+-- More groups than one batch holds, each with several partial
+-- states. Repartitioning the input by the pivot column means the
+-- first aggregation needs no shuffle, so each partition's partial
+-- PivotFirst writes the slots of its own pivot values for nearly
+-- every group, and the final aggregation merges those states.
+-- Without the hint, AQE coalesces the first aggregation's shuffle
+-- into one partition, and each group has a single partial state.
+-- Each (g, k) pair has at most one row, so the result is
+-- deterministic.
+-- ============================================================
+
+statement
+CREATE TABLE pf_many_groups (g int, k string, v int) USING parquet
+
+statement
+INSERT INTO pf_many_groups
+SELECT g, k, v FROM (
+ SELECT CAST(id % 20000 AS INT) AS g,
+ CASE id DIV 20000 WHEN 0 THEN 'a' WHEN 1 THEN 'b' ELSE 'c' END AS k,
+ CASE WHEN id % 11 = 0 THEN NULL ELSE CAST(id AS INT) END AS v
+ FROM range(60000))
+WHERE NOT (k = 'b' AND g % 7 = 0)
+
+query expect_native(pivotfirst)
+SELECT * FROM (SELECT /*+ REPARTITION(4, k) */ g, k, v FROM pf_many_groups)
+ PIVOT (sum(v) FOR k IN ('a', 'b', 'c', 'd'))
+
+-- ============================================================
+-- Pivot column types that fall back. Spark compares binary pivot
+-- values by reference before 4.2, so no row matches there, and by
+-- content from 4.2. It matches arrays and structs through their
+-- interpreted ordering. A pivot on two columns pivots on a struct.
+-- ============================================================
+
+statement
+CREATE TABLE pf_sales_complex (
+ region string, k binary, a array<int>, course string, year int, earnings int)
+ USING parquet
+
+statement
+INSERT INTO pf_sales_complex VALUES
+ ('NA', X'61', array(1, 1), 'dotNET', 2012, 15000),
+ ('NA', X'62', array(2, 2), 'Java', 2013, 48000),
+ ('EU', X'61', array(1, 1), 'dotNET', 2013, 20000)
+
+query expect_fallback(Unsupported pivot column data type: BinaryType)
+SELECT * FROM (SELECT region, k, earnings FROM pf_sales_complex)
+ PIVOT (sum(earnings) FOR k IN (X'61', X'62'))
+ ORDER BY region
+
+query expect_fallback(Unsupported pivot column data type: ArrayType)
+SELECT * FROM (SELECT region, a, earnings FROM pf_sales_complex)
+ PIVOT (sum(earnings) FOR a IN (array(1, 1), array(2, 2)))
+ ORDER BY region
+
+query expect_fallback(Unsupported pivot column data type: StructType)
+SELECT * FROM (SELECT region, course, year, earnings FROM pf_sales_complex)
+ PIVOT (sum(earnings) FOR (course, year) IN (('dotNET', 2012), ('Java', 2013)))
+ ORDER BY region
+
+-- ============================================================
+-- Duplicate pivot values fall back. Spark keeps one buffer slot per
+-- distinct value, so its partial aggregate emits fewer columns than
+-- the list has entries, and the projection reads the missing ones
+-- past the end of the array: NULL with ANSI off, an error with it
+-- on. 0.0 and -0.0 are duplicates under Spark's key equality. Only
+-- rows matching the first entry are used, because Spark writes a
+-- repeated value's row past the end of its buffer.
+-- ============================================================
+
+statement
+SET spark.sql.ansi.enabled=false
+
+query expect_fallback(Pivot values list contains duplicates)
+SELECT * FROM pf_sales_intpivot
+ PIVOT (sum(earnings) FOR year IN (2012, 2014, 2014))
+ ORDER BY region
+
+statement
+CREATE TABLE pf_sales_dblnozero (region string, k double, earnings int) USING parquet
+
+statement
+INSERT INTO pf_sales_dblnozero VALUES
+ ('NA', 1.5D, 15000),
+ ('NA', 2.5D, 48000),
+ ('EU', 1.5D, 20000)
+
+query expect_fallback(Pivot values list contains duplicates)
+SELECT * FROM pf_sales_dblnozero
+ PIVOT (sum(earnings) FOR k IN (1.5D, 0.0D, -0.0D))
+ ORDER BY region
diff --git a/spark/src/test/resources/sql-tests/expressions/aggregate/PivotFirst_collation.sql b/spark/src/test/resources/sql-tests/expressions/aggregate/PivotFirst_collation.sql
new file mode 100644
index 0000000000..2178c5d793
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/aggregate/PivotFirst_collation.sql
@@ -0,0 +1,38 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- From 4.2, Spark matches the values of a pivot column with a non-UTF8_BINARY collation by
+-- that collation, so under UTF8_LCASE the pivot value 'Q1' collects the 'q1' rows. 4.0 and 4.1
+-- match the bytes. The native lookup compares bytes, so PivotFirst falls back for these pivot
+-- columns on every version. Collation syntax requires Spark 4.0+.
+
+-- MinSparkVersion: 4.0
+
+statement
+CREATE TABLE pf_lcase (quarter string COLLATE UTF8_LCASE, course string, earnings int) USING parquet
+
+statement
+INSERT INTO pf_lcase VALUES
+ ('q1', 'dotNET', 10000),
+ ('q2', 'dotNET', 15000),
+ ('q1', 'Java', 20000),
+ ('q2', 'Java', 30000)
+
+query expect_fallback(Unsupported pivot column data type)
+SELECT * FROM pf_lcase
+ PIVOT (sum(earnings) FOR quarter IN ('Q1' AS Q1, 'Q2' AS Q2))
+ ORDER BY course
diff --git a/spark/src/test/scala/org/apache/spark/sql/benchmark/CometPivotBenchmark.scala b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometPivotBenchmark.scala
new file mode 100644
index 0000000000..324e1252c5
--- /dev/null
+++ b/spark/src/test/scala/org/apache/spark/sql/benchmark/CometPivotBenchmark.scala
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.benchmark
+
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression, PivotFirst}
+import org.apache.spark.sql.comet.CometHashAggregateExec
+import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
+
+import org.apache.comet.CometConf
+
+/**
+ * Benchmark for the optimized `PIVOT` plan, whose second aggregation uses Spark's internal
+ * `PivotFirst` aggregate. To run this benchmark:
+ * {{{
+ * SPARK_GENERATE_BENCHMARK_FILES=1 make benchmark-org.apache.spark.sql.benchmark.CometPivotBenchmark
+ * }}}
+ * Results will be written to "spark/benchmarks/CometPivotBenchmark-**results.txt".
+ *
+ * The optional arguments are comma-separated lists of pivot value counts and of group counts,
+ * which default to `4,32,128` and `1000,100000,1000000`:
+ * {{{
+ * make benchmark-org.apache.spark.sql.benchmark.CometPivotBenchmark -- 128 1000000
+ * }}}
+ *
+ * Each case pivots `sum(v)` on a string column with `numPivots` values, grouped by a column with
+ * `numGroups` values, and compares Spark with two Comet runs: one with `PivotFirst` disabled, so
+ * that the aggregates using it fall back to Spark while the rest of the plan stays native, and
+ * one with `PivotFirst` native. The query selects every pivot column and writes the rows to the
+ * noop sink, so no pivot column can be pruned. Before timing, each Comet run is checked for the
+ * plan its name claims and for Spark's answer.
+ */
+object CometPivotBenchmark extends CometBenchmarkBase {
+
+ override def runCometBenchmark(mainArgs: Array[String]): Unit = {
+ require(mainArgs.length <= 2, "Usage: CometPivotBenchmark [pivot-counts] [group-counts]")
+ val pivotCounts = mainArgs.headOption.getOrElse("4,32,128").split(",").map(_.toInt).toSeq
+ val groupCounts =
+ mainArgs.lift(1).getOrElse("1000,100000,1000000").split(",").map(_.toInt).toSeq
+ val numRows = 4L * 1024 * 1024
+ withTempPath { dir =>
+ withTempTable("parquetV1Table") {
+ val columns = groupCounts.map(g => s"CAST(id % $g AS INT) AS g$g") ++
+ pivotCounts.map(p => s"concat('k', CAST(pmod(hash(id), $p) AS STRING)) AS k$p") :+
+ "CAST(pmod(hash(id, 17), 1000) AS INT) AS v"
+ prepareTable(dir, spark.range(numRows).selectExpr(columns: _*))
+ for (numPivots <- pivotCounts; numGroups <- groupCounts) {
+ runPivotBenchmark(numRows, numPivots, numGroups)
+ }
+ }
+ }
+ }
+
+ private def runPivotBenchmark(numRows: Long, numPivots: Int, numGroups: Int): Unit = {
+ val pivotValues = (0 until numPivots).map(i => s"'k$i'").mkString(", ")
+ val query =
+ s"""SELECT * FROM (SELECT g$numGroups AS grp, k$numPivots AS k, v FROM parquetV1Table)
+ |PIVOT (sum(v) FOR k IN ($pivotValues))""".stripMargin
+ val benchmark = new Benchmark(
+ s"Pivot with $numPivots pivot values and $numGroups groups",
+ numRows,
+ output = output)
+
+ val sparkConfigs = Seq(CometConf.COMET_ENABLED.key -> "false")
+ val nativeConfigs =
+ Seq(CometConf.COMET_ENABLED.key -> "true", CometConf.COMET_EXEC_ENABLED.key -> "true")
+ val fallbackConfigs =
+ nativeConfigs :+ (CometConf.getExprEnabledConfigKey("PivotFirst") -> "false")
+
+ // `withSQLConf` returns Unit on Spark 3.x, so the result leaves the block through a var.
+ var expected = 0L
+ withSQLConf(sparkConfigs: _*) { expected = checksum(query)._1 }
+ verify(benchmark, query, fallbackConfigs, expected, pivotFirstNative = false)
+ verify(benchmark, query, nativeConfigs, expected, pivotFirstNative = true)
+
+ benchmark.addCase("Spark") { _ =>
+ withSQLConf(sparkConfigs: _*)(spark.sql(query).noop())
+ }
+ benchmark.addCase("Comet (PivotFirst in Spark)") { _ =>
+ withSQLConf(fallbackConfigs: _*)(spark.sql(query).noop())
+ }
+ benchmark.addCase("Comet (PivotFirst native)") { _ =>
+ withSQLConf(nativeConfigs: _*)(spark.sql(query).noop())
+ }
+ benchmark.run()
+ }
+
+ /**
+ * An order-independent checksum of every column of the query's result, along with the final
+ * plan that computed it.
+ */
+ private def checksum(query: String) = {
+ val df = spark.sql(s"SELECT sum(hash(*)) FROM ($query)")
+ // `collect` runs the Dataset's own query execution, so its plan is the final AQE plan.
+ val sum = df.collect().head.getLong(0)
+ (sum, stripAQEPlan(df.queryExecution.executedPlan))
+ }
+
+ private def verify(
+ benchmark: Benchmark,
+ query: String,
+ configs: Seq[(String, String)],
+ expected: Long,
+ pivotFirstNative: Boolean): Unit = withSQLConf(configs: _*) {
+ val (actual, plan) = checksum(query)
+ require(actual == expected, s"Comet's pivot result differs from Spark's for:\n$query")
+ def hasPivotFirst(aggregates: Seq[AggregateExpression]): Boolean =
+ aggregates.exists(_.aggregateFunction.isInstanceOf[PivotFirst])
+ val pivotFirstRunsNatively = collect(plan) {
+ case agg: CometHashAggregateExec if hasPivotFirst(agg.aggregateExpressions) => true
+ case agg: BaseAggregateExec if hasPivotFirst(agg.aggregateExpressions) => false
+ }
+ val nativeAggregates = collect(plan) { case agg: CometHashAggregateExec => agg }
+ if (pivotFirstRunsNatively.isEmpty || pivotFirstRunsNatively.exists(_ != pivotFirstNative) ||
+ nativeAggregates.isEmpty) {
+ val where = if (pivotFirstNative) "natively" else "in Spark"
+ warn(
+ benchmark,
+ s"""WARNING: expected PivotFirst to run $where and the other aggregates natively, so
+ |this case does not measure what its name says.
+ |Comet plan:""".stripMargin + "\n" + plan.treeString)
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org