You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/03 17:50:37 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-4715-db5a40e9020e94334a8aa4149e1ea9443ec5db19 created (now 569eaa59d0)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-4715-db5a40e9020e94334a8aa4149e1ea9443ec5db19
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 569eaa59d0 feat: support multiply_ym_interval with YearMonth interval codegen dispatch (#4715)
This branch includes the following new commits:
new 569eaa59d0 feat: support multiply_ym_interval with YearMonth interval codegen dispatch (#4715)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: feat: support multiply_ym_interval with YearMonth interval codegen dispatch (#4715)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-4715-db5a40e9020e94334a8aa4149e1ea9443ec5db19
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 569eaa59d032f758669777964aee2d24eb55ebae
Author: Peter Lee <pe...@gmail.com>
AuthorDate: Sat Oct 3 17:50:15 2026 +0000
feat: support multiply_ym_interval with YearMonth interval codegen dispatch (#4715)
* Support multiply_ym_interval with YearMonth interval codegen dispatch
* Fix markdown formatting for interval docs
* Fix markdown formatting for interval docs
* review by andy
* fix md format
* restructure the CometLiteral.getSupportLevel
* Update datatypes.md
* review suggestion
* fix style
* minor change
* md prettier
* Use DataTypeSupport for sink type checks
* address review: keep CalendarIntervalType support intact, group interval expressions
Restore CalendarIntervalType in QueryPlanSerde.supportedDataType (added
deliberately by #4898; removing it narrowed hash, scalar-subquery, and
nested-type gates out of scope for this PR) and revert CometSink to main,
since its isTypeSupported OR existed only to patch sinks around that
removal. CometLiteral now only adds YearMonthIntervalType alongside the
existing DayTimeIntervalType arm. Move MultiplyYMInterval next to
MultiplyDTInterval in the expression map's interval cluster.
Co-Authored-By: Claude Fable 5 <no...@anthropic.com>
* fix: admit year-month intervals in the list-literal gate
Merging upstream/main brought in `listLiteralElementSupported`, which
`CometLiteralSuite` pins against `makeListLiteral`'s arms in both directions.
This branch had already added a `YearMonthIntervalType` arm to the encoder, so
after the merge the two sides disagreed and the pin failed.
Widen the gate rather than drop the encoder arm: `serializeDataType` already
maps the type and recurses through `ArrayType`, and `literal_to_array_ref`
already reads the ints back as an `IntervalYearMonthArray`. An
`array<interval year to month>` literal is now serialized directly instead of
sending the whole projection back to Spark with "Unsupported data type
ArrayType(YearMonthIntervalType(0,1),true)".
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* test: expect native execution for a null year-month interval in abs.sql
`abs.sql` pinned `SELECT abs(CAST(NULL AS INTERVAL YEAR TO MONTH))` as a
fallback because CometLiteral did not admit YearMonthIntervalType literals,
noting that it "flips to a failure when it is fixed". This branch adds that
literal support, so the query now runs natively and the pinned fallback
failed in every [expressions] CI job (Spark 3.4, 3.5, 4.0, 4.1, 4.2). Assert
the answer and native operators instead.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* fix: keep the element type for empty nested list literal children
When every child of a nested list literal level is empty or null, the decoder
built an empty array of the child list type itself instead of its element
type, adding a list level. Concatenating that with a populated sibling failed,
e.g. array(array(CAST(array() AS ARRAY<INTERVAL YEAR TO MONTH>)),
array(array(INTERVAL '1' MONTH))), which year-month interval literals now reach.
Build the empty values array with the element type and cover both a unit case
and the constant-folded query.
* fix: give empty nested list literal levels the field types of populated ones
Spark folds array(array(array()), array(array(array(INTERVAL '1' MONTH)))) into
a literal whose arrays all declare non-nullable elements. literal_to_array_ref
rebuilds every populated level with nullable fields, but built an all-empty
level from the declared element type, so the two siblings differed in nested
field nullability and Arrow refused to concatenate them, failing native
planning. Build the empty values through the same recursion a populated child
takes, so both always agree. Cover it with a decoder unit test and a
constant-folded CometLiteralSuite query, each with an interval and an int leaf.
* test: say the nested empty-array fixture query is not constant-folded
CometSqlFileTestSuite excludes ConstantFolding, so only the inner array() in
that query is a literal and the outer arrays run as native make_array. The
folded form is covered by CometLiteralSuite.
---------
Co-authored-by: Claude Fable 5 <no...@anthropic.com>
---
docs/source/user-guide/latest/expressions.md | 2 +-
native/core/src/execution/planner.rs | 151 ++++++++++++++++++++-
.../codegen/CometBatchKernelCodegenInput.scala | 1 +
.../org/apache/comet/serde/QueryPlanSerde.scala | 1 +
.../scala/org/apache/comet/serde/datetime.scala | 4 +-
.../scala/org/apache/comet/serde/literals.scala | 16 ++-
.../expressions/datetime/multiply_ym_interval.sql | 71 ++++++++++
.../resources/sql-tests/expressions/math/abs.sql | 7 +-
.../org/apache/comet/serde/CometLiteralSuite.scala | 38 ++++++
9 files changed, 274 insertions(+), 17 deletions(-)
diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md
index 7cf2808565..b5abfb8893 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -422,7 +422,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
| Function | Status | Implementation | Notes |
| --- | --- | --- | --- |
| `%` | ✅ | Native | |
-| `*` | ✅ | Native | DayTime interval multiplication routes through the JVM codegen dispatcher; YearMonth and Calendar interval multiplication fall back |
+| `*` | ✅ | Native | YearMonth and DayTime interval multiplication routes through the JVM codegen dispatcher; Calendar interval multiplication falls back |
| `+` | ✅ | Native | |
| `-` | ✅ | Native | |
| `/` | ✅ | Native | DayTime interval division routes through the JVM codegen dispatcher; YearMonth and Calendar interval division fall back |
diff --git a/native/core/src/execution/planner.rs b/native/core/src/execution/planner.rs
index 5e0c3aa3e3..de458eca32 100644
--- a/native/core/src/execution/planner.rs
+++ b/native/core/src/execution/planner.rs
@@ -47,7 +47,7 @@ use crate::execution::{
use crate::jvm_bridge::{jni_call, JVMClasses, ShufflePartitionPusher};
use arrow::compute::CastOptions;
use arrow::datatypes::{
- DataType, Field, FieldRef, Fields, Schema, TimeUnit, DECIMAL128_MAX_PRECISION,
+ DataType, Field, FieldRef, Fields, IntervalUnit, Schema, TimeUnit, DECIMAL128_MAX_PRECISION,
};
use arrow::ffi_stream::{ArrowArrayStreamReader, FFI_ArrowArrayStream};
use datafusion::functions_aggregate::bit_and_or_xor::{bit_and_udaf, bit_or_udaf, bit_xor_udaf};
@@ -114,8 +114,8 @@ use datafusion::physical_expr::LexOrdering;
use crate::parquet::parquet_exec::init_datasource_exec;
use arrow::array::{
- new_empty_array, Array, ArrayRef, BinaryBuilder, BooleanArray, Date32Array, Decimal128Array,
- Float32Array, Float64Array, Int16Array, Int32Array, Int64Array, Int8Array, ListArray,
+ Array, ArrayRef, BinaryBuilder, BooleanArray, Date32Array, Decimal128Array, Float32Array,
+ Float64Array, Int16Array, Int32Array, Int64Array, Int8Array, IntervalYearMonthArray, ListArray,
NullArray, StringBuilder, TimestampMicrosecondArray,
};
use arrow::buffer::{BooleanBuffer, NullBuffer, OffsetBuffer};
@@ -549,6 +549,9 @@ impl PhysicalPlanner {
DataType::Time64(TimeUnit::Nanosecond) => {
ScalarValue::Time64Nanosecond(None)
}
+ DataType::Interval(IntervalUnit::YearMonth) => {
+ ScalarValue::IntervalYearMonth(None)
+ }
DataType::Duration(TimeUnit::Microsecond) => {
ScalarValue::DurationMicrosecond(None)
}
@@ -567,9 +570,12 @@ impl PhysicalPlanner {
Value::IntVal(value) => match data_type {
DataType::Int32 => ScalarValue::Int32(Some(*value)),
DataType::Date32 => ScalarValue::Date32(Some(*value)),
+ DataType::Interval(IntervalUnit::YearMonth) => {
+ ScalarValue::IntervalYearMonth(Some(*value))
+ }
dt => {
return Err(GeneralError(format!(
- "Expected either 'Int32' or 'Date32' for IntVal, but found {dt:?}"
+ "Expected either 'Int32', 'Date32', or 'Interval(YearMonth)' for IntVal, but found {dt:?}"
)))
}
},
@@ -4503,6 +4509,10 @@ fn literal_to_array_ref(
list_literal.int_values.into(),
Some(nulls.clone().into()),
))),
+ DataType::Interval(IntervalUnit::YearMonth) => Ok(Arc::new(IntervalYearMonthArray::new(
+ list_literal.int_values.into(),
+ Some(nulls.clone().into()),
+ ))),
DataType::Timestamp(TimeUnit::Microsecond, None) => {
Ok(Arc::new(TimestampMicrosecondArray::new(
list_literal.long_values.into(),
@@ -4612,8 +4622,13 @@ fn literal_to_array_ref(
let child_refs: Vec<&dyn Array> = child_arrays.iter().map(|a| a.as_ref()).collect();
arrow::compute::concat(&child_refs)?
} else {
- // All entries are null or the list is empty
- new_empty_array(&dt)
+ // All entries are null or empty. Build the empty values through the recursion a
+ // populated entry takes, so they match a populated sibling's values: one list
+ // level below `dt` rather than `dt` itself, e.g. [[[]], [[1]]], and with the
+ // nullable fields every rebuilt level gets rather than the ones `dt` declares,
+ // e.g. a folded [[[]], [[[1]]]] whose arrays Spark declares non-nullable. Either
+ // difference keeps this level from being concatenated with the sibling.
+ literal_to_array_ref(dt, ListLiteral::default())?
};
// Create and return the parent ListArray
@@ -4792,7 +4807,7 @@ mod tests {
Array, ArrayRef, DictionaryArray, Float32Array, Float64Array, Int32Array, Int8Array,
ListArray, RecordBatch, StringArray,
};
- use arrow::datatypes::{DataType, Field, FieldRef, Fields, Schema, SchemaRef};
+ use arrow::datatypes::{DataType, Field, FieldRef, Fields, IntervalUnit, Schema, SchemaRef};
use datafusion::catalog::memory::DataSourceExec;
use datafusion::common::ScalarValue;
use datafusion::config::TableParquetOptions;
@@ -6310,6 +6325,128 @@ mod tests {
Ok(())
}
+ /// A nested list whose children are all empty must decode to the same element type as a
+ /// populated sibling, so the two can be concatenated: `[[[]], [[1]]]`.
+ #[test]
+ fn test_literal_to_list_with_empty_nested_children() -> Result<(), DataFusionError> {
+ for leaf_type in [DataType::Interval(IntervalUnit::YearMonth), DataType::Int32] {
+ let data = ListLiteral {
+ list_values: vec![
+ // [[]]
+ ListLiteral {
+ list_values: vec![ListLiteral::default()],
+ null_mask: vec![true],
+ ..Default::default()
+ },
+ // [[1]]
+ ListLiteral {
+ list_values: vec![ListLiteral {
+ int_values: vec![1],
+ null_mask: vec![true],
+ ..Default::default()
+ }],
+ null_mask: vec![true],
+ ..Default::default()
+ },
+ ],
+ null_mask: vec![true, true],
+ ..Default::default()
+ };
+ let inner = DataType::List(Arc::new(Field::new("item", leaf_type.clone(), true)));
+ let middle = DataType::List(Arc::new(Field::new("item", inner.clone(), true)));
+ let outer = DataType::List(Arc::new(Field::new("item", middle.clone(), true)));
+
+ let array = literal_to_array_ref(outer, data)?;
+
+ // The outer list's values: two lists whose elements are `inner` lists.
+ assert_eq!(array.data_type(), &middle, "{leaf_type}");
+ let lists = array.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(lists.len(), 2);
+
+ let first = lists.value(0);
+ let first = first.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(first.len(), 1);
+ assert!(first.is_valid(0));
+ assert_eq!(first.value(0).len(), 0);
+
+ let second = lists.value(1);
+ let second = second.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(second.len(), 1);
+ assert_eq!(second.value(0).len(), 1);
+ assert_eq!(second.value(0).data_type(), &leaf_type);
+ }
+ Ok(())
+ }
+
+ /// The empty and populated children must also agree on nested field nullability. Spark folds
+ /// `array(array(array()), array(array(array(INTERVAL '1' MONTH))))` into a literal whose
+ /// arrays all declare non-nullable elements, while a populated level is rebuilt with nullable
+ /// fields: `[[[]], [[[1]]]]`.
+ #[test]
+ fn test_literal_to_list_with_empty_nested_children_non_nullable() -> Result<(), DataFusionError>
+ {
+ let list_of = |element: DataType, nullable: bool| {
+ DataType::List(Arc::new(Field::new("item", element, nullable)))
+ };
+ for leaf_type in [DataType::Interval(IntervalUnit::YearMonth), DataType::Int32] {
+ let data = ListLiteral {
+ list_values: vec![
+ // [[]]
+ ListLiteral {
+ list_values: vec![ListLiteral::default()],
+ null_mask: vec![true],
+ ..Default::default()
+ },
+ // [[[1]]]
+ ListLiteral {
+ list_values: vec![ListLiteral {
+ list_values: vec![ListLiteral {
+ int_values: vec![1],
+ null_mask: vec![true],
+ ..Default::default()
+ }],
+ null_mask: vec![true],
+ ..Default::default()
+ }],
+ null_mask: vec![true],
+ ..Default::default()
+ },
+ ],
+ null_mask: vec![true, true],
+ ..Default::default()
+ };
+ let mut declared = leaf_type.clone();
+ for _ in 0..4 {
+ declared = list_of(declared, false);
+ }
+
+ let array = literal_to_array_ref(declared, data)?;
+
+ // The outer list's values: two lists of lists of `leaf_type` lists, nullable at every
+ // level whichever child decoded them.
+ let expected = list_of(list_of(list_of(leaf_type.clone(), true), true), true);
+ assert_eq!(array.data_type(), &expected, "{leaf_type}");
+ let lists = array.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(lists.len(), 2);
+
+ let first = lists.value(0);
+ let first = first.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(first.len(), 1);
+ assert!(first.is_valid(0));
+ assert_eq!(first.value(0).len(), 0);
+
+ let second = lists.value(1);
+ let second = second.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(second.len(), 1);
+ let innermost = second.value(0);
+ let innermost = innermost.as_any().downcast_ref::<ListArray>().unwrap();
+ assert_eq!(innermost.len(), 1);
+ assert_eq!(innermost.value(0).len(), 1);
+ assert_eq!(innermost.value(0).data_type(), &leaf_type);
+ }
+ Ok(())
+ }
+
#[tokio::test]
async fn test_literal_to_list() -> Result<(), DataFusionError> {
/*
diff --git a/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenInput.scala b/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenInput.scala
index 2ed7e33c90..d8c6ef07d0 100644
--- a/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenInput.scala
+++ b/spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenInput.scala
@@ -58,6 +58,7 @@ private[codegen] object CometBatchKernelCodegenInput extends CometTypeShim {
classOf[TinyIntVector],
classOf[SmallIntVector],
classOf[IntVector],
+ classOf[IntervalYearVector],
classOf[BigIntVector],
classOf[Float4Vector],
classOf[Float8Vector],
diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
index bcdb37122b..4a4b9eedbe 100644
--- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
@@ -303,6 +303,7 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
classOf[MakeDTInterval] -> CometMakeDTInterval,
classOf[MakeInterval] -> CometMakeInterval,
classOf[MultiplyDTInterval] -> CometMultiplyDTInterval,
+ classOf[MultiplyYMInterval] -> CometMultiplyYMInterval,
classOf[TimestampAdd] -> CometTimestampAdd,
classOf[TimestampDiff] -> CometTimestampDiff,
classOf[MicrosToTimestamp] -> CometMicrosToTimestamp,
diff --git a/spark/src/main/scala/org/apache/comet/serde/datetime.scala b/spark/src/main/scala/org/apache/comet/serde/datetime.scala
index 664145b3c4..bf27ce2c55 100644
--- a/spark/src/main/scala/org/apache/comet/serde/datetime.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/datetime.scala
@@ -21,7 +21,7 @@ package org.apache.comet.serde
import java.util.Locale
-import org.apache.spark.sql.catalyst.expressions.{AddMonths, Attribute, Cast, ConvertTimezone, DateAdd, DateDiff, DateFormatClass, DateFromUnixDate, DateSub, DayOfMonth, DayOfWeek, DayOfYear, Days, DivideDTInterval, Expression, FromUTCTimestamp, GetDateField, GetTimestamp, Hour, Hours, LastDay, Literal, MakeDate, MakeDTInterval, MakeInterval, MakeTimestamp, MakeYMInterval, MicrosToTimestamp, MillisToTimestamp, Minute, Month, MonthsBetween, MultiplyDTInterval, NextDay, PreciseTimestampCon [...]
+import org.apache.spark.sql.catalyst.expressions.{AddMonths, Attribute, Cast, ConvertTimezone, DateAdd, DateDiff, DateFormatClass, DateFromUnixDate, DateSub, DayOfMonth, DayOfWeek, DayOfYear, Days, DivideDTInterval, Expression, FromUTCTimestamp, GetDateField, GetTimestamp, Hour, Hours, LastDay, Literal, MakeDate, MakeDTInterval, MakeInterval, MakeTimestamp, MakeYMInterval, MicrosToTimestamp, MillisToTimestamp, Minute, Month, MonthsBetween, MultiplyDTInterval, MultiplyYMInterval, NextDay, [...]
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{CalendarIntervalType, DataType, DateType, DoubleType, FloatType, IntegerType, LongType, StringType, TimestampNTZType, TimestampType}
import org.apache.spark.unsafe.types.UTF8String
@@ -894,6 +894,8 @@ object CometGetTimestamp extends CometCodegenDispatch[GetTimestamp]
object CometMakeYMInterval extends CometCodegenDispatch[MakeYMInterval]
+object CometMultiplyYMInterval extends CometCodegenDispatch[MultiplyYMInterval]
+
object CometMakeDTInterval extends CometCodegenDispatch[MakeDTInterval]
object CometDivideDTInterval extends CometCodegenDispatch[DivideDTInterval]
diff --git a/spark/src/main/scala/org/apache/comet/serde/literals.scala b/spark/src/main/scala/org/apache/comet/serde/literals.scala
index 98a3757ffc..0aa03ad6c5 100644
--- a/spark/src/main/scala/org/apache/comet/serde/literals.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/literals.scala
@@ -25,7 +25,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{Attribute, Cast, CreateArray, CreateMap, Expression, KnownNullable, Literal, MapFromArrays}
import org.apache.spark.sql.catalyst.util.{ArrayData, MapData, TypeUtils}
-import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType, ByteType, CalendarIntervalType, DataType, DateType, DayTimeIntervalType, Decimal, DecimalType, DoubleType, FloatType, IntegerType, LongType, MapType, NullType, ShortType, StringType, StructType, TimestampNTZType, TimestampType}
+import org.apache.spark.sql.types.{ArrayType, BinaryType, BooleanType, ByteType, CalendarIntervalType, DataType, DateType, DayTimeIntervalType, Decimal, DecimalType, DoubleType, FloatType, IntegerType, LongType, MapType, NullType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, YearMonthIntervalType}
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import com.google.protobuf.ByteString
@@ -58,7 +58,11 @@ object CometLiteral extends CometExpressionSerde[Literal] with CometTypeShim wit
Unsupported(Some(s"Unsupported literal value for data type ${expr.dataType}"))
} else {
expr.dataType match {
- case _: DayTimeIntervalType => Compatible(None)
+ // ANSI interval types are deliberately kept out of QueryPlanSerde.supportedDataType so
+ // they are not claimed as flowing through arbitrary native operators; literals are
+ // supported here.
+ case _: DayTimeIntervalType | _: YearMonthIntervalType =>
+ Compatible(None)
case dt => Unsupported(Some(s"Unsupported data type $dt"))
}
}
@@ -88,7 +92,8 @@ object CometLiteral extends CometExpressionSerde[Literal] with CometTypeShim wit
case _: BooleanType => exprBuilder.setBoolVal(value.asInstanceOf[Boolean])
case _: ByteType => exprBuilder.setByteVal(value.asInstanceOf[Byte].toInt)
case _: ShortType => exprBuilder.setShortVal(value.asInstanceOf[Short].toInt)
- case _: IntegerType | _: DateType => exprBuilder.setIntVal(value.asInstanceOf[Int])
+ case _: IntegerType | _: DateType | _: YearMonthIntervalType =>
+ exprBuilder.setIntVal(value.asInstanceOf[Int])
case _: LongType | _: TimestampType | _: TimestampNTZType | _: DayTimeIntervalType =>
exprBuilder.setLongVal(value.asInstanceOf[Long])
case dt if isTimeType(dt) =>
@@ -165,7 +170,7 @@ object CometLiteral extends CometExpressionSerde[Literal] with CometTypeShim wit
else null.asInstanceOf[Integer])
listLiteralBuilder.addNullMask(casted != null)
})
- case IntegerType | DateType =>
+ case IntegerType | DateType | _: YearMonthIntervalType =>
array.foreach(v => {
val casted = v.asInstanceOf[Integer]
listLiteralBuilder.addIntValues(casted)
@@ -240,6 +245,9 @@ object CometLiteral extends CometExpressionSerde[Literal] with CometTypeShim wit
TimestampType | TimestampNTZType | FloatType | DoubleType | StringType | BinaryType =>
true
case _: DecimalType => true
+ // Matched as a type rather than a stable identifier: the start/end fields participate in
+ // `equals`, and every (start, end) pair is carried as the same month count.
+ case _: YearMonthIntervalType => true
case ArrayType(elementType, _) => listLiteralElementSupported(elementType)
case _ => false
}
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/multiply_ym_interval.sql b/spark/src/test/resources/sql-tests/expressions/datetime/multiply_ym_interval.sql
new file mode 100644
index 0000000000..6263b34511
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/multiply_ym_interval.sql
@@ -0,0 +1,71 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Routes multiply_ym_interval through the codegen dispatcher; produces YearMonthIntervalType.
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
+
+statement
+CREATE TABLE test_multiply_ym_interval(y int, m int, i int, l long, f float, d double, dec decimal(10,2)) USING parquet
+
+statement
+INSERT INTO test_multiply_ym_interval VALUES
+ (1, 2, 2, CAST(3 AS BIGINT), CAST(1.5 AS FLOAT), CAST(2.5 AS DOUBLE), CAST(2.50 AS DECIMAL(10, 2))),
+ (-1, 1, -2, CAST(-3 AS BIGINT), CAST(-1.5 AS FLOAT), CAST(-2.5 AS DOUBLE), CAST(-2.50 AS DECIMAL(10, 2))),
+ (0, 6, 0, CAST(0 AS BIGINT), CAST(0.5 AS FLOAT), CAST(0.5 AS DOUBLE), CAST(0.50 AS DECIMAL(10, 2))),
+ (2, -6, NULL, NULL, NULL, NULL, NULL)
+
+query
+SELECT
+ make_ym_interval(y, m) * i,
+ make_ym_interval(y, m) * l,
+ make_ym_interval(y, m) * f,
+ make_ym_interval(y, m) * d,
+ make_ym_interval(y, m) * dec
+FROM test_multiply_ym_interval
+
+-- literal interval input
+query
+SELECT INTERVAL '1-2' YEAR TO MONTH * i FROM test_multiply_ym_interval
+
+-- numeric on the left is normalized by Spark to multiply_ym_interval.
+query
+SELECT i * make_ym_interval(y, m), 2 * INTERVAL '1-2' YEAR TO MONTH
+FROM test_multiply_ym_interval
+
+-- literal multipliers, including half-up rounding for fractional months.
+query
+SELECT
+ make_ym_interval(1, 2) * 2,
+ make_ym_interval(1, 2) * 1.5D,
+ make_ym_interval(1, 2) * CAST(1.50 AS DECIMAL(10, 2)),
+ make_ym_interval(-1, 1) * 1.5D
+
+-- null interval input
+query
+SELECT make_ym_interval(NULL, m) * 2 FROM test_multiply_ym_interval
+
+-- a nested interval array whose first branch only holds an empty array. This harness excludes
+-- ConstantFolding, so only the inner array() is a literal here; CometLiteralSuite covers the
+-- folded form.
+query
+SELECT array(array(CAST(array() AS ARRAY<INTERVAL YEAR TO MONTH>)), array(array(INTERVAL '1' MONTH)))
+FROM test_multiply_ym_interval
+
+-- 178956970 years and 7 months is Int.MaxValue months. Multiplication overflows regardless of
+-- ANSI mode; the lowercase pattern matches Spark 3.x and 4.x error messages.
+query expect_error(overflow)
+SELECT make_ym_interval(178956970, 7) * 2
diff --git a/spark/src/test/resources/sql-tests/expressions/math/abs.sql b/spark/src/test/resources/sql-tests/expressions/math/abs.sql
index ff28bfd3e9..c32cadd69e 100644
--- a/spark/src/test/resources/sql-tests/expressions/math/abs.sql
+++ b/spark/src/test/resources/sql-tests/expressions/math/abs.sql
@@ -56,10 +56,9 @@ INSERT INTO test_abs_iv_overflow VALUES (1, 2, 3, 4.5), (-106751991, -4, 0, -54.
query expect_error(overflow)
SELECT abs(make_dt_interval(d, h, m, s)) FROM test_abs_iv_overflow
--- pinned fallback: NullPropagation folds the ym null into a bare typed literal that CometLiteral
--- does not admit (it special-cases only DayTimeIntervalType, literals.scala:63), so the whole
--- projection falls back. Pre-existing gap (#5061); this flips to a failure when it is fixed.
-query expect_fallback(Unsupported data type YearMonthIntervalType)
+-- NullPropagation folds the ym null into a bare typed literal. CometLiteral admits
+-- YearMonthIntervalType literals (one of the gaps tracked in #5061), so it stays native.
+query
SELECT abs(CAST(NULL AS INTERVAL YEAR TO MONTH))
query
diff --git a/spark/src/test/scala/org/apache/comet/serde/CometLiteralSuite.scala b/spark/src/test/scala/org/apache/comet/serde/CometLiteralSuite.scala
index 746cc8e4b8..fd2f3aca61 100644
--- a/spark/src/test/scala/org/apache/comet/serde/CometLiteralSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/serde/CometLiteralSuite.scala
@@ -24,6 +24,7 @@ import scala.util.Try
import org.apache.arrow.vector.types.TimeUnit
import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType}
import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.catalyst.expressions.{Alias, Literal}
import org.apache.spark.sql.catalyst.util.GenericArrayData
import org.apache.spark.sql.comet.util.Utils
import org.apache.spark.sql.types._
@@ -123,4 +124,41 @@ class CometLiteralSuite extends CometTestBase with CometTypeShim {
assert(!CometLiteral.listLiteralElementSupported(collated))
assert(!encoderAcceptsElement(Array.empty, ArrayType(collated)))
}
+
+ // A year-month interval rides in `int_values` as the month count `IntegerType` uses, and
+ // `literal_to_array_ref` reads those ints back as an `IntervalYearMonthArray`, so the whole
+ // array literal is serialized directly instead of sending the projection back to Spark.
+ // `needsExpansion` has no arm for it either, so declining it here is a fallback, not a rewrite.
+ test("a year-month interval array literal is serialized rather than declined") {
+ withParquetTable(Seq((1, 2), (3, 4)), "tbl") {
+ checkSparkAnswerAndOperator(
+ sql("SELECT array(INTERVAL '1-2' YEAR TO MONTH, INTERVAL '-3' MONTH, NULL) FROM tbl"))
+ checkSparkAnswerAndOperator(
+ sql("SELECT array(array(INTERVAL '2-1' YEAR TO MONTH), array()) FROM tbl"))
+ }
+ }
+
+ // Spark folds each projection into one literal whose arrays all declare non-nullable elements,
+ // while `literal_to_array_ref` rebuilds every populated level with nullable fields. The empty
+ // `array()` branch used to keep the declared fields instead, so it could not be concatenated
+ // with its populated sibling and native planning failed. The SQL file tests cannot cover this,
+ // since their harness excludes `ConstantFolding`.
+ test("a folded nested array literal with an empty branch and non-nullable elements") {
+ def nonNullableAtEveryLevel(dataType: DataType): Boolean = dataType match {
+ case ArrayType(elementType, containsNull) =>
+ !containsNull && nonNullableAtEveryLevel(elementType)
+ case _ => true
+ }
+ withParquetTable(Seq((1, 2), (3, 4)), "tbl") {
+ Seq("INTERVAL '1' MONTH", "1").foreach { leaf =>
+ val df = sql(s"SELECT array(array(array()), array(array(array($leaf)))) FROM tbl")
+ val folded = df.queryExecution.optimizedPlan.expressions.exists {
+ case Alias(literal: Literal, _) => nonNullableAtEveryLevel(literal.dataType)
+ case _ => false
+ }
+ assert(folded, s"expected a folded literal with non-nullable arrays for leaf $leaf")
+ checkSparkAnswerAndOperator(df)
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org