You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/05 04:26:43 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-4821-4e23f25d0a06259f4664e65bf101e7c8ff68c15d created (now 811a3b7e20)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-4821-4e23f25d0a06259f4664e65bf101e7c8ff68c15d
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 811a3b7e20 feat: support Spark 4.1 TIME type and expressions via codegen dispatch (#4821)
This branch includes the following new commits:
new 811a3b7e20 feat: support Spark 4.1 TIME type and expressions via codegen dispatch (#4821)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: feat: support Spark 4.1 TIME type and expressions via codegen dispatch (#4821)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-4821-4e23f25d0a06259f4664e65bf101e7c8ff68c15d
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 811a3b7e203ba09782e3d4ff6c870bb382d7c372
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Oct 5 04:26:25 2026 +0000
feat: support Spark 4.1 TIME type and expressions via codegen dispatch (#4821)
* feat: support Spark 4.1 TIME type and expressions via codegen dispatch
Wires the Spark 4.1 TIME data type end-to-end and adds JVM codegen
dispatch for the new TIME expressions. No native implementations are
added: everything routes through the existing emitJvmCodegenDispatch
path so the enclosing projection stays native while Spark's own
doGenCode supplies the semantics.
Adds TimeType to DataTypeSupport, teaches the batch-kernel input and
output emitters, CometScalaUDFCodegen, and ArrowWriters to handle
TimeNanoVector as a Long-backed vector, and updates the shim so
HoursOfTime, MinutesOfTime, SecondsOfTime, SecondsOfTimeWithFraction,
TimeAddInterval, SubtractTimes, TimeDiff, and TimeTrunc reach the
dispatcher via their StaticInvoke(DateTimeUtils, ...) replacement
form. Casts touching TimeType follow the same route via CometCast.
Parquet scans still opt out of TimeType (no native decoder yet); the
fallback reason is now explicit.
* test: cover Spark 4.1 TIME expressions via SQL-file tests
Replaces the initial CometTimeTypeSuite Scala tests with per-expression
sql-tests fixtures under expressions/datetime and expressions/cast, so
coverage runs through the same CometSqlFileTestSuite pipeline as other
datetime tests (constant folding disabled, unified reporting).
Also accept Time64(Nanosecond) in the native LongVal literal path so
constant-folded TIME literals from CometCast (cast literal string ->
TIME evaluated at planning time) round-trip through the native scan.
* review: gate nested TIME at canHandle; add DurationVector spec; scope to_time docs
Restricts isSupportedDataType so ArrayType/StructType/MapType containing TIME return false at plan time, and the operator falls back cleanly instead of crashing when the array/struct/map getters do not yet route TIME to getLong. Adds DurationVector (DayTimeIntervalType) and IntervalYearVector (YearMonthIntervalType) to CometScalaUDFCodegen.specFor and to the getLong ordinal switch in CometBatchKernelCodegenInput so a non-foldable interval attribute passed into timeAddInterval (or any [...]
* test: record that the local scan admits TIME at its supportedDataType boundary
The merge of main gave CometLocalTableScanExec the default allowTimeType, since
TimeNanoWriter makes TIME writable there. The boundary table in QueryPlanSerdeSuite
mirrors each caller's flags, so it still passed while listing TIME as rejected, and the
supportedDataType scaladoc said the same.
* fix: keep TIME columns out of the native CSV scan
The TIME arm in DataTypeSupport reached every operator that inherits it, not just the local scan.
CometBatchScanExec gates the native CSV V2 scan with it, so a CSV read with a TIME column went
native. Spark 4.1's own CSV reader rejects TIME, and Spark 4.2 parses it with the `timeFormat`
option, which the native reader does not take. The arm also opened CometSparkToColumnarExec to
TIME, which nothing tested.
Since the merge of main, the local scan admits TIME through its own supportedDataType flags, so the
arm has no remaining purpose. The new test asserts on the plan, because Spark 4.1 has no answer to
compare against.
* refactor: drop TIME dispatch wiring that main already provides
The 4.1+ shim matched the TIME family's StaticInvoke forms and sent them to the codegen
dispatcher, and CometCast reported every TIME cast as Compatible so that convert could dispatch
it. Main does both already. Since #5692, CometStaticInvoke sends any StaticInvoke it has no
handler for to the dispatcher. CometCast mixes in CodegenDispatchFallback, and every TIME cast
falls through to an Unsupported arm in isSupported, which the mixin dispatches. With both removed,
every TIME fixture passes and reports the same native and dispatched expressions.
Removing the shim arm also keeps it out of the way of #4997, which adds native arms for
getHoursOfTime, getMinutesOfTime and getSecondsOfTime to the same match. Its comment had also
gone stale: getSecondsOfTimeWithFraction runs natively since #5738.
* perf: write TIME through the fixed-width Arrow writer paths
TimeNanoWriter extended plain ArrowFieldWriter, so it missed the unchecked row write and the bulk
column copy that #6566 gave every other fixed-width writer. It now extends
FixedWidthArrowFieldWriter like TimestampNTZWriter, and TimeNanoVector joins the 8-byte arm of
copyValues. CometArrowWriterSuite compares the columnar paths with the row path for TIME on
Spark 4.1+.
* test: pin the TIME fixtures to the dispatcher and widen their coverage
Every fixture's comments said its queries ran through the codegen dispatcher, but the plain
`query` mode cannot tell the dispatcher from a native kernel. Queries that do dispatch now say so
with `expect_dispatch`. Those are the column arguments, and the literal arguments to the TIME
functions. A cast of a literal is folded at planning time and a NULL argument records no
expression, so those stay plain queries. The comments now credit CometStaticInvoke and
CodegenDispatchFallback, which do the routing since the shim and cast wiring were dropped.
- subtract_times.sql pinned `time - time` with `spark_answer_only` on the premise that it fell
back. It runs in Comet now.
- second_time.sql drops its EXTRACT(SECOND FROM TIME) block. That expression runs natively since
#5738, and extract_time.sql covers it.
- time_add_interval.sql adds an interval computed from a column, and the DATETIME_OVERFLOW error
for a result past midnight.
- The local-scan TIME test adds a NULL, TIME(3), and TIME inside an array and a struct, with two
rows to a batch so each scan writes several batches.
* docs: bring the TIME rows up to date
current_time, time_diff and time_trunc gain the spark.sql.timeType.enabled note that the other
TIME rows carry. The two-argument (date, time) forms of make_timestamp_ltz and make_timestamp_ntz
run through codegen dispatch rather than falling back, and so does to_time with a format argument.
The TimeType row in the data types page says that Parquet scans of TIME columns fall back.
---
docs/source/user-guide/latest/datatypes.md | 12 +--
docs/source/user-guide/latest/expressions.md | 12 +--
.../org/apache/comet/rules/CometScanRule.scala | 6 ++
.../org/apache/comet/serde/QueryPlanSerde.scala | 6 +-
.../spark/sql/comet/CometLocalTableScanExec.scala | 6 +-
.../sql/comet/execution/arrow/ArrowWriters.scala | 15 +++-
.../sql-tests/expressions/cast/cast_time.sql | 91 ++++++++++++++++++++++
.../expressions/datetime/current_time.sql | 36 +++++++++
.../sql-tests/expressions/datetime/hour_time.sql | 51 ++++++++++++
.../sql-tests/expressions/datetime/minute_time.sql | 48 ++++++++++++
.../sql-tests/expressions/datetime/second_time.sql | 49 ++++++++++++
.../expressions/datetime/subtract_times.sql | 42 ++++++++++
.../expressions/datetime/time_add_interval.sql | 68 ++++++++++++++++
.../sql-tests/expressions/datetime/time_diff.sql | 65 ++++++++++++++++
.../sql-tests/expressions/datetime/time_trunc.sql | 62 +++++++++++++++
.../apache/comet/csv/CometCsvNativeReadSuite.scala | 30 ++++++-
.../org/apache/comet/exec/CometExecSuite.scala | 23 ++++--
.../apache/comet/serde/QueryPlanSerdeSuite.scala | 5 +-
.../execution/arrow/CometArrowWriterSuite.scala | 12 ++-
19 files changed, 609 insertions(+), 30 deletions(-)
diff --git a/docs/source/user-guide/latest/datatypes.md b/docs/source/user-guide/latest/datatypes.md
index 1b619e67ca..6ffedb1f1e 100644
--- a/docs/source/user-guide/latest/datatypes.md
+++ b/docs/source/user-guide/latest/datatypes.md
@@ -71,12 +71,12 @@ the tables below and may be reconsidered based on demand:
## Datetime
-| Type | Status | Notes |
-| ------------------ | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `DateType` | ✅ | |
-| `TimestampType` | ✅ | |
-| `TimestampNTZType` | ✅ | |
-| `TimeType` | ⚠️ | Spark 4.1+. Native serialization is in place; some operators (sort, min/max) are still being wired up ([#4288](https://github.com/apache/datafusion-comet/issues/4288)). |
+| Type | Status | Notes |
+| ------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `DateType` | ✅ | |
+| `TimestampType` | ✅ | |
+| `TimestampNTZType` | ✅ | |
+| `TimeType` | ⚠️ | Spark 4.1+. Native serialization is in place. Parquet scans of `TIME` columns fall back, and some operators (sort, min/max) are still being wired up ([#4288](https://github.com/apache/datafusion-comet/issues/4288)). |
## Interval
diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md
index b5abfb8893..74232aac65 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -256,7 +256,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `convert_timezone` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) |
| `curdate` | ✅ | — | Constant-folded to a literal (alias of `current_date`) |
| `current_date` | ✅ | — | Constant-folded to a literal before Comet sees the plan |
-| `current_time` | 🔜 | — | Blocked on Spark 4.1 TIME type support ([#4288](https://github.com/apache/datafusion-comet/issues/4288)) |
+| `current_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Constant-folded to a literal before Comet sees the plan |
| `current_timestamp` | ✅ | — | Constant-folded to a literal before Comet sees the plan |
| `current_timezone` | ✅ | — | |
| `date_add` | ✅ | Native | The 2-argument form is native; the `date_add(UNIT, n, ts)` form (Spark 3.5+) parses to `timestampadd` and runs through codegen dispatch |
@@ -285,8 +285,8 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `make_interval` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; intervals outside Arrow's nanosecond range are tracked by [#5279](https://github.com/apache/datafusion-comet/issues/5279); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) |
| `make_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Runs natively; remaining TIME type work is tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
| `make_timestamp` | ✅ | Hybrid | |
-| `make_timestamp_ltz` | ✅ | — | 2-arg TIME form falls back |
-| `make_timestamp_ntz` | ✅ | — | 2-arg TIME form falls back |
+| `make_timestamp_ltz` | ✅ | — | The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch |
+| `make_timestamp_ntz` | ✅ | — | The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch |
| `make_ym_interval` | ✅ | Codegen dispatch | |
| `minute` | ✅ | Native | |
| `month` | ✅ | Native | |
@@ -297,8 +297,8 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `quarter` | ✅ | Native | |
| `second` | ✅ | Native | |
| `session_window` | 🔜 | — | Batch session-window grouping falls back (`UpdatingSessionsExec` is not yet native); tracked by [#4785](https://github.com/apache/datafusion-comet/issues/4785) |
-| `time_diff` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
-| `time_trunc` | 🔜 | — | Spark 4.1 TIME type; tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) |
+| `time_diff` | ✅ | Codegen dispatch | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default |
+| `time_trunc` | ✅ | Codegen dispatch | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default |
| `timediff` | ✅ | — | Spark 4.0+ grammar alias that parses to `timestampdiff`; runs through codegen dispatch |
| `timestamp_micros` | ✅ | Codegen dispatch | |
| `timestamp_millis` | ✅ | Codegen dispatch | |
@@ -306,7 +306,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
| `timestampadd` | ✅ | — | Reached through the grammar rather than the function registry; runs through codegen dispatch |
| `timestampdiff` | ✅ | — | Reached through the grammar rather than the function registry; runs through codegen dispatch |
| `to_date` | ✅ | — | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format) before Comet sees the plan |
-| `to_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. The one-argument form runs natively |
+| `to_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. The one-argument form runs natively, and the form with a format argument runs through codegen dispatch |
| `to_timestamp` | ✅ | — | Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan |
| `to_timestamp_ltz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampType`) |
| `to_timestamp_ntz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampNTZType`) |
diff --git a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala
index 895df2d566..b109fa158e 100644
--- a/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala
+++ b/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala
@@ -1148,6 +1148,12 @@ case class CometScanTypeChecker() extends DataTypeSupport with CometTypeShim {
"native execution if your data does not contain unsigned small integers. " +
CometConf.COMPAT_GUIDE
false
+ case dt if isTimeType(dt) =>
+ // The native Parquet reader has not been taught to decode the TIME logical type into
+ // an Arrow Time64(NANOSECOND) vector, so fall back to Spark for scans that expose one.
+ fallbackReasons += s"Unsupported $name of type $dt (native Parquet scan does not " +
+ "support TIME)"
+ false
case dt if isStringCollationType(dt) =>
// we don't need specific support for collation in scans, but this
// is a convenient place to force the whole query to fall back to Spark for now
diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
index 6bbc3b9f37..430d36289c 100644
--- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
@@ -566,9 +566,9 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
* The defaults preserve expression-serde behavior: primitive types, `CalendarIntervalType`,
* `TimeType`, and all `StringType` variants are accepted, while complex and ANSI interval types
* are rejected. Sinks and native shuffle enable complex and ANSI interval types because their
- * Arrow IPC paths support them. Local scans additionally reject `TimeType` and non-default
- * strings, while JVM columnar shuffle rejects ANSI intervals, calendar intervals, and duplicate
- * struct field names because its unsafe-row-to-Arrow path cannot handle them.
+ * Arrow IPC paths support them. Local scans additionally reject non-default strings, while JVM
+ * columnar shuffle rejects ANSI intervals, calendar intervals, and duplicate struct field names
+ * because its unsafe-row-to-Arrow path cannot handle them.
*
* Note that the option polarity is mixed: `allowComplex` and `allowIntervals` are restrictive
* by default; the other four options are permissive by default.
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometLocalTableScanExec.scala b/spark/src/main/scala/org/apache/spark/sql/comet/CometLocalTableScanExec.scala
index 61d977e1fd..82d5e417c1 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/CometLocalTableScanExec.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometLocalTableScanExec.scala
@@ -133,8 +133,9 @@ object CometLocalTableScanExec extends CometSink[LocalTableScanExec] with DataTy
// downstream expression serdes (issue #4789).
override protected def scanFieldType(dt: DataType): DataType = dt.asNullable
- // RowArrowReader handles NullType and intervals, but not TimeType. Non-default string collations
- // remain unsupported here, matching DataTypeSupport's existing local-scan boundary.
+ // RowArrowReader handles NullType, intervals, and TimeType (through TimeNanoWriter).
+ // Non-default string collations remain unsupported here, matching DataTypeSupport's
+ // existing local-scan boundary.
override def isTypeSupported(
dt: DataType,
name: String,
@@ -143,7 +144,6 @@ object CometLocalTableScanExec extends CometSink[LocalTableScanExec] with DataTy
dt,
allowComplex = true,
allowIntervals = true,
- allowTimeType = false,
allowAnyStringType = false,
// Java Arrow keys struct children by name; declining here hands the struct to
// DataTypeSupport, which records the duplicate-field-name fallback reason.
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowWriters.scala b/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowWriters.scala
index 28e39fa9b4..00c4de38f6 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowWriters.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/ArrowWriters.scala
@@ -77,6 +77,7 @@ private[arrow] object ArrowWriter {
case (DateType, vector: DateDayVector) => new DateWriter(vector)
case (TimestampType, vector: TimeStampMicroTZVector) => new TimestampWriter(vector)
case (TimestampNTZType, vector: TimeStampMicroVector) => new TimestampNTZWriter(vector)
+ case (dt, vector: TimeNanoVector) if Utils.isTimeType(dt) => new TimeNanoWriter(vector)
case (ArrayType(_, _), vector: ListVector) =>
val elementVector = createFieldWriter(vector.getDataVector())
new ArrayWriter(vector, elementVector)
@@ -766,7 +767,7 @@ private[arrow] abstract class FixedWidthArrowFieldWriter extends ArrowFieldWrite
}
}
case _: BigIntVector | _: TimeStampMicroTZVector | _: TimeStampMicroVector |
- _: DurationVector =>
+ _: DurationVector | _: TimeNanoVector =>
if (bulk) {
bulkCopy(input, startRow, numRows, target, 8, input.getLongs(_, _))
} else {
@@ -1368,6 +1369,18 @@ private[arrow] class TimestampNTZWriter(val valueVector: TimeStampMicroVector)
}
}
+private[arrow] class TimeNanoWriter(val valueVector: TimeNanoVector)
+ extends FixedWidthArrowFieldWriter {
+
+ override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {
+ valueVector.setSafe(count, input.getLong(ordinal))
+ }
+
+ override protected def setValueUnsafe(input: SpecializedGetters, ordinal: Int): Unit = {
+ valueVector.set(count, input.getLong(ordinal))
+ }
+}
+
private[arrow] class ArrayWriter(val valueVector: ListVector, val elementWriter: ArrowFieldWriter)
extends ArrowFieldWriter {
diff --git a/spark/src/test/resources/sql-tests/expressions/cast/cast_time.sql b/spark/src/test/resources/sql-tests/expressions/cast/cast_time.sql
new file mode 100644
index 0000000000..378920416f
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/cast/cast_time.sql
@@ -0,0 +1,91 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- Casts to and from TIME have no native lowering in Comet. CometCast reports them Unsupported,
+-- and CodegenDispatchFallback routes them through the JVM codegen dispatcher, so the enclosing
+-- projection stays native. A cast of a literal is folded at planning time instead.
+
+statement
+CREATE TABLE test_cast_time(s string, h int, m int, sec decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_cast_time VALUES
+ ('00:00:00', 0, 0, 0.000000),
+ ('12:34:56', 12, 34, 56.000000),
+ ('23:59:59.999999',23, 59, 59.999999),
+ ('01:00:00', 1, 0, 0.000000),
+ (NULL, NULL, NULL, NULL)
+
+-- string -> TIME (column)
+query expect_dispatch(cast)
+SELECT CAST(s AS TIME) FROM test_cast_time
+
+-- string -> TIME (literals)
+query
+SELECT CAST('00:00:00' AS TIME)
+
+query
+SELECT CAST('23:59:59.999999' AS TIME)
+
+-- TIME -> string (column)
+query expect_dispatch(cast)
+SELECT CAST(make_time(h, m, sec) AS STRING) FROM test_cast_time
+
+-- TIME -> string (literals)
+query
+SELECT CAST(TIME '00:00:00' AS STRING)
+
+query
+SELECT CAST(TIME '13:45:07.123456' AS STRING)
+
+-- TIME -> BIGINT (whole seconds since midnight, via floor(nanos / 1e9))
+query expect_dispatch(cast)
+SELECT CAST(make_time(h, m, sec) AS BIGINT) FROM test_cast_time
+
+query
+SELECT CAST(TIME '00:00:00' AS BIGINT)
+
+query
+SELECT CAST(TIME '01:00:00' AS BIGINT)
+
+query
+SELECT CAST(TIME '23:59:59.999999' AS BIGINT)
+
+-- TIME -> INT (same seconds-truncation semantics)
+query
+SELECT CAST(TIME '13:45:07.999999' AS INT)
+
+-- TIME -> DECIMAL (fractional-seconds-preserving, nanos-based)
+query
+SELECT CAST(TIME '13:45:07.123456' AS DECIMAL(18,6))
+
+-- TIME -> TIME with different precision (truncates fractional digits)
+query
+SELECT CAST(TIME '13:45:07.123456' AS TIME(0))
+
+query
+SELECT CAST(TIME '13:45:07.123456' AS TIME(3))
+
+-- NULL round-trips
+query
+SELECT CAST(CAST(NULL AS STRING) AS TIME)
+
+query
+SELECT CAST(CAST(NULL AS TIME) AS STRING)
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/current_time.sql b/spark/src/test/resources/sql-tests/expressions/datetime/current_time.sql
new file mode 100644
index 0000000000..328160b654
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/current_time.sql
@@ -0,0 +1,36 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- current_time() is CodegenFallback but foldable; Spark's optimizer folds it to a Time
+-- literal before Comet sees the plan, so it never actually hits the codegen dispatcher.
+-- These tests pin behavior rather than a specific timestamp value.
+
+query
+SELECT current_time() IS NOT NULL
+
+query
+SELECT current_time(0) IS NOT NULL
+
+query
+SELECT current_time(6) IS NOT NULL
+
+-- Two calls in the same query return the same value
+query
+SELECT current_time() = current_time()
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/hour_time.sql b/spark/src/test/resources/sql-tests/expressions/datetime/hour_time.sql
new file mode 100644
index 0000000000..6533c97d5b
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/hour_time.sql
@@ -0,0 +1,51 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- hour(TIME) rewrites to HoursOfTime -> StaticInvoke(DateTimeUtils.getHoursOfTime). Comet has
+-- no native handler for it, so CometStaticInvoke routes it through the JVM codegen dispatcher.
+
+statement
+CREATE TABLE test_hour_time(h int, m int, s decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_hour_time VALUES
+ (0, 0, 0.000000),
+ (1, 2, 3.500000),
+ (12, 30, 45.123456),
+ (23, 59, 59.999999),
+ (NULL, 0, 0.000000)
+
+-- column argument built via make_time
+query expect_dispatch(staticinvoke)
+SELECT hour(make_time(h, m, s)) FROM test_hour_time
+
+-- literal TIME arguments
+query expect_dispatch(staticinvoke)
+SELECT hour(TIME '00:00:00')
+
+query expect_dispatch(staticinvoke)
+SELECT hour(TIME '13:45:00')
+
+query expect_dispatch(staticinvoke)
+SELECT hour(TIME '23:59:59.999999')
+
+-- null TIME
+query
+SELECT hour(CAST(NULL AS TIME))
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/minute_time.sql b/spark/src/test/resources/sql-tests/expressions/datetime/minute_time.sql
new file mode 100644
index 0000000000..73ae551ea2
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/minute_time.sql
@@ -0,0 +1,48 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- minute(TIME) rewrites to MinutesOfTime -> StaticInvoke(DateTimeUtils.getMinutesOfTime). Comet
+-- has no native handler for it, so CometStaticInvoke routes it through the JVM codegen dispatcher.
+
+statement
+CREATE TABLE test_minute_time(h int, m int, s decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_minute_time VALUES
+ (0, 0, 0.000000),
+ (1, 2, 3.500000),
+ (12, 30, 45.123456),
+ (23, 59, 59.999999),
+ (0, NULL, 0.000000)
+
+query expect_dispatch(staticinvoke)
+SELECT minute(make_time(h, m, s)) FROM test_minute_time
+
+query expect_dispatch(staticinvoke)
+SELECT minute(TIME '00:00:00')
+
+query expect_dispatch(staticinvoke)
+SELECT minute(TIME '13:45:00')
+
+query expect_dispatch(staticinvoke)
+SELECT minute(TIME '23:59:59.999999')
+
+query
+SELECT minute(CAST(NULL AS TIME))
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/second_time.sql b/spark/src/test/resources/sql-tests/expressions/datetime/second_time.sql
new file mode 100644
index 0000000000..d58f01ac99
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/second_time.sql
@@ -0,0 +1,49 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- second(TIME) rewrites to SecondsOfTime -> StaticInvoke(DateTimeUtils.getSecondsOfTime). Comet
+-- has no native handler for it, so CometStaticInvoke routes it through the JVM codegen dispatcher.
+-- EXTRACT(SECOND FROM TIME) is a different expression that runs natively (extract_time.sql).
+
+statement
+CREATE TABLE test_second_time(h int, m int, s decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_second_time VALUES
+ (0, 0, 0.000000),
+ (1, 2, 3.500000),
+ (12, 30, 45.123456),
+ (23, 59, 59.999999),
+ (0, 0, NULL)
+
+query expect_dispatch(staticinvoke)
+SELECT second(make_time(h, m, s)) FROM test_second_time
+
+query expect_dispatch(staticinvoke)
+SELECT second(TIME '00:00:00')
+
+query expect_dispatch(staticinvoke)
+SELECT second(TIME '13:45:07')
+
+query expect_dispatch(staticinvoke)
+SELECT second(TIME '23:59:59.999999')
+
+query
+SELECT second(CAST(NULL AS TIME))
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/subtract_times.sql b/spark/src/test/resources/sql-tests/expressions/datetime/subtract_times.sql
new file mode 100644
index 0000000000..3c4b429cfe
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/subtract_times.sql
@@ -0,0 +1,42 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- time - time rewrites to SubtractTimes -> StaticInvoke(DateTimeUtils.subtractTimes), which
+-- returns DayTimeIntervalType(HOUR, SECOND). Comet has no native handler for it, so
+-- CometStaticInvoke routes it through the JVM codegen dispatcher.
+
+statement
+CREATE TABLE test_time_sub(h1 int, m1 int, s1 decimal(16,6), h2 int, m2 int, s2 decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_time_sub VALUES
+ (1, 0, 0.000000, 2, 15, 30.000000),
+ (10, 0, 0.000000, 9, 59, 59.500000),
+ (0, 0, 0.000000, 23, 59, 59.999999),
+ (NULL, 0, 0.000000, 0, 0, 0.000000)
+
+query expect_dispatch(staticinvoke)
+SELECT make_time(h2, m2, s2) - make_time(h1, m1, s1) FROM test_time_sub
+
+query expect_dispatch(staticinvoke)
+SELECT TIME '02:15:30' - TIME '01:00:00'
+
+query expect_dispatch(staticinvoke)
+SELECT TIME '09:59:59.5' - TIME '10:00:00'
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/time_add_interval.sql b/spark/src/test/resources/sql-tests/expressions/datetime/time_add_interval.sql
new file mode 100644
index 0000000000..10d874781f
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/time_add_interval.sql
@@ -0,0 +1,68 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- time + day-time interval rewrites to TimeAddInterval ->
+-- StaticInvoke(DateTimeUtils.timeAddInterval). Comet has no native handler for it, so
+-- CometStaticInvoke routes it through the JVM codegen dispatcher.
+
+statement
+CREATE TABLE test_time_add(h int, m int, s decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_time_add VALUES
+ (0, 0, 0.000000),
+ (10, 15, 30.000000),
+ (22, 30, 0.000000),
+ (12, 34, 56.789012),
+ (NULL, 0, 0.000000)
+
+-- add an hour to a TIME column
+query expect_dispatch(staticinvoke)
+SELECT make_time(h, m, s) + INTERVAL '1' HOUR FROM test_time_add
+
+-- add fractional-second interval
+query expect_dispatch(staticinvoke)
+SELECT make_time(h, m, s) + INTERVAL '0.500' SECOND FROM test_time_add
+
+-- literal TIME + literal interval
+query expect_dispatch(staticinvoke)
+SELECT TIME '00:00:00' + INTERVAL '1' HOUR
+
+query expect_dispatch(staticinvoke)
+SELECT TIME '22:30:00' + INTERVAL '30' MINUTE
+
+query expect_dispatch(staticinvoke)
+SELECT TIME '10:00:00' + INTERVAL '0.5' SECOND
+
+-- NULL TIME
+query
+SELECT CAST(NULL AS TIME) + INTERVAL '1' HOUR
+
+-- NULL interval
+query
+SELECT TIME '10:00:00' + CAST(NULL AS INTERVAL HOUR)
+
+-- an interval computed from a column rather than a literal
+query expect_dispatch(staticinvoke)
+SELECT make_time(h, m, s) + make_dt_interval(0, 0, m, 0) FROM test_time_add
+
+-- a result past midnight raises Spark's overflow error
+query expect_error(DATETIME_OVERFLOW)
+SELECT make_time(h, m, s) + INTERVAL '2' HOUR FROM test_time_add WHERE h = 22
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/time_diff.sql b/spark/src/test/resources/sql-tests/expressions/datetime/time_diff.sql
new file mode 100644
index 0000000000..9565cadfaa
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/time_diff.sql
@@ -0,0 +1,65 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- time_diff(unit, start, end) rewrites to TimeDiff -> StaticInvoke(DateTimeUtils.timeDiff), which
+-- returns LongType. Comet has no native handler for it, so CometStaticInvoke routes it through the
+-- JVM codegen dispatcher.
+
+statement
+CREATE TABLE test_time_diff(h1 int, m1 int, s1 decimal(16,6), h2 int, m2 int, s2 decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_time_diff VALUES
+ (20, 30, 29.000000, 21, 30, 29.000000),
+ (0, 0, 0.000000, 0, 0, 1.000000),
+ (20, 30, 29.000000, 12, 0, 0.000000),
+ (0, 0, 0.000000, 23, 59, 59.999999),
+ (NULL, 0, 0.000000, 0, 0, 0.000000)
+
+-- column arguments across all supported units
+query expect_dispatch(staticinvoke)
+SELECT time_diff('HOUR', make_time(h1, m1, s1), make_time(h2, m2, s2)) FROM test_time_diff
+
+query expect_dispatch(staticinvoke)
+SELECT time_diff('MINUTE', make_time(h1, m1, s1), make_time(h2, m2, s2)) FROM test_time_diff
+
+query expect_dispatch(staticinvoke)
+SELECT time_diff('SECOND', make_time(h1, m1, s1), make_time(h2, m2, s2)) FROM test_time_diff
+
+query expect_dispatch(staticinvoke)
+SELECT time_diff('MILLISECOND', make_time(h1, m1, s1), make_time(h2, m2, s2)) FROM test_time_diff
+
+query expect_dispatch(staticinvoke)
+SELECT time_diff('MICROSECOND', make_time(h1, m1, s1), make_time(h2, m2, s2)) FROM test_time_diff
+
+-- literal TIME arguments
+query expect_dispatch(staticinvoke)
+SELECT time_diff('HOUR', TIME '20:30:29', TIME '21:30:29')
+
+query expect_dispatch(staticinvoke)
+SELECT time_diff('SECOND', TIME '00:00:00', TIME '23:59:59.999999')
+
+-- negative difference (start > end)
+query expect_dispatch(staticinvoke)
+SELECT time_diff('HOUR', TIME '20:30:29', TIME '12:00:00')
+
+-- lowercase unit
+query expect_dispatch(staticinvoke)
+SELECT time_diff('second', TIME '00:00:00', TIME '00:00:01')
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/time_trunc.sql b/spark/src/test/resources/sql-tests/expressions/datetime/time_trunc.sql
new file mode 100644
index 0000000000..17e6d2f7dd
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/time_trunc.sql
@@ -0,0 +1,62 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.1
+-- Config: spark.sql.timeType.enabled=true
+
+-- time_trunc(unit, time) rewrites to TimeTrunc -> StaticInvoke(DateTimeUtils.timeTrunc). Comet
+-- has no native handler for it, so CometStaticInvoke routes it through the JVM codegen dispatcher.
+
+statement
+CREATE TABLE test_time_trunc(h int, m int, s decimal(16,6)) USING parquet
+
+statement
+INSERT INTO test_time_trunc VALUES
+ (9, 32, 5.359000),
+ (23, 59, 59.999999),
+ (0, 0, 0.000000),
+ (12, 34, 56.123456),
+ (NULL, 0, 0.000000)
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('HOUR', make_time(h, m, s)) FROM test_time_trunc
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('MINUTE', make_time(h, m, s)) FROM test_time_trunc
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('SECOND', make_time(h, m, s)) FROM test_time_trunc
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('MILLISECOND', make_time(h, m, s)) FROM test_time_trunc
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('MICROSECOND', make_time(h, m, s)) FROM test_time_trunc
+
+-- literal TIME arguments
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('HOUR', TIME '09:32:05.359')
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('MILLISECOND', TIME '09:32:05.123456')
+
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('SECOND', TIME '23:59:59.999999')
+
+-- lowercase unit
+query expect_dispatch(staticinvoke)
+SELECT time_trunc('hour', TIME '13:45:07.999999')
diff --git a/spark/src/test/scala/org/apache/comet/csv/CometCsvNativeReadSuite.scala b/spark/src/test/scala/org/apache/comet/csv/CometCsvNativeReadSuite.scala
index 7ac8faddaf..6e743cfc90 100644
--- a/spark/src/test/scala/org/apache/comet/csv/CometCsvNativeReadSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/csv/CometCsvNativeReadSuite.scala
@@ -23,10 +23,12 @@ import java.nio.charset.StandardCharsets
import java.nio.file.Files
import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.comet.CometCsvNativeScanExec
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{IntegerType, StringType, StructType, TimestampType}
-import org.apache.comet.CometConf
+import org.apache.comet.{CometConf, ExtendedExplainInfo}
+import org.apache.comet.CometSparkSessionExtensions.isSpark41Plus
class CometCsvNativeReadSuite extends CometTestBase {
private val TEST_CSV_PATH_NO_HEADER = "src/test/resources/test-data/csv-test-1.csv"
@@ -119,4 +121,30 @@ class CometCsvNativeReadSuite extends CometTestBase {
}
}
}
+
+ test("Native csv read - TIME columns fall back") {
+ assume(isSpark41Plus, "TimeType requires Spark 4.1+")
+ withTempDir { dir =>
+ Files.write(
+ dir.toPath.resolve("part-0.csv"),
+ "t\n12:34:56\n".getBytes(StandardCharsets.UTF_8))
+ withSQLConf(
+ "spark.sql.timeType.enabled" -> "true",
+ CometConf.COMET_CSV_V2_NATIVE_ENABLED.key -> "true",
+ SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+ // Spark 4.1's CSV reader rejects TIME when it reads the file, and Spark 4.2 parses it with
+ // the `timeFormat` option, which the native reader does not take. Either way the scan has
+ // to stay in Spark. Checked on the plan because Spark 4.1 has no answer to compare with.
+ val plan = spark.read
+ .option("header", "true")
+ .schema("t TIME")
+ .csv(dir.toString)
+ .queryExecution
+ .executedPlan
+ assert(plan.collectFirst { case s: CometCsvNativeScanExec => s }.isEmpty, plan)
+ val reasons = new ExtendedExplainInfo().getFallbackReasons(plan)
+ assert(reasons.exists(_.contains("Unsupported t of type TimeType(6)")), reasons)
+ }
+ }
+ }
}
diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
index 5b78b0badf..82407881b8 100644
--- a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
@@ -4578,7 +4578,7 @@ class CometExecSuite extends CometTestBase {
}
}
- test("CometLocalTableScanExec falls back when schema contains TimeType") {
+ test("CometLocalTableScanExec handles TimeType column") {
assume(
org.apache.comet.CometSparkSessionExtensions.isSpark41Plus,
"TimeType requires Spark 4.1+")
@@ -4586,11 +4586,22 @@ class CometExecSuite extends CometTestBase {
// row encoder accepts TIME (matches Spark's own TimeFunctionsSuiteBase setup).
withSQLConf(
"spark.sql.timeType.enabled" -> "true",
- CometConf.COMET_EXEC_LOCAL_TABLE_SCAN_ENABLED.key -> "true") {
- // VALUES folds to a LocalRelation, exercising the CometLocalTableScanExec convert
- // path; the TimeType column should drive the schema-level fallback.
- val df = spark.sql("SELECT * FROM VALUES (TIME '12:34:56'), (TIME '01:02:03') AS t(c)")
- checkSparkAnswer(df)
+ CometConf.COMET_EXEC_LOCAL_TABLE_SCAN_ENABLED.key -> "true",
+ // Two rows to a batch, so each scan below writes more than one batch.
+ CometConf.COMET_BATCH_SIZE.key -> "2") {
+ // VALUES folds to a LocalRelation, exercising the CometLocalTableScanExec convert path.
+ // TimeType routes through TimeNanoWriter, so the native scan handles it end-to-end.
+ Seq(
+ "SELECT * FROM VALUES (TIME '12:34:56'), (TIME '01:02:03'), (NULL) AS t(c)",
+ // a precision below the default
+ "SELECT * FROM VALUES (CAST(TIME '12:34:56.789' AS TIME(3))), (NULL), " +
+ "(CAST(TIME '00:00:00' AS TIME(3))) AS t(c)",
+ // TIME inside an array and a struct, written by the nested writers
+ "SELECT * FROM VALUES (array(TIME '01:02:03', CAST(NULL AS TIME))), " +
+ "(CAST(NULL AS ARRAY<TIME>)), (array(TIME '23:59:59.999999')) AS t(a)",
+ "SELECT * FROM VALUES (named_struct('x', TIME '01:02:03')), " +
+ "(CAST(NULL AS STRUCT<x: TIME>)), (named_struct('x', CAST(NULL AS TIME))) AS t(s)")
+ .foreach(query => checkSparkAnswerAndOperator(spark.sql(query)))
}
}
diff --git a/spark/src/test/scala/org/apache/comet/serde/QueryPlanSerdeSuite.scala b/spark/src/test/scala/org/apache/comet/serde/QueryPlanSerdeSuite.scala
index e973bbdee3..19476f8e13 100644
--- a/spark/src/test/scala/org/apache/comet/serde/QueryPlanSerdeSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/serde/QueryPlanSerdeSuite.scala
@@ -64,7 +64,6 @@ class QueryPlanSerdeSuite extends AnyFunSuite {
_,
allowComplex = true,
allowIntervals = true,
- allowTimeType = false,
allowAnyStringType = false),
Seq(
IntegerType,
@@ -72,8 +71,8 @@ class QueryPlanSerdeSuite extends AnyFunSuite {
complex,
nestedInterval,
nestedCalendarInterval,
- duplicateFields),
- Seq(emptyStruct) ++ timeTypes ++ collatedStrings),
+ duplicateFields) ++ timeTypes,
+ Seq(emptyStruct) ++ collatedStrings),
(
"native shuffle",
supportedDataType(_, allowComplex = true, allowIntervals = true),
diff --git a/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowWriterSuite.scala b/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowWriterSuite.scala
index 84a504a8ad..294f915298 100644
--- a/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowWriterSuite.scala
+++ b/spark/src/test/scala/org/apache/spark/sql/comet/execution/arrow/CometArrowWriterSuite.scala
@@ -37,6 +37,8 @@ import org.apache.spark.sql.execution.vectorized.{ConstantColumnVector, Dictiona
import org.apache.spark.sql.types._
import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector}
+import org.apache.comet.CometSparkSessionExtensions.isSpark41Plus
+
/**
* The columnar paths of [[ArrowWriter]] copy Spark's vectors in bulk where they can. Whatever
* they take, they must produce what the row path writes for the same rows, which reads every
@@ -46,6 +48,12 @@ class CometArrowWriterSuite extends AnyFunSuite with Matchers {
private val numRows = 300
+ private val nanosPerDay = 24L * 60 * 60 * 1000 * 1000 * 1000
+
+ // `TimeType` only exists from Spark 4.1, so it is parsed from DDL rather than named.
+ private val timeTypes: Seq[DataType] =
+ if (isSpark41Plus) Seq(DataType.fromDDL("TIME")) else Seq.empty
+
private val primitiveTypes: Seq[DataType] = Seq(
BooleanType,
ByteType,
@@ -63,7 +71,7 @@ class CometArrowWriterSuite extends AnyFunSuite with Matchers {
TimestampType,
TimestampNTZType,
YearMonthIntervalType(),
- DayTimeIntervalType())
+ DayTimeIntervalType()) ++ timeTypes
private val nestedTypes: Seq[DataType] = Seq(
ArrayType(IntegerType),
@@ -109,6 +117,8 @@ class CometArrowWriterSuite extends AnyFunSuite with Matchers {
case DoubleType => v.putDouble(row, rnd.nextDouble())
case StringType | BinaryType => v.putByteArray(row, randomBytes(rnd))
case dt: DecimalType => v.putDecimal(row, randomDecimal(rnd, dt), dt.precision)
+ case dt if Utils.isTimeType(dt) =>
+ v.putLong(row, Math.floorMod(rnd.nextLong(), nanosPerDay))
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org