You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/16 07:50:50 UTC
(datafusion-comet) branch main updated (41d6d44838 -> 8c229a703c)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
from 41d6d44838 test: guard native plan equality against omitted parameters (#5953)
new e83d2712d4 perf: skip calendar reconstruction in hour/minute/second and dayofweek/weekday (#5771)
new 8c229a703c test: expand replace compatibility regression coverage (#5409)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../expression-audits/datetime_funcs.md | 20 ++
native/spark-expr/Cargo.toml | 4 +
native/spark-expr/benches/common/mod.rs | 9 +
native/spark-expr/benches/dayofweek_weekday.rs | 204 +++++++++++++++
native/spark-expr/benches/extract_date_part.rs | 265 ++++++++++++++++---
native/spark-expr/src/comet_scalar_funcs.rs | 9 +-
.../spark-expr/src/datetime_funcs/day_of_week.rs | 287 +++++++++++++++++++++
.../src/datetime_funcs/extract_date_part.rs | 174 ++++++++++++-
native/spark-expr/src/datetime_funcs/mod.rs | 2 +
native/spark-expr/src/lib.rs | 6 +-
.../scala/org/apache/comet/serde/datetime.scala | 65 ++---
.../sql-tests/expressions/datetime/hour.sql | 10 +-
.../sql-tests/expressions/datetime/minute.sql | 10 +-
.../sql-tests/expressions/datetime/second.sql | 10 +-
.../expressions/string/string_replace.sql | 22 +-
.../scala/org/apache/comet/CometCodegenSuite.scala | 118 +++++++++
.../org/apache/comet/CometExpressionSuite.scala | 26 ++
17 files changed, 1138 insertions(+), 103 deletions(-)
create mode 100644 native/spark-expr/benches/dayofweek_weekday.rs
create mode 100644 native/spark-expr/src/datetime_funcs/day_of_week.rs
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/02: perf: skip calendar reconstruction in hour/minute/second and dayofweek/weekday (#5771)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit e83d2712d432f957c123bae07d39ecfa775ca209
Author: Peter Lee <pe...@gmail.com>
AuthorDate: Wed Sep 16 00:16:07 2026 +0000
perf: skip calendar reconstruction in hour/minute/second and dayofweek/weekday (#5771)
* perf: skip calendar reconstruction in hour/minute/second and dayofweek/weekday
Both paths rebuilt a chrono calendar datetime per row for a value that is a pure
function of the stored microseconds or the epoch day.
hour/minute/second take an integer fast path when no timezone offset applies:
TimestampNTZ, or a timezone-aware timestamp in a UTC session (array_with_timezone
only re-tags such an array, so the stored value is the UTC instant either way).
Dictionaries, non-microsecond units and offset timezones keep the existing
date_part path. Euclidean division is required so pre-epoch instants, where -1us
is 1969-12-31 23:59:59.999999, do not truncate toward zero.
dayofweek/weekday move to native spark_dayofweek / spark_weekday kernels over
Date32, replacing datepart('dow'|'isodow', ..) plus a separate +1 / -1 arithmetic
node. arrow's date_part builds a NaiveDateTime per row via unary_opt and
recomputes the null mask; both collapse to one modulo of the epoch day.
Measured on 8192-row batches, release profile, aarch64, two samples: -83% to -96%
for the clock fields and 8.4x to 10.2x for dayofweek/weekday, with no regression
on the offset-timezone path.
One behaviour change: the old dayofweek/weekday path returned NULL for epoch days
outside chrono's range; the kernel is correct across the full i32 domain. Spark's
date range does not reach those values.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* docs: record the datetime extraction performance audit
Adds dated Performance lines for hour, minute, second, dayofweek and weekday on
the datetime_funcs audit page, per the optimize-comet-expression workflow, so the
next contributor can see these have been tuned and what technique was used.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* test: set the parquet rebase mode so the ancient date writes on Spark 3.x
The new dayofweek/weekday test inserts date('0001-01-01'), which predates the
Gregorian cutover. Spark 3.x throws WRITE_ANCIENT_DATETIME when writing such a
value to parquet, so the test failed on the Spark 3.4 and 3.5 CI jobs while
passing locally, where the default profile is Spark 4.1 and the rebase mode
already defaults to CORRECTED.
CORRECTED writes the value as-is in the proleptic Gregorian calendar, which is the
calendar the results are compared in, so it does not change what dayofweek or
weekday return. CometIcebergSystemFunctionSuite sets the same conf for the same
reason.
Verified with -Pspark-3.5, the profile that reproduced the failure.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* perf: keep dense-null and low-cardinality dictionary inputs off the slow path
Review of #5771 asked for dense-null and dictionary shapes in the no-regression
comparison. Adding them found two real regressions, both now fixed.
Arrow's `unary_opt`, which the replaced `date_part` path uses, visits only valid
indices via `try_for_each_valid_idx`, while `unary` evaluates every slot. An
all-null batch was therefore nearly free before and was not after: dayofweek on a
flat all-null array measured 0.30x, a 3.3x slowdown. `map_dates` and the clock
fast path now switch on null density. The threshold is half the batch rather than
the mere presence of a null, because `unary` was still ahead at 87.5% nulls and
switching on any null gave up most of the win on a lightly-null column
(sparse dropped from 8.97x to 3.45x before the threshold was tuned).
Arrow's `date_part` maps a dictionary's values and rewraps the keys, so the old
chain's calendar work was proportional to cardinality. Decoding to Date32 first
made a cardinality-8 column about 2x slower (0.51x). The weekday kernels now map
the dictionary values and rewrap, then unpack, matching the old cost profile with
a cheaper kernel; cardinality 1024 improves from 1.19x to about 2.1x as a result.
`map_dates` takes a generic `F: Fn(i32) -> i32 + Copy` rather than a `fn` pointer:
a pointer forces an indirect call per element inside `unary` and cost about 2.5x
(no-null dayofweek fell from 9.0x to 3.7x before this was corrected).
Benchmarks: both targets gain dense-null and all-null shapes plus dictionary
inputs at cardinality 8 and 1024, comparing the real UDF against the complete old
chain including the unpacking cast. The old-path arm also had `Cast` and `Add`
in the wrong order; the serde emitted `Add(Cast(datepart(..), Int32), 1)`, and
arrow rejects `Dictionary(Int32, Int32) + Int32`, so the dictionary comparison
could not have been valid before. The clock generator spanned only 1965-1966
despite its mixed-epoch description and now walks 1875 to 2064 across the epoch.
Equivalence assertions compare nullability per row on every shape.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* test: pin the session timezone for hour/minute/second, fold the clock benchmark in
Review of #5771 raised three things.
Nothing pinned `spark.sql.session.timeZone` for these expressions, so which branch
of the dispatch ran was decided by the ambient JVM zone of whichever runner
executed the suite: on a UTC runner every plain timestamp column took the new
path and the shift-to-session-zone branch was reached only through dictionaries.
hour.sql, minute.sql and second.sql now carry
`-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles`, which runs
each file under both zones, and each gained a 1969-12-31 23:59:59 row so the
Euclidean division is covered end to end rather than only in a Rust unit test.
benches/extract_date_part.rs already covered SparkHour/SparkMinute/SparkSecond, so
the separate extract_clock_fields.rs added here was a second file for the same
three UDFs with its own generator and null-density convention. The new shapes are
folded into the existing file and the duplicate is deleted. It now uses the shared
common/ helpers, sweeps both session zones so each dispatch branch is measured,
adds a dense null ratio alongside NULL_RATIOS, adds dictionary inputs at
cardinality 8 and 1024, and generates instants spanning the epoch rather than a
fixed 2020 base. Upstream's existing shape ids are preserved under the
la_session prefix, with the full ROW_COUNTS sweep kept there and the additions at
8192 rows to bound the matrix.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* fix: make the dense null shape actually dense, and restore the clock equivalence checks
Review of #5771 found that `DENSE_NULLS = (0.875, "dense")` degenerated to an
all-null shape. `common::is_null` derives a stride as `round(1 / ratio)`, which is
1 for any ratio above two thirds, so every row came back null: 8192 nulls and zero
valid rows rather than the intended 7168 and 1024. The dense clock cases were
therefore a second copy of the all-null case and never exercised valid values in a
mostly-null batch.
`is_null` now takes the stride over the valid rows when the ratio is above half,
so 0.875 yields one valid row in eight. Ratios at or below half are unchanged, and
no existing caller passes a ratio in that range -- `NULL_RATIOS` is 0.0, 0.1 and
1.0 -- so no other benchmark's shapes move. `assert_density` checks the realised
null count against the requested ratio before timing, so a shape that silently
degenerates fails instead of being measured.
Folding the clock benchmark into extract_date_part.rs also dropped
`assert_matches_arithmetic` without replacing it, leaving that harness with no
pre-timing comparison, which the PR body incorrectly claimed it had. Every clock
shape now gets a null-aware check: the result must be null exactly where the input
is and every value must sit inside the field's range. Shapes eligible for the fast
path additionally get an exact oracle, recomputed from the generator rather than
from arrow, so it validates the baseline implementation as well as this one.
Dictionary and offset-session controls get the null-aware half, since no
arithmetic oracle applies to them.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
* docs: point the clock audit entries at the surviving benchmark, correct stale figures
Review of #5771 noted three leftovers.
The hour, minute and second entries on the datetime audit page still named
benches/extract_clock_fields.rs, which this PR folded into
benches/extract_date_part.rs and deleted. They also quoted "83-96% faster" from
before the harness repair; the matched results put every fast-path shape with a
baseline of at least 2 us at 82-94% less time, with offset session zones and
dictionary input on the general path and within noise on a repeat.
The dayofweek and weekday entries still carried "8.4-10.2x faster" from before
the dense-null and dictionary work. The final figures are about 9x on flat input
with no or sparse nulls, 3-4x at 87.5% nulls, roughly 1.1x on an all-null batch
or a cardinality-8 dictionary where the replaced path already did little work,
and about 2x at cardinality 1024. The audit page is what the next contributor
reads instead of the PR body, so it should carry the numbers that held up.
Removing `val DayOfWeek = Value("dow")` from CometGetDateField left its two
comment lines behind, so the Sunday-is-0 note sat above `val DayOfYear`.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <no...@anthropic.com>
Co-authored-by: Andy Grove <ag...@apache.org>
---
.../expression-audits/datetime_funcs.md | 20 ++
native/spark-expr/Cargo.toml | 4 +
native/spark-expr/benches/common/mod.rs | 9 +
native/spark-expr/benches/dayofweek_weekday.rs | 204 +++++++++++++++
native/spark-expr/benches/extract_date_part.rs | 265 ++++++++++++++++---
native/spark-expr/src/comet_scalar_funcs.rs | 9 +-
.../spark-expr/src/datetime_funcs/day_of_week.rs | 287 +++++++++++++++++++++
.../src/datetime_funcs/extract_date_part.rs | 174 ++++++++++++-
native/spark-expr/src/datetime_funcs/mod.rs | 2 +
native/spark-expr/src/lib.rs | 6 +-
.../scala/org/apache/comet/serde/datetime.scala | 65 ++---
.../sql-tests/expressions/datetime/hour.sql | 10 +-
.../sql-tests/expressions/datetime/minute.sql | 10 +-
.../sql-tests/expressions/datetime/second.sql | 10 +-
.../org/apache/comet/CometExpressionSuite.scala | 26 ++
15 files changed, 999 insertions(+), 102 deletions(-)
diff --git a/docs/source/contributor-guide/expression-audits/datetime_funcs.md b/docs/source/contributor-guide/expression-audits/datetime_funcs.md
index 9ff3eb3a98..b285c4067b 100644
--- a/docs/source/contributor-guide/expression-audits/datetime_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/datetime_funcs.md
@@ -37,6 +37,10 @@
- Spark 4.0+. Implemented natively: maps a `DateType` value to a fixed US-English abbreviated day name (`DayOfWeek.getDisplayName(TextStyle.SHORT, Locale.US)`), with no session-locale or timezone dependence.
+## dayofweek
+
+- Performance (tuned 2026-09-08, PR [#5771](https://github.com/apache/datafusion-comet/pull/5771)): computed directly from the epoch day (`((days + 4).rem_euclid(7) + 1)`) by the native `spark_dayofweek` kernel, replacing `datepart('dow', ..)` -- which reconstructs a `NaiveDateTime` per row and recomputes the null mask via `unary_opt` -- plus a separate `+ 1` arithmetic node. About 9x faster on flat input with no or sparse nulls, 3-4x at 87.5% nulls, and roughly 1.1x on an all-null batch [...]
+
## from_utc_timestamp
- Spark 3.4.3 (audited 2026-05-12): identical to 3.5.8.
@@ -44,6 +48,10 @@
- Spark 4.0.1 (audited 2026-05-12): `inputTypes` widened to `StringTypeWithCollation`; behaviour unchanged for ASCII timezone strings.
- Marked `Incompatible`: Comet's native timezone parser only accepts IANA zone IDs (e.g. `America/Los_Angeles`) and fixed `+HH:MM` offsets, while Spark also accepts legacy forms (`GMT+1`, `UTC+1`, three-letter abbreviations like `PST`). By default it runs through the codegen dispatcher (Spark-correct) and uses the native path only when incompatible expressions are explicitly allowed, where legacy zone forms throw a native parse error at execution.
+## hour
+
+- Performance (tuned 2026-09-08, PR [#5771](https://github.com/apache/datafusion-comet/pull/5771)): `hour`/`minute`/`second` take an integer fast path when no timezone offset applies -- `TimestampNTZ`, or a timezone-aware timestamp in a UTC session -- computing the field from the stored microseconds with Euclidean division instead of building a `chrono` datetime per row via `date_part`. Dictionaries, non-microsecond units and offset timezones keep the general path. Every fast-path shape [...]
+
## make_timestamp_ltz
- The 6-argument form rewrites to `MakeTimestamp` and runs via the codegen dispatcher. The 2-argument `(date, time)` form requires the Spark 4.1 TIME type and falls back.
@@ -52,6 +60,10 @@
- The 6-argument form rewrites to `MakeTimestamp` and runs via the codegen dispatcher. The 2-argument `(date, time)` form requires the Spark 4.1 TIME type and falls back.
+## minute
+
+- Performance (tuned 2026-09-08, PR [#5771](https://github.com/apache/datafusion-comet/pull/5771)): `hour`/`minute`/`second` take an integer fast path when no timezone offset applies -- `TimestampNTZ`, or a timezone-aware timestamp in a UTC session -- computing the field from the stored microseconds with Euclidean division instead of building a `chrono` datetime per row via `date_part`. Dictionaries, non-microsecond units and offset timezones keep the general path. Every fast-path shape [...]
+
## monthname
- Spark 4.0+. Implemented natively: maps a `DateType` value to a fixed US-English abbreviated month name (`Month.getDisplayName(TextStyle.SHORT, Locale.US)`), with no session-locale or timezone dependence.
@@ -60,6 +72,10 @@
- Alias of `current_timestamp`; constant-folded to a literal by Spark's `ComputeCurrentTime` rule before Comet sees the plan.
+## second
+
+- Performance (tuned 2026-09-08, PR [#5771](https://github.com/apache/datafusion-comet/pull/5771)): `hour`/`minute`/`second` take an integer fast path when no timezone offset applies -- `TimestampNTZ`, or a timezone-aware timestamp in a UTC session -- computing the field from the stored microseconds with Euclidean division instead of building a `chrono` datetime per row via `date_part`. Dictionaries, non-microsecond units and offset timezones keep the general path. Every fast-path shape [...]
+
## to_date
- Rewrites to `Cast` (no format, native) or `Cast(GetTimestamp(...))` (with format, via the codegen dispatcher) before Comet sees the plan.
@@ -96,3 +112,7 @@
- Rewrites to `Cast(..., EvalMode.LEGACY)` (no format, native) or `GetTimestamp(..., failOnError = false)` (with format, via the codegen dispatcher) before Comet sees the plan. In non-ANSI mode the rewritten tree is identical to `to_timestamp`; invalid inputs return NULL to match Spark.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
+
+## weekday
+
+- Performance (tuned 2026-09-08, PR [#5771](https://github.com/apache/datafusion-comet/pull/5771)): computed directly from the epoch day (`(days + 3).rem_euclid(7)`) by the native `spark_weekday` kernel, replacing `datepart('isodow', ..)` plus a `- 1` arithmetic node. About 9x faster on flat input with no or sparse nulls, 3-4x at 87.5% nulls, and roughly 1.1x on an all-null batch or a cardinality-8 dictionary, where the replaced path already did little work (about 2x at cardinality 1024) [...]
diff --git a/native/spark-expr/Cargo.toml b/native/spark-expr/Cargo.toml
index 9019dffbb5..346265f863 100644
--- a/native/spark-expr/Cargo.toml
+++ b/native/spark-expr/Cargo.toml
@@ -402,3 +402,7 @@ harness = false
[[bench]]
name = "iceberg_transforms"
harness = false
+
+[[bench]]
+name = "dayofweek_weekday"
+harness = false
diff --git a/native/spark-expr/benches/common/mod.rs b/native/spark-expr/benches/common/mod.rs
index cc0c6f275f..8d702782ee 100644
--- a/native/spark-expr/benches/common/mod.rs
+++ b/native/spark-expr/benches/common/mod.rs
@@ -37,11 +37,20 @@ pub const ROW_COUNTS: [usize; 3] = [8_192, 65_536, 524_288];
pub const NULL_RATIOS: [(f64, &str); 3] = [(0.0, "no_nulls"), (0.1, "sparse"), (1.0, "all_null")];
+/// Whether row `i` is null for the requested null ratio.
+///
+/// The density is approximated by a stride. Below half, every `round(1/ratio)`-th row is null;
+/// above half the stride is taken over the *valid* rows instead, because `round(1/ratio)`
+/// collapses to 1 for any ratio above 2/3 and would make every row null. A dense ratio such as
+/// 0.875 therefore yields one valid row in eight rather than a second all-null shape.
pub fn is_null(i: usize, null_ratio: f64) -> bool {
if null_ratio <= 0.0 {
false
} else if null_ratio >= 1.0 {
true
+ } else if null_ratio > 0.5 {
+ let stride = (1.0 / (1.0 - null_ratio)).round() as usize;
+ !(stride != 0 && i.is_multiple_of(stride))
} else {
let stride = (1.0 / null_ratio).round() as usize;
stride != 0 && i.is_multiple_of(stride)
diff --git a/native/spark-expr/benches/dayofweek_weekday.rs b/native/spark-expr/benches/dayofweek_weekday.rs
new file mode 100644
index 0000000000..136135e593
--- /dev/null
+++ b/native/spark-expr/benches/dayofweek_weekday.rs
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `dayofweek` / `weekday` over `Date32`, comparing the chain Comet serializes today against a
+//! direct integer kernel on the epoch day.
+//!
+//! The `datepart_*` arms reproduce the chain the serde emitted before this change:
+//! `datepart('dow', child) + 1` for `dayofweek` and `datepart('isodow', child) - 1` for
+//! `weekday`. The `native_*` arms invoke the kernels the serde emits now. Both arms run in the
+//! same process, so this is a direct comparison rather than a cross-run baseline (the native
+//! kernels do not exist on `main`, so a saved baseline could not build them). `datepart` resolves to DataFusion's `date_part`, which for
+//! `Date32` runs `unary_opt(|d| date32_to_datetime(d).map(..))` -- a `NaiveDateTime` per row plus
+//! a recomputed null mask -- and the `+ 1` / `- 1` is a second pass over the result.
+
+use arrow::array::{Array, ArrayRef, Date32Array, DictionaryArray, Int32Array, Scalar};
+use arrow::compute::kernels::numeric::{add_wrapping, sub_wrapping};
+use arrow::compute::{cast, date_part, DatePart};
+use arrow::datatypes::{DataType, Field, Int32Type};
+use criterion::{criterion_group, criterion_main, Criterion, Throughput};
+use datafusion::config::ConfigOptions;
+use datafusion::logical_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
+use datafusion_comet_spark_expr::{SparkDayOfWeek, SparkWeekDay};
+use std::hint::black_box;
+use std::sync::Arc;
+
+const ROWS: usize = 8_192;
+
+/// Null density of an input shape. The old path's `unary_opt` only visits valid slots while the
+/// kernel's `unary` visits every slot, so the dense and all-null shapes are where a replacement
+/// like this is most likely to backfire.
+#[derive(Clone, Copy)]
+enum Nulls {
+ None,
+ Sparse,
+ Dense,
+ All,
+}
+
+impl Nulls {
+ fn tag(self) -> &'static str {
+ match self {
+ Nulls::None => "no_nulls",
+ Nulls::Sparse => "sparse_nulls",
+ Nulls::Dense => "dense_nulls",
+ Nulls::All => "all_nulls",
+ }
+ }
+
+ fn is_null(self, i: usize) -> bool {
+ match self {
+ Nulls::None => false,
+ Nulls::Sparse => i.is_multiple_of(8), // 12.5%
+ Nulls::Dense => !i.is_multiple_of(8), // 87.5%
+ Nulls::All => true,
+ }
+ }
+}
+
+/// Epoch days spread over roughly 1970..2050, the range a date column actually holds.
+fn dates(nulls: Nulls) -> Date32Array {
+ (0..ROWS)
+ .map(|i| {
+ if nulls.is_null(i) {
+ None
+ } else {
+ Some((i as i32).wrapping_mul(3) % 29_220)
+ }
+ })
+ .collect()
+}
+
+/// A dictionary-encoded date column, the shape a partition column arrives in from a Parquet
+/// scan. Cardinality matters: the old path ran the calendar conversion over the dictionary
+/// *values* only and rewrapped the keys, so a low-cardinality column did very little work.
+fn dict_dates(cardinality: usize, nulls: Nulls) -> DictionaryArray<Int32Type> {
+ let values = Arc::new(Date32Array::from(
+ (0..cardinality)
+ .map(|i| (i as i32).wrapping_mul(37) % 29_220)
+ .collect::<Vec<_>>(),
+ )) as ArrayRef;
+ let keys: Int32Array = (0..ROWS)
+ .map(|i| {
+ if nulls.is_null(i) {
+ None
+ } else {
+ Some((i % cardinality) as i32)
+ }
+ })
+ .collect();
+ DictionaryArray::<Int32Type>::new(keys, values)
+}
+
+// ---- the chain the serde emitted before this change ------------------------------------------
+
+/// The serde emitted `Add(Cast(datepart('dow', child), Int32), 1)`. The cast came *first*: it was
+/// an identity no-op for a plain array, but it is what unpacked a dictionary result, and arrow's
+/// arithmetic kernels reject `Dictionary(Int32, Int32) + Int32` outright, so the order matters.
+fn current_dayofweek(array: &ArrayRef) -> ArrayRef {
+ let part = date_part(array.as_ref(), DatePart::DayOfWeekSunday0).unwrap();
+ let unpacked = cast(&part, &DataType::Int32).unwrap();
+ add_wrapping(&unpacked, &Scalar::new(Int32Array::from(vec![1]))).unwrap()
+}
+
+fn current_weekday(array: &ArrayRef) -> ArrayRef {
+ let part = date_part(array.as_ref(), DatePart::DayOfWeekMonday1).unwrap();
+ let unpacked = cast(&part, &DataType::Int32).unwrap();
+ sub_wrapping(&unpacked, &Scalar::new(Int32Array::from(vec![1]))).unwrap()
+}
+
+// ---- native kernels, as the serde emits them now ---------------------------------------------
+
+fn invoke(udf: &dyn ScalarUDFImpl, array: &ArrayRef) -> ArrayRef {
+ udf.invoke_with_args(ScalarFunctionArgs {
+ args: vec![ColumnarValue::Array(Arc::clone(array))],
+ arg_fields: vec![Arc::new(Field::new("d", array.data_type().clone(), true))],
+ number_rows: ROWS,
+ return_field: Arc::new(Field::new(udf.name(), DataType::Int32, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap()
+ .to_array(ROWS)
+ .unwrap()
+}
+
+/// Both arms must agree, values *and* null buffer, before any timing is meaningful.
+fn assert_equivalent(array: &ArrayRef) {
+ for (current, native) in [
+ (
+ current_dayofweek(array),
+ invoke(&SparkDayOfWeek::new(), array),
+ ),
+ (current_weekday(array), invoke(&SparkWeekDay::new(), array)),
+ ] {
+ let a = current.as_any().downcast_ref::<Int32Array>().unwrap();
+ let b = native.as_any().downcast_ref::<Int32Array>().unwrap();
+ // Compare nullability logically: the old path materialises an all-valid null buffer
+ // where the kernel leaves `None`, which is the same thing to every consumer.
+ assert_eq!(a.null_count(), b.null_count(), "null counts differ");
+ for i in 0..a.len() {
+ assert_eq!(a.is_null(i), b.is_null(i), "null flag differs at row {i}");
+ }
+ assert_eq!(a, b, "values differ");
+ }
+}
+
+fn bench_shape(c: &mut Criterion, shape: &str, array: ArrayRef) {
+ assert_equivalent(&array);
+
+ let mut group = c.benchmark_group(format!("dayofweek/{shape}"));
+ group.throughput(Throughput::Elements(ROWS as u64));
+ group.bench_function("datepart_dow_plus_one", |b| {
+ b.iter(|| black_box(current_dayofweek(black_box(&array))))
+ });
+ let dow = SparkDayOfWeek::new();
+ group.bench_function("native_spark_dayofweek", |b| {
+ b.iter(|| black_box(invoke(black_box(&dow), black_box(&array))))
+ });
+ group.finish();
+
+ let mut group = c.benchmark_group(format!("weekday/{shape}"));
+ group.throughput(Throughput::Elements(ROWS as u64));
+ group.bench_function("datepart_isodow_minus_one", |b| {
+ b.iter(|| black_box(current_weekday(black_box(&array))))
+ });
+ let wd = SparkWeekDay::new();
+ group.bench_function("native_spark_weekday", |b| {
+ b.iter(|| black_box(invoke(black_box(&wd), black_box(&array))))
+ });
+ group.finish();
+}
+
+fn criterion_benchmark(c: &mut Criterion) {
+ for nulls in [Nulls::None, Nulls::Sparse, Nulls::Dense, Nulls::All] {
+ bench_shape(c, &format!("flat/{}", nulls.tag()), Arc::new(dates(nulls)));
+ }
+ // Low cardinality is the case the old path handled cheaply: it converted 8 distinct dates
+ // and rewrapped the keys. High cardinality approaches one conversion per row.
+ for cardinality in [8usize, 1024] {
+ for nulls in [Nulls::None, Nulls::Sparse, Nulls::Dense] {
+ bench_shape(
+ c,
+ &format!("dict{cardinality}/{}", nulls.tag()),
+ Arc::new(dict_dates(cardinality, nulls)),
+ );
+ }
+ }
+}
+
+criterion_group!(benches, criterion_benchmark);
+criterion_main!(benches);
diff --git a/native/spark-expr/benches/extract_date_part.rs b/native/spark-expr/benches/extract_date_part.rs
index c98dbdfde5..be50062321 100644
--- a/native/spark-expr/benches/extract_date_part.rs
+++ b/native/spark-expr/benches/extract_date_part.rs
@@ -15,7 +15,9 @@
// specific language governing permissions and limitations
// under the License.
-use arrow::datatypes::{DataType, Field};
+use arrow::array::{Array, ArrayRef, DictionaryArray, Int32Array, TimestampMicrosecondArray};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, Field, Int32Type};
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
use datafusion::common::config::ConfigOptions;
use datafusion::logical_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
@@ -25,54 +27,245 @@ use std::sync::Arc;
#[path = "common/mod.rs"]
mod common;
-use common::{timestamp_micros_array, NULL_RATIOS, ROW_COUNTS};
+use common::{is_null, timestamp_micros_array, NULL_RATIOS, ROW_COUNTS};
const TZ: &str = "America/Los_Angeles";
-const BASE_MICROS: i64 = 1_600_000_000_000_000;
+/// A zero-offset session zone. Only in such a session is a timezone-aware timestamp eligible for
+/// the integer fast path, so each branch of the dispatch needs a session zone of its own.
+const UTC_TZ: &str = "UTC";
+const SMALL_ROWS: usize = 8_192;
+
+/// `NULL_RATIOS` covers none / sparse / all. Dense is added because that is where a kernel which
+/// evaluates every slot can lose to arrow's `unary_opt`, which visits only valid indices.
+const DENSE_NULLS: (f64, &str) = (0.875, "dense");
+
+const MICROS_PER_DAY: i64 = 86_400_000_000;
+
+/// Instants either side of the epoch, roughly 1875 to 2064 at 8192 rows. Pre-epoch values are the
+/// ones that matter: `-1` us is 1969-12-31 23:59:59.999999, so truncating toward zero instead of
+/// dividing Euclidean-style would give the wrong field.
+fn spanning_epoch(i: usize, rows: usize) -> i64 {
+ (i as i64 - (rows as i64) / 2) * 730_000_000_000 + 12_345_678
+}
+
+fn hour_of_day(micros: i64) -> i32 {
+ (micros.rem_euclid(MICROS_PER_DAY) / 3_600_000_000) as i32
+}
+fn minute_of_hour(micros: i64) -> i32 {
+ micros.div_euclid(60_000_000).rem_euclid(60) as i32
+}
+fn second_of_minute(micros: i64) -> i32 {
+ micros.div_euclid(1_000_000).rem_euclid(60) as i32
+}
+
+struct Part {
+ name: &'static str,
+ kernel: fn(i64) -> i32,
+ max: i32,
+}
+
+const PARTS: [Part; 3] = [
+ Part {
+ name: "hour",
+ kernel: hour_of_day,
+ max: 23,
+ },
+ Part {
+ name: "minute",
+ kernel: minute_of_hour,
+ max: 59,
+ },
+ Part {
+ name: "second",
+ kernel: second_of_minute,
+ max: 59,
+ },
+];
+
+fn udf_for(part: &str, session_tz: &str) -> Box<dyn ScalarUDFImpl> {
+ match part {
+ "hour" => Box::new(SparkHour::new(session_tz.to_string())),
+ "minute" => Box::new(SparkMinute::new(session_tz.to_string())),
+ _ => Box::new(SparkSecond::new(session_tz.to_string())),
+ }
+}
+
+fn invoke(udf: &dyn ScalarUDFImpl, args: &[ColumnarValue], rows: usize) -> ArrayRef {
+ udf.invoke_with_args(ScalarFunctionArgs {
+ args: args.to_vec(),
+ arg_fields: vec![],
+ number_rows: rows,
+ return_field: Arc::new(Field::new("result", DataType::Int32, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap()
+ .to_array(rows)
+ .unwrap()
+}
+
+/// A dictionary result stays dictionary-encoded; flatten so every shape is compared the same way.
+fn as_int32(array: &ArrayRef) -> Int32Array {
+ let flat = if matches!(array.data_type(), DataType::Dictionary(_, _)) {
+ cast(array, &DataType::Int32).unwrap()
+ } else {
+ Arc::clone(array)
+ };
+ flat.as_any().downcast_ref::<Int32Array>().unwrap().clone()
+}
+
+/// Validates a shape before it is timed. Every shape gets a null-aware check: the result must be
+/// null exactly where the input is, and every non-null value must be inside the field's range.
+/// Shapes eligible for the fast path additionally get an exact oracle -- the field recomputed
+/// from the generator, independent of arrow -- since no session offset applies to them.
+fn check(
+ part: &Part,
+ array: &ArrayRef,
+ udf: &dyn ScalarUDFImpl,
+ rows: usize,
+ oracle: Option<&dyn Fn(usize) -> Option<i32>>,
+) {
+ let out = as_int32(&invoke(
+ udf,
+ &[ColumnarValue::Array(Arc::clone(array))],
+ rows,
+ ));
+ let input_nulls = array.logical_nulls();
+ assert_eq!(out.len(), rows, "{}: length", part.name);
+ for i in 0..rows {
+ let input_null = input_nulls.as_ref().is_some_and(|n| n.is_null(i));
+ assert_eq!(
+ out.is_null(i),
+ input_null,
+ "{}: null flag at row {i} ({:?})",
+ part.name,
+ array.data_type()
+ );
+ if !input_null {
+ let v = out.value(i);
+ assert!(
+ (0..=part.max).contains(&v),
+ "{}: value {v} out of range at row {i}",
+ part.name
+ );
+ if let Some(f) = oracle {
+ assert_eq!(Some(v), f(i), "{}: value at row {i}", part.name);
+ }
+ }
+ }
+}
+
+fn dict_timestamps(rows: usize, cardinality: usize, null_ratio: f64) -> ArrayRef {
+ let values = Arc::new(TimestampMicrosecondArray::from(
+ (0..cardinality)
+ .map(|i| spanning_epoch(i * 97, cardinality * 97))
+ .collect::<Vec<_>>(),
+ )) as ArrayRef;
+ let keys: Int32Array = (0..rows)
+ .map(|i| {
+ if is_null(i, null_ratio) {
+ None
+ } else {
+ Some((i % cardinality) as i32)
+ }
+ })
+ .collect();
+ Arc::new(DictionaryArray::<Int32Type>::new(keys, values))
+}
+
+fn null_ratios() -> impl Iterator<Item = (f64, &'static str)> {
+ NULL_RATIOS
+ .iter()
+ .copied()
+ .chain(std::iter::once(DENSE_NULLS))
+}
+
+/// The stride approximation in `common::is_null` has to actually produce the density each shape
+/// claims, or a "dense" case silently becomes a second all-null case.
+fn assert_density(rows: usize, null_ratio: f64, tag: &str) {
+ let nulls = (0..rows).filter(|&i| is_null(i, null_ratio)).count();
+ let expected = (rows as f64 * null_ratio).round() as usize;
+ assert!(
+ nulls.abs_diff(expected) <= rows / 100,
+ "{tag}: expected about {expected} nulls of {rows}, generator produced {nulls}"
+ );
+}
fn criterion_benchmark(c: &mut Criterion) {
- // hour/minute/second all route through extract_date_part; sweep the three parts and both the
- // timezone-aware (stored UTC, shifted to session tz) and TimestampNTZ (extracted directly) paths.
- let parts: Vec<(&str, Box<dyn ScalarUDFImpl>)> = vec![
- ("hour", Box::new(SparkHour::new(TZ.to_string()))),
- ("minute", Box::new(SparkMinute::new(TZ.to_string()))),
- ("second", Box::new(SparkSecond::new(TZ.to_string()))),
- ];
- for (part, udf) in &parts {
- let mut group = c.benchmark_group(*part);
- for (tz, tz_tag) in [(Some("UTC"), "tz"), (None, "ntz")] {
- for rows in ROW_COUNTS {
- for (null_ratio, tag) in NULL_RATIOS {
- let arr = timestamp_micros_array(rows, null_ratio, tz, |i| {
- BASE_MICROS + (i as i64) * 3_600_000_000
- });
+ // hour/minute/second all route through extract_date_part. The dispatch takes an integer fast
+ // path when no timezone offset applies -- TimestampNTZ, or a timezone-aware value in a
+ // zero-offset session -- and otherwise shifts to the session zone and reads a calendar
+ // datetime. Both session zones are swept so each branch is measured.
+ for (session_tz, session_tag) in [(TZ, "la_session"), (UTC_TZ, "utc_session")] {
+ for part in &PARTS {
+ let udf = udf_for(part.name, session_tz);
+ let mut group = c.benchmark_group(part.name);
+
+ for (tz, tz_tag) in [(Some("UTC"), "tz"), (None, "ntz")] {
+ // The full row sweep stays on the session zone the benchmark shipped with; the
+ // second zone and the dictionary shapes use one size to bound the matrix.
+ let sizes: &[usize] = if session_tz == TZ {
+ &ROW_COUNTS
+ } else {
+ &[SMALL_ROWS]
+ };
+ // No offset applies to NTZ in any session, nor to a stored UTC instant read in a
+ // zero-offset session, so those shapes have an exact oracle.
+ let eligible = tz.is_none() || session_tz == UTC_TZ;
+ for &rows in sizes {
+ for (null_ratio, tag) in null_ratios() {
+ assert_density(rows, null_ratio, tag);
+ let arr = timestamp_micros_array(rows, null_ratio, tz, |i| {
+ spanning_epoch(i, rows)
+ });
+ let oracle = |i: usize| {
+ if is_null(i, null_ratio) {
+ None
+ } else {
+ Some((part.kernel)(spanning_epoch(i, rows)))
+ }
+ };
+ check(
+ part,
+ &arr,
+ udf.as_ref(),
+ rows,
+ if eligible { Some(&oracle) } else { None },
+ );
+ let args = vec![ColumnarValue::Array(arr)];
+ group.bench_with_input(
+ BenchmarkId::from_parameter(format!(
+ "{session_tag}/{tz_tag}/{rows}/{tag}"
+ )),
+ &args,
+ |b, args| {
+ b.iter(|| black_box(invoke(udf.as_ref(), args, rows)));
+ },
+ );
+ }
+ }
+ }
+
+ for cardinality in [8usize, 1024] {
+ for (null_ratio, tag) in null_ratios() {
+ let arr = dict_timestamps(SMALL_ROWS, cardinality, null_ratio);
+ // Dictionaries stay on the general path in every session, so there is no
+ // arithmetic oracle; the null-aware and range checks still apply.
+ check(part, &arr, udf.as_ref(), SMALL_ROWS, None);
let args = vec![ColumnarValue::Array(arr)];
group.bench_with_input(
- BenchmarkId::from_parameter(format!("{tz_tag}/{rows}/{tag}")),
+ BenchmarkId::from_parameter(format!(
+ "{session_tag}/dict{cardinality}/{SMALL_ROWS}/{tag}"
+ )),
&args,
|b, args| {
- b.iter(|| {
- black_box(
- udf.invoke_with_args(ScalarFunctionArgs {
- args: args.clone(),
- arg_fields: vec![],
- number_rows: rows,
- return_field: Arc::new(Field::new(
- "result",
- DataType::Int32,
- true,
- )),
- config_options: Arc::new(ConfigOptions::default()),
- })
- .unwrap(),
- )
- })
+ b.iter(|| black_box(invoke(udf.as_ref(), args, SMALL_ROWS)));
},
);
}
}
+
+ group.finish();
}
- group.finish();
}
}
diff --git a/native/spark-expr/src/comet_scalar_funcs.rs b/native/spark-expr/src/comet_scalar_funcs.rs
index 7f8e27930b..c9b50a02a0 100644
--- a/native/spark-expr/src/comet_scalar_funcs.rs
+++ b/native/spark-expr/src/comet_scalar_funcs.rs
@@ -29,9 +29,10 @@ use crate::{
spark_isnan, spark_lpad, spark_make_decimal, spark_month_name, spark_read_side_padding,
spark_round, spark_rpad, spark_sequence, spark_to_time, spark_unhex, spark_unscaled_value,
EvalMode, SparkArrayPositionFunc, SparkArraySlice, SparkArraysOverlap, SparkContains,
- SparkDateDiff, SparkDateFromUnixDate, SparkDateTrunc, SparkFlatten, SparkIcebergBucket,
- SparkIcebergTemporalTransform, SparkIcebergTruncate, SparkMakeDate, SparkMakeInterval,
- SparkMakeTime, SparkMapExtract, SparkNextDay, SparkSecondsToTimestamp, SparkSizeFunc,
+ SparkDateDiff, SparkDateFromUnixDate, SparkDateTrunc, SparkDayOfWeek, SparkFlatten,
+ SparkIcebergBucket, SparkIcebergTemporalTransform, SparkIcebergTruncate, SparkMakeDate,
+ SparkMakeInterval, SparkMakeTime, SparkMapExtract, SparkNextDay, SparkSecondsToTimestamp,
+ SparkSizeFunc, SparkWeekDay,
};
use arrow::datatypes::DataType;
use datafusion::common::{DataFusionError, Result as DataFusionResult};
@@ -312,6 +313,8 @@ fn all_scalar_functions() -> Vec<Arc<ScalarUDF>> {
Arc::new(ScalarUDF::new_from_impl(SparkDateDiff::default())),
Arc::new(ScalarUDF::new_from_impl(SparkDateFromUnixDate::default())),
Arc::new(ScalarUDF::new_from_impl(SparkDateTrunc::default())),
+ Arc::new(ScalarUDF::new_from_impl(SparkDayOfWeek::default())),
+ Arc::new(ScalarUDF::new_from_impl(SparkWeekDay::default())),
Arc::new(ScalarUDF::new_from_impl(SparkFlatten::default())),
Arc::new(ScalarUDF::new_from_impl(SparkIcebergBucket::default())),
Arc::new(ScalarUDF::new_from_impl(SparkIcebergTruncate::default())),
diff --git a/native/spark-expr/src/datetime_funcs/day_of_week.rs b/native/spark-expr/src/datetime_funcs/day_of_week.rs
new file mode 100644
index 0000000000..6e5ee9ae3c
--- /dev/null
+++ b/native/spark-expr/src/datetime_funcs/day_of_week.rs
@@ -0,0 +1,287 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Spark's `dayofweek` and `weekday`, computed straight from the epoch day.
+//!
+//! Both fields are a single modulo of the `Date32` value, so there is no need to build a calendar
+//! date per row. Previously the serde emitted `datepart('dow'|'isodow', child)` followed by a
+//! `+ 1` / `- 1` arithmetic node; `date_part` reconstructs a `NaiveDateTime` for every row and the
+//! arithmetic node walks the result a second time.
+
+use arrow::array::{Array, ArrayRef, AsArray, Date32Array, Int32Array};
+use arrow::compute::cast;
+use arrow::datatypes::{DataType, Date32Type, Int32Type};
+use datafusion::common::{utils::take_function_args, DataFusionError, Result};
+use datafusion::logical_expr::{
+ ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark `DayOfWeek`: Sunday = 1, Monday = 2, ..., Saturday = 7.
+///
+/// Epoch day 0 (1970-01-01) is a Thursday, so `+ 4` moves Sunday to 0 before the shift to
+/// 1-based. Widening to `i64` first keeps `i32::MAX` from overflowing.
+#[inline]
+fn day_of_week(days: i32) -> i32 {
+ ((i64::from(days) + 4).rem_euclid(7) + 1) as i32
+}
+
+/// Spark `WeekDay`: Monday = 0, Tuesday = 1, ..., Sunday = 6.
+#[inline]
+fn week_day(days: i32) -> i32 {
+ (i64::from(days) + 3).rem_euclid(7) as i32
+}
+
+/// Applies `kernel` to the dates.
+///
+/// `unary` evaluates every slot and vectorizes; `unary_opt` visits only valid indices but costs
+/// more per element. The path this replaces used `unary_opt`, so an almost entirely null batch
+/// was nearly free there and `unary` would regress it. Skipping only pays off once most of the
+/// batch is null -- at 87.5% nulls `unary` was still ahead -- so switch on density rather than on
+/// the mere presence of a null, which would give up most of the win on a lightly-null column.
+///
+/// `kernel` is generic rather than a `fn` pointer on purpose: a pointer forces an indirect call
+/// per element inside `unary`, which blocks inlining and vectorization and costs roughly 2.5x.
+#[inline]
+fn map_dates<F: Fn(i32) -> i32 + Copy>(dates: &Date32Array, kernel: F) -> Int32Array {
+ if dates.null_count() * 2 <= dates.len() {
+ dates.unary::<_, Int32Type>(kernel)
+ } else {
+ dates.unary_opt::<_, Int32Type>(|d| Some(kernel(d)))
+ }
+}
+
+macro_rules! epoch_day_extractor {
+ ($struct_name:ident, $fn_name:expr, $kernel:ident, $doc:expr) => {
+ #[doc = $doc]
+ #[derive(Debug, PartialEq, Eq, Hash)]
+ pub struct $struct_name {
+ signature: Signature,
+ }
+
+ impl $struct_name {
+ pub fn new() -> Self {
+ Self {
+ // Spark types the child as DateType; a dictionary-encoded date column is
+ // unpacked below.
+ signature: Signature::any(1, Volatility::Immutable),
+ }
+ }
+ }
+
+ impl Default for $struct_name {
+ fn default() -> Self {
+ Self::new()
+ }
+ }
+
+ impl ScalarUDFImpl for $struct_name {
+ fn name(&self) -> &str {
+ $fn_name
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::Int32)
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+ let [date] = take_function_args(self.name(), args.args)?;
+ let array = date.into_array(args.number_rows)?;
+
+ // A date partition column arrives dictionary-encoded from the Parquet scan. Map
+ // the dictionary *values* and rewrap the keys, then unpack, which is what the
+ // previous `Add(Cast(datepart(..), Int32), 1)` chain did: the calendar work is
+ // proportional to the cardinality, not the row count. Decoding to Date32 first
+ // would make a low-cardinality column markedly more expensive than before.
+ if matches!(array.data_type(), DataType::Dictionary(_, _)) {
+ let dict = array.as_any_dictionary();
+ let values =
+ dict.values()
+ .as_primitive_opt::<Date32Type>()
+ .ok_or_else(|| {
+ DataFusionError::Execution(format!(
+ "{} expects a date, got {}",
+ $fn_name,
+ array.data_type()
+ ))
+ })?;
+ let mapped = Arc::new(map_dates(values, $kernel)) as ArrayRef;
+ let rewrapped = dict.with_values(mapped);
+ return Ok(ColumnarValue::Array(cast(&rewrapped, &DataType::Int32)?));
+ }
+
+ let dates = array
+ .as_any()
+ .downcast_ref::<Date32Array>()
+ .ok_or_else(|| {
+ DataFusionError::Execution(format!(
+ "{} expects a date, got {}",
+ $fn_name,
+ array.data_type()
+ ))
+ })?;
+
+ Ok(ColumnarValue::Array(Arc::new(map_dates(dates, $kernel))))
+ }
+ }
+ };
+}
+
+epoch_day_extractor!(
+ SparkDayOfWeek,
+ "spark_dayofweek",
+ day_of_week,
+ "Spark `dayofweek(date)`: Sunday = 1 through Saturday = 7."
+);
+epoch_day_extractor!(
+ SparkWeekDay,
+ "spark_weekday",
+ week_day,
+ "Spark `weekday(date)`: Monday = 0 through Sunday = 6."
+);
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use arrow::array::DictionaryArray;
+ use arrow::datatypes::Field;
+ use datafusion::common::ScalarValue;
+ use datafusion::config::ConfigOptions;
+
+ fn invoke(udf: &dyn ScalarUDFImpl, array: ArrayRef) -> Int32Array {
+ let rows = array.len();
+ let arg_fields = vec![Arc::new(Field::new("d", array.data_type().clone(), true))];
+ let result = udf
+ .invoke_with_args(ScalarFunctionArgs {
+ args: vec![ColumnarValue::Array(array)],
+ arg_fields,
+ number_rows: rows,
+ return_field: Arc::new(Field::new(udf.name(), DataType::Int32, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap()
+ .to_array(rows)
+ .unwrap();
+ result
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap()
+ .clone()
+ }
+
+ /// 1970-01-01 is a Thursday, so the week either side of the epoch pins both numberings.
+ #[test]
+ fn known_weekdays_around_the_epoch() {
+ // days -4..=3 spans Sunday 1969-12-28 through Sunday 1970-01-04.
+ let dates: Date32Array = (-4..=3).collect::<Vec<i32>>().into();
+ let array: ArrayRef = Arc::new(dates);
+ assert_eq!(
+ invoke(&SparkDayOfWeek::new(), Arc::clone(&array)).values(),
+ // Sun Mon Tue Wed Thu Fri Sat Sun
+ &[1, 2, 3, 4, 5, 6, 7, 1]
+ );
+ assert_eq!(
+ invoke(&SparkWeekDay::new(), array).values(),
+ // Sun=6, Mon=0 ... Sat=5, Sun=6
+ &[6, 0, 1, 2, 3, 4, 5, 6]
+ );
+ }
+
+ /// Nulls must pass through in place, and the value under a null must not matter.
+ #[test]
+ fn nulls_are_preserved() {
+ let array: ArrayRef = Arc::new(Date32Array::from(vec![Some(0), None, Some(1), None]));
+ let out = invoke(&SparkDayOfWeek::new(), array);
+ assert_eq!(out.null_count(), 2);
+ assert!(out.is_null(1) && out.is_null(3));
+ assert_eq!(out.value(0), 5);
+ assert_eq!(out.value(2), 6);
+ }
+
+ /// The whole `i32` domain must work: these are far outside chrono's representable range,
+ /// where the previous `date_part` path produced NULL.
+ #[test]
+ fn full_i32_domain() {
+ let array: ArrayRef = Arc::new(Date32Array::from(vec![i32::MIN, i32::MAX]));
+ let dow = invoke(&SparkDayOfWeek::new(), Arc::clone(&array));
+ let wd = invoke(&SparkWeekDay::new(), array);
+ for i in 0..2 {
+ assert!((1..=7).contains(&dow.value(i)), "dayofweek out of range");
+ assert!((0..=6).contains(&wd.value(i)), "weekday out of range");
+ // The two numberings must agree: dayofweek 1 (Sunday) is weekday 6.
+ assert_eq!((dow.value(i) + 5) % 7, wd.value(i));
+ }
+ }
+
+ /// The two numberings agree with each other across a full 400-year Gregorian cycle.
+ #[test]
+ fn numberings_agree_over_a_full_cycle() {
+ for days in -146_097..=146_097 {
+ assert_eq!((day_of_week(days) + 5) % 7, week_day(days), "day {days}");
+ }
+ }
+
+ /// A dictionary-encoded date column (what a Parquet partition column looks like) unpacks to a
+ /// plain Int32 result, matching what the old `datepart` + cast chain produced.
+ #[test]
+ fn dictionary_input_unpacks() {
+ let values = Arc::new(Date32Array::from(vec![0, 1])) as ArrayRef;
+ let keys = Int32Array::from(vec![0, 1, 0]);
+ let dict = DictionaryArray::<Int32Type>::new(keys, values);
+ let out = invoke(&SparkDayOfWeek::new(), Arc::new(dict));
+ assert_eq!(out.values(), &[5, 6, 5]);
+ }
+
+ #[test]
+ fn rejects_non_date_input() {
+ let array: ArrayRef = Arc::new(Int32Array::from(vec![1]));
+ let err = SparkDayOfWeek::new()
+ .invoke_with_args(ScalarFunctionArgs {
+ args: vec![ColumnarValue::Array(Arc::clone(&array))],
+ arg_fields: vec![Arc::new(Field::new("d", DataType::Int32, true))],
+ number_rows: 1,
+ return_field: Arc::new(Field::new("spark_dayofweek", DataType::Int32, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap_err();
+ assert!(err.to_string().contains("expects a date"), "{err}");
+ }
+
+ /// Guard against the scalar path silently producing a wrong-length result.
+ #[test]
+ fn scalar_input_is_supported() {
+ let out = SparkDayOfWeek::new()
+ .invoke_with_args(ScalarFunctionArgs {
+ args: vec![ColumnarValue::Scalar(ScalarValue::Date32(Some(0)))],
+ arg_fields: vec![Arc::new(Field::new("d", DataType::Date32, true))],
+ number_rows: 1,
+ return_field: Arc::new(Field::new("spark_dayofweek", DataType::Int32, true)),
+ config_options: Arc::new(ConfigOptions::default()),
+ })
+ .unwrap()
+ .to_array(1)
+ .unwrap();
+ assert_eq!(
+ out.as_any().downcast_ref::<Int32Array>().unwrap().value(0),
+ 5
+ );
+ }
+}
diff --git a/native/spark-expr/src/datetime_funcs/extract_date_part.rs b/native/spark-expr/src/datetime_funcs/extract_date_part.rs
index 693dae6ffe..7d6e2a50ea 100644
--- a/native/spark-expr/src/datetime_funcs/extract_date_part.rs
+++ b/native/spark-expr/src/datetime_funcs/extract_date_part.rs
@@ -16,13 +16,15 @@
// under the License.
use crate::utils::array_with_timezone;
+use arrow::array::{Array, ArrayRef, Int32Array, TimestampMicrosecondArray};
use arrow::compute::{date_part, DatePart};
-use arrow::datatypes::{DataType, TimeUnit::Microsecond};
+use arrow::datatypes::{DataType, Int32Type, TimeUnit::Microsecond};
use datafusion::common::{internal_datafusion_err, DataFusionError};
use datafusion::logical_expr::{
ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
};
use std::fmt::Debug;
+use std::sync::Arc;
/// Returns true when the type is a timestamp without a timezone (Spark's TimestampNTZType),
/// including when wrapped in a dictionary. Such values store local wall-clock time and must not
@@ -35,8 +37,58 @@ fn is_timestamp_ntz(data_type: &DataType) -> bool {
}
}
+const MICROS_PER_DAY: i64 = 86_400_000_000;
+const MICROS_PER_HOUR: i64 = 3_600_000_000;
+const MICROS_PER_MINUTE: i64 = 60_000_000;
+const MICROS_PER_SECOND: i64 = 1_000_000;
+
+/// Session timezones whose UTC offset is zero at every instant, so a clock field read from the
+/// stored UTC instant needs no shift. The list is deliberately conservative: a name that is not
+/// listed simply takes the general timezone-aware path, which is always correct.
+fn is_utc_timezone(timezone: &str) -> bool {
+ matches!(
+ timezone,
+ "UTC" | "Etc/UTC" | "Etc/GMT" | "GMT" | "Z" | "+00:00" | "-00:00" | "00:00"
+ )
+}
+
+/// Returns the microsecond values when the clock field is a pure function of them, i.e. when no
+/// timezone offset applies. That is the case for `TimestampNTZ` (already local wall-clock) and
+/// for a timezone-aware timestamp read in a UTC session -- `array_with_timezone` only re-tags
+/// such an array, so the stored value is the UTC instant either way.
+///
+/// Dictionaries and non-microsecond units return `None` and keep the general path.
+fn micros_without_offset<'a>(
+ array: &'a ArrayRef,
+ timezone: &str,
+) -> Option<&'a TimestampMicrosecondArray> {
+ match array.data_type() {
+ DataType::Timestamp(Microsecond, None) => {}
+ DataType::Timestamp(Microsecond, Some(_)) if is_utc_timezone(timezone) => {}
+ _ => return None,
+ }
+ array.as_any().downcast_ref::<TimestampMicrosecondArray>()
+}
+
+/// Euclidean division is required: at UTC, `-1` microsecond is `1969-12-31 23:59:59.999999`, so
+/// truncation toward zero would give the wrong field for pre-epoch instants.
+#[inline]
+fn hour_of_day(micros: i64) -> i32 {
+ (micros.rem_euclid(MICROS_PER_DAY) / MICROS_PER_HOUR) as i32
+}
+
+#[inline]
+fn minute_of_hour(micros: i64) -> i32 {
+ micros.div_euclid(MICROS_PER_MINUTE).rem_euclid(60) as i32
+}
+
+#[inline]
+fn second_of_minute(micros: i64) -> i32 {
+ micros.div_euclid(MICROS_PER_SECOND).rem_euclid(60) as i32
+}
+
macro_rules! extract_date_part {
- ($struct_name:ident, $fn_name:expr, $date_part_variant:ident) => {
+ ($struct_name:ident, $fn_name:expr, $date_part_variant:ident, $kernel:ident) => {
#[derive(Debug, PartialEq, Eq, Hash)]
pub struct $struct_name {
signature: Signature,
@@ -82,6 +134,24 @@ macro_rules! extract_date_part {
match args {
[ColumnarValue::Array(array)] => {
+ // Fast path: when no offset applies the field is arithmetic on the stored
+ // microseconds, so no calendar datetime is built per row. `unary` carries
+ // the null buffer over untouched.
+ if let Some(micros) = micros_without_offset(&array, &self.timezone) {
+ // `unary` evaluates every slot and vectorizes; `unary_opt` visits
+ // only valid indices but costs more per element. The `date_part`
+ // path this replaces uses `unary_opt`, so an almost entirely null
+ // batch was nearly free there. Skipping only pays off once most of
+ // the batch is null, so switch on density rather than on the mere
+ // presence of a null.
+ let result: Int32Array = if micros.null_count() * 2 <= micros.len() {
+ micros.unary::<_, Int32Type>($kernel)
+ } else {
+ micros.unary_opt::<_, Int32Type>(|v| Some($kernel(v)))
+ };
+ return Ok(ColumnarValue::Array(Arc::new(result)));
+ }
+
// TimestampNTZ values are stored as local wall-clock time, so the date
// part is extracted directly. Timezone-aware timestamps are stored in UTC
// and must be shifted to the session timezone first.
@@ -113,9 +183,9 @@ macro_rules! extract_date_part {
};
}
-extract_date_part!(SparkHour, "hour", Hour);
-extract_date_part!(SparkMinute, "minute", Minute);
-extract_date_part!(SparkSecond, "second", Second);
+extract_date_part!(SparkHour, "hour", Hour, hour_of_day);
+extract_date_part!(SparkMinute, "minute", Minute, minute_of_hour);
+extract_date_part!(SparkSecond, "second", Second, second_of_minute);
#[cfg(test)]
mod tests {
@@ -150,6 +220,100 @@ mod tests {
Arc::new(TimestampMicrosecondArray::from(vec![Some(MICROS)]))
}
+ /// Invokes a UDF over a whole array and returns every extracted value.
+ fn invoke_all<U: ScalarUDFImpl + ?Sized>(udf: &U, array: ArrayRef) -> Int32Array {
+ let rows = array.len();
+ let return_field = Arc::new(Field::new("v", DataType::Int32, true));
+ let args = ScalarFunctionArgs {
+ args: vec![ColumnarValue::Array(array)],
+ number_rows: rows,
+ return_field,
+ config_options: Arc::new(ConfigOptions::default()),
+ arg_fields: vec![],
+ };
+ match udf.invoke_with_args(args).unwrap() {
+ ColumnarValue::Array(arr) => arr.as_any().downcast_ref::<Int32Array>().unwrap().clone(),
+ _ => panic!("Expected array"),
+ }
+ }
+
+ /// The UTC fast path must agree with the general timezone-aware path. Both are exercised
+ /// here on the same instant: a UTC session takes the fast path, `Etc/GMT+0` is not in the
+ /// allowlist and takes the general path, and both describe the same zero offset.
+ #[test]
+ fn utc_fast_path_matches_general_path() {
+ let array = TimestampMicrosecondArray::from(vec![Some(MICROS)]).with_timezone("UTC");
+ let fast = SparkHour::new("UTC".to_string());
+ let general = SparkHour::new("Etc/GMT+0".to_string());
+ assert!(is_utc_timezone("UTC") && !is_utc_timezone("Etc/GMT+0"));
+ assert_eq!(
+ invoke(&fast, Arc::new(array.clone())),
+ invoke(&general, Arc::new(array))
+ );
+ }
+
+ /// Pre-epoch instants are where truncation toward zero would diverge from Spark: -1 us is
+ /// 1969-12-31 23:59:59.999999, not 1970-01-01 00:00:00.
+ #[test]
+ fn pre_epoch_instants_use_euclidean_division() {
+ let values = vec![Some(-1i64), Some(-MICROS_PER_DAY), Some(-MICROS_PER_SECOND)];
+ for (udf, expected) in [
+ (
+ Box::new(SparkHour::new("UTC".to_string())) as Box<dyn ScalarUDFImpl>,
+ vec![23, 0, 23],
+ ),
+ (
+ Box::new(SparkMinute::new("UTC".to_string())),
+ vec![59, 0, 59],
+ ),
+ (
+ Box::new(SparkSecond::new("UTC".to_string())),
+ vec![59, 0, 59],
+ ),
+ ] {
+ // Both the NTZ shape and the UTC-tagged shape take the fast path.
+ let ntz: ArrayRef = Arc::new(TimestampMicrosecondArray::from(values.clone()));
+ let tagged: ArrayRef =
+ Arc::new(TimestampMicrosecondArray::from(values.clone()).with_timezone("UTC"));
+ for array in [ntz, tagged] {
+ assert_eq!(
+ invoke_all(udf.as_ref(), array).values(),
+ &expected[..],
+ "{} mismatch",
+ udf.name()
+ );
+ }
+ }
+ }
+
+ /// The fast path must carry the null buffer over unchanged, and the value stored under a
+ /// null slot must not influence the result.
+ #[test]
+ fn fast_path_preserves_nulls() {
+ let array: ArrayRef = Arc::new(TimestampMicrosecondArray::from(vec![
+ Some(MICROS),
+ None,
+ Some(-1),
+ ]));
+ let out = invoke_all(&SparkHour::new("UTC".to_string()), array);
+ assert_eq!(out.null_count(), 1);
+ assert!(out.is_null(1));
+ assert_eq!(out.value(0), 18);
+ assert_eq!(out.value(2), 23);
+ }
+
+ /// A session timezone with a real offset must not be routed through the fast path.
+ #[test]
+ fn non_utc_session_is_not_accelerated() {
+ for tz in ["America/Los_Angeles", "Asia/Tokyo", "+05:30"] {
+ assert!(!is_utc_timezone(tz), "{tz} must not be treated as UTC");
+ }
+ let array: ArrayRef =
+ Arc::new(TimestampMicrosecondArray::from(vec![Some(MICROS)]).with_timezone("UTC"));
+ // 18:30 UTC is 03:30 the next day in Tokyo (UTC+9).
+ assert_eq!(invoke(&SparkHour::new("Asia/Tokyo".to_string()), array), 3);
+ }
+
#[test]
fn timestamp_with_timezone_converts_to_session_timezone() {
// A timezone-aware timestamp stores the absolute UTC instant. In a Los Angeles session
diff --git a/native/spark-expr/src/datetime_funcs/mod.rs b/native/spark-expr/src/datetime_funcs/mod.rs
index 05530f29c2..6c8c33b1ef 100644
--- a/native/spark-expr/src/datetime_funcs/mod.rs
+++ b/native/spark-expr/src/datetime_funcs/mod.rs
@@ -19,6 +19,7 @@ mod date_diff;
mod date_from_unix_date;
mod date_trunc;
mod day_month_name;
+mod day_of_week;
mod extract_date_part;
mod hours;
mod make_date;
@@ -34,6 +35,7 @@ pub use date_diff::SparkDateDiff;
pub use date_from_unix_date::SparkDateFromUnixDate;
pub use date_trunc::SparkDateTrunc;
pub use day_month_name::{spark_day_name, spark_month_name};
+pub use day_of_week::{SparkDayOfWeek, SparkWeekDay};
pub use extract_date_part::SparkHour;
pub use extract_date_part::SparkMinute;
pub use extract_date_part::SparkSecond;
diff --git a/native/spark-expr/src/lib.rs b/native/spark-expr/src/lib.rs
index 052e2b5796..e3b77cefab 100644
--- a/native/spark-expr/src/lib.rs
+++ b/native/spark-expr/src/lib.rs
@@ -78,9 +78,9 @@ pub use comet_scalar_funcs::{
pub use csv_funcs::*;
pub use datetime_funcs::{
spark_day_name, spark_month_name, spark_to_time, SparkDateDiff, SparkDateFromUnixDate,
- SparkDateTrunc, SparkHour, SparkHoursTransform, SparkMakeDate, SparkMakeInterval,
- SparkMakeTime, SparkMinute, SparkNextDay, SparkSecond, SparkSecondsToTimestamp,
- SparkUnixTimestamp, TimestampTruncExpr,
+ SparkDateTrunc, SparkDayOfWeek, SparkHour, SparkHoursTransform, SparkMakeDate,
+ SparkMakeInterval, SparkMakeTime, SparkMinute, SparkNextDay, SparkSecond,
+ SparkSecondsToTimestamp, SparkUnixTimestamp, SparkWeekDay, TimestampTruncExpr,
};
pub use error::{decimal_overflow_error, SparkError, SparkErrorWithContext, SparkResult};
pub use hash_funcs::*;
diff --git a/spark/src/main/scala/org/apache/comet/serde/datetime.scala b/spark/src/main/scala/org/apache/comet/serde/datetime.scala
index 8f2b9ab671..ab40b70657 100644
--- a/spark/src/main/scala/org/apache/comet/serde/datetime.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/datetime.scala
@@ -40,12 +40,7 @@ private object CometGetDateField extends Enumeration {
val Year: Value = Value("year")
val Month: Value = Value("month")
val DayOfMonth: Value = Value("day")
- // Datafusion: day of the week where Sunday is 0, but spark sunday is 1 (1 = Sunday,
- // 2 = Monday, ..., 7 = Saturday).
- val DayOfWeek: Value = Value("dow")
val DayOfYear: Value = Value("doy")
- // Datafusion `isodow` is 1..=7 with Monday=1; Spark `WeekDay` is 0..=6 with Monday=0.
- val WeekDay: Value = Value("isodow")
val WeekOfYear: Value = Value("week")
val Quarter: Value = Value("quarter")
}
@@ -111,61 +106,33 @@ object CometDayOfMonth
}
}
-object CometDayOfWeek
- extends CometExpressionSerde[DayOfWeek]
- with CometExprGetDateField[DayOfWeek] {
+/**
+ * Spark `dayofweek` numbers Sunday = 1 through Saturday = 7. The native `spark_dayofweek` kernel
+ * derives that from the epoch day with a single modulo, replacing a `datepart('dow', ..)` call
+ * (which builds a calendar datetime per row) plus a separate `+ 1` arithmetic node.
+ */
+object CometDayOfWeek extends CometExpressionSerde[DayOfWeek] {
override def convert(
expr: DayOfWeek,
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
- // Datafusion: day of the week where Sunday is 0, but spark sunday is 1 (1 = Sunday,
- // 2 = Monday, ..., 7 = Saturday). So we need to add 1 to the result of datepart(dow, ...)
- val optExpr = getDateField(expr, CometGetDateField.DayOfWeek, inputs, binding)
- .zip(exprToProtoInternal(Literal(1), inputs, binding))
- .map { case (left, right) =>
- Expr
- .newBuilder()
- .setAdd(
- ExprOuterClass.MathExpr
- .newBuilder()
- .setLeft(left)
- .setRight(right)
- .setEvalMode(ExprOuterClass.EvalMode.LEGACY)
- .setReturnType(serializeDataType(IntegerType).get)
- .build())
- .build()
- }
- .headOption
- optExpr
+ val childExpr = exprToProtoInternal(expr.child, inputs, binding)
+ scalarFunctionExprToProto("spark_dayofweek", childExpr)
}
}
-object CometWeekDay extends CometExpressionSerde[WeekDay] with CometExprGetDateField[WeekDay] {
+/**
+ * Spark `weekday` numbers Monday = 0 through Sunday = 6, a different convention from
+ * [[CometDayOfWeek]]. The native `spark_weekday` kernel derives it from the epoch day directly,
+ * replacing `datepart('isodow', ..)` plus a `- 1` arithmetic node.
+ */
+object CometWeekDay extends CometExpressionSerde[WeekDay] {
override def convert(
expr: WeekDay,
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
- // Datafusion `isodow` is 1..=7 with Monday=1, but Spark `WeekDay` is 0..=6 with Monday=0,
- // so subtract 1 from the result of datepart(isodow, ...).
- // TODO: fix upstream to avoid substraction
- // https://github.com/apache/datafusion/issues/22599
- val optExpr = getDateField(expr, CometGetDateField.WeekDay, inputs, binding)
- .zip(exprToProtoInternal(Literal(1), inputs, binding))
- .map { case (left, right) =>
- Expr
- .newBuilder()
- .setSubtract(
- ExprOuterClass.MathExpr
- .newBuilder()
- .setLeft(left)
- .setRight(right)
- .setEvalMode(ExprOuterClass.EvalMode.LEGACY)
- .setReturnType(serializeDataType(IntegerType).get)
- .build())
- .build()
- }
- .headOption
- optExpr
+ val childExpr = exprToProtoInternal(expr.child, inputs, binding)
+ scalarFunctionExprToProto("spark_weekday", childExpr)
}
}
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/hour.sql b/spark/src/test/resources/sql-tests/expressions/datetime/hour.sql
index 86d2ef9439..9aa156edef 100644
--- a/spark/src/test/resources/sql-tests/expressions/datetime/hour.sql
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/hour.sql
@@ -15,11 +15,17 @@
-- specific language governing permissions and limitations
-- under the License.
+-- hour/minute/second dispatch on the session timezone: with a zero-offset session the value is
+-- read straight from the stored microseconds, and otherwise it is shifted to the session zone
+-- first. Pinning both zones exercises both branches rather than leaving the choice to the
+-- ambient JVM zone of whichever runner executes this.
+-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles
+
statement
CREATE TABLE test_hour(ts timestamp) USING parquet
statement
-INSERT INTO test_hour VALUES (timestamp('2024-01-15 00:00:00')), (timestamp('2024-01-15 12:30:45')), (timestamp('2024-01-15 23:59:59')), (NULL)
+INSERT INTO test_hour VALUES (timestamp('2024-01-15 00:00:00')), (timestamp('2024-01-15 12:30:45')), (timestamp('2024-01-15 23:59:59')), (timestamp('1969-12-31 23:59:59')), (NULL)
query
SELECT hour(ts) FROM test_hour
@@ -34,7 +40,7 @@ statement
CREATE TABLE test_hour_ntz(ts timestamp_ntz) USING parquet
statement
-INSERT INTO test_hour_ntz VALUES (cast('2024-01-15 00:00:00' as timestamp_ntz)), (cast('2024-01-15 12:30:45' as timestamp_ntz)), (cast('2024-01-15 23:59:59' as timestamp_ntz)), (NULL)
+INSERT INTO test_hour_ntz VALUES (cast('2024-01-15 00:00:00' as timestamp_ntz)), (cast('2024-01-15 12:30:45' as timestamp_ntz)), (cast('2024-01-15 23:59:59' as timestamp_ntz)), (cast('1969-12-31 23:59:59' as timestamp_ntz)), (NULL)
query
SELECT hour(ts) FROM test_hour_ntz
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/minute.sql b/spark/src/test/resources/sql-tests/expressions/datetime/minute.sql
index a1c3a47536..ff638af417 100644
--- a/spark/src/test/resources/sql-tests/expressions/datetime/minute.sql
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/minute.sql
@@ -15,11 +15,17 @@
-- specific language governing permissions and limitations
-- under the License.
+-- hour/minute/second dispatch on the session timezone: with a zero-offset session the value is
+-- read straight from the stored microseconds, and otherwise it is shifted to the session zone
+-- first. Pinning both zones exercises both branches rather than leaving the choice to the
+-- ambient JVM zone of whichever runner executes this.
+-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles
+
statement
CREATE TABLE test_minute(ts timestamp) USING parquet
statement
-INSERT INTO test_minute VALUES (timestamp('2024-01-15 10:00:00')), (timestamp('2024-01-15 10:30:00')), (timestamp('2024-01-15 10:59:59')), (NULL)
+INSERT INTO test_minute VALUES (timestamp('2024-01-15 10:00:00')), (timestamp('2024-01-15 10:30:00')), (timestamp('2024-01-15 10:59:59')), (timestamp('1969-12-31 23:59:59')), (NULL)
query
SELECT minute(ts) FROM test_minute
@@ -34,7 +40,7 @@ statement
CREATE TABLE test_minute_ntz(ts timestamp_ntz) USING parquet
statement
-INSERT INTO test_minute_ntz VALUES (cast('2024-01-15 00:00:00' as timestamp_ntz)), (cast('2024-01-15 12:30:45' as timestamp_ntz)), (cast('2024-01-15 23:59:59' as timestamp_ntz)), (NULL)
+INSERT INTO test_minute_ntz VALUES (cast('2024-01-15 00:00:00' as timestamp_ntz)), (cast('2024-01-15 12:30:45' as timestamp_ntz)), (cast('2024-01-15 23:59:59' as timestamp_ntz)), (cast('1969-12-31 23:59:59' as timestamp_ntz)), (NULL)
query
SELECT minute(ts) FROM test_minute_ntz
diff --git a/spark/src/test/resources/sql-tests/expressions/datetime/second.sql b/spark/src/test/resources/sql-tests/expressions/datetime/second.sql
index d78366b5a7..db1d7bd8cb 100644
--- a/spark/src/test/resources/sql-tests/expressions/datetime/second.sql
+++ b/spark/src/test/resources/sql-tests/expressions/datetime/second.sql
@@ -15,11 +15,17 @@
-- specific language governing permissions and limitations
-- under the License.
+-- hour/minute/second dispatch on the session timezone: with a zero-offset session the value is
+-- read straight from the stored microseconds, and otherwise it is shifted to the session zone
+-- first. Pinning both zones exercises both branches rather than leaving the choice to the
+-- ambient JVM zone of whichever runner executes this.
+-- ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles
+
statement
CREATE TABLE test_second(ts timestamp) USING parquet
statement
-INSERT INTO test_second VALUES (timestamp('2024-01-15 10:30:00')), (timestamp('2024-01-15 10:30:30')), (timestamp('2024-01-15 10:30:59')), (NULL)
+INSERT INTO test_second VALUES (timestamp('2024-01-15 10:30:00')), (timestamp('2024-01-15 10:30:30')), (timestamp('2024-01-15 10:30:59')), (timestamp('1969-12-31 23:59:59')), (NULL)
query
SELECT second(ts) FROM test_second
@@ -34,7 +40,7 @@ statement
CREATE TABLE test_second_ntz(ts timestamp_ntz) USING parquet
statement
-INSERT INTO test_second_ntz VALUES (cast('2024-01-15 00:00:00' as timestamp_ntz)), (cast('2024-01-15 12:30:45' as timestamp_ntz)), (cast('2024-01-15 23:59:59' as timestamp_ntz)), (NULL)
+INSERT INTO test_second_ntz VALUES (cast('2024-01-15 00:00:00' as timestamp_ntz)), (cast('2024-01-15 12:30:45' as timestamp_ntz)), (cast('2024-01-15 23:59:59' as timestamp_ntz)), (cast('1969-12-31 23:59:59' as timestamp_ntz)), (NULL)
query
SELECT second(ts) FROM test_second_ntz
diff --git a/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala
index 620e75b43e..4da9f6caee 100644
--- a/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala
@@ -1881,6 +1881,32 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("dayofweek/weekday over a date column, including ancient dates") {
+ Seq(false, true).foreach { dictionary =>
+ // 0001-01-01 predates the Gregorian cutover, and Spark 3.x refuses to write such a date to
+ // parquet unless the rebase mode is set (Spark 4 already defaults to CORRECTED). CORRECTED
+ // writes the value as-is in the proleptic Gregorian calendar, which is the calendar the
+ // results are compared in, so it does not affect what dayofweek/weekday return.
+ withSQLConf(
+ "parquet.enable.dictionary" -> dictionary.toString,
+ SQLConf.PARQUET_REBASE_MODE_IN_WRITE.key -> "CORRECTED") {
+ val table = "test"
+ withTable(table) {
+ sql(s"create table $table(col date) using parquet")
+ // The week around the epoch pins both numbering conventions (1970-01-01 is a
+ // Thursday); the rest cover a leap day, the Gregorian century rules, and an ancient
+ // date on the far side of the Gregorian cutover.
+ sql(s"""insert into $table values
+ | (date('1969-12-28')), (date('1970-01-01')), (date('1970-01-04')),
+ | (date('1900-01-01')), (date('2000-02-29')), (date('2024-02-29')),
+ | (date('0001-01-01')), (date('9999-12-31')), (null)""".stripMargin)
+ checkSparkAnswerAndOperator(
+ s"SELECT col, dayofweek(col), weekday(col) FROM $table ORDER BY col")
+ }
+ }
+ }
+ }
+
test("from_unixtime") {
Seq(false, true).foreach { dictionary =>
withSQLConf(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 02/02: test: expand replace compatibility regression coverage (#5409)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 8c229a703ccb024a8b5b1b849a56ceef0b66a4bd
Author: sam-1112 <75...@users.noreply.github.com>
AuthorDate: Wed Sep 16 03:28:24 2026 +0000
test: expand replace compatibility regression coverage (#5409)
* test: expand replace compatibility regression coverage
* test: make replace dispatcher assertion robust
---------
Co-authored-by: Hung <hu...@Hungs-Mac-mini.local>
Co-authored-by: Andy Grove <ag...@apache.org>
---
.../expressions/string/string_replace.sql | 22 +++-
.../scala/org/apache/comet/CometCodegenSuite.scala | 118 +++++++++++++++++++++
2 files changed, 139 insertions(+), 1 deletion(-)
diff --git a/spark/src/test/resources/sql-tests/expressions/string/string_replace.sql b/spark/src/test/resources/sql-tests/expressions/string/string_replace.sql
index ad09525b6d..db4800ca57 100644
--- a/spark/src/test/resources/sql-tests/expressions/string/string_replace.sql
+++ b/spark/src/test/resources/sql-tests/expressions/string/string_replace.sql
@@ -19,7 +19,7 @@ statement
CREATE TABLE test_str_replace(s string, search string, replace string) USING parquet
statement
-INSERT INTO test_str_replace VALUES ('hello world', 'world', 'there'), ('aaa', 'a', 'bb'), ('hello', 'xyz', 'abc'), ('', 'a', 'b'), (NULL, 'a', 'b'), ('hello', '', 'x')
+INSERT INTO test_str_replace VALUES ('hello world', 'world', 'there'), ('aaa', 'a', 'bb'), ('hello', 'xyz', 'abc'), ('', 'a', 'b'), (NULL, 'a', 'b'), ('hello', '', 'x'), ('aaaa', 'aa', 'x'), ('你好你好', '你好', 'X'), ('😀a😀', '😀', 'x')
query
SELECT replace(s, search, replace) FROM test_str_replace
@@ -41,6 +41,26 @@ SELECT replace(NULL, '', 'x')
query
SELECT replace('hello', '', NULL)
+-- Overlapping candidates: Spark replaces non-overlapping left-to-right
+-- ('aaaa' + 'aa' -> 'xx'). Compatibility coverage for overlapping candidates.
+query
+SELECT replace(s, 'aa', 'x') FROM test_str_replace WHERE s = 'aaaa'
+
+-- Multi-byte UTF-8 values. Compatibility coverage for multi-byte UTF-8 values.
+query
+SELECT replace(s, '你好', 'X') FROM test_str_replace WHERE s = '你好你好'
+
+query
+SELECT replace(s, '😀', 'x') FROM test_str_replace WHERE s = '😀a😀'
+
+-- Multi-byte replacement.
+query
+SELECT replace(s, '你好', '世界') FROM test_str_replace WHERE s = '你好你好'
+
+-- Replacement contains the search string; replacement is not applied recursively.
+query
+SELECT replace(s, 'aa', 'aaa') FROM test_str_replace WHERE s = 'aaaa'
+
-- column + literal + literal
query
SELECT replace(s, 'world', 'there') FROM test_str_replace
diff --git a/spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala b/spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala
index 403007bca0..149ee7a145 100644
--- a/spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala
@@ -139,6 +139,29 @@ class CometCodegenSuite
}
}
+ /**
+ * Whether `replace` was routed through the JVM codegen dispatcher. Inspects the expression
+ * collection rather than formatted explain text: `rollUpInfoMessages` concatenates sibling
+ * names alphabetically (`cast, divide, ..., replace`), so a substring `"JVM codegen dispatcher:
+ * replace"` misses the case where `replace` is not first.
+ */
+ private def replaceIsDispatched(
+ plan: org.apache.spark.sql.execution.SparkPlan): (Seq[String], String) = {
+ val info = new ExtendedExplainInfo()
+ (info.getCodegenDispatchExpressions(plan), info.generateExtendedInfo(plan))
+ }
+
+ private def assertReplaceDispatch(
+ df: org.apache.spark.sql.DataFrame,
+ expectDispatcher: Boolean,
+ clue: String): Unit = {
+ checkSparkAnswerAndOperator(df)
+ val (dispatched, explain) = replaceIsDispatched(df.queryExecution.executedPlan)
+ assert(
+ dispatched.contains("replace") == expectDispatcher,
+ s"$clue, got dispatched expressions: $dispatched\n$explain")
+ }
+
test("codegen kernel round-trips CalendarIntervalType") {
val input = new IntervalMonthDayNanoVector("in", CometArrowAllocator)
val field =
@@ -568,6 +591,101 @@ class CometCodegenSuite
}
}
+ test("replace compatibility boundary cases stay on JVM codegen dispatcher") {
+ withSQLConf(
+ CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true",
+ CometConf.COMET_EXPLAIN_CODEGEN_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_PROJECT_ENABLED.key -> "true",
+ CometConf.COMET_EXTENDED_EXPLAIN_FORMAT.key ->
+ CometConf.COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE) {
+
+ // Malformed search: CometLiteral would normalize 0xFF to U+FFFD, incorrectly matching
+ // a well-formed U+FFFD in the source.
+ withTable("t") {
+ sql("CREATE TABLE t (s STRING) USING parquet")
+ sql("INSERT INTO t VALUES ('\uFFFD'), ('ok')")
+ assertReplaceDispatch(
+ sql("SELECT replace(s, CAST(X'FF' AS STRING), 'x') FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for malformed search literal")
+ }
+
+ // Malformed replacement has the same serialization hazard.
+ withTable("t") {
+ sql("CREATE TABLE t (s STRING) USING parquet")
+ sql("INSERT INTO t VALUES ('a'), ('b')")
+ assertReplaceDispatch(
+ sql("SELECT replace(s, 'a', CAST(X'FF' AS STRING)) FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for malformed replacement literal")
+ }
+
+ // Spark skips replacement evaluation when src is NULL; native evaluates every child.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("t") {
+ sql("CREATE TABLE t (s STRING, n INT) USING parquet")
+ sql("INSERT INTO t VALUES (NULL, 0), ('a', 1)")
+ assertReplaceDispatch(
+ sql("SELECT replace(s, 'a', CAST(1 / n AS STRING)) FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for throwing replacement expression")
+ }
+ }
+
+ // A 256 KiB scalar replacement overflows Arrow Utf8 offsets when broadcast to 8192 rows.
+ withTable("t") {
+ sql("CREATE TABLE t (s STRING) USING parquet")
+ sql("INSERT INTO t VALUES ('hello')")
+ assertReplaceDispatch(
+ sql("SELECT replace(s, 'notfound', repeat('x', 262144)) FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for oversized replacement literal")
+ }
+
+ // Source is not on the whitelist: Spark short-circuits inside substring when s is NULL.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ withTable("t") {
+ sql("CREATE TABLE t (s STRING, n INT) USING parquet")
+ sql("INSERT INTO t VALUES (NULL, 0), ('a', 1)")
+ assertReplaceDispatch(
+ sql("SELECT replace(substring(s, 1, CAST(1 / n AS INT)), 'a', 'x') FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for throwing expression nested in source")
+ }
+ }
+
+ // Malformed source literal: same CometLiteral byte-normalization as search/replacement.
+ withTable("t") {
+ sql("CREATE TABLE t (r STRING) USING parquet")
+ sql("INSERT INTO t VALUES ('x')")
+ assertReplaceDispatch(
+ sql("SELECT replace(CAST(X'FF' AS STRING), 'a', r) FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for malformed source literal")
+ }
+
+ // Malformed literal nested under concat is still in the source tree.
+ withTable("t") {
+ sql("CREATE TABLE t (r STRING) USING parquet")
+ sql("INSERT INTO t VALUES ('x')")
+ assertReplaceDispatch(
+ sql("SELECT replace(concat(CAST(X'FF' AS STRING), r), 'a', 'x') FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for malformed literal nested in source")
+ }
+
+ // Oversized source literal has the same broadcast / offset-overflow hazard.
+ withTable("t") {
+ sql("CREATE TABLE t (r STRING) USING parquet")
+ sql("INSERT INTO t VALUES ('x')")
+ assertReplaceDispatch(
+ sql("SELECT replace(repeat('x', 262144), 'notfound', r) FROM t"),
+ expectDispatcher = true,
+ "expected dispatcher path for oversized source literal")
+ }
+ }
+ }
+
test("codegen dispatch fallback reasons name the expression") {
// Flag-off short-circuit tags the expression `<name>: <reason>` so distinct expressions
// don't collapse in the `Set[String]` roll-up.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org