You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/23 22:11:47 UTC
[PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
andygrove opened a new pull request, #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014
## Which issue does this PR close?
Closes #5012.
## Rationale for this change
With `spark.sql.ansi.enabled=true`, `CAST(string AS date)` must raise `CAST_INVALID_INPUT` for malformed input. Comet returned `NULL` instead when the string parsed structurally but did not name a real calendar date, e.g. `'2016-13-01'`, `'2016-02-30'`, `'2016-01-32'`.
Beyond the missing exception, this silently changes results: a filter on the cast returns rows for a query that Spark fails outright.
```sql
SELECT count(*) FROM t WHERE cast(s as date) > date'2000-01-01'
-- spark: SparkDateTimeException
-- comet: 1
```
Spark's `SparkDateTimeUtils#stringToDate` returns `None` whenever `LocalDate.of` throws or the narrowing of the epoch day to an `Int` overflows (`localDateToDays` uses `Math.toIntExact`), and `stringToDateAnsi` turns every `None` into `CAST_INVALID_INPUT`. Comet's port had three paths that produced `Ok(None)` without going through `return_result`, which is the only place the `EvalMode::Ansi` error is raised.
## What changes are included in this PR?
In `date_parser` (`native/spark-expr/src/conversion_funcs/string.rs`):
* The canonical `yyyy-mm-dd` fast path now routes an invalid calendar date through `return_result` instead of returning `Ok(None)`.
* The general parser path does the same.
* The `-262143..=262142` year guard is replaced by an explicit `i32::try_from` check on the computed epoch day, so years whose epoch day overflows an `Int` fail exactly where Spark fails. The chrono representability limit is applied afterwards.
Years that Spark accepts but chrono cannot represent (roughly `262143` to `5881580` and their negative counterparts) still return `NULL` in every eval mode. Raising for those would diverge from Spark in the other direction, and Comet's downstream date kernels cannot handle those values, so that limitation is preserved and now documented in the code.
Behaviour that was already correct is unchanged: non-ANSI mode returns `NULL`, `try_cast` returns `NULL` under ANSI, and structurally malformed strings continue to throw.
The `cast(string as boolean)` exception-type discrepancy also mentioned in #5012 is not addressed here and remains open.
## How are these changes tested?
* `date_parser_test` in `native/spark-expr/src/conversion_funcs/string.rs` now asserts an error under `EvalMode::Ansi` for invalid calendar dates in both the fast-path and general-parser shapes, and for years whose epoch day overflows `i32`. The chrono-limit cases keep asserting `Ok(None)` in all three eval modes.
* `CometCastSuite` "cast StringType to DateType" gains ten invalid calendar dates in its `invalidDates` list. That test compares against Spark in both non-ANSI mode (values) and ANSI mode (exceptions), so it covers both directions. The existing fuzz generator uses a `0123456789/` plus whitespace alphabet and so never produced a hyphenated date shape, which is why this went unnoticed.
* New SQL file test `spark/src/test/resources/sql-tests/expressions/cast/cast_string_to_date_ansi.sql` covering the literal casts, `to_date`, the column-backed cast, and the filter case from the issue, plus non-error sentinel queries over valid input so a silent fallback to Spark cannot let the `expect_error` cases pass vacuously. Verified that this fixture fails on the unpatched native library and passes with the fix.
* Full `CometCastSuite` (162 passed), full `CometSqlFileTestSuite` (419 passed), `CometTemporalExpressionSuite` (32 passed), and the full `datafusion-comet-spark-expr` Rust test suite (511 passed) are green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#discussion_r3642907444
##########
native/spark-expr/src/conversion_funcs/string.rs:
##########
@@ -1930,6 +1929,31 @@ fn date_parser(date_str: &str, eval_mode: EvalMode) -> SparkResult<Option<i32>>
Ok(None)
}
}
+
+ /// Turns parsed year/month/day segments into an epoch day. Shared by both parsing paths so
+ /// that the decision of what a segment triple *means* lives in exactly one place.
+ fn finish(
+ year: i64,
+ month: i64,
+ day: i64,
+ date_str: &str,
+ eval_mode: EvalMode,
+ ) -> SparkResult<Option<i32>> {
+ // Spark builds a `LocalDate` and narrows its epoch day to an `Int`, so an invalid
+ // calendar date or an epoch day that overflows `i32` both yield `None` there, which
+ // `stringToDateAnsi` turns into CAST_INVALID_INPUT.
+ let Some(days) = ymd_to_epoch_day(year, month, day).and_then(|d| i32::try_from(d).ok())
+ else {
+ return return_result(date_str, eval_mode);
+ };
+ // Spark accepts years beyond what chrono can represent, and downstream date kernels
+ // cannot handle those values, so Comet keeps returning null for them in every eval mode
+ // rather than raising. This is a Comet limitation, not a malformed input.
Review Comment:
It would be good to add the coresponding chrono issue or its code snippet here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#issuecomment-5071630177
> https://github.com/apache/datafusion-comet/actions/runs/30050199441/job/89355580948?pr=5014
> failed ci looks unrelated https://github.com/apache/datafusion-comet/actions/runs/30050199441/job/89355580948?pr=5014
>
> ```
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x0000000000000000, pid=22862, tid=59915
> #
> # JRE version: OpenJDK Runtime Environment Zulu17.68+17-CA (17.0.20+8) (build 17.0.20+8-LTS)
> # Java VM: OpenJDK 64-Bit Server VM Zulu17.68+17-CA (17.0.20+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
> # Problematic frame:
> # C 0x0000000000000000
> #
> # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /Users/runner/work/datafusion-comet/datafusion-comet/spark/hs_err_pid22862.log
> #
> # If you would like to submit a bug report, please visit:
> # http://www.azul.com/support/
> #
> ```
I've seen this a few times lately. I will file a tracking issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#issuecomment-5072248772
@comphead I went ahead and addressed some of the feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#discussion_r3646339447
##########
native/spark-expr/src/conversion_funcs/string.rs:
##########
@@ -2798,6 +2809,25 @@ mod tests {
);
}
+ /// Malformed input: null in legacy and try mode, CAST_INVALID_INPUT in ANSI mode.
Review Comment:
```
fn assert_dates(
dates: &[&str],
modes: &[EvalMode],
expect_ansi_error: bool,
) {
for &date in dates {
for &mode in modes {
if expect_ansi_error && mode == EvalMode::Ansi {
assert!(date_parser(date, mode).is_err(), "{date}");
} else {
assert_eq!(date_parser(date, mode).unwrap(), None, "{date}");
}
}
}
}
/// Malformed input: null in legacy and try mode, CAST_INVALID_INPUT in ANSI mode.
fn assert_null_or_ansi_error(dates: &[&str]) {
assert_dates(
dates,
&[EvalMode::Legacy, EvalMode::Try, EvalMode::Ansi],
true,
);
}
/// Input Spark parses successfully but Comet cannot represent: null in every eval mode.
fn assert_null_in_all_modes(dates: &[&str]) {
assert_dates(
dates,
&[EvalMode::Legacy, EvalMode::Try, EvalMode::Ansi],
false,
);
}
```
to remove some duplication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#discussion_r3646288812
##########
native/spark-expr/src/conversion_funcs/string.rs:
##########
@@ -1930,6 +1929,31 @@ fn date_parser(date_str: &str, eval_mode: EvalMode) -> SparkResult<Option<i32>>
Ok(None)
}
}
+
+ /// Turns parsed year/month/day segments into an epoch day. Shared by both parsing paths so
+ /// that the decision of what a segment triple *means* lives in exactly one place.
+ fn finish(
Review Comment:
maybe we can rename `finish` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#issuecomment-5066355705
failed ci looks unrelated
https://github.com/apache/datafusion-comet/actions/runs/30050199441/job/89355580948?pr=5014
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=22862, tid=59915
#
# JRE version: OpenJDK Runtime Environment Zulu17.68+17-CA (17.0.20+8) (build 17.0.20+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.68+17-CA (17.0.20+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C 0x0000000000000000
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/runner/work/datafusion-comet/datafusion-comet/spark/hs_err_pid22862.log
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/
#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: raise CAST_INVALID_INPUT for invalid calendar dates in ANSI cast to date [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5014:
URL: https://github.com/apache/datafusion-comet/pull/5014#discussion_r3646298031
##########
native/spark-expr/src/conversion_funcs/string.rs:
##########
@@ -1930,6 +1929,31 @@ fn date_parser(date_str: &str, eval_mode: EvalMode) -> SparkResult<Option<i32>>
Ok(None)
}
}
+
+ /// Turns parsed year/month/day segments into an epoch day. Shared by both parsing paths so
+ /// that the decision of what a segment triple *means* lives in exactly one place.
+ fn finish(
+ year: i64,
Review Comment:
`chrono` usually uses `u32`
https://docs.rs/chrono/latest/chrono/trait.Datelike.html#tymethod.with_day
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org