You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sdf-jkl (via GitHub)" <gi...@apache.org> on 2026/09/03 04:08:13 UTC
[PR] Unify temporal field parsing with DatePart [datafusion]
sdf-jkl opened a new pull request, #24906:
URL: https://github.com/apache/datafusion/pull/24906
## Which issue does this PR close?
- Closes #22033.
## Rationale for this change
Temporal functions currently parse field names independently, which produces inconsistent user-visible behavior. For example, `date_part` accepts aliases such as `mon`, `mons`, `month`, and `months`, while `date_trunc` accepts only `month`.
Using Arrow's `DatePart` parser as the common representation makes these aliases consistent across temporal functions and SQL unparsing.
## What changes are included in this PR?
- Use `DatePart::from_str` directly in `date_part` instead of parsing through `IntervalUnit` and maintaining additional manual mappings.
- Replace the private `DateTruncGranularity` enum and its string parser with `DatePart`.
- Restrict `date_trunc` to the `DatePart` variants that represent valid truncation granularities.
- Update the Spark wrappers to delegate field parsing to the shared implementation while retaining their function-specific semantic restrictions.
- Map Spark's Sunday-based `dayofweek` directly to `dow1` instead of adding one to the result expression.
- Use `DatePart` when canonicalizing temporal fields in the SQL unparser.
## What is the testing strategy for this PR?
SQL logic tests cover the newly shared aliases, including:
- `date_part`: `QUARTERS`, `ISOWEEK`, `DAYOFYEAR`, and `DAYOFWEEK`
- `date_trunc`: `mon`, `months`, `qtr`, and `yrs`
- Spark wrappers: `QUARTERS`, `DAYOFYEAR`, `MONTHS`, and `MINS`
- Unparser canonicalization of aliases such as `MONS` and `YRS`
The following checks passed:
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- Extended workspace tests with `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`
- Spark datetime SQL logic tests
- `cargo test -p datafusion-spark --lib`
## Are there any user-facing changes?
Yes. Temporal functions now consistently accept the field-name aliases recognized by Arrow's `DatePart` parser. This broadens accepted input spellings without changing existing accepted inputs or truncation behavior.
The Spark wrappers inherit these aliases as well, making them slightly more permissive than Spark's exact spelling tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Unify temporal field parsing with DatePart [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24906:
URL: https://github.com/apache/datafusion/pull/24906#issuecomment-5520393517
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24906?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `66.16541%` with `45 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.63%. Comparing base ([`1608ee7`](https://app.codecov.io/gh/apache/datafusion/commit/1608ee70faa06063a38f01e8c57c1a6879708cd6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ea29199`](https://app.codecov.io/gh/apache/datafusion/commit/ea29199f5b87d1fb85d2cd917b54c04be5b0959a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 14 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24906?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/functions/src/datetime/date\_trunc.rs](https://app.codecov.io/gh/apache/datafusion/pull/24906?src=pr&el=tree&filepath=datafusion%2Ffunctions%2Fsrc%2Fdatetime%2Fdate_trunc.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMvc3JjL2RhdGV0aW1lL2RhdGVfdHJ1bmMucnM=) | 62.22% | [34 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24906?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/sql/src/unparser/utils.rs](https://app.codecov.io/gh/apache/datafusion/pull/24906?src=pr&el=tree&filepath=datafusion%2Fsql%2Fsrc%2Funparser%2Futils.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcWwvc3JjL3VucGFyc2VyL3V0aWxzLnJz) | 57.14% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24906?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/functions/src/datetime/date\_part.rs](https://app.codecov.io/gh/apache/datafusion/pull/24906?src=pr&el=tree&filepath=datafusion%2Ffunctions%2Fsrc%2Fdatetime%2Fdate_part.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMvc3JjL2RhdGV0aW1lL2RhdGVfcGFydC5ycw==) | 64.28% | [0 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24906?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24906 +/- ##
==========================================
+ Coverage 81.61% 81.63% +0.01%
==========================================
Files 1123 1123
Lines 409562 409685 +123
Branches 409562 409685 +123
==========================================
+ Hits 334284 334447 +163
+ Misses 55647 55587 -60
- Partials 19631 19651 +20
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24906?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Unify temporal field parsing with DatePart [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #24906:
URL: https://github.com/apache/datafusion/pull/24906#issuecomment-5535406152
Thank you @sdf-jkl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Unify temporal field parsing with DatePart [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal merged PR #24906:
URL: https://github.com/apache/datafusion/pull/24906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org