You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazantsev-maksim (via GitHub)" <gi...@apache.org> on 2025/10/05 11:37:38 UTC
[PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
kazantsev-maksim opened a new pull request, #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523
## Which issue does this PR close?
## Rationale for this change
## What changes are included in this PR?
## How are these changes tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3380048220
The DataFusion implementation of [date_trunc](https://github.com/apache/datafusion/blob/main/datafusion/functions/src/datetime/date_trunc.rs#L279) has a narrower scope than Spark's, as it only accepts TimestampType, while Spark's version additionally supports StringType and DateType.
Comet does not fully support casting DateType to TimestampType yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3377203204
Current Comet performance for `date_trunc` isn't great according to `CometDatetimeExpressionBenchmark`.
```
Running benchmark: Date Truncate - YEAR
Running case: Date Truncate - YEAR - Spark
Stopped after 38 iterations, 2006 ms
Running case: Date Truncate - YEAR - Comet
Stopped after 39 iterations, 2022 ms
OpenJDK 64-Bit Server VM 11.0.28+6-post-Ubuntu-1ubuntu122.04.1 on Linux 6.8.0-84-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Date Truncate - YEAR: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Date Truncate - YEAR - Spark 46 53 6 23.0 43.5 1.0X
Date Truncate - YEAR - Comet 46 52 5 22.6 44.2 1.0X
Running benchmark: Date Truncate - YYYY
Running case: Date Truncate - YYYY - Spark
Stopped after 46 iterations, 2019 ms
Running case: Date Truncate - YYYY - Comet
Stopped after 42 iterations, 2006 ms
OpenJDK 64-Bit Server VM 11.0.28+6-post-Ubuntu-1ubuntu122.04.1 on Linux 6.8.0-84-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Date Truncate - YYYY: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Date Truncate - YYYY - Spark 39 44 6 26.6 37.6 1.0X
Date Truncate - YYYY - Comet 43 48 6 24.4 40.9 0.9X
Running benchmark: Date Truncate - YY
Running case: Date Truncate - YY - Spark
Stopped after 51 iterations, 2010 ms
Running case: Date Truncate - YY - Comet
Stopped after 44 iterations, 2040 ms
OpenJDK 64-Bit Server VM 11.0.28+6-post-Ubuntu-1ubuntu122.04.1 on Linux 6.8.0-84-generic
AMD Ryzen 9 7950X3D 16-Core Processor
Date Truncate - YY: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Date Truncate - YY - Spark 37 39 3 28.6 34.9 1.0X
Date Truncate - YY - Comet 43 46 6 24.5 40.8 0.9X
```
I tried running against this PR but got similar failures to the ones we are seeing in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim closed pull request #2523: Chore: Use DataFusion impl of date_trunc function
URL: https://github.com/apache/datafusion-comet/pull/2523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3372098335
There are some compilation issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3372119077
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 10.44%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a173917`](https://app.codecov.io/gh/apache/datafusion-comet/commit/a173917b2f97e27791796a6730ad135bb065990e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 576 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...c/main/scala/org/apache/comet/serde/datetime.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Fdatetime.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9kYXRldGltZS5zY2FsYQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2523 +/- ##
=============================================
- Coverage 56.12% 10.44% -45.68%
+ Complexity 976 232 -744
=============================================
Files 119 146 +27
Lines 11743 13552 +1809
Branches 2251 2341 +90
=============================================
- Hits 6591 1416 -5175
- Misses 4012 11959 +7947
+ Partials 1140 177 -963
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3377187831
This is a good point @parthchandra we can "steal" date_trunc bench from DF and check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org