You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazantsev-maksim (via GitHub)" <gi...@apache.org> on 2025/10/05 11:37:38 UTC

[PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

kazantsev-maksim opened a new pull request, #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523

   ## Which issue does this PR close?
   
   ## Rationale for this change
   
   ## What changes are included in this PR?
   
   ## How are these changes tested?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3380048220

   The DataFusion implementation of [date_trunc](https://github.com/apache/datafusion/blob/main/datafusion/functions/src/datetime/date_trunc.rs#L279) has a narrower scope than Spark's, as it only accepts TimestampType, while Spark's version additionally supports StringType and DateType.
   
   Comet does not fully support casting DateType to TimestampType yet.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3377203204

   Current Comet performance for `date_trunc` isn't great according to `CometDatetimeExpressionBenchmark`.
   
   ```
   Running benchmark: Date Truncate  - YEAR
     Running case: Date Truncate  - YEAR - Spark 
     Stopped after 38 iterations, 2006 ms
     Running case: Date Truncate  - YEAR - Comet
     Stopped after 39 iterations, 2022 ms
   
   OpenJDK 64-Bit Server VM 11.0.28+6-post-Ubuntu-1ubuntu122.04.1 on Linux 6.8.0-84-generic
   AMD Ryzen 9 7950X3D 16-Core Processor
   Date Truncate  - YEAR:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   Date Truncate  - YEAR - Spark                        46             53           6         23.0          43.5       1.0X
   Date Truncate  - YEAR - Comet                        46             52           5         22.6          44.2       1.0X
   
   Running benchmark: Date Truncate  - YYYY
     Running case: Date Truncate  - YYYY - Spark 
     Stopped after 46 iterations, 2019 ms
     Running case: Date Truncate  - YYYY - Comet
     Stopped after 42 iterations, 2006 ms
   
   OpenJDK 64-Bit Server VM 11.0.28+6-post-Ubuntu-1ubuntu122.04.1 on Linux 6.8.0-84-generic
   AMD Ryzen 9 7950X3D 16-Core Processor
   Date Truncate  - YYYY:                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   Date Truncate  - YYYY - Spark                        39             44           6         26.6          37.6       1.0X
   Date Truncate  - YYYY - Comet                        43             48           6         24.4          40.9       0.9X
   
   Running benchmark: Date Truncate  - YY
     Running case: Date Truncate  - YY - Spark 
     Stopped after 51 iterations, 2010 ms
     Running case: Date Truncate  - YY - Comet
     Stopped after 44 iterations, 2040 ms
   
   OpenJDK 64-Bit Server VM 11.0.28+6-post-Ubuntu-1ubuntu122.04.1 on Linux 6.8.0-84-generic
   AMD Ryzen 9 7950X3D 16-Core Processor
   Date Truncate  - YY:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
   ------------------------------------------------------------------------------------------------------------------------
   Date Truncate  - YY - Spark                          37             39           3         28.6          34.9       1.0X
   Date Truncate  - YY - Comet                          43             46           6         24.5          40.8       0.9X
   ```
   
   I tried running against this PR but got similar failures to the ones we are seeing in CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim closed pull request #2523: Chore: Use DataFusion impl of date_trunc function
URL: https://github.com/apache/datafusion-comet/pull/2523


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3372098335

   There are some compilation issues


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3372119077

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 10.44%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a173917`](https://app.codecov.io/gh/apache/datafusion-comet/commit/a173917b2f97e27791796a6730ad135bb065990e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 576 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...c/main/scala/org/apache/comet/serde/datetime.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Fdatetime.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9kYXRldGltZS5zY2FsYQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #2523       +/-   ##
   =============================================
   - Coverage     56.12%   10.44%   -45.68%     
   + Complexity      976      232      -744     
   =============================================
     Files           119      146       +27     
     Lines         11743    13552     +1809     
     Branches       2251     2341       +90     
   =============================================
   - Hits           6591     1416     -5175     
   - Misses         4012    11959     +7947     
   + Partials       1140      177      -963     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/2523?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Chore: Use DataFusion impl of date_trunc function [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #2523:
URL: https://github.com/apache/datafusion-comet/pull/2523#issuecomment-3377187831

   This is a good point @parthchandra we can "steal" date_trunc bench from DF and check it out. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org