You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/02 04:24:04 UTC

[GitHub] [arrow-datafusion] stuartcarnie opened a new pull request, #3010: feat: Add support for TIME literal values

stuartcarnie opened a new pull request, #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010

   This is a **draft PR**, as it depends on an upgraded arrow-rs crate which contains breaking changes. This PR includes commits to resolve the changes and all tests pass, however, more test coverage for the `TIME` literal is expected, which is the primary intent of this PR.
   
   # Which issue does this PR close?
   
   Closes #2883.
   
    # Rationale for this change
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   # What changes are included in this PR?
   
   TODO(sgc): Write details of all the changes
   
   # Are there any user-facing changes?
   
   This PR teaches DataFusion about `TIME` literals and therefore documentation would need to be revised. 
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1202017034

   # [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/3010?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3010](https://codecov.io/gh/apache/arrow-datafusion/pull/3010?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cc19881) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/e23925c96c97139d101719fa7a456088eeed6ae9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e23925c) will **decrease** coverage by `0.00%`.
   > The diff coverage is `94.30%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #3010      +/-   ##
   ==========================================
   - Coverage   85.84%   85.84%   -0.01%     
   ==========================================
     Files         283      283              
     Lines       51658    51706      +48     
   ==========================================
   + Hits        44346    44387      +41     
   - Misses       7312     7319       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/3010?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [datafusion/core/src/catalog/information\_schema.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9jYXRhbG9nL2luZm9ybWF0aW9uX3NjaGVtYS5ycw==) | `94.42% <0.00%> (ø)` | |
   | [datafusion/core/src/physical\_plan/hash\_utils.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL2hhc2hfdXRpbHMucnM=) | `40.43% <0.00%> (ø)` | |
   | [...tafusion/core/src/physical\_plan/sort\_merge\_join.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL3NvcnRfbWVyZ2Vfam9pbi5ycw==) | `90.36% <0.00%> (ø)` | |
   | [datafusion/core/tests/sql/aggregates.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9hZ2dyZWdhdGVzLnJz) | `99.28% <ø> (ø)` | |
   | [datafusion/core/tests/sql/decimal.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9kZWNpbWFsLnJz) | `100.00% <ø> (ø)` | |
   | [datafusion/core/tests/sql/joins.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9qb2lucy5ycw==) | `99.32% <ø> (ø)` | |
   | [datafusion/expr/src/type\_coercion.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy90eXBlX2NvZXJjaW9uLnJz) | `98.87% <ø> (ø)` | |
   | [datafusion/proto/src/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvZnJvbV9wcm90by5ycw==) | `35.58% <50.00%> (ø)` | |
   | [datafusion/common/src/scalar.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL3NjYWxhci5ycw==) | `84.61% <75.86%> (-0.28%)` | :arrow_down: |
   | [...sion/core/src/physical\_plan/file\_format/parquet.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL2ZpbGVfZm9ybWF0L3BhcnF1ZXQucnM=) | `95.46% <83.33%> (ø)` | |
   | ... and [26 more](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] stuartcarnie commented on a diff in pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
stuartcarnie commented on code in PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#discussion_r945463524


##########
datafusion/sql/src/planner.rs:
##########
@@ -496,7 +496,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
             SQLDataType::Double => Ok(DataType::Float64),
             SQLDataType::Boolean => Ok(DataType::Boolean),
             SQLDataType::Date => Ok(DataType::Date32),
-            SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Millisecond)),
+            SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Nanosecond)),

Review Comment:
   This change constitutes a fix as `Time64(Millisecond)` is invalid. Rather that switch to `Time32(Millisecond)`, I moved to `Time64(Nanosecond)` as PostgreSQL requires at least _microsecond_ precision per [the documentation](https://www.postgresql.org/docs/14/datatype-datetime.html#DATATYPE-DATETIME-TABLE). Given Nanosecond comes at no additional cost, it seems reasonable to support the increase in precision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] waitingkuo commented on pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
waitingkuo commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1215447603

   thank you @stuartcarnie !
   
   note that time64 cannot be compared yet as there's no implementation in the [kernel](https://github.com/apache/arrow-rs/blob/master/arrow/src/compute/kernels/comparison.rs#L1897). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] ursabot commented on pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1215617319

   Benchmark runs are scheduled for baseline = 49a3b005ca3e0415e920b2459b1121dad729b743 and contender = 15a9a4becc4bb41262b10f1d5395fc1d026de753. 15a9a4becc4bb41262b10f1d5395fc1d026de753 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5298704331ec4dba95a102800f9a3b04...e9f6a367c13947928130c535ee53f178/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/6bc2c6e33d6645a7ac708ac026083112...bf13703729ec493e9d6a6a1ef31ef9f4/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/83524bee421d45dba52acfbe21dd8e5e...73cd9461330a472594d1ca911ccda049/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/0ff8d6ef51944af8803b3356be3c0c3d...275b0bb678d149ecaa1b9c48fbe6cf2d/)
   Buildkite builds:
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] avantgardnerio commented on pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1202781087

   Thank you SO much for this! I struggled for a few days on this issue. Your help is very appreciated!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#discussion_r946032481


##########
datafusion/sql/src/planner.rs:
##########
@@ -496,7 +496,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
             SQLDataType::Double => Ok(DataType::Float64),
             SQLDataType::Boolean => Ok(DataType::Boolean),
             SQLDataType::Date => Ok(DataType::Date32),
-            SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Millisecond)),
+            SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Nanosecond)),

Review Comment:
   I think that is 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb merged pull request #3010: feat: Add support for TIME literal values

Posted by GitBox <gi...@apache.org>.
alamb merged PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org