You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/02 04:24:04 UTC
[GitHub] [arrow-datafusion] stuartcarnie opened a new pull request, #3010: feat: Add support for TIME literal values
stuartcarnie opened a new pull request, #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010
This is a **draft PR**, as it depends on an upgraded arrow-rs crate which contains breaking changes. This PR includes commits to resolve the changes and all tests pass, however, more test coverage for the `TIME` literal is expected, which is the primary intent of this PR.
# Which issue does this PR close?
Closes #2883.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
TODO(sgc): Write details of all the changes
# Are there any user-facing changes?
This PR teaches DataFusion about `TIME` literals and therefore documentation would need to be revised.
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1202017034
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/3010?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3010](https://codecov.io/gh/apache/arrow-datafusion/pull/3010?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cc19881) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/e23925c96c97139d101719fa7a456088eeed6ae9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e23925c) will **decrease** coverage by `0.00%`.
> The diff coverage is `94.30%`.
```diff
@@ Coverage Diff @@
## master #3010 +/- ##
==========================================
- Coverage 85.84% 85.84% -0.01%
==========================================
Files 283 283
Lines 51658 51706 +48
==========================================
+ Hits 44346 44387 +41
- Misses 7312 7319 +7
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/3010?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [datafusion/core/src/catalog/information\_schema.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9jYXRhbG9nL2luZm9ybWF0aW9uX3NjaGVtYS5ycw==) | `94.42% <0.00%> (ø)` | |
| [datafusion/core/src/physical\_plan/hash\_utils.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL2hhc2hfdXRpbHMucnM=) | `40.43% <0.00%> (ø)` | |
| [...tafusion/core/src/physical\_plan/sort\_merge\_join.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL3NvcnRfbWVyZ2Vfam9pbi5ycw==) | `90.36% <0.00%> (ø)` | |
| [datafusion/core/tests/sql/aggregates.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9hZ2dyZWdhdGVzLnJz) | `99.28% <ø> (ø)` | |
| [datafusion/core/tests/sql/decimal.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9kZWNpbWFsLnJz) | `100.00% <ø> (ø)` | |
| [datafusion/core/tests/sql/joins.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9qb2lucy5ycw==) | `99.32% <ø> (ø)` | |
| [datafusion/expr/src/type\_coercion.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy90eXBlX2NvZXJjaW9uLnJz) | `98.87% <ø> (ø)` | |
| [datafusion/proto/src/from\_proto.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvZnJvbV9wcm90by5ycw==) | `35.58% <50.00%> (ø)` | |
| [datafusion/common/src/scalar.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb21tb24vc3JjL3NjYWxhci5ycw==) | `84.61% <75.86%> (-0.28%)` | :arrow_down: |
| [...sion/core/src/physical\_plan/file\_format/parquet.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFuL2ZpbGVfZm9ybWF0L3BhcnF1ZXQucnM=) | `95.46% <83.33%> (ø)` | |
| ... and [26 more](https://codecov.io/gh/apache/arrow-datafusion/pull/3010/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] stuartcarnie commented on a diff in pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
stuartcarnie commented on code in PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#discussion_r945463524
##########
datafusion/sql/src/planner.rs:
##########
@@ -496,7 +496,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
SQLDataType::Double => Ok(DataType::Float64),
SQLDataType::Boolean => Ok(DataType::Boolean),
SQLDataType::Date => Ok(DataType::Date32),
- SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Millisecond)),
+ SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Nanosecond)),
Review Comment:
This change constitutes a fix as `Time64(Millisecond)` is invalid. Rather that switch to `Time32(Millisecond)`, I moved to `Time64(Nanosecond)` as PostgreSQL requires at least _microsecond_ precision per [the documentation](https://www.postgresql.org/docs/14/datatype-datetime.html#DATATYPE-DATETIME-TABLE). Given Nanosecond comes at no additional cost, it seems reasonable to support the increase in precision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1215447603
thank you @stuartcarnie !
note that time64 cannot be compared yet as there's no implementation in the [kernel](https://github.com/apache/arrow-rs/blob/master/arrow/src/compute/kernels/comparison.rs#L1897).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1215617319
Benchmark runs are scheduled for baseline = 49a3b005ca3e0415e920b2459b1121dad729b743 and contender = 15a9a4becc4bb41262b10f1d5395fc1d026de753. 15a9a4becc4bb41262b10f1d5395fc1d026de753 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5298704331ec4dba95a102800f9a3b04...e9f6a367c13947928130c535ee53f178/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/6bc2c6e33d6645a7ac708ac026083112...bf13703729ec493e9d6a6a1ef31ef9f4/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/83524bee421d45dba52acfbe21dd8e5e...73cd9461330a472594d1ca911ccda049/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/0ff8d6ef51944af8803b3356be3c0c3d...275b0bb678d149ecaa1b9c48fbe6cf2d/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] avantgardnerio commented on pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
avantgardnerio commented on PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#issuecomment-1202781087
Thank you SO much for this! I struggled for a few days on this issue. Your help is very appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010#discussion_r946032481
##########
datafusion/sql/src/planner.rs:
##########
@@ -496,7 +496,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
SQLDataType::Double => Ok(DataType::Float64),
SQLDataType::Boolean => Ok(DataType::Boolean),
SQLDataType::Date => Ok(DataType::Date32),
- SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Millisecond)),
+ SQLDataType::Time => Ok(DataType::Time64(TimeUnit::Nanosecond)),
Review Comment:
I think that is 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #3010: feat: Add support for TIME literal values
Posted by GitBox <gi...@apache.org>.
alamb merged PR #3010:
URL: https://github.com/apache/arrow-datafusion/pull/3010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org