You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/21 19:15:09 UTC
[GitHub] [arrow-datafusion] andre-cc-natzka opened a new pull request, #4312: Support type coercion for timestamp and utf8
andre-cc-natzka opened a new pull request, #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312
# Which issue does this PR close?
Closes [#431](https://github.com/apache/arrow-datafusion/issues/4311.
# Rationale for this change
Currently DataFusion supports type coercion for (`DataType::Time`, `DataType::Utf8`) and (`DataType::Date`, `DataType::Utf8`) pairs, but is still missing type coercion for (`DataType::Timestamp`, `DataType::Utf8`).
# What changes are included in this PR?
Two lines of code are added in the `temporal_coercion` function to account for type coercion for a (`DataType::Timestamp`, `DataType::Utf8`) pair. The output type is `DataType::Timestamp(TimeUnit::Nanosecond, _)`, because it is the only time unit supported by Arrow, which forces us to change the original time unit to nanoseconds. Although not ideal, this is not a problem, as both the left and right hand side of an expression are converted into `DataType::Timestamp(TimeUnit::Nanosecond, _)` and the expression can be properly evaluated.
# Are these changes tested?
Yes, corresponding test cases are added to the test function `test_type_coercion` from `datafusion/expr/src/type_coercion/binary.rs`.
# Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1338929873
hi @andre-cc-natzka thank you, it looks good to me.
I suggest that add some SQL level tests
e.g.
```bash
❯ select (timestamp '2000-01-01T00:00:00') = '2000-01-01T00:00:00';
+-----------------------------------------------------------+
| Utf8("2000-01-01T00:00:00") = Utf8("2000-01-01T00:00:00") |
+-----------------------------------------------------------+
| true |
+-----------------------------------------------------------+
1 row in set. Query took 0.002 seconds.
```
and this should raise error until we could cast TimestampTz to Utf8
```bash
❯ select (timestamptz '2000-01-01T00:00:00+00:00') = '2000-01-01T00:00:00+08:00';
NotImplemented("Unsupported CAST from Utf8 to Timestamp(Nanosecond, Some(\"+00:00\"))")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1342638288
Thank you for the contribution @andre-cc-natzka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andre-cc-natzka commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
andre-cc-natzka commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1342513732
Hello again @alamb , @waitingkuo. Thank you very much for your second revision and for merging the PR!
Thank you @alamb also for adding the SQL tests that test this new functionality, they look really nice!
All the best,
André.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1341450982
Thanks again @andre-cc-natzka -- here are some SQL level tests https://github.com/apache/arrow-datafusion/pull/4545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1342904436
thank you @andre-cc-natzka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
alamb merged PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1325461836
Marking this PR as draft as it still needs some work prior to passing CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1341398337
I plan to write some SQL level tests for this feature as they will help us in IOx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on a diff in pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on code in PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#discussion_r1028536829
##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -830,10 +844,12 @@ message ScalarValue{
ScalarDictionaryValue dictionary_value = 27;
bytes binary_value = 28;
bytes large_binary_value = 29;
+
ScalarTime64Value time64_value = 30;
IntervalMonthDayNanoValue interval_month_day_nano = 31;
StructValue struct_value = 32;
ScalarFixedSizeBinary fixed_size_binary_value = 34;
+>>>>>>> upstream/master
Review Comment:
```suggestion
```
##########
datafusion/core/tests/sql/select.rs:
##########
@@ -814,6 +814,8 @@ async fn query_on_string_dictionary() -> Result<()> {
];
assert_batches_eq!(expected, &actual);
+ // filtering with Time32 and Time64 types
+
Review Comment:
not quite sure what it is, i guess we need to delete ths
##########
datafusion/proto/proto/datafusion.proto:
##########
@@ -770,6 +770,20 @@ message ScalarTimestampValue {
string timezone = 5;
}
+message ScalarTime32Value {
+ oneof value {
+ int32 time32_second_value = 1;
+ int32 time32_millisecond_value = 2;
+ };
+}
+
+message ScalarTime64Value {
+ oneof value {
+ int64 time64_microsecond_value = 1;
+ int64 time64_nanosecond_value = 2;
+ };
+}
+
Review Comment:
i think you should rebase master, the pr for time32/64 is merged
##########
datafusion/core/tests/sql/timestamp.rs:
##########
@@ -1555,7 +1555,7 @@ async fn cast_timestamp_to_timestamptz() -> Result<()> {
#[tokio::test]
async fn test_cast_to_time() -> Result<()> {
let ctx = SessionContext::new();
- let sql = "SELECT 0::TIME";
+ let sql = "SELECT 0::TIME64";
Review Comment:
i'm not sure why do we have this. but `select 0::time64` doesn't work for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andre-cc-natzka commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
andre-cc-natzka commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1336990762
Hello @waitingkuo, @alamb. Thanks for your feedback!
It's been a long time, I apologize for that. I took some days of vacation actually. I am also sorry for the state of this PR, I should indeed have rebased master in advance. I have been through the process now and I believe the problems pointed out by @waitingkuo have been solved.
In fact, the changes I implemented here are very simple, and are limited to the ones in [datafusion/expr/src/type_coercion/binary.rs](https://github.com/apache/arrow-datafusion/pull/4312/files#diff-6a1eb5571196f1bd9f3defbbd97ed99e0e527e4ede426ce7e802ab949f1c431e), which aim at enabling type coercion to a timestamp (to nanosecond accuracy, since it is the only one supported by Arrow) when a string and a timestamp (with arbitrary precision) are provided. This could be very useful in my opinion.
Thank you again,
André.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #4312: Support type coercion for timestamp and utf8
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4312:
URL: https://github.com/apache/arrow-datafusion/pull/4312#issuecomment-1341468837
Benchmark runs are scheduled for baseline = fbadebb894672f61327a30f77cda2ee88a343b2a and contender = 8547fd8a562ed9071d765a6b8ddcaec0e9461503. 8547fd8a562ed9071d765a6b8ddcaec0e9461503 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/99cdb805ca644c52ace79d4c1689c546...647f934a587441fca9064eba7e3d6913/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/547d401eb0104c73aae0c4607ecdcbc0...8e5b57af9a06403dbd00bd158d462710/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/b997e251e3874a54a40bfc67737796d0...991961d660174184ae060acc163dc02a/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/c2de7d6bf7b4412386e2526837bb8c5f...37396be563a44a589d186fa8df98e883/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org