You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/06/15 19:45:16 UTC
[GitHub] [arrow-datafusion] velvia opened a new pull request #567: `to_timestamp_millis()`, `to_timestamp_micros()`, `to_timestamp_seconds()`
velvia opened a new pull request #567:
URL: https://github.com/apache/arrow-datafusion/pull/567
# Which issue does this PR close?
Closes #355 .
# Rationale for this change
This PR implements the timestamp functions above in #355, basically extending the following capabilities for all Timestamp types (specifically for Millis, micros, and seconds) via the above named functions:
- Ability to convert timestamp strings to Timestamp type
- Ability to cast Int64 numeric arrays into equivalent Timestamp types
# What changes are included in this PR?
Pretty straightforward additions to `functions.rs`, `datetime_expressions` etc.
What is NOT included so far but could be depending on discussion:
- Not sure where to add documentation for these functions
- It might be useful to users to be able to cast different Timestamp types into others, for example, Timestamp-Nanos to Timestamp-Millis, and that would be straightforward to add as well.
# Are there any user-facing changes?
Three new user-facing SQL functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a change in pull request #567: `to_timestamp_millis()`, `to_timestamp_micros()`, `to_timestamp_seconds()`
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #567:
URL: https://github.com/apache/arrow-datafusion/pull/567#discussion_r654782968
##########
File path: docs/user-guide/src/sql/datafusion-functions.md
##########
@@ -0,0 +1,86 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Datafusion-Specific Functions
+
+These SQL functions are specific to DataFusion, or they are well known and have functionality which is specific to DataFusion. Specifically, the `to_timestamp_xx()` functions exist due to Arrow's support for multiple timestamp resolutions.
+
+## `to_timestamp`
+
+`to_timestamp()` is similar to the standard SQL function. It performs conversions to type `Timestamp(Nanoseconds, None)`, from:
Review comment:
Great docs
##########
File path: README.md
##########
@@ -320,31 +324,31 @@ execution. The SQL types from
[sqlparser-rs](https://github.com/ballista-compute/sqlparser-rs/blob/main/src/ast/data_type.rs#L57)
are mapped to Arrow types according to the following table
-| SQL Data Type | Arrow DataType |
-| ------------- | ------------------------------- |
-| `CHAR` | `Utf8` |
-| `VARCHAR` | `Utf8` |
-| `UUID` | _Not yet supported_ |
-| `CLOB` | _Not yet supported_ |
-| `BINARY` | _Not yet supported_ |
-| `VARBINARY` | _Not yet supported_ |
-| `DECIMAL` | `Float64` |
-| `FLOAT` | `Float32` |
-| `SMALLINT` | `Int16` |
-| `INT` | `Int32` |
-| `BIGINT` | `Int64` |
-| `REAL` | `Float64` |
-| `DOUBLE` | `Float64` |
-| `BOOLEAN` | `Boolean` |
-| `DATE` | `Date32` |
-| `TIME` | `Time64(TimeUnit::Millisecond)` |
-| `TIMESTAMP` | `Date64` |
-| `INTERVAL` | _Not yet supported_ |
-| `REGCLASS` | _Not yet supported_ |
-| `TEXT` | _Not yet supported_ |
-| `BYTEA` | _Not yet supported_ |
-| `CUSTOM` | _Not yet supported_ |
-| `ARRAY` | _Not yet supported_ |
+| SQL Data Type | Arrow DataType |
+| ------------- | --------------------------------- |
+| `CHAR` | `Utf8` |
+| `VARCHAR` | `Utf8` |
+| `UUID` | _Not yet supported_ |
+| `CLOB` | _Not yet supported_ |
+| `BINARY` | _Not yet supported_ |
+| `VARBINARY` | _Not yet supported_ |
+| `DECIMAL` | `Float64` |
+| `FLOAT` | `Float32` |
+| `SMALLINT` | `Int16` |
+| `INT` | `Int32` |
+| `BIGINT` | `Int64` |
+| `REAL` | `Float64` |
+| `DOUBLE` | `Float64` |
+| `BOOLEAN` | `Boolean` |
+| `DATE` | `Date32` |
+| `TIME` | `Time64(TimeUnit::Millisecond)` |
+| `TIMESTAMP` | `Timestamp(TimeUnit::Nanosecond)` |
Review comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] velvia edited a comment on pull request #567: `to_timestamp_millis()`, `to_timestamp_micros()`, `to_timestamp_seconds()`
Posted by GitBox <gi...@apache.org>.
velvia edited a comment on pull request #567:
URL: https://github.com/apache/arrow-datafusion/pull/567#issuecomment-862806454
@alamb I believe all changes and feedback have been addressed. I added the capability of to_timestamp_xx() to convert from other timestamp types for convenience, and also added documentation. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #567: `to_timestamp_millis()`, `to_timestamp_micros()`, `to_timestamp_seconds()`
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #567:
URL: https://github.com/apache/arrow-datafusion/pull/567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] velvia commented on pull request #567: `to_timestamp_millis()`, `to_timestamp_micros()`, `to_timestamp_seconds()`
Posted by GitBox <gi...@apache.org>.
velvia commented on pull request #567:
URL: https://github.com/apache/arrow-datafusion/pull/567#issuecomment-861783228
/cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #567: `to_timestamp_millis()`, `to_timestamp_micros()`, `to_timestamp_seconds()`
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #567:
URL: https://github.com/apache/arrow-datafusion/pull/567#issuecomment-864391436
Thanks @velvia -- this looks great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org