You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/09/08 21:28:04 UTC
[GitHub] [arrow] alamb opened a new pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
alamb opened a new pull request #8142:
URL: https://github.com/apache/arrow/pull/8142
This PR adds:
1. An implementation of the 1 argument form of the `TO_TIMESTAMP` function that takes a single argument (string) and converts it into a nanosecond precision timestamp, as described in the proposal (doc)[https://docs.google.com/document/d/18O9YPRyJ3u7-58J02NtNVYb6TDWBzi3mIQC58VhwxUk/edit#heading=h.2nfzn4ggyl34]
2. An explicit dependence on the `chrono` library. Note that this does not add any *actual* new dependencies for datafusion because depends on `arrow` which already depends on chrono [source link](https://github.com/apache/arrow/blob/master/rust/arrow/Cargo.toml#L49).
fyi @jorgecarleitao @andygrove @emkornfield @jhorstmann
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
andygrove commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485275485
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
+}
+
+/// convert an array of strings into `Timestamp(Nanosecond, None)`
+pub fn to_timestamp(args: &[ArrayRef]) -> Result<TimestampNanosecondArray> {
+ let num_rows = args[0].len();
+ let mut ts_builder = TimestampNanosecondArray::builder(num_rows);
+ let string_args = &args[0]
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("input cast to StringArray failed");
Review comment:
nit: Although the planner and type coercion logic should prevent errors here, we could actually return an `Err` result rather than panic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#issuecomment-689861246
@andygrove -- I have rebased this PR and I believe I have implemented all suggestions. There is one more lingering thing regarding parsing local timestamps that I want to check out but I can also do that after merge if you prefer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jhorstmann commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
jhorstmann commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485468664
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
Review comment:
I'd be interested in a benchmark of this function or the kernel, maybe one for the happy case with `T` and `Z` and one for the last fallback. From my experience with Java, [parsing timestamps can be rather slow][1] and it might be worth writing a specialized implementation.
The behaviour regarding time zones should also be documented, does it default to a local time zone or to always to UTC?
Lastly, storing nanoseconds as i64 gives a range of about year 1677 to 2262, this might be a limitation for currently very few usecases but should also be mentioned.
[1]: https://github.com/jhorstmann/packedtime#benchmarks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485333061
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
+}
+
+/// convert an array of strings into `Timestamp(Nanosecond, None)`
+pub fn to_timestamp(args: &[ArrayRef]) -> Result<TimestampNanosecondArray> {
+ let num_rows = args[0].len();
+ let mut ts_builder = TimestampNanosecondArray::builder(num_rows);
+ let string_args = &args[0]
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("input cast to StringArray failed");
+
+ for i in 0..string_args.len() {
Review comment:
I think that we get a massive performance boost if we build `ArrayData` here (learning from @nevi-me :P).
@nevi-me did this recently [here](https://github.com/apache/arrow/commit/9ea24092064205b9966c4e08da50ea344bf042e5#diff-084bc9b19a2397f6ba80602e2d136833R33), e.g.
```rust
let num_rows = args[0].len();
let string_args = &args[0]
.as_any()
.downcast_ref::<StringArray>()
.expect("input cast to StringArray failed");
let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;
let data = ArrayData::new(
DataType::Timestamp(TimeUnit::Nanosecond, None),
num_rows,
Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),
0,
vec![Buffer::from(result.to_byte_slice())],
vec![],
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#issuecomment-689147235
https://issues.apache.org/jira/browse/ARROW-9944
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485950446
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
Review comment:
I filed https://issues.apache.org/jira/browse/ARROW-9955 to track the (excellent) suggestion for benchmark and possible improvement based on results. I agree efficient timestamp parsing is a bit of an art form.
I will also document both the timezone behavior as well as the date range limitations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] alamb commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485935565
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
+}
+
+/// convert an array of strings into `Timestamp(Nanosecond, None)`
+pub fn to_timestamp(args: &[ArrayRef]) -> Result<TimestampNanosecondArray> {
+ let num_rows = args[0].len();
+ let mut ts_builder = TimestampNanosecondArray::builder(num_rows);
+ let string_args = &args[0]
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("input cast to StringArray failed");
Review comment:
Will do
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
Review comment:
good idea -- I will do so
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove closed pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
andygrove closed pull request #8142:
URL: https://github.com/apache/arrow/pull/8142
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485333061
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
+}
+
+/// convert an array of strings into `Timestamp(Nanosecond, None)`
+pub fn to_timestamp(args: &[ArrayRef]) -> Result<TimestampNanosecondArray> {
+ let num_rows = args[0].len();
+ let mut ts_builder = TimestampNanosecondArray::builder(num_rows);
+ let string_args = &args[0]
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("input cast to StringArray failed");
+
+ for i in 0..string_args.len() {
Review comment:
I think that we get a massive performance boost if we build `ArrayData` here (learning from @nevi-me :P).
@nevi-me did this recently [here](https://github.com/apache/arrow/commit/9ea24092064205b9966c4e08da50ea344bf042e5#diff-084bc9b19a2397f6ba80602e2d136833R33), e.g.
```rust
let num_rows = args[0].len();
let string_args = &args[0]
.as_any()
.downcast_ref::<StringArray>()
.expect("input cast to StringArray failed");
let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;
let data = ArrayData::new(
DataType::Timestamp(TimeUnit::Nanosecond, None),
num_rows,
Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),
0,
vec![Buffer::from(result.to_byte_slice())],
vec![],
);
Ok(make_array(Arc::new(data)))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8142: ARROW-9944: [Rust][DataFusion] Implement to_timestamp function
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8142:
URL: https://github.com/apache/arrow/pull/8142#discussion_r485330796
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
Review comment:
add `s` that failed, or just a slice of it, here?
##########
File path: rust/datafusion/src/physical_plan/datetime_expressions.rs
##########
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! DateTime expressions
+
+use crate::error::{ExecutionError, Result};
+use arrow::array::{Array, ArrayRef, StringArray, TimestampNanosecondArray};
+use chrono::prelude::*;
+
+#[inline]
+fn string_to_timestamp_nanos(s: &str) -> Result<i64> {
+ // Fast path: RFC3339 timestamp (with a T)
+ // Example: 2020-09-08T13:42:29.190855Z
+ if let Ok(ts) = DateTime::parse_from_rfc3339(s) {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Implement quasi-RFC3339 support by trying to parse the
+ // timestamp with various other format specifiers to to support
+ // separating the date and time with a space ' ' rather than 'T' to be
+ // (more) compatible with Apache Spark SQL
+
+ // timezone offset, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855-05:00
+ if let Ok(ts) = DateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f%:z") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // with an explict Z, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29Z
+ if let Ok(ts) = Utc.datetime_from_str(s, "%Y-%m-%d %H:%M:%S%.fZ") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Support timestamps without an explicit timezone offset, again
+ // to be compatible with what Apache Spark SQL does.
+
+ // without a timezone specifier as a local time, using T as a separator
+ // Example: 2020-09-08T13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%dT%H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // without a timezone specifier as a local time, using ' ' as a separator
+ // Example: 2020-09-08 13:42:29.190855
+ if let Ok(ts) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S.%f") {
+ return Ok(ts.timestamp_nanos());
+ }
+
+ // Note we don't pass along the error message from the underlying
+ // chrono parsing because we tried several different format
+ // strings and we don't know which the user was trying to
+ // match. Ths any of the specific error messages is likely to be
+ // be more confusing than helpful
+ Err(ExecutionError::General("Timestamp parse error".into()))
+}
+
+/// convert an array of strings into `Timestamp(Nanosecond, None)`
+pub fn to_timestamp(args: &[ArrayRef]) -> Result<TimestampNanosecondArray> {
+ let num_rows = args[0].len();
+ let mut ts_builder = TimestampNanosecondArray::builder(num_rows);
+ let string_args = &args[0]
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .expect("input cast to StringArray failed");
+
+ for i in 0..string_args.len() {
Review comment:
I think that we get a massive performance boost if we build `ArrayData` here (learning from @nevi-me :P).
@nevi-me did this recently [here](https://github.com/apache/arrow/commit/9ea24092064205b9966c4e08da50ea344bf042e5#diff-084bc9b19a2397f6ba80602e2d136833R33), e.g.
```rust
let num_rows = args[0].len();
let string_args = &args[0]
.as_any()
.downcast_ref::<StringArray>()
.expect("input cast to StringArray failed");
let result = (0..num_rows).map(|i| string_to_timestamp_nanos(string_args.value(i))).collect::<Result<Vec<i64>>>()?;
let data = ArrayData::new(
DataType::Timestamp(DataType::TimeUnit::Nanosecond, None),
num_rows,
Some(string_args.null_count()),
string_args.data().null_buffer().cloned(),
0,
vec![Buffer::from(result.to_byte_slice())],
vec![],
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org