You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/17 07:01:10 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
HyukjinKwon opened a new pull request #35887:
URL: https://github.com/apache/spark/pull/35887
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/commit/fab4ceb157baac870f6d50b942084bb9b2cd4ad2 that mistakenly removed the numeric type support in `to_timestamp(...)`. This PR restores the behavior back.
### Why are the changes needed?
To keep the previous behavior.
### Does this PR introduce _any_ user-facing change?
To end users, no because the change has not been released yet.
### How was this patch tested?
Unit test was added, and manually tested:
```scala
spark.range(1).selectExpr("to_timestamp(id)").show()
```
**Before**
```
+----------------+
|to_timestamp(id)|
+----------------+
| null|
+----------------+
```
**After**
```
+-------------------+
| to_timestamp(id)|
+-------------------+
|1970-01-01 09:00:00|
+-------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35887:
URL: https://github.com/apache/spark/pull/35887#issuecomment-1072107281
Merged to master and branch-3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35887:
URL: https://github.com/apache/spark/pull/35887#issuecomment-1070386059
cc @cloud-fan and @xinrong-databricks FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #35887:
URL: https://github.com/apache/spark/pull/35887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35887:
URL: https://github.com/apache/spark/pull/35887#issuecomment-1072107281
Merged to master and branch-3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #35887:
URL: https://github.com/apache/spark/pull/35887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #35887:
URL: https://github.com/apache/spark/pull/35887#discussion_r828823945
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -2080,8 +2080,10 @@ case class ParseToTimestamp(
override def inputTypes: Seq[AbstractDataType] = {
// Note: ideally this function should only take string input, but we allow more types here to
// be backward compatible.
- TypeCollection(StringType, DateType, TimestampType, TimestampNTZType) +:
- format.map(_ => StringType).toSeq
+ val types = Seq(StringType, DateType, TimestampType, TimestampNTZType)
Review comment:
BTW, it's from `Cast` logic - we used `Cast` here before https://github.com/apache/spark/commit/fab4ceb157baac870f6d50b942084bb9b2cd4ad2:
https://github.com/apache/spark/blob/c1e5e8a27595c711c53bb3571ea391ef700d531a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L71
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #35887: [SPARK-38583][SQL] Restore the behavior of `to_timestamp` that allows numeric types
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #35887:
URL: https://github.com/apache/spark/pull/35887#issuecomment-1073523495
Thanks @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org