You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/02 20:15:33 UTC
[GitHub] [spark] MaxGekk opened a new pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
MaxGekk opened a new pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438
### What changes were proposed in this pull request?
In the PR, I propose to change the description of the `to_timestamp()` function and the `ParseToTimestamp` expression regarding the precision of their result, and change the pattern in the example.
### Why are the changes needed?
To inform users about actual behavior of the `to_timestamp` function.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581172977
**[Test build #117752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117752/testReport)** for PR 27438 at commit [`f6a7060`](https://github.com/apache/spark/commit/f6a7060be07eaa80ffb07fcb4751935a0b9e1e44).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581267606
@MaxGekk . Every committer's suggestion is his/her own criteria for his/her acceptance. I am also able to merge only what I can agree. And, as you know, the other committers also have different opinion and they will merge this if they agree with AS-IS status more. It always does. In addition, I don't complain about the other committer's decision when I understand it's on the edge. Since this PR is yours, it's up to you always~ @MaxGekk .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373965439
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
> Since SimpleDateFormat/FastDateFormat cannot parsing in microseconds precision
It has been there from the very first place. I think it's not clear if we should take it as a bug to backport.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581189491
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117752/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374271828
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
Review comment:
If you don't mind, I would like to say here that currently supported pattern for seconds fractions is `SSS` only. The second s fractions can be parsed but will be ignored while casting to TimestampType.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581369045
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581837701
OK then this doc change LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581189393
**[Test build #117752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117752/testReport)** for PR 27438 at commit [`f6a7060`](https://github.com/apache/spark/commit/f6a7060be07eaa80ffb07fcb4751935a0b9e1e44).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374061420
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
what's wrong with ".SSSS"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374202127
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
SPARK-29904 is merged at 2.4.5. In the worst case, we can revert @MaxGekk 's #26507 because of this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373944831
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
Review comment:
Of course, you can do whatever you want with much more details, but I want to minimize the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581268053
Anyway, thank you always for you active contribution. Apache Spark community really needs that. 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581189488
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373940008
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -1370,7 +1370,7 @@ case class ParseToDate(left: Expression, format: Option[Expression], child: Expr
}
/**
- * Parses a column to a timestamp based on the supplied format.
+ * Parses a column to a timestamp in the seconds precision based on the supplied format.
Review comment:
Let's not touch this file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581369045
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373962994
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
Since SimpleDateFormat/FastDateFormat cannot parsing in microseconds precision, I would propose to switch `to_timestamp` on the parser introduced by https://github.com/apache/spark/pull/26507 and port back https://github.com/apache/spark/pull/24420 to 2.4. At the moment, the behavior of `to_timestamp` doesn't seem consistent. /cc @cloud-fan WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373942625
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
Review comment:
May I ask you to explain why we shouldn't clarify function's behavior here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581294021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22529/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581293414
**[Test build #117767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117767/testReport)** for PR 27438 at commit [`c7f75a9`](https://github.com/apache/spark/commit/c7f75a95bc18886161f2986d10ddcfa729755b86).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373944890
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2962,12 +2962,12 @@ object functions {
}
/**
- * Converts time string with the given pattern to timestamp.
+ * Converts time string with the given pattern to timestamp in the seconds precision.
Review comment:
Same with the others.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581173094
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22514/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the timestamp pattern in the example for `to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581837184
> maybe we can just fix the issue in to_timestamp after RC2 passes?
@cloud-fan I have tried to port all functions on `TimestampParser` by rewriting `DateTimeUtils.newDateFormat`:
```scala
def newDateFormat(formatString: String, timeZone: TimeZone): TimestampParser = {
new TimestampParser(FastDateFormat.getInstance(formatString, timeZone))
}
```
from
```scala
def newDateFormat(formatString: String, timeZone: TimeZone): DateFormat = {
val sdf = new SimpleDateFormat(formatString, Locale.US)
sdf.setTimeZone(timeZone)
// Enable strict parsing, if the input date/format is invalid, it will throw an exception.
// e.g. to parse invalid date '2016-13-12', or '2016-01-12' with invalid format 'yyyy-aa-dd',
// an exception will be throwed.
sdf.setLenient(false)
sdf
}
```
but I got a few test failures due to strict mode set via **`sdf.setLenient(false)`**. I haven't found a way to set similar mode for `FastDateFormat`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374204648
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
> for 1), is it fixed in 2.4 or not?
@cloud-fan It is fixed in JSON and CSV datasources only but not for `to_timestamp()` and other functions.
> for 2), have we documented it?
I have tried to document it here in the PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581369057
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117767/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373940062
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
Review comment:
Let's not touch this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373954840
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
@MaxGekk, sorry if I missed some contexts but do you mean the ones below don't work? Seems working fine in Spark 2.4.4:
```
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
...
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").show(false)
+----------------------------------------+
|to_timestamp('2011-01-01 12:12:12.4321')|
+----------------------------------------+
|2011-01-01 12:12:12.4321 |
+----------------------------------------+
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").printSchema()
root
|-- to_timestamp('2011-01-01 12:12:12.4321'): timestamp (nullable = true)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581189488
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374227010
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
@MaxGekk . To be clear, I want to make it sure we don't have any regression at 2.4.5 as a release manager. You raised this issue and we are investigating the relevant PRs. That's all. We are considering the scope and effect. So far, we didn't make any decision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581294021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22529/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the timestamp pattern in the example for `to_timestamp`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581293414
**[Test build #117767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117767/testReport)** for PR 27438 at commit [`c7f75a9`](https://github.com/apache/spark/commit/c7f75a95bc18886161f2986d10ddcfa729755b86).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373954840
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
@MaxGekk, sorry if I missed some contexts but do you mean the ones below don't work? Seems working find in Spark 2.4.4:
```
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
...
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").show(false)
+----------------------------------------+
|to_timestamp('2011-01-01 12:12:12.4321')|
+----------------------------------------+
|2011-01-01 12:12:12.4321 |
+----------------------------------------+
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").printSchema()
root
|-- to_timestamp('2011-01-01 12:12:12.4321'): timestamp (nullable = true)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581172977
**[Test build #117752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117752/testReport)** for PR 27438 at commit [`f6a7060`](https://github.com/apache/spark/commit/f6a7060be07eaa80ffb07fcb4751935a0b9e1e44).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581725693
maybe we can just fix the issue in `to_timestamp` after RC2 passes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the timestamp pattern in the example for `to_timestamp`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581368586
**[Test build #117767 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117767/testReport)** for PR 27438 at commit [`c7f75a9`](https://github.com/apache/spark/commit/c7f75a95bc18886161f2986d10ddcfa729755b86).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581173093
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581294016
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581172618
@dongjoon-hyun Please, take a look at this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581262355
In general, this seems to be a follow-up of SPARK-23792 (at 2.4.0). Since it's too old JIRA, we had better use another JIRA or use `[MINOR]` like you. I support your choice. I believe that one thing we need is that `yyyy-MM-dd HH:mm:ss.SSSS` -> `yyyy-MM-dd HH:mm:ss` change. I recommend to focus on the above change only.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581294016
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373942941
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2962,12 +2962,12 @@ object functions {
}
/**
- * Converts time string with the given pattern to timestamp.
+ * Converts time string with the given pattern to timestamp in the seconds precision.
Review comment:
Just in case, are you ok with this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374095137
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
for 1), is it fixed in 2.4 or not?
for 2), have we documented it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581173094
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22514/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373957826
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
Review comment:
Usually it's better to make the changes minimised if we have to do it to old branches to reduce the potential conflicts when backporting. Seems this is the case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27438: [MINOR][SQL][DOCS][2.4]
Fix the timestamp pattern in the example for `to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-583123284
@dongjoon-hyun Kindly remind you about this minor fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374207683
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
@cloud-fan I tried to document the restriction of `to_timestmp()` in the PR but @dongjoon-hyun convinced me to do not do that here, see https://github.com/apache/spark/pull/27438#issuecomment-581262355 " I believe that one thing we need is that yyyy-MM-dd HH:mm:ss.SSSS -> yyyy-MM-dd HH:mm:ss change. I recommend to focus on the above change only."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581369057
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117767/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581192957
It's too late for RC2, @MaxGekk . Let's consider this PR after RC2 vote.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374079736
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
1. The result is incorrect, look at the PR which fixes that https://github.com/apache/spark/pull/26507, and here is an example:
```
val sdf = new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSS")
val res = sdf.parse("1970-01-01 00:00:00.1234")
println(sdf.format(res))
1970-01-01 00:00:01.0234
```
2. And the result of `to_timestamp` is truncated to seconds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-583123902
Sure, @MaxGekk . Sorry for making you wait. Merged to `branch-2.4`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581189491
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117752/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374205903
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
> In the worst case, we can revert @MaxGekk 's #26507 because of this.
@dongjoon-hyun Why? What's the worst case? I do think we should apply the fix in other places but not reverting it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r374204648
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2967,7 +2967,7 @@ object functions {
* See [[java.text.SimpleDateFormat]] for valid date and time format patterns
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
Review comment:
> for 1), is it fixed in 2.4 or not?
@cloud-fan It is fixed in JSON and CSV datasources but for `to_timestamp()` and other functions.
> for 2), have we documented it?
I have tried to document it here in the PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373960161
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
Right, it works due to your changes https://github.com/apache/spark/pull/17901 . When the `format` parameter is omitted, `to_timestamp` uses another parsing mechanism via `DateTimeUtils.stringToTimestamp` and the result is not truncated to seconds as we do when the `format` is provided. Weird behavior. :-(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373968203
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
Anyway, I think it's better to discuss about this out of this place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373954840
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
@MaxGekk, sorry if I missed some contexts but do you mind the ones below don't work? Seems working find in Spark 2.4.4:
```
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
...
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").show(false)
+----------------------------------------+
|to_timestamp('2011-01-01 12:12:12.4321')|
+----------------------------------------+
|2011-01-01 12:12:12.4321 |
+----------------------------------------+
scala> sql("select to_timestamp('2011-01-01 12:12:12.4321')").printSchema()
root
|-- to_timestamp('2011-01-01 12:12:12.4321'): timestamp (nullable = true)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#discussion_r373965439
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -2949,10 +2949,10 @@ object functions {
def unix_timestamp(s: Column, p: String): Column = withExpr { UnixTimestamp(s.expr, Literal(p)) }
/**
- * Converts to a timestamp by casting rules to `TimestampType`.
+ * Converts to a timestamp by casting rules to `TimestampType` in the seconds precision.
*
* @param s A date, timestamp or string. If a string, the data must be in a format that can be
- * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+ * cast to a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss`
Review comment:
> Since SimpleDateFormat/FastDateFormat cannot parsing in microseconds precision
It has been there from the very first place. I think it's not clear if we should take it as a bug fix to backport.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and
`ParseToTimestamp`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the descriptions of `to_timestamp` and `ParseToTimestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-581173093
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27438:
[MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for
`to_timestamp`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27438: [MINOR][SQL][DOCS][2.4] Fix the timestamp pattern in the example for `to_timestamp`
URL: https://github.com/apache/spark/pull/27438#issuecomment-582112886
Thank you for the investigation, @MaxGekk . And, thank you for the conclusion, @cloud-fan .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org