You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/29 12:08:53 UTC
[GitHub] [spark] TJX2014 opened a new pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
TJX2014 opened a new pull request #28946:
URL: https://github.com/apache/spark/pull/28946
### What changes were proposed in this pull request?
1.Use local timezone instead of server side session timezone to show `Date`.
2.Add a unit test.
### Why are the changes needed?
Users may need to get a unify date result in client side.
Before this patch, people will get different result from `df.toPandas()` and `df.collect()` as follows：
>>> spark.conf.set("spark.sql.session.timeZone","UTC")
>>> df = spark.createDataFrame([("2018-06-01 01:00:00",)], ["ts"])
>>> df = df.withColumn("ts", df["ts"].astype("timestamp"))
>>> print(df.toPandas().iloc[0,0])
2018-06-01 01:00:00
>>> print(df.collect()[0][0])
2018-06-01 09:00:00
(tips: Client side timezone is UTC+8)
### Does this PR introduce _any_ user-facing change?
Yes，
Users will follow local timezone as `collect()` when use `toPandas`.
### How was this patch tested?
Unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28946:
URL: https://github.com/apache/spark/pull/28946#issuecomment-705902803
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #28946:
URL: https://github.com/apache/spark/pull/28946
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #28946:
URL: https://github.com/apache/spark/pull/28946#issuecomment-705902803
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tharradine commented on pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
tharradine commented on pull request #28946:
URL: https://github.com/apache/spark/pull/28946#issuecomment-651455471
This isn't exactly the SPARK-32123 fix I was expecting, I was expecting the behaviour mentioned in the [docs](https://spark.apache.org/docs/2.3.0/sql-programming-guide.html#timestamp-with-time-zone-semantics) to be adhered to, specifically:
> When timestamp data is exported or displayed in Spark, the session time zone is used to localize the timestamp values.
Is it not preferable to perform this localization whilst converting from/to internal timestamp datatypes, and preserve the documented behaviour?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #28946:
URL: https://github.com/apache/spark/pull/28946#issuecomment-651413402
Jenkins ok to test
cc @BryanCutler
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28946:
URL: https://github.com/apache/spark/pull/28946#issuecomment-651096568
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28946: [SPARK-32123][PYSPARK] Setting `spark.sql.session.timeZone` only partially respected
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28946:
URL: https://github.com/apache/spark/pull/28946#issuecomment-651096568
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org