You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/25 16:49:07 UTC
[GitHub] [spark] MaxGekk opened a new pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
MaxGekk opened a new pull request #33838:
URL: https://github.com/apache/spark/pull/33838
### What changes were proposed in this pull request?
In the PR, I propose to use the session time zone ( see the SQL config `spark.sql.session.timeZone`) instead of JVM default time zone while converting of special timestamp_ntz strings such as "today", "tomorrow" and so on.
### Why are the changes needed?
Current implementation is based on the system time zone, and it controverses to other functions/classes that use the session time zone. For example, Spark doesn't respects user's settings:
```sql
$ export TZ="Europe/Amsterdam"
$ ./bin/spark-sql -S
spark-sql> select timestamp_ntz'now';
2021-08-25 18:12:36.233
spark-sql> set spark.sql.session.timeZone=America/Los_Angeles;
spark.sql.session.timeZone America/Los_Angeles
spark-sql> select timestamp_ntz'now';
2021-08-25 18:14:40.547
```
### Does this PR introduce _any_ user-facing change?
Yes. For the example above, after the changes:
```sql
spark-sql> select timestamp_ntz'now';
2021-08-25 18:47:46.832
spark-sql> set spark.sql.session.timeZone=America/Los_Angeles;
spark.sql.session.timeZone America/Los_Angeles
spark-sql> select timestamp_ntz'now';
2021-08-25 09:48:05.211
```
### How was this patch tested?
By running the affected test suites:
```
$ build/sbt "test:testOnly *DateTimeUtilsSuite"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #33838:
URL: https://github.com/apache/spark/pull/33838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905890982
**[Test build #142770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142770/testReport)** for PR 33838 at commit [`f5e3eb4`](https://github.com/apache/spark/commit/f5e3eb44df99030efbaf97b41d6df9140af56a14).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905775091
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905786743
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905704800
**[Test build #142770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142770/testReport)** for PR 33838 at commit [`f5e3eb4`](https://github.com/apache/spark/commit/f5e3eb44df99030efbaf97b41d6df9140af56a14).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905900109
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142770/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905704800
**[Test build #142770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142770/testReport)** for PR 33838 at commit [`f5e3eb4`](https://github.com/apache/spark/commit/f5e3eb44df99030efbaf97b41d6df9140af56a14).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905786743
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-906018845
Thanks, merging to master/3.2! (3.2 hides NTZ type but we merge the code to avoid future code conflicts)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905900109
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142770/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33838: [SPARK-36590][SQL] Convert special timestamp_ntz values in the session time zone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33838:
URL: https://github.com/apache/spark/pull/33838#issuecomment-905737420
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org