You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/26 15:19:53 UTC
[GitHub] [spark] EnricoMi opened a new pull request #27711:
[SPARK-30815][SQL] Add function to format timestamp with time zone
EnricoMi opened a new pull request #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711
### What changes were proposed in this pull request?
Whenever timestamps are turned into strings (`Column.cast(StringType)`, `date_format(timestamp, format)`, `Dataset.show()`) the default time zone is used to format the string. This default time zone is either the java default zone `user.timezone`, the Spark default zone `spark.sql.session.timeZone` or the default DataFrameWriter zone `timeZone`. Currently there is no way to format a single column in a different timezone.
### Why are the changes needed?
Formatting timestamps in other timezones than the default one requires a UDF where a very similar operation `date_format` with all the code already exists.
### Does this PR introduce any user-facing change?
Yes, it introduces a new SQL operation `date_format_tz`. I could not get the existing `date_format` take an optional third `tz` argument.
### How was this patch tested?
Unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27711:
[SPARK-30815][SQL] Add function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-591482201
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] aray commented on issue #27711: [SPARK-30815][SQL] Add
function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
aray commented on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-614314452
I'm interested in getting this PR in. Although its absolutely true that it can be done as @MaxGekk suggests with a Scala UDF, the same does not apply in SQL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27711: [SPARK-30815][SQL]
Add function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-591482201
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on issue #27711: [SPARK-30815][SQL] Add
function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-591986170
This also holds for `date_format`, still it exists. This should be added for the sake of completeness.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27711:
[SPARK-30815][SQL] Add function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-591481584
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27711: [SPARK-30815][SQL]
Add function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-613159617
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27711: [SPARK-30815][SQL]
Add function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-614330903
@aray @EnricoMi @MaxGekk How about the other data processing/management systems? Do they have the built-in functions that provide the same functionality?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27711: [SPARK-30815][SQL]
Add function to format timestamp with time zone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27711: [SPARK-30815][SQL] Add function to format timestamp with time zone
URL: https://github.com/apache/spark/pull/27711#issuecomment-591481584
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org