You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/31 10:42:43 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
HyukjinKwon opened a new pull request #33875:
URL: https://github.com/apache/spark/pull/33875
### What changes were proposed in this pull request?
This PR proposes to add the support of `TimestampNTZType` in Arrow APIs.
Now, Arrow can write `TimestampNTZType` as Timestamp with `null` timezone in Arrow.
### Why are the changes needed?
To complete the support of `TimestampNTZType` in Apache Spark.
### Does this PR introduce _any_ user-facing change?
Yes, the Arrow APIs (`ArrowColumnVector`) can now write `TimestampNTZType`
### How was this patch tested?
Unittests were added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909218289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909125844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909125844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909784623
> So is this to support time-zone naive timestamps as described in https://github.com/apache/arrow/blob/master/format/Schema.fbs#L255 ?
Yes, I double checked the semantic, and I think there isn't any hole in the current implementation.
> I'm trying to remember way back when we first did timestamps and I thought we needed to always conform to time zone in "spark.sql.session.timeZone" which defaults to the local time zone?
Yes. This specific `TimestampNTZType` is an exception (to comply ANSI standard). In Spark code side, it computes with UTC as well so I think we're all good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33875:
URL: https://github.com/apache/spark/pull/33875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BryanCutler commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909681001
So is this to support time-zone naive timestamps as described in https://github.com/apache/arrow/blob/master/format/Schema.fbs#L255 ?
This sounds fine to me, but I'm trying to remember way back when we first did timestamps and I thought we needed to always conform to time zone in "spark.sql.session.timeZone" which defaults to the local time zone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909218289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
Just to clarify a bit more, Arrow specification describes as below (previously it was documented a local datetime)
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an \*unknown\* timezone.
> In particular, it is \*not\* possible to interpret an unset or empty timezone as the same as "UTC"
which I believe is inspired from naive datetime vs aware datetime: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects:
> Because naive `datetime` objects are treated by many `datetime` methods as local times
Here is Spark's take:
- With `TimestampType`, we will interpret naive `datetime` as a local time (a.k.a. `TIMESTAMP WITH TIME ZONE`)
- With `TimestampNTZType`, we will also interpret naive `datetime` as a time in an unknown timezone (a.k.a. `TIMESTAMP WITHOUT LOCAL TIME ZONE`), and computes them without caring the local (session) timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909897615
+1, late LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
Just to clarify a bit more, Arrow specification describes as below (previously it was documented a local datetime)
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an \*unknown\* timezone.
> In particular, it is \*not\* possible to interpret an unset or empty timezone as the same as "UTC"
which I believe is inspired from naive datetime vs aware datetime: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects:
> Because naive `datetime` objects are treated by many `datetime` methods as local times
Here is Spark's take:
- With `TimestampType`, we will interpret `datetime` as a local time (a.k.a. `TIMESTAMP WITH TIME ZONE`)
- With `TimestampNTZType`, we will also interpret `datetime` as a time in an unknown timezone (a.k.a. `TIMESTAMP WITHOUT LOCAL TIME ZONE`), and computes them without caring the local (session) timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
Just to clarify a bit more, Arrow specification describes as below (previously it was documented a local datetime)
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in the \*UTC\* timezone
> In particular, it is \*not\* possible to interpret an unset or empty timezone as the same as "UTC"
which I believe is inspired from naive datetime vs aware datetime: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects:
> Because naive `datetime` objects are treated by many `datetime` methods as local times
Here is Spark's take:
- With `TimestampType`, we will interpret `datetime` as a local times (a.k.a. `TIMESTAMP WITH TIME ZONE`)
- With `TimestampNTZType`, we will also interpret `datetime` in an unknown timezone (a.k.a. `TIMESTAMP WITHOUT LOCAL TIME ZONE`), and computes them without caring the local (session) timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
Just to clarify a bit more, Arrow specification describes as below (previously it was documented a local datetime)
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an \*unknown\* timezone.
> In particular, it is \*not\* possible to interpret an unset or empty timezone as the same as "UTC"
which I believe is inspired from other places like naive datetime vs aware datetime: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects:
> Because naive `datetime` objects are treated by many `datetime` methods as local times
Here is Spark's take:
- With `TimestampType`, we will interpret naive `datetime` as a local time (a.k.a. `TIMESTAMP WITH TIME ZONE`)
- With `TimestampNTZType`, we will also interpret naive `datetime` as a time in an unknown timezone (a.k.a. `TIMESTAMP WITHOUT LOCAL TIME ZONE`), and computes them without caring the local (session) timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909218289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909899584
Late LGTM. Thank you @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909899584
Late LGTM. Thank you @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909125844
cc @gengliangwang @ueshin @BryanCutler FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909209162
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
Just to clarify a bit more, Arrow specification describes as below (previously it was documented a local time)
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an \*unknown\* timezone.
> In particular, it is \*not\* possible to interpret an unset or empty timezone as the same as "UTC"
which I believe is inspired from other places like naive datetime vs aware datetime: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects:
> Because naive `datetime` objects are treated by many `datetime` methods as local times
Here is Spark's take:
- With `TimestampType`, we will interpret naive `datetime` as a local time (a.k.a. `TIMESTAMP WITH TIME ZONE`)
- With `TimestampNTZType`, we will also interpret naive `datetime` as a time in an unknown timezone (a.k.a. `TIMESTAMP WITHOUT LOCAL TIME ZONE`), and computes them without caring the local (session) timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909785009
I am pretty confident on this - let me get this in but let me know if there's any more concern @BryanCutler. I am open to revert 👍 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909784623
> So is this to support time-zone naive timestamps as described in https://github.com/apache/arrow/blob/master/format/Schema.fbs#L255 ?
Yes, I double checked the semantic, and I think there isn't any hole in the current implementation.
> I'm trying to remember way back when we first did timestamps and I thought we needed to always conform to time zone in "spark.sql.session.timeZone" which defaults to the local time zone?
Yes. This specific `TimestampNTZType` is an exception (to comply ANSI standard). In Spark code side, it computes with UTC as well so I think we're all good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909897615
+1, late LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909125844
cc @gengliangwang @ueshin @BryanCutler @viirya FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909218289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909137652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909137652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909785369
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909137652
**[Test build #142885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142885/testReport)** for PR 33875 at commit [`cb09f40`](https://github.com/apache/spark/commit/cb09f404ccd2016ac0e9c3afa6a818224165c550).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909218289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
Just to clarify a bit more, Arrow specification describes as below (previously it was documented a local datetime)
> If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an \*unknown\* timezone.
> In particular, it is \*not\* possible to interpret an unset or empty timezone as the same as "UTC"
which I believe is inspired from naive datetime vs aware datetime: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects:
> Because naive `datetime` objects are treated by many `datetime` methods as local times
Here is Spark's take:
- With `TimestampType`, we will interpret `datetime` as a local times (a.k.a. `TIMESTAMP WITH TIME ZONE`)
- With `TimestampNTZType`, we will also interpret `datetime` in an unknown timezone (a.k.a. `TIMESTAMP WITHOUT LOCAL TIME ZONE`), and computes them without caring the local (session) timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909834254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909168429
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909125844
cc @gengliangwang @ueshin @BryanCutler FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909218289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47388/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33875:
URL: https://github.com/apache/spark/pull/33875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909125844
cc @gengliangwang @ueshin @BryanCutler @viirya FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] BryanCutler commented on pull request #33875: [SPARK-36608][SQL] Support TimestampNTZ in Arrow
Posted by GitBox <gi...@apache.org>.
BryanCutler commented on pull request #33875:
URL: https://github.com/apache/spark/pull/33875#issuecomment-909681001
So is this to support time-zone naive timestamps as described in https://github.com/apache/arrow/blob/master/format/Schema.fbs#L255 ?
This sounds fine to me, but I'm trying to remember way back when we first did timestamps and I thought we needed to always conform to time zone in "spark.sql.session.timeZone" which defaults to the local time zone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org