You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/10 13:32:03 UTC
[GitHub] [spark] beliefer opened a new pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
beliefer opened a new pull request #35803:
URL: https://github.com/apache/spark/pull/35803
### What changes were proposed in this pull request?
Based discussion on https://github.com/apache/spark/pull/34984#discussion_r823685909.
This PR don't allow reading NTZ as LTZ for now.
### Why are the changes needed?
Forbid the data error when read TimestampNTZ as TimestampLTZ.
### Does this PR introduce _any_ user-facing change?
'Yes'.
This PR don't allow reading NTZ as LTZ for now explicitly.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35803:
URL: https://github.com/apache/spark/pull/35803#discussion_r823722234
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
##########
@@ -294,4 +295,30 @@ class QueryExecutionErrorsSuite extends QueryTest
assert(e.getMessage ===
"Datetime operation overflow: add 1000000 YEAR to '2022-03-09T09:02:03Z'.")
}
+
+ test("UNSUPPORTED_OPERATION - SPARK-38504: can't read TimestampNTZ as TimestampLTZ") {
+ val data = (1 to 10).map { i =>
+ // The second parameter is `nanoOfSecond`, while java.sql.Timestamp accepts milliseconds
+ // as input. So here we multiple the `nanoOfSecond` by NANOS_PER_MILLIS
+ val ts = LocalDateTime.ofEpochSecond(0, i * 1000000, ZoneOffset.UTC)
Review comment:
does the value matter? We can just use 0 and remove the code comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35803:
URL: https://github.com/apache/spark/pull/35803#issuecomment-1066417362
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #35803:
URL: https://github.com/apache/spark/pull/35803#discussion_r825305085
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
##########
@@ -294,4 +295,28 @@ class QueryExecutionErrorsSuite extends QueryTest
assert(e.getMessage ===
"Datetime operation overflow: add 1000000 YEAR to '2022-03-09T09:02:03Z'.")
}
+
+ test("UNSUPPORTED_OPERATION - SPARK-38504: can't read TimestampNTZ as TimestampLTZ") {
Review comment:
You could slightly simplify the test:
```scala
withTempPath { file =>
sql("select timestamp_ntz'2019-03-21 00:02:03'").write.orc(file.getCanonicalPath)
withAllNativeOrcReaders {
val e = intercept[SparkException] {
spark.read.schema("time timestamp_ltz").orc(file.getCanonicalPath).collect()
}.getCause.asInstanceOf[SparkUnsupportedOperationException]
assert(e.getErrorClass === "UNSUPPORTED_OPERATION")
assert(e.getMessage === "The operation is not supported: " +
"Unable to convert timestamp ntz of Orc to data type 'timestamp_ltz'")
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #35803:
URL: https://github.com/apache/spark/pull/35803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #35803:
URL: https://github.com/apache/spark/pull/35803#issuecomment-1065822318
ping @gengliangwang @MaxGekk cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #35803:
URL: https://github.com/apache/spark/pull/35803#discussion_r824335792
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
##########
@@ -294,4 +295,30 @@ class QueryExecutionErrorsSuite extends QueryTest
assert(e.getMessage ===
"Datetime operation overflow: add 1000000 YEAR to '2022-03-09T09:02:03Z'.")
}
+
+ test("UNSUPPORTED_OPERATION - SPARK-38504: can't read TimestampNTZ as TimestampLTZ") {
+ val data = (1 to 10).map { i =>
+ // The second parameter is `nanoOfSecond`, while java.sql.Timestamp accepts milliseconds
+ // as input. So here we multiple the `nanoOfSecond` by NANOS_PER_MILLIS
+ val ts = LocalDateTime.ofEpochSecond(0, i * 1000000, ZoneOffset.UTC)
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #35803:
URL: https://github.com/apache/spark/pull/35803#discussion_r825370534
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
##########
@@ -294,4 +295,28 @@ class QueryExecutionErrorsSuite extends QueryTest
assert(e.getMessage ===
"Datetime operation overflow: add 1000000 YEAR to '2022-03-09T09:02:03Z'.")
}
+
+ test("UNSUPPORTED_OPERATION - SPARK-38504: can't read TimestampNTZ as TimestampLTZ") {
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #35803: [SPARK-38504][SQL] Cannot read TimestampNTZ as TimestampLTZ
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #35803:
URL: https://github.com/apache/spark/pull/35803#issuecomment-1066449370
@cloud-fan @MaxGekk Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org