You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/12 16:31:24 UTC
[GitHub] [spark] qiuwenxiang0103 opened a new pull request #29420: [SPARK-32602][SQL] Fix issue where data with date type are saved into…
qiuwenxiang0103 opened a new pull request #29420:
URL: https://github.com/apache/spark/pull/29420
[SPARK-32602][SQL] Fix issue where data with date type are saved into hive table with wrong value '1970-01-01'
### What changes were proposed in this pull request?
Data with date type are saved into hive table with wrong value '1970-01-01'
scala> spark.sql("create table t1(d date)")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("insert into table t1 values(cast('2020-08-09' as date))")
res3: org.apache.spark.sql.DataFrame = []
scala> spark.sql("select d from t1").show
+----------+
| d|
+----------+
|1970-01-01|
+----------+
Spark 3.0 introduced DaysWritable which extends DateWrite from hive to handle date type. DaysWritable.toString() is called to write its value into hive table. DateWrite.toString() is defined as:
@Override
public String toString() {
// For toString, the time does not matter
return get(false).toString();
}
public Date get(boolean doesTimeMatter) {
return new Date(daysToMillis(daysSinceEpoch, doesTimeMatter));
}
DaysWritable didn't override toString(), neither get(boolean doesTimeMatter)。It did override get():
override def get(): Date = new Date(DateWritable.daysToMillis(julianDays))
but this didn't help with toString(), so with daysSinceEpoch in DateWritable always as 0, calls to DaysWritable.toString() will always return '1970-01-01', and as a result date value stored into hive table will always have value '1970-01-01'。
This pull request overrides DaysWritable.get(boolean doesTimeMatter) so that it's toString() behaves properly.
### Why are the changes needed?
Fix the correctness issue describe above.
### Does this PR introduce _any_ user-facing change?
'No'.
### How was this patch tested?
New test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29420: [SPARK-32602][SQL] Fix issue where data with date type are saved into…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29420:
URL: https://github.com/apache/spark/pull/29420#issuecomment-672984005
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29420: [SPARK-32602][SQL] Fix issue where data with date type are saved into…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29420:
URL: https://github.com/apache/spark/pull/29420#issuecomment-672983464
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] qiuwenxiang0103 closed pull request #29420: [SPARK-32602][SQL] Fix issue where data with date type are saved into…
Posted by GitBox <gi...@apache.org>.
qiuwenxiang0103 closed pull request #29420:
URL: https://github.com/apache/spark/pull/29420
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29420: [SPARK-32602][SQL] Fix issue where data with date type are saved into…
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29420:
URL: https://github.com/apache/spark/pull/29420#issuecomment-672983464
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org