You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "karim-ramadan (via GitHub)" <gi...@apache.org> on 2024/11/20 11:07:04 UTC
[PR] [SPARK-50364] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
karim-ramadan opened a new pull request, #48905:
URL: https://github.com/apache/spark/pull/48905
### What changes were proposed in this pull request?
In this Pull request, I propose to add a LocalDateTime serializer to the Row.jsonValue method
`
case (d: LocalDateTime, _) => JString(timestampFormatter.format(d))
`
In order to enable JSON serialization of _TimestampNTZType_ columns
### Why are the changes needed?
Currently trying to serialize a Row containing a _TimestampNTZType_ column results in an error:
`[FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G
org.apache.spark.SparkIllegalArgumentException: [FAILED_ROW_TO_JSON] Failed to convert the row value '2018-05-14T12:13' of the class class java.time.LocalDateTime to the target SQL type "TIMESTAMPNTZTYPE" in the JSON format. SQLSTATE: 2203G
at org.apache.spark.sql.Row.toJson$1(Row.scala:663)
at org.apache.spark.sql.Row.toJson$1(Row.scala:651)
at org.apache.spark.sql.Row.jsonValue(Row.scala:665)
at org.apache.spark.sql.Row.jsonValue$(Row.scala:598)
at org.apache.spark.sql.catalyst.expressions.GenericRow.jsonValue(rows.scala:28)
at org.apache.spark.sql.RowJsonSuite.$anonfun$testJson$1(RowJsonSuite.scala:41)
`
How to reproduce the issue:
`
import org.apache.spark.sql.Row
import java.time.LocalDateTime
val r = Row.fromSeq(LocalDateTime.of(2024,8,10,12,33) :: Nil)
r.json
r.prettyJson
`
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tests were added to the already existing RowJsonSuite.scala class
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
Posted by "krm95 (via GitHub)" <gi...@apache.org>.
krm95 commented on PR #48905:
URL: https://github.com/apache/spark/pull/48905#issuecomment-2489035866
.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #48905: [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue
URL: https://github.com/apache/spark/pull/48905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50364] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
Posted by "karim-ramadan (via GitHub)" <gi...@apache.org>.
karim-ramadan commented on PR #48905:
URL: https://github.com/apache/spark/pull/48905#issuecomment-2488303194
Related issue [SPARK-50364](https://issues.apache.org/jira/browse/SPARK-50364)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
Posted by "karim-ramadan (via GitHub)" <gi...@apache.org>.
karim-ramadan commented on PR #48905:
URL: https://github.com/apache/spark/pull/48905#issuecomment-2489013041
@MaxGekk thanks for the fast response and review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #48905:
URL: https://github.com/apache/spark/pull/48905#issuecomment-2488971475
+1, LGTM. All GAs passed. Merging to master.
Thank you, @karim-ramadan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50364][SQL] Implement serialization for LocalDateTime type in Row.jsonValue [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #48905:
URL: https://github.com/apache/spark/pull/48905#issuecomment-2488977299
@karim-ramadan Congratulations with your first contribution to Apache Spark!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org