You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "mihailom-db (via GitHub)" <gi...@apache.org> on 2025/02/25 10:10:24 UTC
[PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
mihailom-db opened a new pull request, #50079:
URL: https://github.com/apache/spark/pull/50079
### What changes were proposed in this pull request?
Making createDataFrame use lenient Encoder for serialization.
### Why are the changes needed?
Passing Rows is something defined by users, so if we do not allow this, we would fail queries like this.
```
val schema = new org.apache.spark.sql.types.StructType().add("a", "date")
val rdd = spark.sparkContext.parallelize(
org.apache.spark.sql.Row(java.time.LocalDate.of(2020, 5, 13)) :: Nil)
spark.createDataFrame(rdd, schema).collect()
```
### Does this PR introduce _any_ user-facing change?
Yes.
The change is that we fix failing queries when we try to read Date/LocalDate independent of java8API config value.
### How was this patch tested?
Added appropriate tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #50079: [SPARK-51312][SQL] Fix createDataFrame from RDD[Row]
URL: https://github.com/apache/spark/pull/50079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #50079:
URL: https://github.com/apache/spark/pull/50079#discussion_r1969956302
##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2779,6 +2779,23 @@ class DatasetSuite extends QueryTest
}
}
+ test("SPARK-51312: createDataFrame should work with both Date and LocalDate") {
+ val testCases = Seq(
+ ("true", Array(Row(java.time.LocalDate.of(2020, 5, 13)),
+ Row(java.time.LocalDate.of(2020, 5, 13)))),
Review Comment:
I agree, don't need to collect dates explicitly back and behavior of the SQL flag, just check
```
checkAnswer(spark.createDataFrame(rdd, schema), sql("select date'2020-05-13', date'2020-05-13'"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #50079:
URL: https://github.com/apache/spark/pull/50079#discussion_r1969956302
##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2779,6 +2779,23 @@ class DatasetSuite extends QueryTest
}
}
+ test("SPARK-51312: createDataFrame should work with both Date and LocalDate") {
+ val testCases = Seq(
+ ("true", Array(Row(java.time.LocalDate.of(2020, 5, 13)),
+ Row(java.time.LocalDate.of(2020, 5, 13)))),
Review Comment:
I agree, don't need to collect dates backs, just check
```
checkAnswer(spark.createDataFrame(rdd, schema), sql("select date'2020-05-13', date'2020-05-13'"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50079:
URL: https://github.com/apache/spark/pull/50079#discussion_r1969887781
##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2779,6 +2779,23 @@ class DatasetSuite extends QueryTest
}
}
+ test("SPARK-51312: createDataFrame should work with both Date and LocalDate") {
+ val testCases = Seq(
+ ("true", Array(Row(java.time.LocalDate.of(2020, 5, 13)),
+ Row(java.time.LocalDate.of(2020, 5, 13)))),
Review Comment:
let's write the test this way:
- the input row has two columns: one is `LocalDate` and one is `Date`
- we run the test twice with the java8 datetime api on and off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #50079:
URL: https://github.com/apache/spark/pull/50079#issuecomment-2684443652
+1, LGTM. Merging to master.
Thank you, @mihailom-db and @hvanhovell @cloud-fan for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50079:
URL: https://github.com/apache/spark/pull/50079#discussion_r1969887781
##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2779,6 +2779,23 @@ class DatasetSuite extends QueryTest
}
}
+ test("SPARK-51312: createDataFrame should work with both Date and LocalDate") {
+ val testCases = Seq(
+ ("true", Array(Row(java.time.LocalDate.of(2020, 5, 13)),
+ Row(java.time.LocalDate.of(2020, 5, 13)))),
Review Comment:
let's write the test this way:
- the input row has two columns: one is `LocalDate` and one is `Date`
- we create the DF from the input row twice, with the java8 datetime api on and off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50079:
URL: https://github.com/apache/spark/pull/50079#discussion_r1969891707
##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2779,6 +2779,23 @@ class DatasetSuite extends QueryTest
}
}
+ test("SPARK-51312: createDataFrame should work with both Date and LocalDate") {
+ val testCases = Seq(
+ ("true", Array(Row(java.time.LocalDate.of(2020, 5, 13)),
+ Row(java.time.LocalDate.of(2020, 5, 13)))),
Review Comment:
oh it's already this way, but why do we need the `testCases`? The expected result is simply `sql("SELECT DATE'2020-05-13', ...")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51312][SQL] Fix createDataFrame from RDD[Row] [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on PR #50079:
URL: https://github.com/apache/spark/pull/50079#issuecomment-2681445658
@cloud-fan @MaxGekk @HyukjinKwon could you please review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org