[GitHub] [spark] HyukjinKwon commented on a diff in pull request #33436: [SPARK-35912][SQL] Fix nullability of `spark.read.json/spark.read.csv`

[GitBox <gi...@apache.org>]

HyukjinKwon commented on code in PR #33436:
URL: https://github.com/apache/spark/pull/33436#discussion_r863271189


##########
docs/sql-migration-guide.md:
##########
@@ -22,6 +22,10 @@ license: |
 * Table of contents
 {:toc}
 
+## Upgrading from Spark SQL 3.2 to 3.3
+
+  - Since Spark 3.3, Spark turns a non-nullable schema into nullable for API `DataFrameReader.schema(schema: StructType).json(jsonDataset: Dataset[String])` and `DataFrameReader.schema(schema: StructType).csv(csvDataset: Dataset[String])` when the schema is specified by the user and contains non-nullable fields.

Review Comment:
   I actually underestimated this problem. It can actually change the results:
   
   
   ```scala
   import org.apache.spark.sql.types._
   val ds = Seq("a,", "a,b").toDS
   spark.read.schema(
     StructType(
       StructField("f1", StringType, nullable = false) ::
       StructField("f2", StringType, nullable = false) :: Nil)
     ).option("mode", "FAILFAST").csv(ds).show()
   ```
   
   Before:
   
   ```
   +---+---+
   | f1| f2|
   +---+---+
   |  a|  b|
   +---+---+
   ```
   
   After:
   
   ```
   +---+----+
   | f1|  f2|
   +---+----+
   |  a|null|
   |  a|   b|
   +---+----+
   ```
   
   I think we should at least add a legacy configuration .. let me make a quick followup.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org