[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37011: [SPARK-39625][SPARK-38904][SQL] Add Dataset.as(StructType)

[GitBox <gi...@apache.org>]

HyukjinKwon commented on code in PR #37011:
URL: https://github.com/apache/spark/pull/37011#discussion_r913371314


##########
sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -464,6 +464,25 @@ class Dataset[T] private[sql](
    */
   def as[U : Encoder]: Dataset[U] = Dataset[U](sparkSession, logicalPlan)
 
+  /**
+   * Returns a new DataFrame where each row is reconciled to match the specified schema. Spark will:
+   * - Reorder columns and/or inner fields by name to match the specified schema.
+   * - Project away columns and/or inner fields that are not needed by the specified schema. Missing
+   *   columns and/or inner fields lead to failures.
+   * - Cast the columns and/or inner fields to match the data types in the specified schema, if the
+   *   types are compatible, e.g., numeric to numeric (error if overflows), but not string to int.
+   * - The columns and/or inner fields will carry the metadata from the specified schema, while
+   *   still keep their own metadata if not overwritten by the specified schema.
+   * - Fail if the nullability are not compatible. For example, the column and/or inner field is
+   *   nullable but the specified schema requires them to be not nullable.

Review Comment:
   Are we talking about this? This sounds same to me.
   Do we fail if the column is non-nullable but the specified schema is nullable? That should work I think. e.g.)
   
   ```scala
   scala> spark.range(1).toDF.union(spark.createDataFrame(Tuple1(None) :: Tuple1(Some(1)) :: Nil)).printSchema()
   root
    |-- id: long (nullable = true)
   
   
   scala> spark.range(1).toDF.printSchema()
   root
    |-- id: long (nullable = false)
   
   
   scala> spark.createDataFrame(Tuple1(None) :: Tuple1(Some(1)) :: Nil).printSchema()
   root
    |-- _1: integer (nullable = true)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org