[GitHub] [spark] ravwojdyla commented on pull request #37011: [SPARK-39625][SPARK-38904][SQL] Add Dataset.as(StructType)

[GitBox <gi...@apache.org>]

ravwojdyla commented on PR #37011:
URL: https://github.com/apache/spark/pull/37011#issuecomment-1174145196

   Thanks for the updates/fixes @cloud-fan!
   
   >> afait if the new schema has field nullable, while the incoming DF is not-nullable, the resulting DF will still have not-nullable field
   
   > I think stricter nullability is always better? Also I don't find an existing expression to turn a non-nullable input to nullable.
   
   @cloud-fan well that depends on the use case, if as a user supplied a **specific** schema they want the DF to be cast to, they might choose to use nullable fields for a use-case specific reason. We already have a functionality like `as` internally, and being able to "cast" to exact schema is an important part of SPARK-38904 (including being able to control nullability).
   
   > Also I don't find an existing expression to turn a non-nullable input to nullable.
   
   Can we just use `withNullability` in `reconcileColumnType` with config/flags?
   
   > Usually, we add python API in a separated PR. Right?
   
   Do you have this convention documented somewhere?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org