Re: [PR] [SPARK-47546][SQL] Improve validation when reading Variant from Parquet [spark]

["cashmand (via GitHub)" <gi...@apache.org>]

cashmand commented on PR #45703:
URL: https://github.com/apache/spark/pull/45703#issuecomment-2018438251

   Hi @cloud-fan and @dongjoon-hyun,
   
   I updated the title, and made a few more changes:
   1) Refactored the Variant checks into a separate function, similar to what already exists for scalar and struct types.
   2) Added a few more test cases.
   3) Added a check that the value and metadata fields in parquet are non-nullable. This is also one that is a bit up for debate - without this check, we should fail somewhere downstream with a null pointer exception if we encounter a null value, but it seems clearer to enforce it at the parquet metadata level.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org