[GitHub] [incubator-iceberg] rdblue commented on issue #590: Allow spark.read.schema() to be set.

[GitBox <gi...@apache.org>]

rdblue commented on issue #590: Allow spark.read.schema() to be set.
URL: https://github.com/apache/incubator-iceberg/pull/590#issuecomment-548125375
 
 
   I don't think this is a correct use of the `schema` method. That is intended for formats that don't have type information, like CSV, so you can set the type instead of getting nothing but strings back.
   
   But Iceberg keeps type information, so it doesn't make sense to use the schema method. Instead, you should use normal cast and select methods to convert to different types and select columns in your dataframe. Usually when people want to use this `schema` method, the intent is actually to configure a projection, but using Spark's dataframe methods for that is the right way to go so that you get the benefit of the planner handling it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org