[GitHub] [spark] cloud-fan commented on issue #26913: [SPARK-29219][SQL] Introduce SupportsCatalogOptions for TableProvider

[GitBox <gi...@apache.org>]

cloud-fan commented on issue #26913: [SPARK-29219][SQL] Introduce SupportsCatalogOptions for TableProvider
URL: https://github.com/apache/spark/pull/26913#issuecomment-568389494
 
 
   It's good to see that we are looking at the big picture, to support user-specified schema, catalog in options, and schema/partition in builtin metastore.
   
   For "1 TableProvider", agree with the expectation here, and I'd say `getTable(properties)` is the simplest to satisfy the expectation.
   
   For "2 SupportsExternalMetadata", I think itself can be used as a marker and we don't need `SupportsUserSpecifiedSchema`:
   ```
   - spark.table(...): Metastore schema + partitioning info + properties is passed in to create Table
   - spark.read.load(...): Call inferSchema + inferPartitioning then pass in inferred schema + partitioning + df options to create Table
   - spark.read.schema().load(...): Call inferPartitioning, pass in schema + inferred partitioning + df options to create Table
   ```
   
   For data source, I don't think they care about where the schema/partitioning come from. It can be inferred by themselves, or specified by end-users, or from Spark's builtin catalog. In `getTable(schema, partitioning, properties)`, they need to validate the schema/partitioning anyway.
   
   For "3 SupportsCatalogOptions", sounds good to me, but looks better if we can fail with user-specified schema not supported like `TableProvider`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org