[GitHub] [iceberg] aokolnychyi commented on issue #2270: Spark: Session level Iceberg table config defaults

[GitBox <gi...@apache.org>]

aokolnychyi commented on issue #2270:
URL: https://github.com/apache/iceberg/issues/2270#issuecomment-786255694


   I think it will be useful. For example, I've frequently seen projects where folks create a number of tables and want to use more or less the same properties across tables within that project. Instead of specifying the same props over and over again, they could set them in their session config template. In another project, they may have a set of different props.
   
   I believe Spark has does that in some cases already. For instance, properties like `spark.datasource.ds_name.key` are propagated into sources during some operations. I'd vote for a catalog-native solution in Spark 3. That way, it will become a standard way of representing this in Spark.
   
   The session conf can look like this:
   
   ```
   spark.sql.catalog.cat_name.table_defaults.key1 = value1
   spark.sql.catalog.cat_name.table_defaults.key2 = value2
   ```
   
   Then individual property values in create statements will override what is in the session conf.
   
   ```
   CREATE TABLE ...
   TBLPROPERTIES (
     'k2' 'custom_value'
   )
   ````
   
   In the future, we can also support read and write options.
   
   ```
   // the name is arbitrary
   interface SupportsSessionConfigDefaults extends TableCatalog {
       String tablePropertiesPrefix();
   }
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org