[GitHub] [iceberg] jackye1995 commented on issue #3994: Allow table properties defaults to be configured at catalog level

[GitBox <gi...@apache.org>]

jackye1995 commented on issue #3994:
URL: https://github.com/apache/iceberg/issues/3994#issuecomment-1023493891


   we actually have a similar use case that EMR clusters would benefit from some cluster level configs that could be applied to all CREATE TABLE operations, and it’s not hard to achieve.
   
   I would imagine that a Spark cluster admin could do something like the following (I use CLI example, but this could be in the Spark default config file):
   
   ```
   spark-sql \
       --conf spark.sql.catalog.glue=org.apache.iceberg.spark.SparkCatalog \
       --conf spark.sql.catalog.glue.warehouse=s3://bucket/glue \
       --conf spark.sql.catalog.glue.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
       --conf spark.sql.catalog.glue.table.create.default.write.parquet.compression-codec=zstd
       --conf spark.sql.catalog.glue.table.create.default.write.parquet.dict-size-bytes=4194304
   ```
   
   And everything under `table.create.default` are initialized in the catalog implementation to be used as the default when a table is created. If user provides the same property, user can win.
   
   Another property we can introduce is `table.create.required`, in that case if user provides the same property, it has no effect and system default wins.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org