[GitHub] [iceberg] jackye1995 commented on pull request #1844: AWS: support custom client configuration

[GitBox <gi...@apache.org>]

jackye1995 commented on pull request #1844:
URL: https://github.com/apache/iceberg/pull/1844#issuecomment-742363762


   >     * CatalogProperties: I'm not sure what these are? 
   
   In the context of Spark, it is the `Map<String, String> argument` of spark source. It is just generalized also for Flink (and any other use case that can accept a map).
   
   > Should a file io be created just for the json file and then a new file io be created for the other part of the table after reading the table properties? (manifest list/ manifest + data files?)
   
   That is why FileIO is treated as a catalog property instead of a table property, to avoid this complication.
    
   > Where should the kms key id go? Is that a per table thing to write new data files (table properties) or is it a user/system defined property which should go in the arguments?
   
   With the same reasoning as FileIO, if you put KMS key as a table property, then you need to create a table to know the KMS key, which needs to write to S3 and cause a circular dependency. 
   
   I do agree there is a use case for one KMS key per table, that can be added in the future if there is such a request, with the KMS key in catalog property as default, and the one in table property as an override.
   
   Please let me know if these answer your question.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org