Re: [I] Support for Shallow Clone / Zero Copy Cloning in Apache Iceberg [iceberg]

["JarroVGIT (via GitHub)" <gi...@apache.org>]

JarroVGIT commented on issue #12263:
URL: https://github.com/apache/iceberg/issues/12263#issuecomment-3730570582

   @RussellSpitzer I think I may have found an avenue that does not require any change to the Table spec, nor the REST spec, but with limitations. Like mentioned above, the key requirements are:
   - Possibility to separate RBAC (in the catalog)
   - Separate location (preferred, due to vended credentials)
   - Completely independent lifecycle (e.g. schema changes/drops/maintenance should not interfere).
   
   Limitations: I haven't thought it through for clone-of-clone scenario's. 
   
   The key learning today was that the REST catalog specifies that multiple credentials[>1] (for different path prefixes) can be returned in a `GET <table>` (or `loadTable`) request. In the implementation of a catalog I looked at, it (logically) only returned a single credential, but this is something that could be changed (on the implementation side). Not sure if readers can deal with it, but that is a tomorrow-problem for now. 
   
   Then, on the catalog side, I could:
   - Add an endpoint to do a clone, which does quite a bit, e.g.: 
     - copy metadata to new location and strip it from history, new UUID, etc.
     - add some custom property like `table.clone.from`,
     - put tags on source snapshot and target snapshot (prevent maintenance)
     - register clone table.
   - Change logic of creating credentials based on existence of `table.clone.from` to generate more than one vended credential.
   - Additional checks and logic on `DROP` requests
   
   It isn't perfect, and it puts a lot of burden on the catalog implementation, but I think this could work. The most important thing is that it doesn't require a specification change and (assuming readers can handle multiple credentials for a table) doesn't change the reader requirements. 
   
   [>1]: I did find the document where this specification of the REST spec was discussed, but I couldn't find a rationale of why this was an array and not just a single `credential` field. I'll have to look tomorrow how readers would handle this though.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org