[GitHub] [iceberg] anuragmantri edited a comment on issue #1617: Support relative paths in Table Metadata

[GitBox <gi...@apache.org>]

anuragmantri edited a comment on issue #1617:
URL: https://github.com/apache/iceberg/issues/1617#issuecomment-880068307


   Catching up on the comments from past couple of days. Thanks everyone for providing inputs.
   
   @flyrain - Although it will increase the scope of this design to include bi-directional replication, we should consider covering that here. 
   
   Overall, I agree with @jackye1995's proposal. Thanks for the detailed explanation. 
   
   > Then these 4 properties are used to get 2 absolute root paths:
   > 
   > root path to write data: the LocationProvider would read some fields in this list to determine the base root path, and generate absolute paths to write data files.
   > root path to write metadata: similarly, the TableOperations.metadataFileLocation reads some fields above to determine the base root path, and generate paths for metadata files.
   
   In the initial design doc, my thinking was the writes mentioned above need to change at all. However, reading the comments from @pvary above, we may indeed  have to store a `root` location somewhere for cases where we want to read inaccessible paths. Whether we want to have explicit method to set this root before accessing a table or have a mechanism for replicated tables to register themselves as replicas and then use a retry logic can be discussed.
   
   My takeaways from this are:
   
   1. Metadata writers need to know
      a) Relative path boolean 
      b) True root of the table
      c) Metadata path if set
      Additionally, they will also convert paths to relative in metadata files.
      
   2. Data writers need to know
      a) Data path if set
      b) True root of the table 
      
   An update transaction will change all the above except relative path boolean. Please correct me if my understanding is incorrect.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org