[GitHub] [iceberg] anuragmantri commented on issue #1617: Support relative paths in Table Metadata

[GitBox <gi...@apache.org>]

anuragmantri commented on issue #1617:
URL: https://github.com/apache/iceberg/issues/1617#issuecomment-880068307


   Catching up on the comments from past couple of days. Thanks everyone for providing inputs.
   
   @flyrain - Although it will increase the scope of this design to include bi-directional replication, we should consider covering that here. 
   
   Overall, I agree with @jackye1995's proposal. Thanks for the detailed explanation. 
   
   > Then these 4 properties are used to get 2 absolute root paths:
   > 
   > root path to write data: the LocationProvider would read some fields in this list to determine the base root path, and generate absolute paths to write data files.
   > root path to write metadata: similarly, the TableOperations.metadataFileLocation reads some fields above to determine the base root path, and generate paths for metadata files.
   
   In the initial design doc, my thinking was the writes mentioned above need to change at all. However, reading the comments from @pvary above, we may indeed  have to store a `root` location somewhere for cases where we want to read inaccessible paths. Whether we want to have explicit method to set this root before accessing a table or have a mechanism for replicated tables to register themselves as replicas and then use a retry logic can be discussed.
   
   > Why not expose these methods instead:
   > 
   > // initialize the provider with the root paths and the `useRelativePath` config,
   > // or have different implementations like:
   > // - AbsoluteLocationProvider,
   > // - RelativeLocationProvider?
   > LocationProvider(Properties tableProperties);
   > 
   > // generate the absolute path for the metadata files
   > Path metaDataLocation(String file);
   > // generate the absolute path for the metadata files
   > Path dataLocation(String file);
   > 
   > // generate the value we put into the avro files - absolute or relative path
   > String metaDataKey(Path metadataFile);
   > // generate the value we put into the avro files - absolute or relative path
   > String dataKey(Path metadataFile);
   
   @pvary - This may make the code change larger. Other than that, I don't see why this cannot be done unless I'm missing something.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org