[GitHub] [iceberg] jackye1995 commented on issue #1617: Support relative paths in Table Metadata

[GitBox <gi...@apache.org>]

jackye1995 commented on issue #1617:
URL: https://github.com/apache/iceberg/issues/1617#issuecomment-878856794


   Just trying to catch up with the thread, and reread the design doc. 
   
   For the design doc, I think we should not make `HiveTable` a separated case to discuss, all tables that are retrieved through a `Catalog` implementation should operate the same if we introduce relative path, we should not try to break the abstraction layer and treat Hive tables separately.
   
   To have a brief summary (and also potentially answer Peter's question), from my understanding we are leading towards a solution that following items to keep track of the true table roots, and should be absolute, while all other paths can be relative:
   1. `location`
   2. `write.metadata.path`
   3. `write.folder-storage.path`
   4. `write.object-storage.path`
   
   Then these 4 properties are used to get 2 absolute root paths:
   1. root path to write data: the `LocationProvider` would read some fields in this list to determine the base root path, and generate absolute paths to write data files. 
   2. root path to write metadata: similarly, the `TableOperations.metadataFileLocation` reads some fields above to determine the base root path, and generate paths for metadata files.
   
   These probably should be exposed as new methods to make sure the same location derivation logic is used everywhere.
   
   Once the content of a file is written, the paths are written to one metadata layer above with/without absolute path based on a flag proposed in the design doc, and the writer of this path needs to know what are the true root paths used in the 2 places above. On the read side, the absolute path is created based on the 2 absolute root paths. Those operations are done inside each specific IO method by passing the relative/absolute path, the correct absolute root path, and the boolean flag, to make sure operations are transparent above the IO level.
   
   All updates to the 4 locations on the top can be done in a UpdateProperties + UpdateLocation transaction, and when the metadata path is updated, the `metadata_location` in catalog will be automatically updated with the new file root path to write and then read the latest metadata json file that is now in anther root path. Therefore, we need to make sure that when this value is updated, the metadata json file writer uses the value in the update instead of the existing value. This should be able to switch a reader of Iceberg table from one replica to another explicitly, or implement try-and-error logic to switch to the replica after certain retries. I am not sure if there is a use case such as choosing a random replica to read file, but I think that should be solved by directly registering all replicas as different tables and round-robin on the reader side.
   
   I think if we go with this route then it sounds good to me, and we can try to hash out some more details in PRs. Please let me know if this summary is accurate, or if I have any misunderstanding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org