[GitHub] [iceberg] RussellSpitzer commented on pull request #1471: RemoveOrphanFiles: consider only path to compare and delete, avoid authority and scheme

[GitBox <gi...@apache.org>]

RussellSpitzer commented on pull request #1471:
URL: https://github.com/apache/iceberg/pull/1471#issuecomment-949747196


   > In case of `S3FileIO`, default scheme is `s3://`. Writes happening from different clients, will have schemes based on `io-impl` property. The manifest might have mix of `s3://`, `s3a://` etc But the file listing(in DeleteOrphanFiles), will have only only a single prefix(which is determined by the Client Hadoop configuration). This will result in orphan files not being cleaned. When the user is aware that the scheme can be ignored, I think we should provide a configuration to do that.
   > 
   > I am not able to come up with a concrete case for the authority(may be HDFS with and with authority), but that could also be a configuration.
   > 
   > @RussellSpitzer @aokolnychyi @rdblue @flyrain @raptond Your thoughts on this?
   
   As we've talked about this before offline, I am a strong believer that not deleting files that we aren't sure about. So if for example a user has a files in
   ```
   hdfs://earth/sgcommand 
   hdfs://alphasite/sgcommand
   ```
   
   And the metadata has any of the following
   ```
   a: hdfs:///sgcommand/
   or
   b: hdfs://alphasite/sgcommand/
   or
   c: hdfs://earth/sgcommand/
   ```
   
   Our default behavior should be to not delete both original files. I think if we want we can add a optional flag for "exact matches"
   which would 
   ```
   case a: delete both files
   case b: delete only earth
   case c: delete only alphasite
   ```
   
   So by default I believe we should only compare the `path` portion of any uri assuming that everything else can be strongly dependent on the FileSystem implementation actually doing the file listing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org