[GitHub] [iceberg] openinx commented on issue #2525: The spark's remove_orphan_files procedure cannot expire the orphan files that located in remote object storage services

[GitBox <gi...@apache.org>]

openinx commented on issue #2525:
URL: https://github.com/apache/iceberg/issues/2525#issuecomment-827375959


   Looks like the current `remove_orphan_files` procedure are depending on the [hadoop fs](https://github.com/apache/iceberg/blob/1f77257fc24891f3e66a6c162ac239afd6ae8d72/spark/src/main/java/org/apache/iceberg/spark/actions/BaseRemoveOrphanFilesSparkAction.java#L178) to get all the files,  while if we did not add the `oss` hadoop fs implementation, then the spark could not remove those orphan files.  Then for the people that want to put their data on the cloud storage services,  they have to implement both the `FileIO` and `org.apache.hadoop.fs.FileSystem` interfaces,  seems much redundant.   Is it possible to add a  `list` interfaces in the `FileIO` classes and it is only allowed to used for data management procedures  (it isn't allowed to used for table read/write path) ?  In this way, we could remove the hadoop fs dependency I think.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org