[PR] fix(io): skip subdirectory markers in blobfs WalkDir [iceberg-go]

["Revanth14 (via GitHub)" <gi...@apache.org>]

Revanth14 opened a new pull request, #2187:
URL: https://github.com/apache/iceberg-go/pull/2187

   ## Problem
   
   On flat object stores (S3, GCS, in-memory), `blobfs.FileIO.WalkDir` reports a subdirectory's trailing-slash marker as a file named after that subdirectory. Walking `tbl` with a `tbl/data/` marker yields `tbl/data/data`, a path that doesn't exist. Orphan cleanup would report it as an orphan and try to delete it.
   
   ## Fix
   
   `walkFS.ReadDir` now skips the listed object whose key equals the prefix of the directory being read. Under a `/` delimiter, that object can only be the directory's own `name/` marker. A directory that holds only its marker reads as empty rather than missing, so walking an empty marker directory doesn't fail.
   
   This check also covers the walk root's marker, so `isDirectoryMarker` and its skip in the `WalkDir` callback are removed.
   
   A non-empty object stored at a `dir/` key is no longer reported. Previously it appeared under a path that doesn't exist, which cleanup couldn't act on anyway.
   
   ## Testing
   
   Added `TestBlobFileIOWalkDirSkipsSubdirectoryMarkers`, covering a populated subdirectory marker, an empty one (`tbl/empty/`), and a directory with no marker. Removing either the marker skip or the empty-directory guard makes it fail, along with the existing `TestBlobFileIOWalkDirSkipsDirectoryMarker`.
   
   Closes #2185
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] fix(io): skip subdirectory markers in blobfs WalkDir [iceberg-go]

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade merged PR #2187:
URL: https://github.com/apache/iceberg-go/pull/2187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org