You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "alessandro-nori (via GitHub)" <gi...@apache.org> on 2026/07/31 11:19:48 UTC
[PR] Spark: Fix orphan file matching for sibling table prefixes [iceberg]
alessandro-nori opened a new pull request, #17448:
URL: https://github.com/apache/iceberg/pull/17448
## Summary
- In `DeleteOrphanFilesSparkAction.filteredCompareToFileList()`, when `compareToFileList` is used, files are filtered with `startsWith(location)`. If `location` doesn't end with `/`, this can also match sibling table paths (e.g. `/ns/my_table` matches `/ns/my_table_2`), incorrectly marking their files as orphans of the wrong table.
- Normalize `location` to always end with `/` before filtering, mirroring the behavior of the default directory-listing path.
- Scoped to Spark 4.1 (latest supported version per repo policy); older Spark versions can be addressed in a follow-up if this is approved.
## Motivation
Prevents `deleteOrphanFiles` with a custom `compareToFileList` from deleting files belonging to a different table that merely shares a path prefix with the target table's location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Fix orphan file matching for sibling table prefixes [iceberg]
Posted by "alessandro-nori (via GitHub)" <gi...@apache.org>.
alessandro-nori commented on PR #17448:
URL: https://github.com/apache/iceberg/pull/17448#issuecomment-5142338270
Repro-only PR (test without the fix, expected to fail CI) demonstrating the bug: #17449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Fix orphan file matching for sibling table prefixes [iceberg]
Posted by "alessandro-nori (via GitHub)" <gi...@apache.org>.
alessandro-nori closed pull request #17448: Spark: Fix orphan file matching for sibling table prefixes
URL: https://github.com/apache/iceberg/pull/17448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Fix orphan file matching for sibling table prefixes [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #17448:
URL: https://github.com/apache/iceberg/pull/17448#issuecomment-5144273490
cc @wombatu-kun ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org