You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tom-s-powell (via GitHub)" <gi...@apache.org> on 2026/09/08 11:22:45 UTC
[PR] Core, Spark: Reduce remove orphan files join data [iceberg]
tom-s-powell opened a new pull request, #18013:
URL: https://github.com/apache/iceberg/pull/18013
Separate normalized file identity from the original URI. Spark now uses `FileIdentifier` for the valid-file side of the orphan-file join, so it shuffles only `path`, `scheme`, and `authority`.
## Why
The valid-file side previously used `FileURI`, including `uriAsString`. Matching does not use this duplicate URI string, but Spark still transferred it during the join. This adds network and sort data for tables with many files.
## How it works
`FileIdentifier` contains the normalized fields required for matching. `FileURI` extends it and retains the original URI string required for deletion and output.
The valid-file dataset now uses `Dataset<FileIdentifier>`, while listed files continue to use `Dataset<FileURI>`. The change is applied to Spark 3.5, 4.0, 4.1, and 4.2.
## Testing
- Full remove-orphan action tests for Spark 3.5–4.2
- Core `FileIdentifier` and `FileURI` tests
- Core Revapi compatibility check
- Flink 2.3 compilation
- Spotless checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Reduce remove orphan files join data [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #18013:
URL: https://github.com/apache/iceberg/pull/18013#issuecomment-5775332083
@tom-s-powell Please find and ping relevant committers right here on github, in your PR ^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Reduce remove orphan files join data [iceberg]
Posted by "tom-s-powell (via GitHub)" <gi...@apache.org>.
tom-s-powell commented on PR #18013:
URL: https://github.com/apache/iceberg/pull/18013#issuecomment-6440416613
Or perhaps @kevinjqliu ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Reduce remove orphan files join data [iceberg]
Posted by "tom-s-powell (via GitHub)" <gi...@apache.org>.
tom-s-powell commented on PR #18013:
URL: https://github.com/apache/iceberg/pull/18013#issuecomment-6290914828
@huaxingao @singhpk234 don't suppose either of you would be able to do a final review here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Reduce remove orphan files join data [iceberg]
Posted by "tom-s-powell (via GitHub)" <gi...@apache.org>.
tom-s-powell commented on PR #18013:
URL: https://github.com/apache/iceberg/pull/18013#issuecomment-5668911527
@uros-b can I check what it was you were wanting to see? A change to the PR title/description itself or some additional information in the javadoc?
@uros-b @yangshangqing95 is Slack the best place to ping for committers for discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Spark: Reduce remove orphan files join data [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #18013:
URL: https://github.com/apache/iceberg/pull/18013#issuecomment-5668746703
> Please ping committers for further discussion.
In any chase, I definitely agree with this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org