[I] Reduce redundant location validation for historical Iceberg metadata references [polaris]

["fivetran-ashokborra (via GitHub)" <gi...@apache.org>]

fivetran-ashokborra opened a new issue, #5581:
URL: https://github.com/apache/polaris/issues/5581

   ## Problem\n\nPolaris validates direct Iceberg metadata references during table commits. These references include manifest-list, statistics, and partition-statistics file paths.\n\nFor tables with long snapshot histories, many historical references resolve to the same metadata directory. Validating each file path independently creates substantial redundant work on every commit.\n\n## Impact\n\nThe validation cost grows with snapshot history rather than with the number of distinct storage locations involved in the commit. This can cause severe commit latency and resource pressure for large tables.\n\n## Proposed direction\n\nNormalize direct metadata file references to their containing directories before location validation, then deduplicate the resulting paths.\n\nThe newly written table metadata file should continue to be validated separately at file level.\n\n## Compatibility and security\n\nHistorical references must continue to be checked so that changes to allowed storage loc
 ations cannot bypass restrictions. The optimization should preserve the existing allowed-location semantics while reducing duplicate directory checks.\n\nA pull request with focused tests is available from a fork branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Reduce redundant location validation for historical Iceberg metadata references [polaris]

["durgadasepalli (via GitHub) via issues" <is...@polaris.apache.org>]

durgadasepalli commented on issue #5581:
URL: https://github.com/apache/polaris/issues/5581#issuecomment-6590243070

   Hi @fivetran-ashokborra @dimas-b — I see #5582 was closed by the stale bot. If you’re not planning to pick it back up, I’d like to carry this forward. Following @dimas-b’s review, I’d split it into two PRs:
   
   Add parent-location support to StorageLocation, using string-based logic rather than URI.create so S3 keys that aren’t valid URIs still work. Tests would cover S3, Azure, GCS, and file locations, plus the s3://b/b edge case from the review.
   Rebase the dedup from #5582 on top of it, with the CHANGELOG note about the error message and credit to the original work.
   
   Does that sound good? Happy to defer if you’d rather finish it yourself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Reduce redundant location validation for historical Iceberg metadata references [polaris]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #5581:
URL: https://github.com/apache/polaris/issues/5581#issuecomment-6355392441

   This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org