You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tom-s-powell (via GitHub)" <gi...@apache.org> on 2026/10/06 19:27:22 UTC
[PR] Fix ADLSFileIO listPrefix FileInfo location. [iceberg]
tom-s-powell opened a new pull request, #18400:
URL: https://github.com/apache/iceberg/pull/18400
`SupportsPrefixOperations.listPrefix` is a generic API whose callers feed each `FileInfo.location()` straight back into a `FileIO` (or use it to compare against Iceberg metadata locations). The two other cloud implementations honour this contract by returning fully-qualified URIs:
* `S3FileIO.listPrefix` → `s3://bucket/key`
* `GCSFileIO.listPrefix` → `gs://bucket/blob`
`ADLSFileIO.listPrefix`, however, was returning the raw `PathItem.getName()` from the Azure SDK, which is only the container-relative path (e.g. dir/file). That means the location it emits:
* cannot be re-parsed by `ADLSLocation` (its regex requires an abfs[s]/wasb[s] scheme, so anything passed back through `newInputFile`/`deleteFile` throws a `ValidationException`),
* is not round-trippable through `deletePrefix` the way S3/GCS are,
* silently misbehaves in generic prefix walkers like `FileSystemWalker.listDirRecursivelyWithFileIO` and downstream actions like `DeleteOrphanFilesSparkAction` / `RemoveOrphanFilesProcedure`, which rely on `FileInfo.location()` being a usable location string.
### What changed
* `ADLSLocation` now captures and exposes the URI scheme (abfs, abfss, wasb, wasbs) so it can be reconstituted on the way out.
* `ADLSFileIO.listPrefix` reconstructs a fully-qualified URI (`<scheme>://<container>@<host>/<path>`, or `<scheme>://<host>/<path>` when no container is present) from the input prefix's scheme/host/container and the `PathItem` name, bringing it in line with `S3FileIO` and `GCSFileIO`.
* Tests updated to assert the fully-qualified form and to cover `ADLSLocation#scheme()`.
### Compatibility
This is a behaviour change for any caller that was relying on the previous container-relative return value. Given the existing behaviour was inconsistent with the interface contract and with every other `DelegateFileIO` implementation - and would break in-tree callers such as orphan-file cleanup on ADLS - this is a bug fix rather than a semantic break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Azure: Return fully-qualified locations from ADLSFileIO.listPrefix [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on PR #18400:
URL: https://github.com/apache/iceberg/pull/18400#issuecomment-6304987325
@mxm, @singhpk234 - Could you please take a look as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Azure: Return fully-qualified locations from ADLSFileIO.listPrefix [iceberg]
Posted by "tom-s-powell (via GitHub)" <gi...@apache.org>.
tom-s-powell commented on PR #18400:
URL: https://github.com/apache/iceberg/pull/18400#issuecomment-6250450565
@anuragmantri this should be good for another review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Azure: Return fully-qualified locations from ADLSFileIO.listPrefix [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #18400:
URL: https://github.com/apache/iceberg/pull/18400#issuecomment-6306164843
Thank you @tom-s-powell and thanks for the review @anuragmantri
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Azure: Return fully-qualified locations from ADLSFileIO.listPrefix [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #18400:
URL: https://github.com/apache/iceberg/pull/18400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix ADLSFileIO listPrefix FileInfo location. [iceberg]
Posted by "anuragmantri (via GitHub)" <gi...@apache.org>.
anuragmantri commented on code in PR #18400:
URL: https://github.com/apache/iceberg/pull/18400#discussion_r4263349870
##########
azure/src/integration/java/org/apache/iceberg/azure/adlsv2/TestADLSFileIO.java:
##########
@@ -169,7 +169,8 @@ public void testListPrefixOperations() {
// assert that only files were returned and not directories
FileInfo fileInfo = result.next();
- assertThat(fileInfo.location()).isEqualTo("dir/file");
+ assertThat(fileInfo.location())
Review Comment:
This now covers `abfs` with a container. Could the test be parameterized over scheme (`abfs`, `abfss`, `wasb`, `wasbs`) and container presence? The no-container branch of `fullyQualifiedLocation` has no coverage, and asserting that each result parses back through `new ADLSLocation(...)` would cover the round trip. I tried those combinations locally with a mocked client and they all work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Azure: Return fully-qualified locations from ADLSFileIO.listPrefix [iceberg]
Posted by "tom-s-powell (via GitHub)" <gi...@apache.org>.
tom-s-powell commented on code in PR #18400:
URL: https://github.com/apache/iceberg/pull/18400#discussion_r4269514802
##########
azure/src/integration/java/org/apache/iceberg/azure/adlsv2/TestADLSFileIO.java:
##########
@@ -169,7 +169,8 @@ public void testListPrefixOperations() {
// assert that only files were returned and not directories
FileInfo fileInfo = result.next();
- assertThat(fileInfo.location()).isEqualTo("dir/file");
+ assertThat(fileInfo.location())
Review Comment:
Made this `@ParameterizedTest` with coverage for the different combinations of scheme and container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org