You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "krishnakaanchan-png (via GitHub)" <gi...@apache.org> on 2026/08/31 19:02:18 UTC
[PR] PyArrow: Keep the storage account out of ADLS paths in `parse_location` [iceberg-python]
krishnakaanchan-png opened a new pull request, #3884:
URL: https://github.com/apache/iceberg-python/pull/3884
# Rationale for this change
`parse_location` builds the path as `f"{uri.netloc}{uri.path}"`. That is correct for S3, where the netloc is the bucket. For Azure the netloc is `container@account.dfs.core.windows.net`, so the account ends up inside the path and PyArrow reads that whole first segment as the container name. Every `PyArrowFileIO` call on a canonical ABFS location is therefore pointed at a container that cannot exist, since container names allow only lowercase letters, numbers and hyphens.
PyArrow itself handles these locations correctly. `FileSystem.from_uri("abfss://myfs@myacct.dfs.core.windows.net/wh/d.parquet")` returns `myfs/wh/d.parquet`, so the problem is only on our side. After this change `parse_location` returns the same path that PyArrow does.
The full trace and the Azure documentation references are in #2698.
This is the path half only. The other half is that `_initialize_azure_fs` takes no netloc, so the account can come only from `adls.account-name` and never from the location itself. I will send that separately, it needs a call on precedence when the property and the location disagree.
Related to #2698. Not closing it here, since the account derivation is still pending.
## Are these changes tested?
Yes. Two new parametrised tests over `abfs`, `abfss`, `wasb` and `wasbs`, one for the account qualified form and one for the container only form.
I also added an S3 case to `test_parse_location`. That function has to keep serving both shapes and there was nothing pinning the S3 behaviour that the new branch has to preserve.
`make lint` is clean and `tests/io/test_pyarrow.py` passes.
## Are there any user-facing changes?
Yes. Locations of the form `abfs[s]://<container>@<account>.dfs.core.windows.net/<path>` now resolve to the correct container under `PyArrowFileIO`. Earlier they resolved to a container named after the entire netloc. S3, HDFS and the container only Azure form are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: Keep the storage account out of ADLS paths in `parse_location` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #3884:
URL: https://github.com/apache/iceberg-python/pull/3884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: Keep the storage account out of ADLS paths in `parse_location` [iceberg-python]
Posted by "krishnakaanchan-png (via GitHub)" <gi...@apache.org>.
krishnakaanchan-png commented on PR #3884:
URL: https://github.com/apache/iceberg-python/pull/3884#issuecomment-5647272475
This one is green and approved now, and the CI problem from earlier is sorted after the rebase.
The rebase turned off the auto merge that @Fokko had enabled, so it is just sitting here. Could someone enable it again or merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: Keep the storage account out of ADLS paths in `parse_location` [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #3884:
URL: https://github.com/apache/iceberg-python/pull/3884#issuecomment-5735271784
Hey @krishnakaanchan-png Sorry for the long wait, just added it to the queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] PyArrow: Keep the storage account out of ADLS paths in `parse_location` [iceberg-python]
Posted by "krishnakaanchan-png (via GitHub)" <gi...@apache.org>.
krishnakaanchan-png commented on PR #3884:
URL: https://github.com/apache/iceberg-python/pull/3884#issuecomment-5592016449
The three required checks here were never reporting. `markdown-link-check`, `python-ci-required` and `python-integration-required` were all sitting at "Expected, waiting for status to be reported" and the commit had no statuses at all, so auto merge had nothing to wait on. The head commit was from 31 Aug and predates those jobs becoming required, so they never ran for it.
I have rebased on main and pushed. All three report now, and `markdown-link-check` and `python-ci-required` are green.
`python-integration-required` is red, but only because `integration-test` failed while building the hive image:
```
target hive: failed to solve: process "/bin/sh -c apt-get update -qq &&
apt-get -qq -y install --no-install-recommends curl ..." did not complete
successfully: exit code: 100
```
That is an apt failure in the container build, nothing from the test suite ran. A re-run should clear it.
Could someone re-run `integration-test`, and the force push has also turned off auto merge so that needs enabling again. Sorry for the churn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org