You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "NikitaMatskevich (via GitHub)" <gi...@apache.org> on 2025/06/17 13:46:15 UTC
[I] Support Azure FileSystem with Pyarrow FileIO [iceberg-python]
NikitaMatskevich opened a new issue, #2112:
URL: https://github.com/apache/iceberg-python/issues/2112
### Feature Request / Improvement
# Rationale for this change
PyArrow is the [default IO](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L366-L369) for Pyiceberg catalogs. In Azure environment it handles wider spectrum of auth strategies then Fsspec, including, for instance, [Managed Identities](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). Also, prior to this PR https://github.com/apache/iceberg-python/pull/1663 (that is not merged yet) there was no support for wasb(s) with Fsspec.
Direct impact on Pyiceberg users:
- Pyiceberg is usable in services with Managed Identities auth strategy.
- Pyiceberg is usable with wasb(s) schemes in Azure.
Right now the only way to use Managed identities with pyiceberg is extremely hacky:
```
import os
from fsspec import AbstractFileSystem
from pyiceberg.io.fsspec import FsspecFileIO
from pyiceberg.catalog.rest import RestCatalog
from typing import Any
ADLS_ANON = "adls.anon"
ADLS_CONNECTION_STRING = "adls.connection-string"
ADLS_ACCOUNT_NAME = "adls.account-name"
ADLS_ACCOUNT_KEY = "adls.account-key"
ADLS_SAS_TOKEN = "adls.sas-token"
ADLS_TENANT_ID = "adls.tenant-id"
ADLS_CLIENT_ID = "adls.client-id"
ADLS_CLIENT_SECRET = "adls.client-secret"
ADLS_ACCOUNT_HOST = "adls.account-host"
Properties = dict[str, Any]
def my_adls(properties: Properties) -> AbstractFileSystem:
from adlfs import AzureBlobFileSystem
for key, sas_token in {
key.replace(f"{ADLS_SAS_TOKEN}.", ""): value for key, value in properties.items() if key.startswith(ADLS_SAS_TOKEN)
}.items():
if ADLS_ACCOUNT_NAME not in properties:
properties[ADLS_ACCOUNT_NAME] = key.split(".")[0]
if ADLS_SAS_TOKEN not in properties:
properties[ADLS_SAS_TOKEN] = sas_token
return AzureBlobFileSystem(
connection_string=properties.get(ADLS_CONNECTION_STRING),
anon=properties.get(ADLS_ANON),
account_name=properties.get(ADLS_ACCOUNT_NAME),
account_key=properties.get(ADLS_ACCOUNT_KEY),
sas_token=properties.get(ADLS_SAS_TOKEN),
tenant_id=properties.get(ADLS_TENANT_ID),
client_id=properties.get(ADLS_CLIENT_ID),
client_secret=properties.get(ADLS_CLIENT_SECRET),
account_host=properties.get(ADLS_ACCOUNT_HOST),
)
injected_file_io = FsspecFileIO(properties={ADLS_ANON: False, ADLS_ACCOUNT_NAME: "my-account"})
injected_file_io.get_fs = lambda scheme: my_adls(injected_file_io.properties)
catalog = RestCatalog(
name="test_catalog",
uri="https://my-url/internal/catalog",
properties={
"io-impl": "pyiceberg.io.fsspec.FsspecFileIO",
}
)
catalog.file_io = injected_file_io
table = catalog.load_table("test.my_test_table")
table.io = injected_file_io
table.scan(limit=100).to_pandas()
```
As you can see, at least the "anon" flag must be passed to AzureBlobFileSystem, which is not currently done by Pyiceberg. Also, IO must be injected. With this PR it can be reduced to normal workflow:
```
catalog = load_catalog(uri="https://my-url/internal/catalog")
table = catalog.load_table("test.my_test_table")
table.scan(limit=100).to_pandas()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Azure FileSystem with Pyarrow FileIO [iceberg-python]
Posted by "NikitaMatskevich (via GitHub)" <gi...@apache.org>.
NikitaMatskevich closed issue #2112: Support Azure FileSystem with Pyarrow FileIO
URL: https://github.com/apache/iceberg-python/issues/2112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Azure FileSystem with Pyarrow FileIO [iceberg-python]
Posted by "NikitaMatskevich (via GitHub)" <gi...@apache.org>.
NikitaMatskevich commented on issue #2112:
URL: https://github.com/apache/iceberg-python/issues/2112#issuecomment-2980455610
[https://github.com/apache/iceberg-python/pull/2111](url)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support Azure FileSystem with Pyarrow FileIO [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2112:
URL: https://github.com/apache/iceberg-python/issues/2112#issuecomment-2994853060
#2111 is now merged. @NikitaMatskevich is this good to close?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org