You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "daviewales (via GitHub)" <gi...@apache.org> on 2025/02/26 23:10:03 UTC
[I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
daviewales opened a new issue, #7204:
URL: https://github.com/apache/arrow-rs/issues/7204
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
Currently, `object_store` automatically detects Azure credentials from environment variables and IMDS. `object_store` also supports retrieving credentials from Azure CLI, however, it does not automatically try this.
If you want to retrieve credentials from Azure CLI, you need to manually specify `use_azure_cli=True`.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
Ideally, `object_store` would try `use_azure_cli=True` if no environment or IMDS credentials can be found. This makes it possible to easily test code on your local machine (based on your Azure CLI credentials), then deploy the code unchanged to an Azure VM, and it will automatically use the environment or IMDS credentials, without requiring any code changes.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
A workaround is to add `AZURE_USE_AZURE_CLI=True` to your local environment variables while testing. However, this is an extra step.
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
Most Azure libraries fall back to the Azure CLI credential if environment or IMDS credentials are not found. For example, `fsspec/adlfs`, `azure.identity.DefaultAzureCredential`, `polars`, `pandas`.
I'm aware of #6470, but I'm wondering if simply adding a fallback to an already supported authentication method is more in scope than completely replicating `DefaultAzureCredential`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #7204:
URL: https://github.com/apache/arrow-rs/issues/7204#issuecomment-2689375154
I think that's a fair rationale for keeping the default here as false.
While I want the Python bindings to minimize the differences from `object_store` for best maintainability, I think this is a place where it might be reasonable to override the default on the application side. https://github.com/developmentseed/obstore/pull/292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #7204:
URL: https://github.com/apache/arrow-rs/issues/7204#issuecomment-2704206356
> That being said, applications wanting this behaviour by default could override the default options
I implemented this in obstore in https://github.com/developmentseed/obstore/pull/292, however I had initially expected that it would try the Azure CLI, and if it didn't exist or couldn't access credentials, then it would still try instance metadata credentials.
But rereading
https://github.com/apache/arrow-rs/blob/95c42a7e235dbc689dbf50636aea514331c714c9/object_store/src/azure/builder.rs#L1015-L1028
shows that I was wrong; if `use_azure_cli` is set to `True` by default, then instance credentials will _never_ be checked unless the end user explicitly sets `use_azure_cli` to `False`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #7204:
URL: https://github.com/apache/arrow-rs/issues/7204#issuecomment-2688828451
My 2 cents is I would not be expecting an application to be shelling out unless I told it to do so, especially given IIRC the azure binary doesn't have a very collision resistant name. I'm not familiar enough with Azure tooling to know if this is an unusual expectation.
That being said, applications wanting this behaviour by default could override the default options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #7204:
URL: https://github.com/apache/arrow-rs/issues/7204#issuecomment-2688789449
Thanks for creating this issue.
I assume this would be considered a breaking change, and so it would need a resolution [within the next few days](https://github.com/apache/arrow-rs/issues/6903) (unless we want to wait a while)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
Posted by "sugibuchi (via GitHub)" <gi...@apache.org>.
sugibuchi commented on issue #7204:
URL: https://github.com/apache/arrow-rs/issues/7204#issuecomment-2730451128
> If shelling out is the only concern, it should be possible to inspect the tokens stored in $AZURE_CONFIG_DIR/accessTokens.json, where $AZURE_CONFIG_DIR is ~/.azure by [default](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration#cli-configuration-file).
This is not a feasible solution.
* `.azure/accessTokens.json` is an old location. The current version of Azure CLI uses `.azure/msal_token_cache.json`.
* `.azure/msal_token_cache.json` is encrypted in some environments (for example, Windows)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Automatically fall back to use_azure_cli=True if no other credentials can be found [arrow-rs]
Posted by "daviewales (via GitHub)" <gi...@apache.org>.
daviewales commented on issue #7204:
URL: https://github.com/apache/arrow-rs/issues/7204#issuecomment-2692971261
Thanks. I understand the hesitation to shell out to Azure CLI. On the other hand, this is a common behaviour of many Azure tools.
Looking at the code, I think that the only Azure CLI command used is `az account get-access-token`.
If shelling out is the only concern, it should be possible to inspect the tokens stored in `$AZURE_CONFIG_DIR/accessTokens.json`, where `$AZURE_CONFIG_DIR` is `~/.azure` by [default](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration#cli-configuration-file).
[This article](https://mikhail.io/2019/07/how-azure-cli-manages-access-tokens/) notes that this file contains both the latest `"accessToken"`, (60 minute expiry) and also a long lived (90 day) `"refreshToken"`.
If the access token is expired, it should be possible to `POST` the refresh token to the `"_authority"` to get a fresh access token. For example, if the `"_authority"` is `https://login.microsoftonline.com/common`, then you should be able to get a new 60 minute access token by `POST`ing the refresh token to `https://login.microsoftonline.com/common/oauth2/v2.0/token`. Other parameters from the `accessTokens.json` may also be required. See this [SO post](https://stackoverflow.com/a/74076836) for an example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org