You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "gianarb (via GitHub)" <gi...@apache.org> on 2023/11/22 20:47:56 UTC
[I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
gianarb opened a new issue, #5115:
URL: https://github.com/apache/arrow-rs/issues/5115
**Which part is this question about**
object_store/azure
**Describe your question**
I use the method described here to authenticate containers running in Azure AKS:
https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet
In practice I have a service account that looks like this:
```
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: afgd
azure.workload.identity/tenant-id: 44hb
creationTimestamp: "2023-11-22T11:47:33Z"
labels:
environment: test
name: blob-access
namespace: default
```
And I annotation the pod I want to authenticate as described in the documentation.
It worked out of the box for the containers written in other language (Go) but not for the one I have in Rust that uses datafusion.
This is the way I build the azure client:
```rust
MicrosoftAzureBuilder::from_env()
.with_url(c.account)
.with_container_name(c.container_name)
.build()
.unwrap()
```
I can't figure out the right way to make this authentication method to work
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5115:
URL: https://github.com/apache/arrow-rs/issues/5115#issuecomment-1823670572
It might be possible this would work - https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_federated_token_file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5115:
URL: https://github.com/apache/arrow-rs/issues/5115#issuecomment-1868285590
Closing this one as I believe the question has been answered, feel free to reopen / file follow on tickets as required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5115:
URL: https://github.com/apache/arrow-rs/issues/5115#issuecomment-1823662193
I don't believe this credential provider is currently supported by object_store. I would be happy to review a PR adding support for it, it looks very similar to the AWS EKS WebIdentityProvider, but I do not intend to work on this myself.
Another option is if there is a Rust SDK that supports this you could hook in a custom CredentialProvider for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5115: object_store: Support for Azure Microsoft Entra for authentication from AKS
URL: https://github.com/apache/arrow-rs/issues/5115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "gianarb (via GitHub)" <gi...@apache.org>.
gianarb commented on issue #5115:
URL: https://github.com/apache/arrow-rs/issues/5115#issuecomment-1823883296
Thank you for your explanation! I am not currently setting the federated token file. So I need to figure out how to get it. I see:
```
azure.workload.identity/inject-proxy-sidecar
Injects a proxy init container and proxy sidecar into the pod. The proxy sidecar is used to intercept token requests to IMDS and acquire a Microsoft Entra token on behalf of the user with federated identity credential.
```
Is on by default so it should acquire the token, just need to figure out how to close the missing part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "gianarb (via GitHub)" <gi...@apache.org>.
gianarb commented on issue #5115:
URL: https://github.com/apache/arrow-rs/issues/5115#issuecomment-1823530484
I realized I was not using the latest datafusion/arrow-rs so I updated to arrow 0.48 and datafusion 0.33 with object_store 0.7.0.
And now I get a different error
```
Server failed with Object Store error: Generic MicrosoftAzure error: Error performing list request: response error "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
```
My feeling is that it is not getting the credentials from the service account now. Because I am pretty sure I am using the same user I use for the other application with access to this blob store container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] object_store: Support for Azure Microsoft Entra for authentication from AKS [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5115:
URL: https://github.com/apache/arrow-rs/issues/5115#issuecomment-1823680286
It looks like this should be supported by https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html.
You will need to specify the following:
* https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_tenant_id
* https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_client_id
* https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html#method.with_federated_token_file
Or alternatively these can be specified as environment variables:
* AZURE_TENANT_ID
* AZURE_CLIENT_ID
* AZURE_FEDERATED_TOKEN_FILE
These values should correspond to the service account annotations - https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet#service-account-annotations
There might be a nicer way to project these in, AWS EKS has an admission hook that automatically sets some environment variables, but setting them manually may be good enough to prove it works first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org