You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "yaxxie-piramidal (via GitHub)" <gi...@apache.org> on 2025/10/21 08:19:36 UTC
[I] S3FileSystem Profile Support [arrow]
yaxxie-piramidal opened a new issue, #47880:
URL: https://github.com/apache/arrow/issues/47880
### Describe the enhancement requested
`pyarrow.fs.S3FileSystem` works fine if you're willing to pull temporary credentials and create this client manually
This comes with inconvenience such as having to recreate this client periodically as the temporary credentials expire.
In particular it is something that is necessary to do when set up using multiple profiles, as currently there is no way to select a profile and use the underlying SDK machinery to use/refresh the credentials given the profile.
Please add support for this
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Add Profile support to S3FileSystem [arrow]
Posted by "yaxxie-piramidal (via GitHub)" <gi...@apache.org>.
yaxxie-piramidal commented on issue #47880:
URL: https://github.com/apache/arrow/issues/47880#issuecomment-3425972547
Thanks for adding this; that issue is certainly related, but not the same. I'm asking for a "profile" argument to be exposed.
When you do `aws configure` you can have multiple named profiles set up, and you can swap between them by supplying the profile argument
What I was doing was assuming that pyarrow already taps into the refreshing credentials machinery available (which from that issue it appears not to be the case), and suggesting that we can expose the profile parameter and get to tap into that machinery.
[Would this resource be enough?](https://docs.aws.amazon.com/sdk-for-cpp/latest/api/aws-cpp-sdk-core/html/_a_w_s_credentials_provider_8h_source.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Add Profile support to S3FileSystem [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #47880:
URL: https://github.com/apache/arrow/issues/47880#issuecomment-3427116583
Yes, from the looks of `cpp/src/arrow/filesystem/s3fs.cc` it does seem that the `ProfileConfigFileAWSCredentialsProvider` is not implemented as a possible `S3Options.credentials_provider`:
https://github.com/apache/arrow/blob/459248654d4f2e4403fb9206fd628ba2f71cd93c/cpp/src/arrow/filesystem/s3fs.cc#L237-L273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] S3FileSystem Profile Support [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #47880:
URL: https://github.com/apache/arrow/issues/47880#issuecomment-3425843807
Thanks for opening the issue. Is this related to the following?
- https://github.com/apache/arrow/issues/18888
If not, could you add details of AWS documentation for this specific underlying SDK and the feature. This documentation will be a necessary resource if someone can work on this in the future. I am not that familiar with all the AWS S3 authentication methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][C++] Add Profile support to S3FileSystem [arrow]
Posted by "thomas-pfeiffer (via GitHub)" <gi...@apache.org>.
thomas-pfeiffer commented on issue #47880:
URL: https://github.com/apache/arrow/issues/47880#issuecomment-3922745598
It would be awesome to have this feature implemented, so that PyIceberg can add AWS profile configuration support when using its PyArrow backend as well (see https://github.com/apache/iceberg-python/issues/3065). FYI - For the other PyIceberg backend `Fsspec` AWS profile support is already implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org