You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/12/20 19:52:16 UTC
[GitHub] [arrow] balancap opened a new pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
balancap opened a new pull request #8977:
URL: https://github.com/apache/arrow/pull/8977
Add support for AWS STS web identity.
The AWS SDK has made the strange choice to force users to pass web identity arguments through environment variables (see https://github.com/aws/aws-sdk-cpp/blob/2be13177875e944151132d90305dbc46e80bf8e3/aws-cpp-sdk-core/source/auth/STSCredentialsProvider.cpp), and does not provide a way of directly passing these values.
I am not sure the design choice made in this PR is so great, but I have failed to find a nicer way to integrate, considering the limitations of the AWS sdk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-852333338
Closing this, since it was superseded by #10088.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-768141638
No problem, take your time! (but a reminder can sometimes help ;))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sahil1105 commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
sahil1105 commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-820657340
> @sahil1105 Perhaps you would be interested in this PR?
In reviewing it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] balancap commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
balancap commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-755570648
@pitrou Thanks for checking the MR! I'll push some modifications following your comments, and look more closely at the CI failures in the next couple of days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-820602860
@sahil1105 Perhaps you would be interested in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-748664534
https://issues.apache.org/jira/browse/ARROW-10675
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#discussion_r551385876
##########
File path: python/pyarrow/_s3fs.pyx
##########
@@ -149,6 +157,8 @@ cdef class S3FileSystem(FileSystem):
)
elif anonymous:
options = CS3Options.Anonymous()
+ elif use_web_identity:
+ options = CS3Options.FromAssumeRoleWithWebIdentity()
Review comment:
If `use_web_identity`, `anonymous`, `role_arn` and (`access_key` or `secret_key`) are all exclusive, we should probably test for such cases and raise an error.
##########
File path: python/pyarrow/_s3fs.pyx
##########
@@ -82,6 +82,8 @@ cdef class S3FileSystem(FileSystem):
external_id: str, default None
An optional unique identifier that might be required when you assume
a role in another account.
+ use_web_identity: bool, default False
+ ...
Review comment:
Can you add the argument description here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-820711984
@sahil1105 Reviewing, and potentially suggesting updates (or submitting a fresh new PR) if you are motivated enough :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-766843489
@balancap do you have time to update this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #8977:
URL: https://github.com/apache/arrow/pull/8977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] sahil1105 commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
sahil1105 commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-820815084
> @sahil1105 Reviewing, and potentially suggesting updates (or submitting a fresh new PR) if you are motivated enough :-)
Sure. I'll take a crack at it this weekend (or early next week) and submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] balancap commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
balancap commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-768139574
@jorgecarleitao @pitrou I am really sorry I did not take the time to update this PR. I'll get that done for the end of the week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #8977: ARROW-10675: [C++][Python] Support AWS S3 web identity credentials.
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #8977:
URL: https://github.com/apache/arrow/pull/8977#issuecomment-766843489
@balancap do you have time to update this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org