You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "jaidisido (via GitHub)" <gi...@apache.org> on 2023/02/18 16:59:57 UTC

[GitHub] [arrow] jaidisido opened a new issue, #34253: Pass extra kwargs to S3 Pyarrow Filesystem

jaidisido opened a new issue, #34253:
URL: https://github.com/apache/arrow/issues/34253

   ### Describe the enhancement requested
   
   AFAIK current implementation of [Pyarrow s3fs](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) does not support passing extra kwargs to the various APIs. One example is [`x-amz-request-payer`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax) (i.e. Requester pays) which cannot be set in `get_file_info` or `open_input_file`.
   
   #10251 has addressed this for one particular use case (ACL) but a more general approach would be preferred
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Python] Pass extra kwargs to S3 Pyarrow Filesystem [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.

thisisnic commented on issue #34253:
URL: https://github.com/apache/arrow/issues/34253#issuecomment-3921695767

   This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] tomsimpkins commented on issue #34253: [C++][Python] Pass extra kwargs to S3 Pyarrow Filesystem

Posted by "tomsimpkins (via GitHub)" <gi...@apache.org>.

tomsimpkins commented on issue #34253:
URL: https://github.com/apache/arrow/issues/34253#issuecomment-1507274729

   Also marking my interest in being able to provide additional arguments to the underlying SDK.  My particular use case is around setting the SSECustomerAlgorithm and SSECustomerKey in [get_object](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) 
   
   As noted in https://github.com/apache/arrow/issues/10251, `s3fs` uses `additional_kwargs` and forwards these to the underlying botocore client.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] westonpace commented on issue #34253: [C++][Python] Pass extra kwargs to S3 Pyarrow Filesystem

Posted by "westonpace (via GitHub)" <gi...@apache.org>.

westonpace commented on issue #34253:
URL: https://github.com/apache/arrow/issues/34253#issuecomment-1439124196

   Do you know of a more generic capability that exists in the S3 SDK?  We use that to make requests and so our ability to tackle things generically is somewhat limited by the SDK's APIs.  That being said, I don't have full knowledge of the SDK though and so maybe there is something that exists that I'm just not aware of.
   
   For example, even though `x-amz-request-payer` is (conceptually) "just another HTTP header" the API exposes it as a property with getter and setter methods.  https://sdk.amazonaws.com/cpp/api/LATEST/aws-cpp-sdk-s3/html/class_aws_1_1_s3_1_1_model_1_1_put_object_request.html#a63072f3d8080c36ea6ad97851b22d255
   
   I suppose we could perhaps expose the underlying HTTP headers though that seems a very crude tool.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jaidisido commented on issue #34253: [C++][Python] Pass extra kwargs to S3 Pyarrow Filesystem

Posted by "jaidisido (via GitHub)" <gi...@apache.org>.

jaidisido commented on issue #34253:
URL: https://github.com/apache/arrow/issues/34253#issuecomment-1439715639

   I am not aware of a more generic capability. As you say the C++ SDK exposes the arguments as getter/setter and the Python APIs as [additional arguments](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2). I agree that passing them as HTTP headers wouldn't be recommended


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org