[GitHub] [arrow-rs] wjones127 opened a new issue, #3213: Object store (AWS): Support dynamically resolving S3 bucket

[GitBox <gi...@apache.org>]

wjones127 opened a new issue, #3213:
URL: https://github.com/apache/arrow-rs/issues/3213

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   We currently return an error if the S3 bucket isn't in the region we are configured for, which can be annoying to some users. We might want to support automatically resolving the region.
   
   
   
   **Describe the solution you'd like**
   
   ```rust
   let s3: Result<AmazonS3, object_store::aws::Error> = AmazonS3Builder::new()
     .with_bucket_name("my_bucket")
     .resolve_and_build()?;
   ```
   
   `resolve_and_build()` would call the [Head Bucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) endpoint. It would return a result, with the following errors:
   
    * `BucketNotFound`: the request returned 404
    * `PermissionDenied`: the request returned 403 error; credentials do not have permission to perform `s3:ListBucket`.
    * `HttpError`: Http request failed
   
   **Describe alternatives you've considered**
   
   
   **Additional context**
   
   Issue reported in delta-rs: https://github.com/delta-io/delta-rs/issues/798#issuecomment-1328614080
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold closed issue #3213: Object store (AWS): Support dynamically resolving S3 bucket

["tustvold (via GitHub)" <gi...@apache.org>]

tustvold closed issue #3213: Object store (AWS): Support dynamically resolving S3 bucket
URL: https://github.com/apache/arrow-rs/issues/3213


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] mr-brobot commented on issue #3213: Object store (AWS): Support dynamically resolving S3 bucket

["mr-brobot (via GitHub)" <gi...@apache.org>]

mr-brobot commented on issue #3213:
URL: https://github.com/apache/arrow-rs/issues/3213#issuecomment-1540553959

   Using AWS SigV4 for authentication, I think we need the region to calculate both the [string to sign](https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html#create-string-to-sign) and the [signature](https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html#calculate-signature).
   
   The good news is that, even with a 403 Forbidden, the HeadBucket API returns the bucket region in the response headers. This means we can infer the region using an anonymous HeadBucket request (as long as the bucket exists), simply ignoring 403 errors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org