You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/28 09:22:29 UTC
[GitHub] [arrow-rs] tustvold opened a new issue, #2795: Erratic Behaviour if Incorrect S3 Region Configured
tustvold opened a new issue, #2795:
URL: https://github.com/apache/arrow-rs/issues/2795
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
S3 appears to have a somewhat perverse behaviour when a request is made to a bucket in the wrong region, specifically it returns a 301 redirect missing a LOCATION header. FWIW this appears to be in contravention of the HTTP specification - https://www.rfc-editor.org/rfc/rfc2616#section-10.3.2.
As this is missing a LOCATION header, it fails to trigger the redirect logic [here](https://github.com/seanmonstar/reqwest/blob/master/src/async_impl/client.rs#L1944) which causes it to consider this a "valid" request. I'm not sure can be considered an upstream bug, as for its purposes it is unclear what the correct behaviour for redirects that aren't actually redirects is :sweat_smile:.
[reqwest::error_for_status_by_ref](https://docs.rs/reqwest/latest/reqwest/struct.Response.html#method.error_for_status_ref) then treats this response as not an error. Normally failed redirects would result in an error higher up, but as this response lacks a LOCATION header, reqwest doesn't consider it a well-formed redirect
The result is that:
* Requests with no response body will "succeed" - despite not doing anything
* Requests with a response body will likely fail to parse correctly
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
Configure an S3 integration test with the wrong region
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
We should return an error clearly indicating what has gone wrong
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2741686666
Migrating from arrow-rs issue [#2795](https://github.com/apache/arrow-rs/issues/2795)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #2795: Erratic Behaviour if Incorrect S3 Region Configured
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-1263396887
`label_issue.py` automatically added labels {'object-store'} from #2796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "johnpyp (via GitHub)" <gi...@apache.org>.
johnpyp commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285045976
Yeah, I think I agree in principle particularly for users who are directly consuming `object_store` with their own well-known configuration.
In cases like the `datafusion-cli` where it may be more desirable for ease-of-use reasons to swallow the issue (or warn but continue gracefully), it seems better to not require the additional `HEAD` request, so exposing the corrective-region on the error would at least help avoid needing *three* requests to auto-correct instead of two.
Would you be opposed to adding a `redirect_region` (and maybe `redirect_url`) field to the `BareRedirect` enum member for that usecase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #2795: Erratic Behaviour if Incorrect S3 Region Configured
Posted by GitBox <gi...@apache.org>.
tustvold closed issue #2795: Erratic Behaviour if Incorrect S3 Region Configured
URL: https://github.com/apache/arrow-rs/issues/2795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "johnpyp (via GitHub)" <gi...@apache.org>.
johnpyp commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285034841
Ah, nice. This *could* be implemented automatically in the response path without the need for an additional `HEAD` request like in the aforementioned SDKs (because the `GET` returns that region header too). Alternatively, that header data could be included in the `ObjectStore` error to have the requester retry it themselves, as opposed to today where the context is minimal (`Error: ObjectStore(Generic { store: "S3", source: BareRedirect })`) - not sure which is preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285037461
As an incorrect region implies a misconfiguration my preference is to surface this as an error to the user, and they can choose to act on it as they deem fit. Silently following a redirect and potentially doubling the latency of every request seems not ideal. Uses who explicitly don't know what region their bucket is in, should use the linked API to determine it as part of configuring the builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "johnpyp (via GitHub)" <gi...@apache.org>.
johnpyp commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285064694
> If you can figure out a way to expose this without needing to expose currently private crate details, I don't see a major issue, but I'm also not sure how this would be possible.
Ah I see, it's erased by the `Generic` error. Yeah, seems difficult in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285029594
See https://github.com/apache/arrow-rs/pull/4188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "johnpyp (via GitHub)" <gi...@apache.org>.
johnpyp commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285028221
The bucket to redirect to is actually provided in the response headers, as `x-amz-bucket-region`.
The botocore library and thus awscli v2 use this to automatically do the redirect ([source](https://github.com/boto/botocore/blob/3e85531776532a9de1accbb86180698017bacc6f/botocore/utils.py#L1706)).
And the latest aws-sdk-js-v3 library does too ([source](https://github.com/aws/aws-sdk-js-v3/blob/1a479dc8a26dfacef950b4d86631f168b55f3561/packages/middleware-sdk-s3/src/region-redirect-middleware.ts#L44))
And the aws-sdk-java-v2 library does ([source](https://github.com/aws/aws-sdk-java-v2/blob/431d022aad26c6639a9b5db4b463b8aba1362988/services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/crossregion/S3CrossRegionAsyncClient.java#L98)), when enabled.
This isn't quite consistent among all the SDKs, like you pointed out the Go SDK doesn't have it, and neither does the Rust one from a brief glance.
This seems like it would be a nice option to at least be configurable if not the default, for use-cases like the `datafusion-cli` where this would be quite convenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Erratic Behaviour if Incorrect S3 Region Configured [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #2795:
URL: https://github.com/apache/arrow-rs/issues/2795#issuecomment-2285051887
> Would you be opposed to adding a redirect_region (and maybe redirect_url) field to the BareRedirect enum member for that usecase?
If you can figure out a way to expose this without needing to expose currently private crate details, I don't see a major issue, but I'm also not sure how this would be possible.
> In cases like the datafusion-cli where it may be more desirable for ease-of-use reasons to swallow the issue
I think we will have to agree to disagree on this, silently swallowing issues IMO is worse for usability than returning a clear error, systems IMO should fail-fast, but appreciate this mentality isn't universal, especially when it comes to certain ecosystems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org