You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ritchie46 (via GitHub)" <gi...@apache.org> on 2023/12/15 10:10:47 UTC
[I] Make aws region optional. [arrow-rs]
ritchie46 opened a new issue, #5211:
URL: https://github.com/apache/arrow-rs/issues/5211
We get this request upstream: https://github.com/pola-rs/polars/issues/13042
Where we cause an error because we try to infer the region. However it turns out any region would suffice because they have a private s3.
To be able to tailor to that use case I assume it would be better to make region optional. We can hack around this by just setting a region for the time being.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-1870299416
> We intercept such redirects and return a more helpful error, I therefore think defaulting to us-east-1 is reasonable. If incorrect, we just defer the error, but if valid everything just works 🎉
Sound great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-1870297572
We intercept such redirects and return a more helpful error, I therefore think defaulting to us-east-1 is reasonable. If incorrect, we just defer the error, but if valid everything just works 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-1870276146
> Not sure here, as you said,It might not work for all end-points. 🤔
Yep, it doesn't work for all endpoints. Even minio users could set their own regions like `org-us-1`. In severe instances, an incorrect region can lead to more perplexing errors that are significantly harder for users to resolve.
For example, accessing s3 from wrong region could got:
```shell
{ code: "PermanentRedirect", message: "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.", resource: "", request_id: "MN9M36MVQ5AHKDQ2" }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-1857768944
The region forms part of the sigv4 signature so we can't just ignore it, although many systems like cloudflare R2 accept a region of us-east-1 or auto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-1878523751
`label_issue.py` automatically added labels {'object-store'} from #5244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "ritchie46 (via GitHub)" <gi...@apache.org>.
ritchie46 commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-1859637375
The `"auto"` seems ok. :+1:
It is perfectly solvable by end-users of `object-store`. Maybe we we could inform them of this option in an error message.
Not sure here, as you said,It might not work for all end-points. :thinking:
Feel free to close. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5211: [object-store] Make aws region optional.
URL: https://github.com/apache/arrow-rs/issues/5211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [object-store] Make aws region optional. [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5211:
URL: https://github.com/apache/arrow-rs/issues/5211#issuecomment-2741682561
Migrating from arrow-rs issue [#5211](https://github.com/apache/arrow-rs/issues/5211)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org