You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "carols10cents (via GitHub)" <gi...@apache.org> on 2023/09/28 20:33:56 UTC
[GitHub] [arrow-rs] carols10cents opened a new pull request, #4876: WIP: Adding AWS presigned URL support
carols10cents opened a new pull request, #4876:
URL: https://github.com/apache/arrow-rs/pull/4876
NOTE: This is currently a work in progress, and I'd love general feedback at this point to know that I'm going in the right direction! I think there's more refactoring I could do and tests I could add before this is ready for merging; and I'm not sure if this is the best public API or not.
Also I haven't actually tested this against a live S3 bucket yet 😳
# Which issue does this PR close?
Closes #4763.
# Rationale for this change
Enable reusing the authorization and signing code to generate signed URLs that can be used to give permission to upload or view particular objects.
# What changes are included in this PR?
A new public method on `AwsAuthorizer`, currently called `sign` (as opposed to the existing `authorize` method) that adds AWS signature query parameters to a specified `Url` so that it can be given to another user to authorize uploading or viewing a particular resource.
This shares a lot of the signing code with the existing `authorize` method, so I've pulled out a few private methods, but I don't think the current structure is ideal yet. Thoughts on how far to take this are very welcome, such as if there's another struct or three starting to cry to be let out of `AwsAuthorizer`, if these should be free functions, etc.
# Are there any user-facing changes?
Yes, one new method, and I'd like to add an example of how to use this when you have an `AmazonS3` instance before merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1751425604
Ok, I think I'm happy-ish with this? I have [an iox branch working](https://github.com/influxdata/influxdb_iox/pull/8927) that I think isn't terrible, maybe?
CI is failing with something about ring, I don't know why :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on code in PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#discussion_r1342780895
##########
object_store/src/aws/credential.rs:
##########
@@ -154,21 +155,140 @@ impl<'a> AwsAuthorizer<'a> {
let header_digest = HeaderValue::from_str(&digest).unwrap();
request.headers_mut().insert(HASH_HEADER, header_digest);
- // Each path segment must be URI-encoded twice (except for Amazon S3 which only gets URI-encoded once).
+ let (signed_headers, canonical_headers) = canonicalize_headers(request.headers());
+
+ let scope = self.scope(date);
+
+ let string_to_sign = self.string_to_sign(
+ date,
+ &scope,
+ request.method(),
+ request.url(),
+ &canonical_headers,
+ &signed_headers,
+ &digest,
+ );
+
+ // sign the string
+ let signature =
+ self.credential
+ .sign(&string_to_sign, date, self.region, self.service);
+
+ // build the actual auth header
+ let authorisation = format!(
+ "{} Credential={}/{}, SignedHeaders={}, Signature={}",
+ ALGORITHM, self.credential.key_id, scope, signed_headers, signature
+ );
+
+ let authorization_val = HeaderValue::from_str(&authorisation).unwrap();
+ request.headers_mut().insert(AUTH_HEADER, authorization_val);
+ }
+
+ /// Authorize a request via `method` to `url` valid for the duration specified in `expires_in`
+ /// by attaching the relevant [AWS SigV4] query parameters.
+ ///
+ /// [AWS SigV4]: https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
+ ///
+ /// # Example
+ ///
+ /// This example modifies `url` to add the signing parameters needed to enable a user to upload
+ /// a file to "some-folder/some-file.txt" in the next hour.
+ ///
+ /// ```
+ /// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
+ /// use object_store::{aws::{AmazonS3Builder, AwsAuthorizer}, path::Path};
+ /// use reqwest::Method;
+ /// use std::time::Duration;
+ /// use url::Url;
+ ///
+ /// let region = "us-east-1";
+ /// let s3 = AmazonS3Builder::new()
+ /// .with_region(region)
+ /// .with_bucket_name("my-bucket")
+ /// .with_access_key_id("my-access-key-id")
+ /// .with_secret_access_key("my-secret-access-key")
+ /// .build()?;
+ /// let credential = s3
+ /// .credentials()
+ /// .get_credential()
+ /// .await?;
+ /// let authorizer = AwsAuthorizer::new(&credential, "s3", region);
+ /// let mut url = Url::parse(&s3.path_url(&Path::from("some-folder/some-file.txt")))?;
+ /// authorizer.sign(Method::PUT, &mut url, Duration::from_secs(60 * 60));
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub fn sign(&self, method: Method, url: &mut Url, expires_in: Duration) {
+ let date = self.date.unwrap_or_else(Utc::now);
+ let scope = self.scope(date);
+
+ // https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
+ url.query_pairs_mut()
+ .append_pair("X-Amz-Algorithm", ALGORITHM)
+ .append_pair(
+ "X-Amz-Credential",
+ &format!("{}/{}", self.credential.key_id, scope),
+ )
+ .append_pair("X-Amz-Date", &date.format("%Y%m%dT%H%M%SZ").to_string())
+ .append_pair("X-Amz-Expires", &expires_in.as_secs().to_string())
+ .append_pair("X-Amz-SignedHeaders", "host");
+
+ // TODO: For S3, you must include the X-Amz-Security-Token query parameter in the URL if
Review Comment:
and done ✅
##########
object_store/src/aws/credential.rs:
##########
@@ -154,21 +155,140 @@ impl<'a> AwsAuthorizer<'a> {
let header_digest = HeaderValue::from_str(&digest).unwrap();
request.headers_mut().insert(HASH_HEADER, header_digest);
- // Each path segment must be URI-encoded twice (except for Amazon S3 which only gets URI-encoded once).
+ let (signed_headers, canonical_headers) = canonicalize_headers(request.headers());
+
+ let scope = self.scope(date);
+
+ let string_to_sign = self.string_to_sign(
+ date,
+ &scope,
+ request.method(),
+ request.url(),
+ &canonical_headers,
+ &signed_headers,
+ &digest,
+ );
+
+ // sign the string
+ let signature =
+ self.credential
+ .sign(&string_to_sign, date, self.region, self.service);
+
+ // build the actual auth header
+ let authorisation = format!(
+ "{} Credential={}/{}, SignedHeaders={}, Signature={}",
+ ALGORITHM, self.credential.key_id, scope, signed_headers, signature
+ );
+
+ let authorization_val = HeaderValue::from_str(&authorisation).unwrap();
+ request.headers_mut().insert(AUTH_HEADER, authorization_val);
+ }
+
+ /// Authorize a request via `method` to `url` valid for the duration specified in `expires_in`
+ /// by attaching the relevant [AWS SigV4] query parameters.
+ ///
+ /// [AWS SigV4]: https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
+ ///
+ /// # Example
+ ///
+ /// This example modifies `url` to add the signing parameters needed to enable a user to upload
+ /// a file to "some-folder/some-file.txt" in the next hour.
+ ///
+ /// ```
+ /// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
+ /// use object_store::{aws::{AmazonS3Builder, AwsAuthorizer}, path::Path};
+ /// use reqwest::Method;
+ /// use std::time::Duration;
+ /// use url::Url;
+ ///
+ /// let region = "us-east-1";
+ /// let s3 = AmazonS3Builder::new()
+ /// .with_region(region)
+ /// .with_bucket_name("my-bucket")
+ /// .with_access_key_id("my-access-key-id")
+ /// .with_secret_access_key("my-secret-access-key")
+ /// .build()?;
+ /// let credential = s3
+ /// .credentials()
+ /// .get_credential()
+ /// .await?;
+ /// let authorizer = AwsAuthorizer::new(&credential, "s3", region);
+ /// let mut url = Url::parse(&s3.path_url(&Path::from("some-folder/some-file.txt")))?;
+ /// authorizer.sign(Method::PUT, &mut url, Duration::from_secs(60 * 60));
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub fn sign(&self, method: Method, url: &mut Url, expires_in: Duration) {
+ let date = self.date.unwrap_or_else(Utc::now);
+ let scope = self.scope(date);
+
+ // https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
+ url.query_pairs_mut()
+ .append_pair("X-Amz-Algorithm", ALGORITHM)
+ .append_pair(
+ "X-Amz-Credential",
+ &format!("{}/{}", self.credential.key_id, scope),
+ )
+ .append_pair("X-Amz-Date", &date.format("%Y%m%dT%H%M%SZ").to_string())
+ .append_pair("X-Amz-Expires", &expires_in.as_secs().to_string())
+ .append_pair("X-Amz-SignedHeaders", "host");
+
+ // TODO: For S3, you must include the X-Amz-Security-Token query parameter in the URL if
Review Comment:
and done ✅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1757809694
@tustvold fixed, and CI is passing now!! 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1747528081
> I think I might be missing some context on why the pre-signed URL functionality needs to be conflated with the ObjectStore trait.
TL;DR IOx; I'm happy to make changes to IOx to do whatever.
That's purely due to the current IOx implementation that has a `clap_blocks` function for configuring any object store and currently returns an `Arc<DynObjectStore>`, thus erasing any type information after that point.
For the bulk import functionality, what we've planned for a piece of it is a gRPC service implemented in the router that has an action that returns a pre-signed URL. Currently, the router uses the aforementioned `clap_blocks` to configure the object store and has the `Arc<DynObjectStore>` plumbed through it everywhere-- I'd like to avoid changing all that if possible, and supporting presigned URLs through other object stores if those implementations gain the signing capability in the future... does that make more sense? I am relying on the IOx context you have, let me know if I glossed over too much :)
> I would have perhaps expected a `dyn Signer` that could be passed around to the places that need it.
Right, but who defines `trait Signer`? I think it'd be nice if it was in `object_store` such that as more object store implementations gain that functionality, users wouldn't have to add the implementations themselves. Would that be ok or do you think it doesn't belong here?
----
Do you have any general comments about this PR as-is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1749129998
> In IOx terms, the router will have something that can produce signed urls (configured using the code in clap_blocks, https://github.com/influxdata/influxdb_iox/pull/8927) and will pass that full URL to a bulk import tool
This looks sensible to me :+1:
> So I think the ObjectStore trait will need methods like fn put_presigned(&self, location: &Url, bytes: Bytes) and fn get_presigned(&self, location: &Url), and these should be on the trait so that the adapters like LimitStore and ThrottledStore can support them as well.
> that will also have an object store configured using the code in clap_blocks (that currently returns an Arc<DynObjectStore> which is a LimitStore wrapping an AmazonS3 in this case) so that we don't have to reimplement all those settings and code handling the upload request
The upload request is a simple PUT request with the object payload as the body. I'm not sure I follow why this would benefit from being ObjectStore-ified?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1747407319
I think I might be missing some context on why the pre-signed URL functionality needs to be conflated with the ObjectStore trait.
I would have perhaps expected a `dyn Signer` that could be passed around to the places that need it. This is based on the potentially incorrect belief that these would be largely disjoint from those that actually need ObjectStore?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1748683523
Codes looks good, will continue to think about how we expose this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1752868909
Ring failure was likely - https://github.com/briansmith/ring/issues/1707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1747377693
Hm. I'm trying to put this into use in IOx and running into a bit of a problem... ideally, we'd like to not have to change much about the rest of the codebase that uses `Arc<object_store::DynObjectStore>` pretty much everywhere, so Dom and I were thinking of defining a new trait for presigned URLs (that would be a supertrait of `object_store::ObjectStore` and the bulk import could say it needed that, `AmazonS3` could implement it, and the other object store types could implement the new trait but return an error (at some level TBD).
The problem with that I'm running into is that we're not actually using `AmazonS3`, we're using a `LimitStore` adapting an `AmazonS3`, and I can't `impl<T: TheNewTrait> TheNewTrait for `LimitStore<T>` because `LimitStore` (or for that matter `ThrottledStore`) doesn't expose access to the inner `T`.
The solutions I'm seeing to this need to be done in `object_store`, I think:
- Add `pub fn inner(&self) -> &T` to `LimitStore` and `ThrottledStore` to enable implementing traits that delegate to these adapters' inner object stores, which could enable other object store extensions in the future
- Define this new trait I tried to write within `object_store` itself, and have all the implementers other than `AmazonS3` return an error saying it's not supported
- ??? Something else I'm not seeing?
I'm going to try going down these paths to see if I run into different issues, but I'd love thoughts on the best way to solve this from the `object_store` perspective!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1749375449
> n IOx terms, the router will have something that can produce signed urls (configured using the code in clap_blocks, https://github.com/influxdata/influxdb_iox/pull/8927) and will pass that full URL to a bulk import tool, that will also have an object store configured using the code in clap_blocks (that currently returns an Arc<DynObjectStore> which is a LimitStore wrapping an AmazonS3 in this case) so that we don't have to reimplement all those settings and code handling the upload request.
As Raphael and I were chatting, I realized this statement of mine was exactly wrong, as Raphael also commented-- the whole point of the thing using the signed URLs is that it DOESN'T have the `object_store` credentials etc configured-- everything needed will be in the URL. So yeah, ignore this-- the actual upload shouldn't happen through the `object_store` crate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#discussion_r1350210330
##########
object_store/Cargo.toml:
##########
@@ -45,6 +45,7 @@ walkdir = "2"
# Cloud storage support
base64 = { version = "0.21", default-features = false, features = ["std"], optional = true }
+http = { version = "0.2", default_features = false, optional = true }
Review Comment:
FWIW reqwest re-exports `http::Method` which appears to be the only thing this is used for
##########
object_store/src/aws/mod.rs:
##########
@@ -209,6 +210,65 @@ impl AmazonS3 {
pub fn credentials(&self) -> &AwsCredentialProvider {
&self.client.config().credentials
}
+
+ /// Create a full URL to the resource specified by `path` with this instance's configuration.
+ fn path_url(&self, path: &Path) -> String {
+ self.client.config().path_url(path)
+ }
+}
+
+#[async_trait]
+impl Signer for AmazonS3 {
+ /// Create a URL containing the relevant [AWS SigV4] query parameters that authorize a request
+ /// via `method` to the resource at `path` valid for the duration specified in `expires_in`.
+ ///
+ /// [AWS SigV4]: https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
+ ///
+ /// # Example
+ ///
+ /// This example returns a URL that will enable a user to upload a file to
+ /// "some-folder/some-file.txt" in the next hour.
+ ///
+ /// ```
+ /// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
+ /// use object_store::{aws::AmazonS3Builder, path::Path, signer::Signer};
+ /// use http::Method;
+ /// use std::time::Duration;
+ ///
Review Comment:
```suggestion
/// # use object_store::{aws::AmazonS3Builder, path::Path, signer::Signer};
/// # use http::Method;
/// # use std::time::Duration;
/// #
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on code in PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#discussion_r1342762468
##########
object_store/src/aws/credential.rs:
##########
@@ -154,21 +155,140 @@ impl<'a> AwsAuthorizer<'a> {
let header_digest = HeaderValue::from_str(&digest).unwrap();
request.headers_mut().insert(HASH_HEADER, header_digest);
- // Each path segment must be URI-encoded twice (except for Amazon S3 which only gets URI-encoded once).
+ let (signed_headers, canonical_headers) = canonicalize_headers(request.headers());
+
+ let scope = self.scope(date);
+
+ let string_to_sign = self.string_to_sign(
+ date,
+ &scope,
+ request.method(),
+ request.url(),
+ &canonical_headers,
+ &signed_headers,
+ &digest,
+ );
+
+ // sign the string
+ let signature =
+ self.credential
+ .sign(&string_to_sign, date, self.region, self.service);
+
+ // build the actual auth header
+ let authorisation = format!(
+ "{} Credential={}/{}, SignedHeaders={}, Signature={}",
+ ALGORITHM, self.credential.key_id, scope, signed_headers, signature
+ );
+
+ let authorization_val = HeaderValue::from_str(&authorisation).unwrap();
+ request.headers_mut().insert(AUTH_HEADER, authorization_val);
+ }
+
+ /// Authorize a request via `method` to `url` valid for the duration specified in `expires_in`
+ /// by attaching the relevant [AWS SigV4] query parameters.
+ ///
+ /// [AWS SigV4]: https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
+ ///
+ /// # Example
+ ///
+ /// This example modifies `url` to add the signing parameters needed to enable a user to upload
+ /// a file to "some-folder/some-file.txt" in the next hour.
+ ///
+ /// ```
+ /// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
+ /// use object_store::{aws::{AmazonS3Builder, AwsAuthorizer}, path::Path};
+ /// use reqwest::Method;
+ /// use std::time::Duration;
+ /// use url::Url;
+ ///
+ /// let region = "us-east-1";
+ /// let s3 = AmazonS3Builder::new()
+ /// .with_region(region)
+ /// .with_bucket_name("my-bucket")
+ /// .with_access_key_id("my-access-key-id")
+ /// .with_secret_access_key("my-secret-access-key")
+ /// .build()?;
+ /// let credential = s3
+ /// .credentials()
+ /// .get_credential()
+ /// .await?;
+ /// let authorizer = AwsAuthorizer::new(&credential, "s3", region);
+ /// let mut url = Url::parse(&s3.path_url(&Path::from("some-folder/some-file.txt")))?;
+ /// authorizer.sign(Method::PUT, &mut url, Duration::from_secs(60 * 60));
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub fn sign(&self, method: Method, url: &mut Url, expires_in: Duration) {
+ let date = self.date.unwrap_or_else(Utc::now);
+ let scope = self.scope(date);
+
+ // https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
+ url.query_pairs_mut()
+ .append_pair("X-Amz-Algorithm", ALGORITHM)
+ .append_pair(
+ "X-Amz-Credential",
+ &format!("{}/{}", self.credential.key_id, scope),
+ )
+ .append_pair("X-Amz-Date", &date.format("%Y%m%dT%H%M%SZ").to_string())
+ .append_pair("X-Amz-Expires", &expires_in.as_secs().to_string())
+ .append_pair("X-Amz-SignedHeaders", "host");
+
+ // TODO: For S3, you must include the X-Amz-Security-Token query parameter in the URL if
Review Comment:
WHOOPS at some point this weekend I remembered this TODO was in here... taking care of it now! 😅
##########
object_store/src/aws/credential.rs:
##########
@@ -154,21 +155,140 @@ impl<'a> AwsAuthorizer<'a> {
let header_digest = HeaderValue::from_str(&digest).unwrap();
request.headers_mut().insert(HASH_HEADER, header_digest);
- // Each path segment must be URI-encoded twice (except for Amazon S3 which only gets URI-encoded once).
+ let (signed_headers, canonical_headers) = canonicalize_headers(request.headers());
+
+ let scope = self.scope(date);
+
+ let string_to_sign = self.string_to_sign(
+ date,
+ &scope,
+ request.method(),
+ request.url(),
+ &canonical_headers,
+ &signed_headers,
+ &digest,
+ );
+
+ // sign the string
+ let signature =
+ self.credential
+ .sign(&string_to_sign, date, self.region, self.service);
+
+ // build the actual auth header
+ let authorisation = format!(
+ "{} Credential={}/{}, SignedHeaders={}, Signature={}",
+ ALGORITHM, self.credential.key_id, scope, signed_headers, signature
+ );
+
+ let authorization_val = HeaderValue::from_str(&authorisation).unwrap();
+ request.headers_mut().insert(AUTH_HEADER, authorization_val);
+ }
+
+ /// Authorize a request via `method` to `url` valid for the duration specified in `expires_in`
+ /// by attaching the relevant [AWS SigV4] query parameters.
+ ///
+ /// [AWS SigV4]: https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
+ ///
+ /// # Example
+ ///
+ /// This example modifies `url` to add the signing parameters needed to enable a user to upload
+ /// a file to "some-folder/some-file.txt" in the next hour.
+ ///
+ /// ```
+ /// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
+ /// use object_store::{aws::{AmazonS3Builder, AwsAuthorizer}, path::Path};
+ /// use reqwest::Method;
+ /// use std::time::Duration;
+ /// use url::Url;
+ ///
+ /// let region = "us-east-1";
+ /// let s3 = AmazonS3Builder::new()
+ /// .with_region(region)
+ /// .with_bucket_name("my-bucket")
+ /// .with_access_key_id("my-access-key-id")
+ /// .with_secret_access_key("my-secret-access-key")
+ /// .build()?;
+ /// let credential = s3
+ /// .credentials()
+ /// .get_credential()
+ /// .await?;
+ /// let authorizer = AwsAuthorizer::new(&credential, "s3", region);
+ /// let mut url = Url::parse(&s3.path_url(&Path::from("some-folder/some-file.txt")))?;
+ /// authorizer.sign(Method::PUT, &mut url, Duration::from_secs(60 * 60));
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub fn sign(&self, method: Method, url: &mut Url, expires_in: Duration) {
+ let date = self.date.unwrap_or_else(Utc::now);
+ let scope = self.scope(date);
+
+ // https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
+ url.query_pairs_mut()
+ .append_pair("X-Amz-Algorithm", ALGORITHM)
+ .append_pair(
+ "X-Amz-Credential",
+ &format!("{}/{}", self.credential.key_id, scope),
+ )
+ .append_pair("X-Amz-Date", &date.format("%Y%m%dT%H%M%SZ").to_string())
+ .append_pair("X-Amz-Expires", &expires_in.as_secs().to_string())
+ .append_pair("X-Amz-SignedHeaders", "host");
+
+ // TODO: For S3, you must include the X-Amz-Security-Token query parameter in the URL if
Review Comment:
WHOOPS at some point this weekend I remembered this TODO was in here... taking care of it now! 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1747750880
I need to sleep on it, but perhaps we just add it to the ObjectStore trait with a default not implemented impl, it wouldn't be the weirdest thing there - I'm looking at you append 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add AWS presigned URL support [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on PR #4876:
URL: https://github.com/apache/arrow-rs/pull/4876#issuecomment-1749015043
Ok, so here's another wrinkle.
The URL signing API should return a full URL so that if the user wants to, they can pass the URL to something that isn't using the `object_store` crate and use it with `cURL`, `wget`, a web browser, what have you.
However, if the user _is_ doing all upload/download through the `object_store` crate, and particularly through wrappers like `LimitStore` and such, they _will_ want to send the request using the presigned URL through their `object_store`, but currently `put_request` and friends take a `Path` that gets concatenated onto the object store implementation's full URL (which wouldn't be what we'd want in this case).
So I think the `ObjectStore` trait will need methods like `fn put_presigned(&self, location: &Url, bytes: Bytes)` and `fn get_presigned(&self, location: &Url)`, and these should be on the trait so that the adapters like `LimitStore` and `ThrottledStore` can support them as well.
In IOx terms, the router will have something that can produce signed urls (configured using the code in clap_blocks, [I think I've figured out a good way to do that](https://github.com/influxdata/influxdb_iox/pull/8927)) and will pass that full URL to a bulk import tool, that will also have an object store configured using the code in clap_blocks (that currently returns an `Arc<DynObjectStore>` which is a `LimitStore` wrapping an `AmazonS3` in this case) so that we don't have to reimplement all those settings and code handling the upload request.
Does that sound right to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org