You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "singhsaabir (via GitHub)" <gi...@apache.org> on 2025/09/09 08:02:54 UTC
[I] AWS/S3 Default Headers are not considered for signature calculation [arrow-rs-object-store]
singhsaabir opened a new issue, #484:
URL: https://github.com/apache/arrow-rs-object-store/issues/484
**Describe the bug**
S3 supports quite a few headers in system defined metadata and user defined metadata described in https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#SysMetadata. When setting these headers as default headers, they are not considered in the signature calculation.
As a result, S3 throws an error along the lines of
```
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<Error>
<Code>AccessDenied</Code>
<Message>There were headers present in the request which were not signed</Message>
<HeadersNotSigned>x-amz-meta-s1, x-amz-tagging</HeadersNotSigned>
...
</Error>
```
**To Reproduce**
```
let s3 = AmazonS3Builder::new()
.with_bucket_name(bucket)
.with_client_options({
ClientOptions::new().with_default_headers({
let mut headers = HeaderMap::new();
headers.insert("x-amz-meta-s1", "t1".parse().unwrap());
headers.insert("x-amz-tagging", "Project=blue&Classification=confidential".parse().unwrap());
headers
})
})
.build()?;
let store: Arc<dyn ObjectStore> = Arc::new(s3);
store.put(&key.clone().into(), "content".into()).await?;
println!("Uploaded file with key: {}", key);
```
**Expected behavior**
The request should succeed and S3 HeadObject should echo these headers back.
**Additional context**
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS/S3 Default Headers are not considered for signature calculation [arrow-rs-object-store]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #484:
URL: https://github.com/apache/arrow-rs-object-store/issues/484#issuecomment-3271017337
I think the better approach is to try to use [`Attributes`](https://docs.rs/object_store/latest/object_store/enum.Attribute.html) here. I'm not sure if there's a way to pass along custom S3 headers there, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS/S3 Default Headers are not considered for signature calculation [arrow-rs-object-store]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #484:
URL: https://github.com/apache/arrow-rs-object-store/issues/484#issuecomment-3271554894
I think `Attributes` as it stands are intended to be cross-platform. I'm not sure how best to set AWS-specific attributes there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS/S3 Default Headers are not considered for signature calculation [arrow-rs-object-store]
Posted by "singhsaabir (via GitHub)" <gi...@apache.org>.
singhsaabir commented on issue #484:
URL: https://github.com/apache/arrow-rs-object-store/issues/484#issuecomment-3271407802
Thank you for your quick reply. TIL, maybe [Attributes](https://docs.rs/object_store/latest/object_store/enum.Attribute.html) is a better use case for metadata. I was trying to think of building something generic here since S3 can supports quite a few headers.
Specifically, AWS requires internal services to set `x-amz-source-account` and `x-amz-source-arn` headers to prevent Confused Deputy attacks. They enable the `SourceAccount` and `SourceArn` variables in IAM policies. More details about these in https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html. I was thinking I could use `default_headers` for all of these use cases. But seeing [Attributes](https://docs.rs/object_store/latest/object_store/enum.Attribute.html) and [with_request_payer](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_request_payer) in the AmazonS3Builder, maybe it would be better to have `with_source_account` and `with_source_arn` for this purpose.
I am happy to contribute with a PR for this. Looking for direction/guidance on the approach we could take :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] AWS/S3 Default Headers are not considered for signature calculation [arrow-rs-object-store]
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum closed issue #484: AWS/S3 Default Headers are not considered for signature calculation
URL: https://github.com/apache/arrow-rs-object-store/issues/484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org