You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "quartox (via GitHub)" <gi...@apache.org> on 2023/11/17 01:17:41 UTC
[I] Can I use S3 server side encryption [arrow-rs]
quartox opened a new issue, #5087:
URL: https://github.com/apache/arrow-rs/issues/5087
**Which part is this question about**
Library API
**Describe your question**
I need to use [server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-s3-encryption.html) when uploading to S3. The upload is rejected without specifying this option, e.g. from aws cli `--server-side-encryption AES256`. Based on the AmazonS3ConfigKey and ClientKey enums it looks like this is not supported but I am not certain if I understand this correctly. Is there currently a way to specify this argument when uploading?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-1910727763
I plan to work on adding support for this and related use-cases such as https://github.com/apache/arrow-rs/issues/5329 tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-1910687807
I'm looking at this for SSE-KMS support, and it looks like the problem with default header will be that AWS S3 accepts the encryption headers for PUT requests, but will return 400 Bad Request if you pass them along to GET and HEAD requests:
> If your object uses SSE-KMS, don't send encryption request headers for GET requests and HEAD requests, or you’ll get an HTTP 400 BadRequest error.[^1]
So I think these are likely configurations that need to be added to the `AmazonS3` store.
[^1]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-1916850790
I think whilst #5334 will potentially provide a way to configure this on a per-operation basis, there is also a use-case for being able to configure this for an entire bucket through properties on the builder.
I think this would be a relatively straightforward addition, and I would be happy to review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-1943782579
> GCP's XML API always uses headers regardless of mode.
We only use the XML API
> Keys are managed by a KMS
I suspect this is what the vast majority of users are after, and it might be worth starting with just this, unless you specifically require user-provided key support
> Potential Shared API
I wonder if we can just lean on the existing configuration system for this, i.e. just allow the different stores to have their own configurations for encryption keys, etc... This isn't materially different from any store specific configuration IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-2741682693
Migrating from arrow-rs issue [#5087](https://github.com/apache/arrow-rs/issues/5087)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5087: Can I use S3 server side encryption
URL: https://github.com/apache/arrow-rs/issues/5087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-1815901369
There isn't great support for this, but you could possibly use https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_default_headers to specify the necessary headers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Can I use S3 server side encryption [arrow-rs]
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on issue #5087:
URL: https://github.com/apache/arrow-rs/issues/5087#issuecomment-1942897487
In keeping with the goals of this crate, I thought I might see if there is a general pattern across object store implementations for how encryption is handled. Indeed there does seem to be a pattern.
In general, there are two modes:
1. Keys are managed by a KMS, and object store can directly reach out and get keys from it. Here, they only need to be provided when writing. On read, the object store can lookup which key it needs to decrypt with from metadata and fetch as needed.
2. Keys are provided by the user, and must be provided on ever write **and read** request. The object store does not keep the keys, just a digest of them.
For the most part, encryption information is provided as headers as part of requests. The one exception to this is GCS uses query parameters, but only for it's JSON API when in KMS mode. User-provided keys are always passed via headers, and GCP's XML API always uses headers regardless of mode. 😵
For all three object stores, user-provided keys are always AES-256 keys. They are always accompanied by a digest. AWS uses MD5, while Azure and GCP use sha256.
## Potential Shared API
This makes me think there might be a general configuration API that could be passed into builders.
```rust
// TODO: do not derive debug so encryption keys aren't accidentally printed.
enum EncryptionMode {
/// Use a key management store
KMS {
/// If not provided, will attempt to use a default key, if any.
key_id: Option<String>,
},
/// Provide a key with each request
Key {
/// The encryption key to use. Most services require this to be
key: Bytes,
/// The digest of the key. If this is not provided, this will be computed
/// from the key based on the object stores preferred algorithm, if known.
/// This is MD5 for AWS S3 and sha256 for others.
digest: Option<Bytes>,
}
}
```
Or perhaps this should be more of a struct like [ClientOptions](https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html), which is the best parallel I can think of to this situation.
I'll think about this a little more, and likely submit something just implementing AWS for now. But I do think it would be nice to have an API that would work well for other object stores that support server-side encryption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org