[jira] [Commented] (HADOOP-19197) S3A: Support AWS KMS Encryption Context

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-19197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18052580#comment-18052580 ] 

ASF GitHub Bot commented on HADOOP-19197:
-----------------------------------------

raphaelazzolini opened a new pull request, #8185:
URL: https://github.com/apache/hadoop/pull/8185

   Followup the main HADOOP-19197 patch to address serialization and compilation issues
   
   * Recreate serialization ID
   * Restore two arg constructor
   * Define DEFAULT_S3_ENCRYPTION_CONTEXT to specify what the default value is (just "", but being explicit)
   * Restore ability to unmarshal old version encryption secrets.
   * Tests
   
   This allows for YARN services to load DTs supplied by older releases.
   
   If they marshall the secrets again the fact they were the older version is lost, they get upgraded. This may complicate any worker node launch where the DT list is modified before passing to the launched process
   
   #### This PR squases the commits from https://github.com/apache/hadoop/pull/7830 and adds testUnmarshallUnknownEncryptionSecretsVersion for testing EncryptionSecrets with unknown serialize version
   
   ### How was this patch tested?
   Unit tests. ITests failures are meaningless until https://github.com/apache/hadoop/pull/7814 is in.
   
   ### For code changes:
   
   - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?




> S3A: Support AWS KMS Encryption Context
> ---------------------------------------
>
>                 Key: HADOOP-19197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19197
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 3.4.0
>            Reporter: Raphael Azzolini
>            Assignee: Raphael Azzolini
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 3.5.0
>
>
> S3A properties allow users to choose the AWS KMS key ({_}fs.s3a.encryption.key{_}) and S3 encryption algorithm to be used (f{_}s.s3a.encryption.algorithm{_}). In addition to the AWS KMS Key, an encryption context can be used as non-secret data that adds additional integrity and authenticity to check the encrypted data. However, there is no option to specify the [AWS KMS Encryption Context|https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context] in S3A.
> In AWS SDK v2 the encryption context in S3 requests is set by the parameter [ssekmsEncryptionContext.|https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/model/CreateMultipartUploadRequest.Builder.html#ssekmsEncryptionContext(java.lang.String)] It receives a base64-encoded UTF-8 string holding JSON with the encryption context key-value pairs. The value of this parameter could be set by the user in a new property {_}*fs.s3a.encryption.context*{_}, and be stored in the [EncryptionSecrets|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecrets.java] to later be used when setting the encryption parameters in [RequestFactoryImpl|https://github.com/apache/hadoop/blob/f92a8ab8ae54f11946412904973eb60404dee7ff/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org