You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2024/12/30 17:30:00 UTC

[jira] [Commented] (HADOOP-19378) S3A: terasort test fail with CSE-kMS enabled and london region

    [ https://issues.apache.org/jira/browse/HADOOP-19378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17908920#comment-17908920 ] 

Steve Loughran commented on HADOOP-19378:
-----------------------------------------

[~srahman] over to you. I guess I hadn't run the terasort tests (only kicked off with -Dscale).

you'll need to include all CSE-KMS settings in the marshalled secrets, the goal being that work submitted into a cluster can collect on the client all auth and encryption details, rather have them in the cluster configuration.

I'd have expect this to have surfaced first in {{ITestSessionDelegationInFilesystem.testDelegatedFileSystem().testDelegatedFileSystem}}, though it didn't

* make sure that test is removing all config options related to cse; it must pick up all it needs from that delegation token
* once it is good, worry about the terasort. That sticks all its logs deep under {{target/test/data}} and not something you can debug

> S3A: terasort test fail with CSE-kMS enabled and london region
> --------------------------------------------------------------
>
>                 Key: HADOOP-19378
>                 URL: https://issues.apache.org/jira/browse/HADOOP-19378
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 3.4.2
>         Environment:  test bucket with cse-kms both bucket and kms in eu-west-2 region
>            Reporter: Steve Loughran
>            Assignee: Syed Shameerur Rahman
>            Priority: Major
>
> terasort test runs failing; job logs show the process yarn spawns is trying to use the default region provider list
> {code}
>  Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain
> {code}
> these are spawned processes; all login info and encryption secrets are passed in the marshalled delegation token {{AbstractS3ATokenIdentifier}}, with the encryption stuff in {{EncryptionSecrets}}
> This is probably going to need to be extended with information about the KMS region, with this info extracted during instantiation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org