You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2023/08/21 13:41:00 UTC
[jira] [Updated] (HADOOP-18857) AWS v2 SDK: fail meaningfully if legacy v2 signing requested
[ https://issues.apache.org/jira/browse/HADOOP-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-18857:
------------------------------------
Description:
if the v2 signer is requested on an AWS SDKv2 build, you get an NPE.
proposed: special recognition of fs.s3a.signing-algorithm=S3SignerType and fail with a message that it isn't supported
{code}
> bin/hadoop jar $CLOUDSTORE storediag -D fs.s3a.signing-algorithm=S3SignerType -w s3a://stevel-london/
2023-08-21 14:36:41,229 [main] INFO diag.StoreDiag (StoreDurationInfo.java:close(137)) - Duration of Creating filesystem for s3a://stevel-london/: 0:00:357
java.lang.NullPointerException
at org.apache.hadoop.fs.s3a.auth.SignerFactory.createSigner(SignerFactory.java:104)
at org.apache.hadoop.fs.s3a.impl.AWSClientConfig.createClientConfigBuilder(AWSClientConfig.java:90)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createClientOverrideConfiguration(DefaultS3ClientFactory.java:163)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:147)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:89)
at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:989)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:610)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
{code}
was:
if the v2 signer is requested on an AWS SDKv2 build, you get an NPE.
proposed: special recogni
{code}
2023-08-21 14:36:41,229 [main] INFO diag.StoreDiag (StoreDurationInfo.java:close(137)) - Duration of Creating filesystem for s3a://stevel-london/: 0:00:357
java.lang.NullPointerException
at org.apache.hadoop.fs.s3a.auth.SignerFactory.createSigner(SignerFactory.java:104)
at org.apache.hadoop.fs.s3a.impl.AWSClientConfig.createClientConfigBuilder(AWSClientConfig.java:90)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createClientOverrideConfiguration(DefaultS3ClientFactory.java:163)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:147)
at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:89)
at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:989)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:610)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
{code}
> AWS v2 SDK: fail meaningfully if legacy v2 signing requested
> ------------------------------------------------------------
>
> Key: HADOOP-18857
> URL: https://issues.apache.org/jira/browse/HADOOP-18857
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Priority: Major
>
> if the v2 signer is requested on an AWS SDKv2 build, you get an NPE.
> proposed: special recognition of fs.s3a.signing-algorithm=S3SignerType and fail with a message that it isn't supported
> {code}
> > bin/hadoop jar $CLOUDSTORE storediag -D fs.s3a.signing-algorithm=S3SignerType -w s3a://stevel-london/
> 2023-08-21 14:36:41,229 [main] INFO diag.StoreDiag (StoreDurationInfo.java:close(137)) - Duration of Creating filesystem for s3a://stevel-london/: 0:00:357
> java.lang.NullPointerException
> at org.apache.hadoop.fs.s3a.auth.SignerFactory.createSigner(SignerFactory.java:104)
> at org.apache.hadoop.fs.s3a.impl.AWSClientConfig.createClientConfigBuilder(AWSClientConfig.java:90)
> at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createClientOverrideConfiguration(DefaultS3ClientFactory.java:163)
> at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.configureClientBuilder(DefaultS3ClientFactory.java:147)
> at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:89)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:989)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:610)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3601)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org