You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2019/10/02 17:47:00 UTC
[jira] [Comment Edited] (HADOOP-16393) S3Guard init command uses
global settings, not those of target bucket
[ https://issues.apache.org/jira/browse/HADOOP-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943031#comment-16943031 ]
Steve Loughran edited comment on HADOOP-16393 at 10/2/19 5:46 PM:
------------------------------------------------------------------
still seeing this
{{```}}
{{ {bin/hadoop s3guard init s3a://hwdev-steve-ireland-new/}}
{{ 2019-10-02 18:42:22,019 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:run(1758)) - Executing command init}}
{{ 2019-10-02 18:42:22,045 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1137)) - Propagating entries under fs.s3a.bucket.hwdev-steve-ireland-new.}}
{{ 2019-10-02 18:42:22,047 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.committer.name from [core-site.xml]}}
{{ 2019-10-02 18:42:22,048 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.endpoint from [core-site.xml]}}
{{ 2019-10-02 18:42:22,048 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.committer.magic.enabled from [core-site.xml]}}
{{ 2019-10-02 18:42:22,048 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.metadatastore.impl from [core-site.xml]}}
{{ java.lang.IllegalArgumentException: No DynamoDB table name configured}}
{{ at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)}}
{{ at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:484)}}
{{ at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.initMetadataStore(S3GuardTool.java:320)}}
{{ at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Init.run(S3GuardTool.java:542)}}
{{ at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:427)}}
{{ at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)}}
{{ at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1797)}}
{{ at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1806)}}
{{ 2019-10-02 18:42:22,071 [main] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status -1: java.lang.IllegalArgumentException: No DynamoDB table name configured}}
{{ }}}}
{{ ```}}
was (Author: stevel@apache.org):
still seeing this
```
bin/hadoop s3guard init s3a://hwdev-steve-ireland-new/
2019-10-02 18:42:22,019 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:run(1758)) - Executing command init
2019-10-02 18:42:22,045 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1137)) - Propagating entries under fs.s3a.bucket.hwdev-steve-ireland-new.
2019-10-02 18:42:22,047 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.committer.name from [core-site.xml]
2019-10-02 18:42:22,048 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.endpoint from [core-site.xml]
2019-10-02 18:42:22,048 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.committer.magic.enabled from [core-site.xml]
2019-10-02 18:42:22,048 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1158)) - Updating fs.s3a.metadatastore.impl from [core-site.xml]
java.lang.IllegalArgumentException: No DynamoDB table name configured
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:484)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.initMetadataStore(S3GuardTool.java:320)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Init.run(S3GuardTool.java:542)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:427)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1797)
at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1806)
2019-10-02 18:42:22,071 [main] INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status -1: java.lang.IllegalArgumentException: No DynamoDB table name configured
```
> S3Guard init command uses global settings, not those of target bucket
> ---------------------------------------------------------------------
>
> Key: HADOOP-16393
> URL: https://issues.apache.org/jira/browse/HADOOP-16393
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Fix For: 3.3.0
>
>
> If you call {{s3guard init s3a://name/}} then the custom bucket options of fs.s3a.bucket.name are not picked up, instead the global value is used.
> Fix: take the name of the bucket and use that to eval properties and patch the config used for the init command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org