You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Bharat Viswanadham (Jira)" <ji...@apache.org> on 2020/09/15 00:05:00 UTC
[jira] [Commented] (HDDS-4243) Get API not working from S3A
filesystem with Ozone S3
[ https://issues.apache.org/jira/browse/HDDS-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195782#comment-17195782 ]
Bharat Viswanadham commented on HDDS-4243:
------------------------------------------
With below config changes, able to read file from Ozone S3buckets with S3A filesystem from OzoneS3
{code:java}
<property>
<name>fs.s3a.change.detection.version.required</name>
<value>false</value>
</property>
<property>
<name>fs.s3a.change.detection.mode</name>
<value>none</value>
</property>
{code}
From my understanding one reason for this might be OzoneS3 does not support ETag feature, in HeadObject we returned ETag as Key Modification time, in GetObject ETag is not returned, and this has caused to fail with exception.
*S3ACode:*
{code:java}
default:
// mode == Client or Server; will trigger on version failures
// of getObjectMetadata even on server.
return new ImmutablePair<>(true,
new RemoteFileChangedException(uri,
operation,
String.format("%s "
+ CHANGE_DETECTED
+ " during %s%s."
+ " Expected %s got %s",
getSource(), operation, positionText, revisionId, newRevisionId)));
{code}
> Get API not working from S3A filesystem with Ozone S3
> -----------------------------------------------------
>
> Key: HDDS-4243
> URL: https://issues.apache.org/jira/browse/HDDS-4243
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Major
> Labels: OzoneS3, S3A
>
> TroubleShooting S3A mentions S3 compatible servers that donot support Etags will see this server
> Refer [link|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html] and look for below section content.
> Using a third-party S3 implementation that doesn’t support eTags might result in the following error.
> org.apache.hadoop.fs.s3a.NoVersionAttributeException: `s3a://my-bucket/test/file.txt':
> Change detection policy requires ETag
> at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:153)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:200)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:346)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:195)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:193)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:215)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:339)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:372)
> {code:java}
> org.apache.hadoop.fs.s3a.NoVersionAttributeException: `s3a://sept14/dir1/dir2/dir3/key1': Change detection policy requires ETag
> at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processNewRevision(ChangeTracker.java:275)
> at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processMetadata(ChangeTracker.java:261)
> at org.apache.hadoop.fs.s3a.impl.ChangeTracker.processResponse(ChangeTracker.java:195)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:208)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:359)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:223)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$5(Invoker.java:347)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407)
> at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:343)
> at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:221)
> at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:265)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:351)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:464)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129)
> at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:494)
> at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:416)
> at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:351)
> at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:286)
> at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:271)
> at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:367)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:304)
> at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:266)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
> at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:237)
> at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
> get: `s3a://sept14/dir1/dir2/dir3/key1': Change detection policy requires ETag
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org