You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/07/14 12:15:00 UTC
[jira] [Commented] (HADOOP-17802) S3A Xattr/getXAttr to handle
directories without markers
[ https://issues.apache.org/jira/browse/HADOOP-17802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380545#comment-17380545 ]
Steve Loughran commented on HADOOP-17802:
-----------------------------------------
{code}
bin/hadoop fs -getfattr -d s3a://stevel-london/user/stevel/target/test/data
# file: s3a://stevel-london/user/stevel/target/test/data
2021-07-14 13:10:53,783 [main] DEBUG shell.Command (Command.java:displayError(464)) - getfattr failure
java.io.FileNotFoundException: getObjectMetadata on user/stevel/target/test/data/: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: MJ7KHEZ2R82NT179; S3 Extended Request ID: jWOXiMtgA8hmuVvwg4daFEwWnG0sTUa9g5+5nJ24GTlKTn1QDBSSnoXDylM6HYisdKjAI7c6nfE=; Proxy: null), S3 Extended Request ID: jWOXiMtgA8hmuVvwg4daFEwWnG0sTUa9g5+5nJ24GTlKTn1QDBSSnoXDylM6HYisdKjAI7c6nfE=:404 Not Found
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:261)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:119)
at org.apache.hadoop.fs.s3a.S3AFileSystem$HeaderProcessingCallbacksImpl.getObjectMetadata(S3AFileSystem.java:4774)
at org.apache.hadoop.fs.s3a.impl.HeaderProcessing.lambda$retrieveHeaders$1(HeaderProcessing.java:287)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.impl.HeaderProcessing.retrieveHeaders(HeaderProcessing.java:286)
at org.apache.hadoop.fs.s3a.impl.HeaderProcessing.getXAttrs(HeaderProcessing.java:405)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getXAttrs(S3AFileSystem.java:4727)
at org.apache.hadoop.fs.shell.XAttrCommands$GetfattrCommand.processPath(XAttrCommands.java:107)
at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:370)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:333)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:306)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:288)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:272)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
at org.apache.hadoop.fs.shell.Command.run(Command.java:179)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: MJ7KHEZ2R82NT179; S3 Extended Request ID: jWOXiMtgA8hmuVvwg4daFEwWnG0sTUa9g5+5nJ24GTlKTn1QDBSSnoXDylM6HYisdKjAI7c6nfE=; Proxy: null), S3 Extended Request ID: jWOXiMtgA8hmuVvwg4daFEwWnG0sTUa9g5+5nJ24GTlKTn1QDBSSnoXDylM6HYisdKjAI7c6nfE=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1367)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:2462)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:377)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2450)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2430)
at org.apache.hadoop.fs.s3a.S3AFileSystem$HeaderProcessingCallbacksImpl.lambda$getObjectMetadata$0(S3AFileSystem.java:4775)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
... 19 more
getfattr: getObjectMetadata on user/stevel/target/test/data/: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: MJ7KHEZ2R82NT179; S3 Extended Request ID: jWOXiMtgA8hmuVvwg4daFEwWnG0sTUa9g5+5nJ24GTlKTn1QDBSSnoXDylM6HYisdKjAI7c6nfE=; Proxy: null), S3 Extended Request ID: jWOXiMtgA8hmuVvwg4daFEwWnG0sTUa9g5+5nJ24GTlKTn1QDBSSnoXDylM6HYisdKjAI7c6nfE=:404 Not Found
{code}
> S3A Xattr/getXAttr to handle directories without markers
> --------------------------------------------------------
>
> Key: HADOOP-17802
> URL: https://issues.apache.org/jira/browse/HADOOP-17802
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Priority: Minor
>
> The XAttr support in S3 raises FNFE if given a directory as a path when that directory has children but no marker.
> It should somehow downgrade (e.g. FNFE -> probe for a dir, and if found return an empty set of headers)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org