[jira] [Created] (HADOOP-13572) fs.s3native.mkdirs does not work if the user is only authorized to a subdirectory

["Marcin Zukowski (JIRA)" <ji...@apache.org>]

Marcin Zukowski created HADOOP-13572:
----------------------------------------

             Summary: fs.s3native.mkdirs does not work if the user is only authorized to a subdirectory
                 Key: HADOOP-13572
                 URL: https://issues.apache.org/jira/browse/HADOOP-13572
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
            Reporter: Marcin Zukowski


Noticed that when working with Spark. I have an S3 bucket with top directories having protected access, and a dedicated open directory deeper in the tree for Spark temporary data.

Writing to this directory fails with the following stack
{noformat}
[info]   org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/SPARK-SNOWFLAKEDB' - ResponseCode=403, ResponseMessage=Forbidden
[info]   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245)
[info]   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:119)
[info]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[info]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[info]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[info]   at java.lang.reflect.Method.invoke(Method.java:497)
[info]   at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
[info]   at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
[info]   at org.apache.hadoop.fs.s3native.$Proxy34.retrieveMetadata(Unknown Source)
[info]   at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:414)
[info]   at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:539)
[info]   at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:532)
[info]   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1933)
[info]   at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:291)
[info]   at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:131)
{noformat}

I believe this is because mkdirs in NativeS3FileSystem.java tries to create directories starting "from the root", and so if the process can't "list" objects on a given level, it fails. Perhaps it should accept this kind of failures, or go "from the leaf" first to find the level from which it needs to start creating directories. That might also be good for performance assuming the directories exist most of the time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org