You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/09/01 18:21:20 UTC
[jira] [Resolved] (HADOOP-13572) fs.s3native.mkdirs does not work
if the user is only authorized to a subdirectory
[ https://issues.apache.org/jira/browse/HADOOP-13572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-13572.
-------------------------------------
Resolution: Duplicate
> fs.s3native.mkdirs does not work if the user is only authorized to a subdirectory
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-13572
> URL: https://issues.apache.org/jira/browse/HADOOP-13572
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.2
> Reporter: Marcin Zukowski
> Priority: Minor
>
> Noticed that when working with Spark. I have an S3 bucket with top directories having protected access, and a dedicated open directory deeper in the tree for Spark temporary data.
> Writing to this directory fails with the following stack
> {noformat}
> [info] org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/SPARK-SNOWFLAKEDB' - ResponseCode=403, ResponseMessage=Forbidden
> [info] at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245)
> [info] at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:119)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [info] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [info] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [info] at java.lang.reflect.Method.invoke(Method.java:497)
> [info] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> [info] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> [info] at org.apache.hadoop.fs.s3native.$Proxy34.retrieveMetadata(Unknown Source)
> [info] at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:414)
> [info] at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdir(NativeS3FileSystem.java:539)
> [info] at org.apache.hadoop.fs.s3native.NativeS3FileSystem.mkdirs(NativeS3FileSystem.java:532)
> [info] at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1933)
> [info] at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:291)
> [info] at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:131)
> {noformat}
> I believe this is because mkdirs in NativeS3FileSystem.java tries to create directories starting "from the root", and so if the process can't "list" objects on a given level, it fails. Perhaps it should accept this kind of failures, or go "from the leaf" first to find the level from which it needs to start creating directories. That might also be good for performance assuming the directories exist most of the time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org