You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "vaibhav beriwala (JIRA)" <ji...@apache.org> on 2019/01/30 04:36:00 UTC
[jira] [Comment Edited] (HADOOP-13402) S3A should allow renaming to
a pre-existing destination directory to move the source path under that
directory, similar to HDFS.
[ https://issues.apache.org/jira/browse/HADOOP-13402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755668#comment-16755668 ]
vaibhav beriwala edited comment on HADOOP-13402 at 1/30/19 4:35 AM:
--------------------------------------------------------------------
The following two scenarios in rename use to work for NativeS3FileSystem but does not work for S3AFilesystem
1) initial directory layout is of form : a/b/c/file
rename parameters are - *src*: a/b/c/file and *dst*: a/
expected final directory structure: a/file
2) initial directory layout is of form : a/b/c/file
rename parameters- *src*: a/b/c and *dst*: a/
expected final directory structure: a/c/file
The reason the above used to work in NativeS3FileSystem was because of the following logic in its code where it used to append src.getName() to dst:
{code:java}
// Move to within the existent directory
dstKey = pathToKey(makeAbsolute(new Path(dst, src.getName())));{code}
Could we have a similar logic in S3AFilesystem to make the above 2 scenarios work.
Something like:-
{code:java}
S3AFileStatus dstStatus = null;
try { dstStatus = innerGetFileStatus(dst, true);
if (dstStatus.isDirectory() && !src.getName().equals(dst.getName())){
dstKey = pathToKey(new Path(dst, src.getName()));
dst = new Path(dst, src.getName());
dstStatus = innerGetFileStatus(dst, true);
}
.
.
.
....{code}
was (Author: vaibhavb):
The following two scenarios in rename use to work for NativeS3FileSystem but does not work for S3AFilesystem
1) initial directory layout is of form : a/b/c/file
rename parameters are - *src*: a/b/c/file and *dst*: a/
expected final directory structure: a/file
2) initial directory layout is of form : a/b/c/file
rename parameters- *src*: a/b/c and *dst*: a/
expected final directory structure: a/c/file
Thee reason the above used to work in NativeS3FileSystem was because of the following logic in its code where it used to append src.getName() to dst:
{code:java}
// Move to within the existent directory
dstKey = pathToKey(makeAbsolute(new Path(dst, src.getName())));{code}
Could we have a similar logic in S3AFilesystem to make the above 2 scenarios work.
Something like:-
{code:java}
S3AFileStatus dstStatus = null;
try { dstStatus = innerGetFileStatus(dst, true);
if (dstStatus.isDirectory() && !src.getName().equals(dst.getName())){
dstKey = pathToKey(new Path(dst, src.getName()));
dst = new Path(dst, src.getName());
dstStatus = innerGetFileStatus(dst, true);
}
.
.
.
....{code}
> S3A should allow renaming to a pre-existing destination directory to move the source path under that directory, similar to HDFS.
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13402
> URL: https://issues.apache.org/jira/browse/HADOOP-13402
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Priority: Minor
>
> In HDFS, a rename to a destination path that is a pre-existing directory is interpreted as moving the source path relative to that pre-existing directory. In S3A, this operation currently fails (does nothing and returns {{false}}), unless that destination directory is empty. This issue proposes to change S3A to allow this behavior, so that it more closely matches the semantics of HDFS and other file systems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org