[jira] [Commented] (HADOOP-13402) S3A should allow renaming to a pre-existing destination directory to move the source path under that directory, similar to HDFS.

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16760902#comment-16760902 ] 

Steve Loughran commented on HADOOP-13402:
-----------------------------------------

[~vaibhavb] its a running joke that "everybody gets rename wrong". Which is why I'm not going to say you are wrong here, but at the same time, I'm not going to say "just because s3n did it it should be considered good"

S3n was broken because you could go
{code}
hadoop fs rename s3a://bucket/a/b s3a://bucket/a/b/c 
{code}
everything in path /a/b would be renamed under a/b/c *and then everything under /a/b deleted*

So no, we don't want to blindly replicate s3n's behaviour

Regarding the issues you've covered here.you can't currently rename a source directory into a destination directory which is non-empty. At least, that is what FileSystem.rename/3 forbids, and which S3A also blocks
{code}
      if (srcStatus.isDirectory()) {
        if (dstStatus.isFile()) {
          throw new RenameFailedException(src, dst,
              "source is a directory and dest is a file")
              .withExitCode(srcStatus.isFile());
        } else if (dstStatus.isEmptyDirectory() != Tristate.TRUE) {
          throw new RenameFailedException(src, dst,
              "Destination is a non-empty directory")
              .withExitCode(false);
        }
{code}

It's posix's rule, because it gets complicated with issues like "what if a source file of the same name as the test exists?"

Before
{code}
isDir("/a")
isFile("/a/b/c/file)
isFile("/a/file")
{code}

then you go "rename("/a/b/c","/a")

What is the state of ("/a/file") after? Unchanged? Original? 

What if the condition was isDir(/a/b/c/file)? Would the after state be isDir("a/file")?

Its too messy, Posix forbids it for a reason. And if you propose some sort of "lets check first", you are up against the cost of that check *and* The fact it wouldn't be atomic anyway.

Afraid this says "WONTFIX" to me. Sorry

> S3A should allow renaming to a pre-existing destination directory to move the source path under that directory, similar to HDFS.
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13402
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>
> In HDFS, a rename to a destination path that is a pre-existing directory is interpreted as moving the source path relative to that pre-existing directory.  In S3A, this operation currently fails (does nothing and returns {{false}}), unless that destination directory is empty.  This issue proposes to change S3A to allow this behavior, so that it more closely matches the semantics of HDFS and other file systems.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org