You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Karel Kolman (JIRA)" <ji...@apache.org> on 2012/08/06 13:09:02 UTC
[jira] [Created] (HADOOP-8653) FTPFileSystem rename broken
Karel Kolman created HADOOP-8653:
------------------------------------
Summary: FTPFileSystem rename broken
Key: HADOOP-8653
URL: https://issues.apache.org/jira/browse/HADOOP-8653
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0-alpha, 0.20.2
Reporter: Karel Kolman
The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
{noformat}
INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
at java.lang.Thread.run(Thread.java:680)
{noformat}
The solution would be this:
{noformat}
--- a/FTPFileSystem.java
+++ b/FTPFileSystem.java
@@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
throw new IOException("Destination path " + dst
+ " already exist, cannot rename!");
}
- String parentSrc = absoluteSrc.getParent().toUri().toString();
- String parentDst = absoluteDst.getParent().toUri().toString();
+ URI parentSrc = absoluteSrc.getParent().toUri();
+ URI parentDst = absoluteDst.getParent().toUri();
String from = src.getName();
String to = dst.getName();
- if (!parentSrc.equals(parentDst)) {
+ if (!parentSrc.toString().equals(parentDst.toString())) {
throw new IOException("Cannot rename parent(source): " + parentSrc
+ ", parent(destination): " + parentDst);
}
- client.changeWorkingDirectory(parentSrc);
+ client.changeWorkingDirectory(parentSrc.getPath().toString());
boolean renamed = client.rename(from, to);
return renamed;
}
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Karel Kolman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karel Kolman updated HADOOP-8653:
---------------------------------
Fix Version/s: 2.0.1-alpha
Target Version/s: 2.0.1-alpha
Status: Patch Available (was: Open)
patch for FTPFileSystem rename method fix
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.0.0-alpha, 0.20.2
> Reporter: Karel Kolman
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-8653-1.patch
>
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429192#comment-13429192 ]
Daryn Sharp commented on HADOOP-8653:
-------------------------------------
You mention that file:// is being added to the path, but in the stack trace it appears the problem is {{ftp://localhost:61246/tmp/myfile}} is having a leading slash prepended because that's probably the cwd of the server. Thus the real fix in your patch is calling {{path.getPath}} to remove the scheme+authority.
I may suggest implementing {{setWorkingDirectory(Path)}} to pass only {{path#getPath()}} through to the server, although I question why many of the methods are implicitly changing the cwd?
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429258#comment-13429258 ]
Daryn Sharp commented on HADOOP-8653:
-------------------------------------
My apologies if I was confusing. We're saying the same thing - the fully qualified path, not the fully qualified uri, should be passed. I suggested a common method since many places seem to do the same logic to invoke with just the path.
Note this may just be a test case bug: the next line after the CWD you cite is {noformat}org.mockftpserver.fake.filesystem.FileSystemException: /ftp://...{noformat} which means it may just be the mock prepending the /?
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430565#comment-13430565 ]
Daryn Sharp commented on HADOOP-8653:
-------------------------------------
Ok, that's what I thought. Does the test fail if run against a live {{FTPFileSystem}}? It looks like it's maybe a mock problem, although the patch is an improvement. BTW, you need to attach patches as a file to the jira and submit them for the pre-commit build to test it.
Related, but perhaps for another jira: you may consider having {{makeAbsolute(Path)}} return {{new Path(null, null, makeQualified(path).getPath())}}. That will return just the absolute path of the URI, which will allow the code in rename and many other methods to be simplified.
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-8653:
----------------------------------
Fix Version/s: (was: 2.0.1-alpha)
3.0.0
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
> Fix For: 3.0.0
>
> Attachments: HDFS-8653-1.patch
>
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430999#comment-13430999 ]
Hadoop QA commented on HADOOP-8653:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539796/HDFS-8653-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1264//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1264//console
This message is automatically generated.
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-8653-1.patch
>
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Karel Kolman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429224#comment-13429224 ]
Karel Kolman commented on HADOOP-8653:
--------------------------------------
@Daryn i don't think you are right here, the first log line from ftp server reads:
{noformat}
Received command: [CWD ftp://localhost:61246/tmp/myfile]
{noformat}
while it should receive only [CWD /tmp/myfile] from hadoop's FTPFileSystem
Also all other calls to changeWorkingDirectory in FTPFileSystem use a variant of parent.toUri().getPath() (parent.toUri().getPath().toString())
{noformat}
FTPFileSystem.java-6393- // FSDataInputStream.
FTPFileSystem.java:6419: client.changeWorkingDirectory(parent.toUri().getPath());
--
FTPFileSystem.java-8320- // FSDataOutputStream.
FTPFileSystem.java:8347: client.changeWorkingDirectory(parent.toUri().getPath());
--
FTPFileSystem.java-17201- String parentDir = parent.toUri().getPath();
FTPFileSystem.java:17254: client.changeWorkingDirectory(parentDir);
--
FTPFileSystem.java-19688- }
FTPFileSystem.java:19694: client.changeWorkingDirectory(parentSrc);
{noformat}
so i think it should be patched like this
I'm sorry im not providing a unit/integration testcase, is there one ? Basically this was caught by our integration tests trying to use FTPFileSystem for some file renaming.
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Karel Kolman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430976#comment-13430976 ]
Karel Kolman commented on HADOOP-8653:
--------------------------------------
well testing on live ftp, the "CWD file://path" is still being sent to it (and of course failing on the ftp server side), no mock problem, ftp server commands have no clue about schemes...
attaching the patch against trunk
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-8653-1.patch
>
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Karel Kolman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karel Kolman updated HADOOP-8653:
---------------------------------
Attachment: HDFS-8653-1.patch
patch making the argument to changeWorkingDirectory() in rename() correct - without ftp:// scheme prefix
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
> Attachments: HDFS-8653-1.patch
>
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8653) FTPFileSystem rename broken
Posted by "Karel Kolman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430346#comment-13430346 ]
Karel Kolman commented on HADOOP-8653:
--------------------------------------
the "/" is the ftp user's home directory, which is set to / in the mock server setup. Changing the home to "/home/test/", the next error line reads
{noformat}
org.mockftpserver.fake.filesystem.FileSystemException: /home/test/ftp://localhost:61246/tmp/myfile
{noformat}
/home/test/ftp://localhost:61246/tmp/myfile
I'm not so sure it makes sense to create an additional method for this, changeWorkingDirectory(String) method is a public one and
I don't really have a clue about hadoop's file systems, so all that Path to URI to Path conversion happening in this close is a mystery to me.
The patch had a unneeded toString() at parentSrc.getPath().toString()
{noformat}
- client.changeWorkingDirectory(parentSrc);
+ client.changeWorkingDirectory(parentSrc.getPath());
{noformat}
> FTPFileSystem rename broken
> ---------------------------
>
> Key: HADOOP-8653
> URL: https://issues.apache.org/jira/browse/HADOOP-8653
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.20.2, 2.0.0-alpha
> Reporter: Karel Kolman
>
> The FTPFileSystem.rename(FTPClient client, Path src, Path dst) method is broken.
> The changeWorkingDirectory command underneath is being passed a string with file:// uri prefix (which FTP server does not understand obviously)
> {noformat}
> INFO [2012-08-06 12:59:39] (DefaultSession.java:297) - Received command: [CWD ftp://localhost:61246/tmp/myfile]
> WARN [2012-08-06 12:59:39] (AbstractFakeCommandHandler.java:213) - Error handling command: Command[CWD:[ftp://localhost:61246/tmp/myfile]]; org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> org.mockftpserver.fake.filesystem.FileSystemException: /ftp://localhost:61246/tmp/myfile
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.verifyFileSystemCondition(AbstractFakeCommandHandler.java:264)
> at org.mockftpserver.fake.command.CwdCommandHandler.handle(CwdCommandHandler.java:44)
> at org.mockftpserver.fake.command.AbstractFakeCommandHandler.handleCommand(AbstractFakeCommandHandler.java:76)
> at org.mockftpserver.core.session.DefaultSession.readAndProcessCommand(DefaultSession.java:421)
> at org.mockftpserver.core.session.DefaultSession.run(DefaultSession.java:384)
> at java.lang.Thread.run(Thread.java:680)
> {noformat}
> The solution would be this:
> {noformat}
> --- a/FTPFileSystem.java
> +++ b/FTPFileSystem.java
> @@ -549,15 +549,15 @@ public class FTPFileSystem extends FileSystem {
> throw new IOException("Destination path " + dst
> + " already exist, cannot rename!");
> }
> - String parentSrc = absoluteSrc.getParent().toUri().toString();
> - String parentDst = absoluteDst.getParent().toUri().toString();
> + URI parentSrc = absoluteSrc.getParent().toUri();
> + URI parentDst = absoluteDst.getParent().toUri();
> String from = src.getName();
> String to = dst.getName();
> - if (!parentSrc.equals(parentDst)) {
> + if (!parentSrc.toString().equals(parentDst.toString())) {
> throw new IOException("Cannot rename parent(source): " + parentSrc
> + ", parent(destination): " + parentDst);
> }
> - client.changeWorkingDirectory(parentSrc);
> + client.changeWorkingDirectory(parentSrc.getPath().toString());
> boolean renamed = client.rename(from, to);
> return renamed;
> }
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira