You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Bengt Rodehav <be...@rodehav.com> on 2010/11/03 09:31:32 UTC
sftp "broken" in Camel 2.5
I have been using Camel 2.4 for sftp up until now. For several excellent
reasons I now want to upgrade to Camel 2.5 but it seems that the sftp
support has been "broken" - or at least it has changed from the previous
version. I have seen posts about the sftp changes on this list but I haven't
tested the changes until now (which I of course should have).
I've read the following thread
http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-td2806817.html#a2807611
to
try to understand what has been done. However, the thread isn't that easy to
follow and I need to know:
1. What was the problem in the first place?
2. What has been changed in Camel 2.5 compared to 2.4?
3. What is not backwards compatible?
My attempt to understand this lead me to the following answers (please
correct me if I'm wrong):
1. Some server(s) do not allow retrieval of files from other directories
than the working directory.
2. Camel now changes directory before retrieving the file and it does so in
a "stepwise" manner.
3. Presently I know that my sftp test against Serv-U (on Windows) doesn't
work. I'll elaborate a little bit about what seem to happen:
First of all, I haven't yet tested to GET files just to PUT them. Regardless
if I'm using a relative path like
*sftp://user@localhost/sample/sample/file2sftp*
or an absolute path like
*sftp://user@localhost//temp/servu/user/sample/sample/file2sftp*
I always get
*org.apache.camel.component.file.GenericFileOperationFailedException:
Cannot change directory to: /*
If I change a setting in Serv-U to "lock user in home directory" (which is
not the default), then Serv-U will translate "/" to the home directory and
it will work. However, I need to support situations where "/" is not
automatically translated to the home directory since this is very server
dependent. Currently I can't seem to do that. Without having looked at the
source code yet, it seems like Camel is always changing directory to "/"
before doing anything else. Why is that?
Note that since Serv-U is running on Windows, when not "locking the user to
home directory", the absolute path "/" doesn't even exist. When using
Filezilla (client) to access Serv-U I notice that the syntax it uses for
changing directory (if I ask it to) is:
/C:/temp/servu/user/sample/sample/file2sftp
The topmost directory that can be "changed to" is
/C:/temp
Since Camel now seems to always start by changing to "/" it will never
succeed. Don't know if this is specific to Serv-U or of Windows sftp servers
in general.
A couple of comments:
- If some servers need to change the directory before retrieving (or
putting) files, then the behavior must be configurable since there will also
be servers that will not allow you to change the directory in this way. When
choosing between behaviors, go with what is already supported otherwise
existing applications will break. New behavior is best supported by adding
additional configuration options.
- If a change of directory scheme is to be supported, then it must be
configurable whether to change the directory in "one go" or "stepwise"
(first to the top, then to the next level and so on). "The top" isn't always
"/".
- I don't see how to use a relative path (which I have been doing before).
Even if I specify a relative path, Camel will try to change the directory to
"/" anyway. This is a backwards incompatible change. Is there a workaround I
can use if I upgrade to Camel 2.5?
/Bengt
Re: sftp "broken" in Camel 2.5
Posted by Bengt Rodehav <be...@rodehav.com>.
I've gone through the code now and it seems that the problems I've seen
basically has to do with the "stepwise" changing of directories. It
manifests itself in two ways:
1. Serv-U reports absolute paths in a "Windows way". E g
"/C:/temp/servu/user/sample/file2sftp". Since the top folder is not "/" but
"/C:" the logic for dividing a path into its subdirectories fail.
2. Even if the user has acess to the absolute path, he/she doesn't necessary
have access to all the intermediate folders.
The above could be handled if we, via configuration, could tell camel
whether to use "stepwise" changing of directory or do it in one go. That's
what I have in mind right now. Curiously, the "stepwise" approach is not
used everywhere. E g in the buildDirectory() methods in SftpOperations and
FtpOperations, the directory is changed in one go. The problems I
encountered were related to the "changing back to original directory" at the
end of these methods since that is being done stepwise.
What do you think Claus? A configuration property that controls how
directories should be changed (stepwise or not). I would argue that "not
stepwise" should be the default. What do you think?
/Bengt
2010/11/3 Bengt Rodehav <be...@rodehav.com>
> I will have a look at the code and see what I can do.
>
> A follow-up question: When using relative paths, why does Camel still start
> with a CD? It makes sense to have that option when using absolute paths but
> I don't really see the rational when using relative paths. By definition (I
> think), the home directory is where you end up after login. A relative path
> is relative to the home directory (which is where you end up initially)
> which makes the CD command unnecessary.
>
> /Bengt
>
> 2010/11/3 Claus Ibsen <cl...@gmail.com>
>
> On Wed, Nov 3, 2010 at 9:31 AM, Bengt Rodehav <be...@rodehav.com> wrote:
>> > I have been using Camel 2.4 for sftp up until now. For several excellent
>> > reasons I now want to upgrade to Camel 2.5 but it seems that the sftp
>> > support has been "broken" - or at least it has changed from the previous
>> > version. I have seen posts about the sftp changes on this list but I
>> haven't
>> > tested the changes until now (which I of course should have).
>> >
>>
>> Yeah we called out for people to test it months ago. And also the RC
>> for 2.5 is for people to test.
>>
>>
>> > I've read the following thread
>> >
>> http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-td2806817.html#a2807611
>> > to
>> > try to understand what has been done. However, the thread isn't that
>> easy to
>> > follow and I need to know:
>> >
>> > 1. What was the problem in the first place?
>>
>> camel-ftp did not work with all FTP servers (sftp or plain ftp)
>>
>> > 2. What has been changed in Camel 2.5 compared to 2.4?
>>
>> It now uses same "stepwise" manner as we do in Camel 1.x.
>> For example some FTP server can only do authorization using this approach.
>> Others can only traverse folders using this approach.
>>
>> > 3. What is not backwards compatible?
>> >
>>
>> Only if your FTP server now has a problem. As said its hard to
>> communicate with FTP servers across and have it working well.
>>
>>
>> > My attempt to understand this lead me to the following answers (please
>> > correct me if I'm wrong):
>> >
>> > 1. Some server(s) do not allow retrieval of files from other directories
>> > than the working directory.
>> >
>> > 2. Camel now changes directory before retrieving the file and it does so
>> in
>> > a "stepwise" manner.
>> >
>> > 3. Presently I know that my sftp test against Serv-U (on Windows)
>> doesn't
>> > work. I'll elaborate a little bit about what seem to happen:
>> >
>> > First of all, I haven't yet tested to GET files just to PUT them.
>> Regardless
>> > if I'm using a relative path like
>> >
>> > *sftp://user@localhost/sample/sample/file2sftp*
>> >
>> > or an absolute path like
>> >
>> > *sftp://user@localhost//temp/servu/user/sample/sample/file2sftp*
>> >
>> > I always get
>> >
>> > *org.apache.camel.component.file.GenericFileOperationFailedException:
>> > Cannot change directory to: /*
>> >
>>
>> Camel uses the API of JSCH or Commons Net to get the home directory.
>> And they may return "/" as home directory.
>>
>> An idea would be to introduce an option to control if Camel should CD
>> into the home directory when it starts polling
>> or not.
>>
>> It would be great if you could try to change the code in camel-ftp and
>> see if you can get this fixed on your system.
>> Then we can introduce such an option.
>>
>>
>> > If I change a setting in Serv-U to "lock user in home directory" (which
>> is
>> > not the default), then Serv-U will translate "/" to the home directory
>> and
>> > it will work. However, I need to support situations where "/" is not
>> > automatically translated to the home directory since this is very server
>> > dependent. Currently I can't seem to do that. Without having looked at
>> the
>> > source code yet, it seems like Camel is always changing directory to "/"
>> > before doing anything else. Why is that?
>> >
>> > Note that since Serv-U is running on Windows, when not "locking the user
>> to
>> > home directory", the absolute path "/" doesn't even exist. When using
>> > Filezilla (client) to access Serv-U I notice that the syntax it uses for
>> > changing directory (if I ask it to) is:
>> >
>> > /C:/temp/servu/user/sample/sample/file2sftp
>> >
>> > The topmost directory that can be "changed to" is
>> >
>> > /C:/temp
>> >
>> > Since Camel now seems to always start by changing to "/" it will never
>> > succeed. Don't know if this is specific to Serv-U or of Windows sftp
>> servers
>> > in general.
>> >
>> > A couple of comments:
>> >
>> > - If some servers need to change the directory before retrieving (or
>> > putting) files, then the behavior must be configurable since there will
>> also
>> > be servers that will not allow you to change the directory in this way.
>> When
>> > choosing between behaviors, go with what is already supported otherwise
>> > existing applications will break. New behavior is best supported by
>> adding
>> > additional configuration options.
>> >
>> > - If a change of directory scheme is to be supported, then it must be
>> > configurable whether to change the directory in "one go" or "stepwise"
>> > (first to the top, then to the next level and so on). "The top" isn't
>> always
>> > "/".
>> >
>> > - I don't see how to use a relative path (which I have been doing
>> before).
>> > Even if I specify a relative path, Camel will try to change the
>> directory to
>> > "/" anyway. This is a backwards incompatible change. Is there a
>> workaround I
>> > can use if I upgrade to Camel 2.5?
>> >
>> > /Bengt
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
>
Re: sftp "broken" in Camel 2.5
Posted by Bengt Rodehav <be...@rodehav.com>.
I will have a look at the code and see what I can do.
A follow-up question: When using relative paths, why does Camel still start
with a CD? It makes sense to have that option when using absolute paths but
I don't really see the rational when using relative paths. By definition (I
think), the home directory is where you end up after login. A relative path
is relative to the home directory (which is where you end up initially)
which makes the CD command unnecessary.
/Bengt
2010/11/3 Claus Ibsen <cl...@gmail.com>
> On Wed, Nov 3, 2010 at 9:31 AM, Bengt Rodehav <be...@rodehav.com> wrote:
> > I have been using Camel 2.4 for sftp up until now. For several excellent
> > reasons I now want to upgrade to Camel 2.5 but it seems that the sftp
> > support has been "broken" - or at least it has changed from the previous
> > version. I have seen posts about the sftp changes on this list but I
> haven't
> > tested the changes until now (which I of course should have).
> >
>
> Yeah we called out for people to test it months ago. And also the RC
> for 2.5 is for people to test.
>
>
> > I've read the following thread
> >
> http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-td2806817.html#a2807611
> > to
> > try to understand what has been done. However, the thread isn't that easy
> to
> > follow and I need to know:
> >
> > 1. What was the problem in the first place?
>
> camel-ftp did not work with all FTP servers (sftp or plain ftp)
>
> > 2. What has been changed in Camel 2.5 compared to 2.4?
>
> It now uses same "stepwise" manner as we do in Camel 1.x.
> For example some FTP server can only do authorization using this approach.
> Others can only traverse folders using this approach.
>
> > 3. What is not backwards compatible?
> >
>
> Only if your FTP server now has a problem. As said its hard to
> communicate with FTP servers across and have it working well.
>
>
> > My attempt to understand this lead me to the following answers (please
> > correct me if I'm wrong):
> >
> > 1. Some server(s) do not allow retrieval of files from other directories
> > than the working directory.
> >
> > 2. Camel now changes directory before retrieving the file and it does so
> in
> > a "stepwise" manner.
> >
> > 3. Presently I know that my sftp test against Serv-U (on Windows) doesn't
> > work. I'll elaborate a little bit about what seem to happen:
> >
> > First of all, I haven't yet tested to GET files just to PUT them.
> Regardless
> > if I'm using a relative path like
> >
> > *sftp://user@localhost/sample/sample/file2sftp*
> >
> > or an absolute path like
> >
> > *sftp://user@localhost//temp/servu/user/sample/sample/file2sftp*
> >
> > I always get
> >
> > *org.apache.camel.component.file.GenericFileOperationFailedException:
> > Cannot change directory to: /*
> >
>
> Camel uses the API of JSCH or Commons Net to get the home directory.
> And they may return "/" as home directory.
>
> An idea would be to introduce an option to control if Camel should CD
> into the home directory when it starts polling
> or not.
>
> It would be great if you could try to change the code in camel-ftp and
> see if you can get this fixed on your system.
> Then we can introduce such an option.
>
>
> > If I change a setting in Serv-U to "lock user in home directory" (which
> is
> > not the default), then Serv-U will translate "/" to the home directory
> and
> > it will work. However, I need to support situations where "/" is not
> > automatically translated to the home directory since this is very server
> > dependent. Currently I can't seem to do that. Without having looked at
> the
> > source code yet, it seems like Camel is always changing directory to "/"
> > before doing anything else. Why is that?
> >
> > Note that since Serv-U is running on Windows, when not "locking the user
> to
> > home directory", the absolute path "/" doesn't even exist. When using
> > Filezilla (client) to access Serv-U I notice that the syntax it uses for
> > changing directory (if I ask it to) is:
> >
> > /C:/temp/servu/user/sample/sample/file2sftp
> >
> > The topmost directory that can be "changed to" is
> >
> > /C:/temp
> >
> > Since Camel now seems to always start by changing to "/" it will never
> > succeed. Don't know if this is specific to Serv-U or of Windows sftp
> servers
> > in general.
> >
> > A couple of comments:
> >
> > - If some servers need to change the directory before retrieving (or
> > putting) files, then the behavior must be configurable since there will
> also
> > be servers that will not allow you to change the directory in this way.
> When
> > choosing between behaviors, go with what is already supported otherwise
> > existing applications will break. New behavior is best supported by
> adding
> > additional configuration options.
> >
> > - If a change of directory scheme is to be supported, then it must be
> > configurable whether to change the directory in "one go" or "stepwise"
> > (first to the top, then to the next level and so on). "The top" isn't
> always
> > "/".
> >
> > - I don't see how to use a relative path (which I have been doing
> before).
> > Even if I specify a relative path, Camel will try to change the directory
> to
> > "/" anyway. This is a backwards incompatible change. Is there a
> workaround I
> > can use if I upgrade to Camel 2.5?
> >
> > /Bengt
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: sftp "broken" in Camel 2.5
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Nov 3, 2010 at 9:31 AM, Bengt Rodehav <be...@rodehav.com> wrote:
> I have been using Camel 2.4 for sftp up until now. For several excellent
> reasons I now want to upgrade to Camel 2.5 but it seems that the sftp
> support has been "broken" - or at least it has changed from the previous
> version. I have seen posts about the sftp changes on this list but I haven't
> tested the changes until now (which I of course should have).
>
Yeah we called out for people to test it months ago. And also the RC
for 2.5 is for people to test.
> I've read the following thread
> http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-td2806817.html#a2807611
> to
> try to understand what has been done. However, the thread isn't that easy to
> follow and I need to know:
>
> 1. What was the problem in the first place?
camel-ftp did not work with all FTP servers (sftp or plain ftp)
> 2. What has been changed in Camel 2.5 compared to 2.4?
It now uses same "stepwise" manner as we do in Camel 1.x.
For example some FTP server can only do authorization using this approach.
Others can only traverse folders using this approach.
> 3. What is not backwards compatible?
>
Only if your FTP server now has a problem. As said its hard to
communicate with FTP servers across and have it working well.
> My attempt to understand this lead me to the following answers (please
> correct me if I'm wrong):
>
> 1. Some server(s) do not allow retrieval of files from other directories
> than the working directory.
>
> 2. Camel now changes directory before retrieving the file and it does so in
> a "stepwise" manner.
>
> 3. Presently I know that my sftp test against Serv-U (on Windows) doesn't
> work. I'll elaborate a little bit about what seem to happen:
>
> First of all, I haven't yet tested to GET files just to PUT them. Regardless
> if I'm using a relative path like
>
> *sftp://user@localhost/sample/sample/file2sftp*
>
> or an absolute path like
>
> *sftp://user@localhost//temp/servu/user/sample/sample/file2sftp*
>
> I always get
>
> *org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot change directory to: /*
>
Camel uses the API of JSCH or Commons Net to get the home directory.
And they may return "/" as home directory.
An idea would be to introduce an option to control if Camel should CD
into the home directory when it starts polling
or not.
It would be great if you could try to change the code in camel-ftp and
see if you can get this fixed on your system.
Then we can introduce such an option.
> If I change a setting in Serv-U to "lock user in home directory" (which is
> not the default), then Serv-U will translate "/" to the home directory and
> it will work. However, I need to support situations where "/" is not
> automatically translated to the home directory since this is very server
> dependent. Currently I can't seem to do that. Without having looked at the
> source code yet, it seems like Camel is always changing directory to "/"
> before doing anything else. Why is that?
>
> Note that since Serv-U is running on Windows, when not "locking the user to
> home directory", the absolute path "/" doesn't even exist. When using
> Filezilla (client) to access Serv-U I notice that the syntax it uses for
> changing directory (if I ask it to) is:
>
> /C:/temp/servu/user/sample/sample/file2sftp
>
> The topmost directory that can be "changed to" is
>
> /C:/temp
>
> Since Camel now seems to always start by changing to "/" it will never
> succeed. Don't know if this is specific to Serv-U or of Windows sftp servers
> in general.
>
> A couple of comments:
>
> - If some servers need to change the directory before retrieving (or
> putting) files, then the behavior must be configurable since there will also
> be servers that will not allow you to change the directory in this way. When
> choosing between behaviors, go with what is already supported otherwise
> existing applications will break. New behavior is best supported by adding
> additional configuration options.
>
> - If a change of directory scheme is to be supported, then it must be
> configurable whether to change the directory in "one go" or "stepwise"
> (first to the top, then to the next level and so on). "The top" isn't always
> "/".
>
> - I don't see how to use a relative path (which I have been doing before).
> Even if I specify a relative path, Camel will try to change the directory to
> "/" anyway. This is a backwards incompatible change. Is there a workaround I
> can use if I upgrade to Camel 2.5?
>
> /Bengt
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: sftp "broken" in Camel 2.5
Posted by Bengt Rodehav <be...@rodehav.com>.
I have done some more research regarding our production requirements. It's a
bit complicated to test the new Camel version against a production sftp
server. However, I connected the Filezilla client to a production sftp
server to see whether I could change directory the way it seems like sftp in
Camel 2.5 does. Note that I do not know what sftp server is being used.
It turns out that I can change directory into my home directory (of course)
and to some other directories but I cannot change into the intermediate
directories from the root down to my home directory. E g if my home
directory is "/users/myuser" then I can do "cd /users/myuser" but I cannot
do "cd /users".
This makes me unsure whether I can use Camel 2.5 or not. Still a bit
confused...
/Bengt
2010/11/3 Bengt Rodehav <be...@rodehav.com>
> I have been using Camel 2.4 for sftp up until now. For several excellent
> reasons I now want to upgrade to Camel 2.5 but it seems that the sftp
> support has been "broken" - or at least it has changed from the previous
> version. I have seen posts about the sftp changes on this list but I haven't
> tested the changes until now (which I of course should have).
>
> I've read the following thread
> http://camel.465427.n5.nabble.com/How-to-change-directory-while-using-sftp-component-td2806817.html#a2807611 to
> try to understand what has been done. However, the thread isn't that easy to
> follow and I need to know:
>
> 1. What was the problem in the first place?
> 2. What has been changed in Camel 2.5 compared to 2.4?
> 3. What is not backwards compatible?
>
> My attempt to understand this lead me to the following answers (please
> correct me if I'm wrong):
>
> 1. Some server(s) do not allow retrieval of files from other directories
> than the working directory.
>
> 2. Camel now changes directory before retrieving the file and it does so in
> a "stepwise" manner.
>
> 3. Presently I know that my sftp test against Serv-U (on Windows) doesn't
> work. I'll elaborate a little bit about what seem to happen:
>
> First of all, I haven't yet tested to GET files just to PUT them.
> Regardless if I'm using a relative path like
>
> *sftp://user@localhost/sample/sample/file2sftp*
>
> or an absolute path like
>
> *sftp://user@localhost//temp/servu/user/sample/sample/file2sftp*
>
> I always get
>
> *org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot change directory to: /*
>
> If I change a setting in Serv-U to "lock user in home directory" (which is
> not the default), then Serv-U will translate "/" to the home directory and
> it will work. However, I need to support situations where "/" is not
> automatically translated to the home directory since this is very server
> dependent. Currently I can't seem to do that. Without having looked at the
> source code yet, it seems like Camel is always changing directory to "/"
> before doing anything else. Why is that?
>
> Note that since Serv-U is running on Windows, when not "locking the user to
> home directory", the absolute path "/" doesn't even exist. When using
> Filezilla (client) to access Serv-U I notice that the syntax it uses for
> changing directory (if I ask it to) is:
>
> /C:/temp/servu/user/sample/sample/file2sftp
>
> The topmost directory that can be "changed to" is
>
> /C:/temp
>
> Since Camel now seems to always start by changing to "/" it will never
> succeed. Don't know if this is specific to Serv-U or of Windows sftp servers
> in general.
>
> A couple of comments:
>
> - If some servers need to change the directory before retrieving (or
> putting) files, then the behavior must be configurable since there will also
> be servers that will not allow you to change the directory in this way. When
> choosing between behaviors, go with what is already supported otherwise
> existing applications will break. New behavior is best supported by adding
> additional configuration options.
>
> - If a change of directory scheme is to be supported, then it must be
> configurable whether to change the directory in "one go" or "stepwise"
> (first to the top, then to the next level and so on). "The top" isn't always
> "/".
>
> - I don't see how to use a relative path (which I have been doing before).
> Even if I specify a relative path, Camel will try to change the directory to
> "/" anyway. This is a backwards incompatible change. Is there a workaround I
> can use if I upgrade to Camel 2.5?
>
> /Bengt
>
>
>
>
>