You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by johper <ka...@hotmail.com> on 2007/03/28 09:24:07 UTC
sftp hints
Hi,
Using LW-SA and FTPSender..works fine with a few patches.
Now we need to use the SFTP-protocol and I dont know how to migrate LW-SA
FTPSender?
Have someone made an SFTP solution with ServiceMix or can point in a
direction ..?
VFS looks great but I not sure how to incorporate it in SMX?
http://jakarta.apache.org/commons/vfs/apidocs/index.html
tia
--
View this message in context: http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9708395
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: sftp hints
Posted by Bruce Snyder <br...@gmail.com>.
On 3/28/07, johper <ka...@hotmail.com> wrote:
>
> "native" SFTP with Apache VFS goes like:
> FileSystemManager fsManager = VFS.getManager();
> FileObject ftpDir = fsManager.resolveFile(
> "sftp://myuser:mypass@myhost/inbox-dir-on-server/");
>
> would it not be reasonable to believe SMX VFS can handle this out of the box
> with :
>
> <sm:activationSpec componentName="sftpSender"
> service="postman:sftpSender">
> <sm:component>
> <bean class="org.apache.servicemix.components.vfs.FileWriter">
> <property name="path"
> value="sftp://myuser:mypass@myhost/inbox-on-server/"/>
> <property name="marshaler">
> <bean
> class="org.apache.servicemix.components.util.BinaryFileMarshaler">
> <property name="fileName">
> <bean class="org.apache.servicemix.expression.JAXPXPathExpression">
> <constructor-arg
> value="concat(substring($org.apache.servicemix.file.name, 0,
> string-length($org.apache.servicemix.file.name) - 3), '.emm')" />
> </bean>
> </property>
> </bean>
> </property>
> </bean>
> </sm:component>
> </sm:activationSpec>
Guillaume's point is that the lightweight component for Commons VFS
that comes with ServiceMix is not JBI compliant and that it would be
much nicer to have JBI compliant component for Commons VFS.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache Geronimo - http://geronimo.apache.org/
Apache ActiveMQ - http://activemq.org/
Apache ServiceMix - http://servicemix.org/
Castor - http://castor.org/
Re: sftp hints
Posted by johper <ka...@hotmail.com>.
"native" SFTP with Apache VFS goes like:
FileSystemManager fsManager = VFS.getManager();
FileObject ftpDir = fsManager.resolveFile(
"sftp://myuser:mypass@myhost/inbox-dir-on-server/");
would it not be reasonable to believe SMX VFS can handle this out of the box
with :
<sm:activationSpec componentName="sftpSender"
service="postman:sftpSender">
<sm:component>
<bean class="org.apache.servicemix.components.vfs.FileWriter">
<property name="path"
value="sftp://myuser:mypass@myhost/inbox-on-server/"/>
<property name="marshaler">
<bean
class="org.apache.servicemix.components.util.BinaryFileMarshaler">
<property name="fileName">
<bean class="org.apache.servicemix.expression.JAXPXPathExpression">
<constructor-arg
value="concat(substring($org.apache.servicemix.file.name, 0,
string-length($org.apache.servicemix.file.name) - 3), '.emm')" />
</bean>
</property>
</bean>
</property>
</bean>
</sm:component>
</sm:activationSpec>
--
View this message in context: http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9710816
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: sftp hints
Posted by johper <ka...@hotmail.com>.
You are right. Due to unstable Apache VFS and some extra features needed we
built our own SFTP-JBI-BC (on top of JSCH) and FTP-JBI-BC (with date
archiving in both poll and send).
--
View this message in context: http://www.nabble.com/sftp-hints-tf3478697s12049.html#a10156476
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: sftp hints
Posted by Steve Kondik <sh...@chemlab.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Guillaume Nodet wrote:
> On 4/12/07, johper <ka...@hotmail.com> wrote:
>>
>>
>> Build and tried the mergence trunk. FTP poller and FTP sender works fine
>> but
>> when doing an archive directory on the polling ftp-server it does not
>> work.
>> Problem occur when using same userid/password for both ftp polling och
>> sending on same ftp-server. When moving archiving to another
>> ftp-server it
>> works fine...
>> Will try to get around this with different ftp-users on same
>> ftp-server...or
>> another workaround...
>>
>> Thanks for sharing your trunk, don't you think VFS will obsolete
>> JBI-FILE-comp and JBI-FTP-comp? Or at least a JBI-VFS-comp would be able
>> to
>> do both ftp and file?
>
>
> Well, maybe, but not sure. commons-vfs uses an abstract file system
> Api, so
> i guess some features won't be able to be implemented, such as file
> renaming
> after download, os things like that ....
>
I'm just using FileObject.moveTo which seems to work fine for SFTP and
local filesystems. I'll see if I can figure out a workaround.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGHm17MrBfzfMVwMcRAtmlAJ4oWkO9bdr0nDBCDo/0Q7Omn/WHIACdGydp
vrtqllr4dynW9qx+BbU0Xu4=
=Flpz
-----END PGP SIGNATURE-----
Re: sftp hints
Posted by Guillaume Nodet <gn...@gmail.com>.
On 4/12/07, johper <ka...@hotmail.com> wrote:
>
>
> Build and tried the mergence trunk. FTP poller and FTP sender works fine
> but
> when doing an archive directory on the polling ftp-server it does not
> work.
> Problem occur when using same userid/password for both ftp polling och
> sending on same ftp-server. When moving archiving to another ftp-server it
> works fine...
> Will try to get around this with different ftp-users on same
> ftp-server...or
> another workaround...
>
> Thanks for sharing your trunk, don't you think VFS will obsolete
> JBI-FILE-comp and JBI-FTP-comp? Or at least a JBI-VFS-comp would be able
> to
> do both ftp and file?
Well, maybe, but not sure. commons-vfs uses an abstract file system Api, so
i guess some features won't be able to be implemented, such as file renaming
after download, os things like that ....
=============================================
> DEBUG - DeliveryChannelImpl - Notified: ID:mw-51550-3:1(4c2849)
> in DeliveryChannel{mergence-vfs} from sendSync
> 13:23:42.163 [pool-component.mergence-vfs-thread-3] ERROR
> c.h.m.c.vfs.VFSPollerEndpoint - Post processing failed: Could not rename
> "ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil" to
> "ftp://u:p@10.10.10.10/businesspartner/ARCHIVE/2007-04-12/cs000016.fil".
> org.apache.commons.vfs.FileSystemException: Could not rename
> "ftp://u:p@10.47.225.11/businesspartner/out/cs000016.fil" to
> "ftp://u:p@10.10.10.10/businesspartner/ARCHIVE/2007-04-12/cs000016.fil".
> at
> org.apache.commons.vfs.provider.AbstractFileObject.moveTo(
> AbstractFileObject.java:970)
> at
> com.hmsinc.mergence.components.vfs.VFSPollerEndpoint.postProcessFile(
> VFSPollerEndpoint.java:227)
> at
> com.hmsinc.mergence.components.vfs.VFSPollerEndpoint$1.run(
> VFSPollerEndpoint.java:177)
> at
>
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:665)
> at
>
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:690)
> at java.lang.Thread.run(Thread.java:595)
> Caused by: org.apache.commons.vfs.FileSystemException: Could not rename
> FTP
> file "ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil" to
> "ftp://u:p@10.10.10.10/businesspartner/ARCHIVE/2007-04-12/cs000016.fil".
> at
> org.apache.commons.vfs.provider.ftp.FtpFileObject.doRename(
> FtpFileObject.java:432)
> at
> org.apache.commons.vfs.provider.AbstractFileObject.moveTo(
> AbstractFileObject.java:956)
> ... 5 common frames omitted
> 13:23:44.918 [pool-component.mergence-vfs-thread-4] DEBUG
> c.h.m.c.vfs.VFSPollerEndpoint - Found file:
> ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil
> 13:23:44.918 [pool-component.mergence-vfs-thread-4] INFO
> c.h.m.c.vfs.VFSPollerEndpoint - Scheduling file
> ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil for processing
> DEBUG - DeliveryChannelImpl - SendSync ID:1176376964894-3:2 in
> DeliveryChannel{mergence-vfs}
> DEBUG - SedaFlow - Called Flow send
> DEBUG - DeliveryChannelImpl - Waiting for exchange
> ID:964894-3:2
> (177039) to be answered in DeliveryChannel{mergence-vfs} from sendSync
> 13:23:44.929 [pool-flow.seda.mergence-vfs-thread-1] DEBUG
> o.a.s.jbi.nmr.flow.seda.SedaQueue -
> org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@c8c6d9 dequeued
> exchange: InOnly[
> id: ID:64894-3:2
> status: Active
> role: provider
> service: {http://company.com/filemover/businesspartner/1.0}send
> endpoint: endpoint
> in: null
> ]
> 13:23:44.930 [pool-flow.seda.mergence-vfs-thread-1] DEBUG
> c.h.m.components.vfs.VFSComponent - Received exchange: status: Active,
> role:
> provider
> 13:23:44.931 [pool-flow.seda.mergence-vfs-thread-1] DEBUG
> c.h.m.components.vfs.VFSComponent - Retrieved correlation id: null
> --
> View this message in context:
> http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9961549
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
LogicBlaze
Principal Engineer, IONA
Blog: http://gnodet.blogspot.com/
Re: sftp hints
Posted by johper <ka...@hotmail.com>.
Build and tried the mergence trunk. FTP poller and FTP sender works fine but
when doing an archive directory on the polling ftp-server it does not work.
Problem occur when using same userid/password for both ftp polling och
sending on same ftp-server. When moving archiving to another ftp-server it
works fine...
Will try to get around this with different ftp-users on same ftp-server...or
another workaround...
Thanks for sharing your trunk, don't you think VFS will obsolete
JBI-FILE-comp and JBI-FTP-comp? Or at least a JBI-VFS-comp would be able to
do both ftp and file?
=============================================
DEBUG - DeliveryChannelImpl - Notified: ID:mw-51550-3:1(4c2849)
in DeliveryChannel{mergence-vfs} from sendSync
13:23:42.163 [pool-component.mergence-vfs-thread-3] ERROR
c.h.m.c.vfs.VFSPollerEndpoint - Post processing failed: Could not rename
"ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil" to
"ftp://u:p@10.10.10.10/businesspartner/ARCHIVE/2007-04-12/cs000016.fil".
org.apache.commons.vfs.FileSystemException: Could not rename
"ftp://u:p@10.47.225.11/businesspartner/out/cs000016.fil" to
"ftp://u:p@10.10.10.10/businesspartner/ARCHIVE/2007-04-12/cs000016.fil".
at
org.apache.commons.vfs.provider.AbstractFileObject.moveTo(AbstractFileObject.java:970)
at
com.hmsinc.mergence.components.vfs.VFSPollerEndpoint.postProcessFile(VFSPollerEndpoint.java:227)
at
com.hmsinc.mergence.components.vfs.VFSPollerEndpoint$1.run(VFSPollerEndpoint.java:177)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.apache.commons.vfs.FileSystemException: Could not rename FTP
file "ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil" to
"ftp://u:p@10.10.10.10/businesspartner/ARCHIVE/2007-04-12/cs000016.fil".
at
org.apache.commons.vfs.provider.ftp.FtpFileObject.doRename(FtpFileObject.java:432)
at
org.apache.commons.vfs.provider.AbstractFileObject.moveTo(AbstractFileObject.java:956)
... 5 common frames omitted
13:23:44.918 [pool-component.mergence-vfs-thread-4] DEBUG
c.h.m.c.vfs.VFSPollerEndpoint - Found file:
ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil
13:23:44.918 [pool-component.mergence-vfs-thread-4] INFO
c.h.m.c.vfs.VFSPollerEndpoint - Scheduling file
ftp://u:p@10.10.10.10/businesspartner/out/cs000016.fil for processing
DEBUG - DeliveryChannelImpl - SendSync ID:1176376964894-3:2 in
DeliveryChannel{mergence-vfs}
DEBUG - SedaFlow - Called Flow send
DEBUG - DeliveryChannelImpl - Waiting for exchange ID:964894-3:2
(177039) to be answered in DeliveryChannel{mergence-vfs} from sendSync
13:23:44.929 [pool-flow.seda.mergence-vfs-thread-1] DEBUG
o.a.s.jbi.nmr.flow.seda.SedaQueue -
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1@c8c6d9 dequeued
exchange: InOnly[
id: ID:64894-3:2
status: Active
role: provider
service: {http://company.com/filemover/businesspartner/1.0}send
endpoint: endpoint
in: null
]
13:23:44.930 [pool-flow.seda.mergence-vfs-thread-1] DEBUG
c.h.m.components.vfs.VFSComponent - Received exchange: status: Active, role:
provider
13:23:44.931 [pool-flow.seda.mergence-vfs-thread-1] DEBUG
c.h.m.components.vfs.VFSComponent - Retrieved correlation id: null
--
View this message in context: http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9961549
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: sftp hints
Posted by Steve Kondik <sh...@chemlab.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Bruce Snyder wrote:
> On 3/30/07, Steve Kondik <sh...@chemlab.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I have a commons-vfs component that supports a lot of things that people
>> are asking for (compress after, rename, etc). It's not well documented
>> yet, but it's very similar to the ftp/file components. The only problem
>> I have with it, is that there seems to be no way to specify a relative
>> path on the local filesystem with commons-vfs. I haven't tested it in
>> any kind of clustered situation yet (although it uses locking the same
>> way the SM components do).
>>
>> It depends on some non-ServiceMix stuff, to deal with "chunking" of very
>> large files. I can clean it up and contribute it to SM if you'd like.
>>
>> The source is here:
>>
>> https://internal.hmsinc.com/svn/mergence/trunk/mergence-vfs
>
> It would be great to have your contribution, including the
> ChunkingFileMarshaler if you can. I think it would be best to add your
> features to the servicemix-ftp component so that it's more complete.
> Have you considered taking this route?
>
> BTW, information on contributing patches is available here:
>
> http://incubator.apache.org/servicemix/contributing.html#Contributing-SubmittingPatches
>
>
> Bruce
The code is modeled after the FTP component (uses the executors,
LockManager, etc). It's being used in a semi-production way right now,
so it's fairly well tested.
I'll try to get it submitted sometime next week.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGDxohMrBfzfMVwMcRAvRYAJwJ13MxETHpQC5h43l5Kej655pQGgCfYmrf
hjyhAFs6dv8LL1DVAUE5Kf8=
=s15e
-----END PGP SIGNATURE-----
Re: sftp hints
Posted by Bruce Snyder <br...@gmail.com>.
On 3/30/07, Steve Kondik <sh...@chemlab.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have a commons-vfs component that supports a lot of things that people
> are asking for (compress after, rename, etc). It's not well documented
> yet, but it's very similar to the ftp/file components. The only problem
> I have with it, is that there seems to be no way to specify a relative
> path on the local filesystem with commons-vfs. I haven't tested it in
> any kind of clustered situation yet (although it uses locking the same
> way the SM components do).
>
> It depends on some non-ServiceMix stuff, to deal with "chunking" of very
> large files. I can clean it up and contribute it to SM if you'd like.
>
> The source is here:
>
> https://internal.hmsinc.com/svn/mergence/trunk/mergence-vfs
It would be great to have your contribution, including the
ChunkingFileMarshaler if you can. I think it would be best to add your
features to the servicemix-ftp component so that it's more complete.
Have you considered taking this route?
BTW, information on contributing patches is available here:
http://incubator.apache.org/servicemix/contributing.html#Contributing-SubmittingPatches
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache Geronimo - http://geronimo.apache.org/
Apache ActiveMQ - http://activemq.org/
Apache ServiceMix - http://servicemix.org/
Castor - http://castor.org/
Re: sftp hints
Posted by Guillaume Nodet <gn...@gmail.com>.
This is caused by a missing reference to servicemix-shared library.
You can add:
<dependency>
<groupId>org.apache.servicemix</groupId>
<artifactId>servicemix-shared</artifactId>
<version>${servicemix-version}</version>
</dependency>
to the pom.xml and it should work.
On 4/3/07, johper <ka...@hotmail.com> wrote:
>
>
> Building mergence-vfs-1.0-SNAPSHOT-installer.zip OK, but when installing
> it :
>
> Caused by: javax.jbi.management.DeploymentException:
> java.lang.NoClassDefFoundError: org/a
> pache/servicemix/common/DefaultComponent
>
> The DefaultBootstrap class is in classpath...
>
> --
> View this message in context:
> http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9805448
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Architect, LogicBlaze (http://www.logicblaze.com/)
Blog: http://gnodet.blogspot.com/
Re: sftp hints
Posted by johper <ka...@hotmail.com>.
Building mergence-vfs-1.0-SNAPSHOT-installer.zip OK, but when installing it :
Caused by: javax.jbi.management.DeploymentException:
java.lang.NoClassDefFoundError: org/a
pache/servicemix/common/DefaultComponent
The DefaultBootstrap class is in classpath...
--
View this message in context: http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9805448
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: sftp hints
Posted by Steve Kondik <sh...@chemlab.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I have a commons-vfs component that supports a lot of things that people
are asking for (compress after, rename, etc). It's not well documented
yet, but it's very similar to the ftp/file components. The only problem
I have with it, is that there seems to be no way to specify a relative
path on the local filesystem with commons-vfs. I haven't tested it in
any kind of clustered situation yet (although it uses locking the same
way the SM components do).
It depends on some non-ServiceMix stuff, to deal with "chunking" of very
large files. I can clean it up and contribute it to SM if you'd like.
The source is here:
https://internal.hmsinc.com/svn/mergence/trunk/mergence-vfs
Guillaume Nodet wrote:
> There is a lightweight component using vfs that you can try,
> but a JBI compliant one would be much better imo.
> Anyone want to try writing one ? It should be easy
> starting from servicemix-ftp or servicemix-file ...
>
> On 3/28/07, johper <ka...@hotmail.com> wrote:
>>
>>
>> Hi,
>> Using LW-SA and FTPSender..works fine with a few patches.
>>
>> Now we need to use the SFTP-protocol and I dont know how to migrate LW-SA
>> FTPSender?
>>
>> Have someone made an SFTP solution with ServiceMix or can point in a
>> direction ..?
>>
>> VFS looks great but I not sure how to incorporate it in SMX?
>> http://jakarta.apache.org/commons/vfs/apidocs/index.html
>>
>> tia
>>
>> --
>> View this message in context:
>> http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9708395
>> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>>
>>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGDQYkMrBfzfMVwMcRAtF+AJ9fpuT/qSoNi0YOeraRaVis1O+rYACeKxKh
Y+O+DMHDIiGaDTE6MErO/Wk=
=QabS
-----END PGP SIGNATURE-----
Re: sftp hints
Posted by Guillaume Nodet <gn...@gmail.com>.
There is a lightweight component using vfs that you can try,
but a JBI compliant one would be much better imo.
Anyone want to try writing one ? It should be easy
starting from servicemix-ftp or servicemix-file ...
On 3/28/07, johper <ka...@hotmail.com> wrote:
>
>
> Hi,
> Using LW-SA and FTPSender..works fine with a few patches.
>
> Now we need to use the SFTP-protocol and I dont know how to migrate LW-SA
> FTPSender?
>
> Have someone made an SFTP solution with ServiceMix or can point in a
> direction ..?
>
> VFS looks great but I not sure how to incorporate it in SMX?
> http://jakarta.apache.org/commons/vfs/apidocs/index.html
>
> tia
>
> --
> View this message in context:
> http://www.nabble.com/sftp-hints-tf3478697s12049.html#a9708395
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Architect, LogicBlaze (http://www.logicblaze.com/)
Blog: http://gnodet.blogspot.com/