You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by GarethHughes <ga...@hp.com> on 2013/03/13 16:49:22 UTC
Performance problem uploading to remote FTP using Stream
Can anyone point me in the right direction to solve this performance problem.
We need to move large files around our system and ultimately upload them to
a remote FTP server. I am using the Claim Check EIP to move the files around
the system so immediately prior to performing the FTP upload to the remote
server I extract the file from an Oracle database to the local file system
using the largeFileService bean as shown below:
So as you can see I pass a stream to the next stage of processing which is
to upload to the remote FTP server thus....
I also have streamCache enabled on my Camel context.
Problem is, the FTP transfer takes an inordinately long time. A manual
transfer of the same 33Mb file to the same FTP server takes 2.88 seconds.
Whereas the Camel route(s) above managed to transfer just 4Mb in 15 minutes
before I kill it off.
Am I missing something blindingly obvious here? It feels like a buffering
issue.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Performance-problem-uploading-to-remote-FTP-using-Stream-tp5729125.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Performance problem uploading to remote FTP using Stream
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Not sure if setting a sentBufferSize on the client helps?
ftpClient.sentBufferSize=8192
Its -1 by default.
Check the API of Commons Net FTP to see if it has any impact.
org.apache.commons.net.ftp.FTPClient
And maybe there is other options you can set on the ftpClient to speedup upload.
Also you can enable TRACE logging on the
org.apache.camel.component.file.remote name to ensure that the file is
using a stream to upload it.
On Wed, Mar 13, 2013 at 4:49 PM, GarethHughes <ga...@hp.com> wrote:
> Can anyone point me in the right direction to solve this performance problem.
> We need to move large files around our system and ultimately upload them to
> a remote FTP server. I am using the Claim Check EIP to move the files around
> the system so immediately prior to performing the FTP upload to the remote
> server I extract the file from an Oracle database to the local file system
> using the largeFileService bean as shown below:
>
>
>
> So as you can see I pass a stream to the next stage of processing which is
> to upload to the remote FTP server thus....
>
>
>
> I also have streamCache enabled on my Camel context.
>
> Problem is, the FTP transfer takes an inordinately long time. A manual
> transfer of the same 33Mb file to the same FTP server takes 2.88 seconds.
> Whereas the Camel route(s) above managed to transfer just 4Mb in 15 minutes
> before I kill it off.
>
> Am I missing something blindingly obvious here? It feels like a buffering
> issue.
>
> Thanks
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Performance-problem-uploading-to-remote-FTP-using-Stream-tp5729125.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Performance problem uploading to remote FTP using Stream
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I added a debug logging that logs how long time it takes to upload the
file in Camel 2.11.
As well what input stream is being used as source.
Though in the trace logging below there is no logs about using a
charset, as you upload binary data.
Though if you force use a special charset then Camel would have to
read the stream and write with that charset,
which could be a reason for being slower. But there is nothing in your logs.
On Wed, Mar 13, 2013 at 6:35 PM, GarethHughes <ga...@hp.com> wrote:
> The only options that I could see on the ftpClient
> http://commons.apache.org/proper/commons-net//apidocs/org/apache/commons/net/ftp/FTPClient.html
> <http://commons.apache.org/proper/commons-net//apidocs/org/apache/commons/net/ftp/FTPClient.html>
> to adjust the buffer size are 'setBufferSize' and
> 'setSendDataSocketBufferSize'. I tried various combinations/values but
> nothing had any noticeable impact. I Assume I am setting these
> appropriately in the URI?:
>
>
> .to("ftp:"+host.getUsername()+"@"+host.getAddress()+"/camel_hosts/"+host.getDirectory()+"/"+host.getHostName()+"/in"
> +
> "?password=" + host.getPassword() +
> "&tempFileName=../temp/${file:onlyname}" +
> "&binary=true" +
> "&ftpClient.dataTimeout=3000" +
> "&ftpClient.bufferSize=8192" +
> "&ftpClient.sendDataSocketBufferSize=8192" +
> "&stepwise=false")
>
> TRACE level logging was inconclusive (as least from my layman's point of
> view):
>
> 13/Mar/2013 17:31:47,795 [onsumer[outbound_files_DEVH11]] RemoteFileProducer
> TRACE Writing using tempNameFile:
> camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr
> 13/Mar/2013 17:31:47,795 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE
> existsFile(camel_hosts/.//DEVH11/in/D0265_001_POC2_B_POC2_B_OPER_L.usr)
> 13/Mar/2013 17:31:47,983 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE
> existsFile(camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr)
> 13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE buildDirectory(camel_hosts\.\DEVH11\in\..\temp)
> 13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE changeCurrentDirectory(/devh11)
> 13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE Changing directory: /devh11
> 13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] RemoteFileProducer
> TRACE About to write
> [camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr] to
> [Endpoint[ftp://devh11@devgw02/camel_hosts/.//DEVH11/in?binary=true&ftpClient.bufferSize=8192&ftpClient.dataTimeout=3000&ftpClient.sendDataSocketBufferSize=8192&password=******&stepwise=false&tempFileName=..%2Ftemp%2F%24%7Bfile%3Aonlyname%7D]]
> from exchange [Exchange[JmsMessage[JmsMessageID:
> ID:ukmdselk024-3936-1363195885202-1:2:1:2:1]]]
> 13/Mar/2013 17:31:48,217 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE
> storeFile(camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr)
> 13/Mar/2013 17:31:48,217 [onsumer[outbound_files_DEVH11]] FtpOperations
> TRACE
> doStoreFile(camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr)
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Performance-problem-uploading-to-remote-FTP-using-Stream-tp5729125p5729135.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Performance problem uploading to remote FTP using Stream
Posted by GarethHughes <ga...@hp.com>.
The only options that I could see on the ftpClient
http://commons.apache.org/proper/commons-net//apidocs/org/apache/commons/net/ftp/FTPClient.html
<http://commons.apache.org/proper/commons-net//apidocs/org/apache/commons/net/ftp/FTPClient.html>
to adjust the buffer size are 'setBufferSize' and
'setSendDataSocketBufferSize'. I tried various combinations/values but
nothing had any noticeable impact. I Assume I am setting these
appropriately in the URI?:
.to("ftp:"+host.getUsername()+"@"+host.getAddress()+"/camel_hosts/"+host.getDirectory()+"/"+host.getHostName()+"/in"
+
"?password=" + host.getPassword() +
"&tempFileName=../temp/${file:onlyname}" +
"&binary=true" +
"&ftpClient.dataTimeout=3000" +
"&ftpClient.bufferSize=8192" +
"&ftpClient.sendDataSocketBufferSize=8192" +
"&stepwise=false")
TRACE level logging was inconclusive (as least from my layman's point of
view):
13/Mar/2013 17:31:47,795 [onsumer[outbound_files_DEVH11]] RemoteFileProducer
TRACE Writing using tempNameFile:
camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr
13/Mar/2013 17:31:47,795 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE
existsFile(camel_hosts/.//DEVH11/in/D0265_001_POC2_B_POC2_B_OPER_L.usr)
13/Mar/2013 17:31:47,983 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE
existsFile(camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr)
13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE buildDirectory(camel_hosts\.\DEVH11\in\..\temp)
13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE changeCurrentDirectory(/devh11)
13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE Changing directory: /devh11
13/Mar/2013 17:31:48,201 [onsumer[outbound_files_DEVH11]] RemoteFileProducer
TRACE About to write
[camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr] to
[Endpoint[ftp://devh11@devgw02/camel_hosts/.//DEVH11/in?binary=true&ftpClient.bufferSize=8192&ftpClient.dataTimeout=3000&ftpClient.sendDataSocketBufferSize=8192&password=******&stepwise=false&tempFileName=..%2Ftemp%2F%24%7Bfile%3Aonlyname%7D]]
from exchange [Exchange[JmsMessage[JmsMessageID:
ID:ukmdselk024-3936-1363195885202-1:2:1:2:1]]]
13/Mar/2013 17:31:48,217 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE
storeFile(camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr)
13/Mar/2013 17:31:48,217 [onsumer[outbound_files_DEVH11]] FtpOperations
TRACE
doStoreFile(camel_hosts/.//DEVH11/in/../temp/D0265_001_POC2_B_POC2_B_OPER_L.usr)
--
View this message in context: http://camel.465427.n5.nabble.com/Performance-problem-uploading-to-remote-FTP-using-Stream-tp5729125p5729135.html
Sent from the Camel - Users mailing list archive at Nabble.com.