You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Claus Ibsen (JIRA)" <ji...@apache.org> on 2017/04/04 07:36:41 UTC
[jira] [Resolved] (CAMEL-11104) Performance issue in camel sftp
[ https://issues.apache.org/jira/browse/CAMEL-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-11104.
---------------------------------
Resolution: Invalid
Use the user forum or user mailing list to ask for help
> Performance issue in camel sftp
> -------------------------------
>
> Key: CAMEL-11104
> URL: https://issues.apache.org/jira/browse/CAMEL-11104
> Project: Camel
> Issue Type: Bug
> Components: camel-sftp
> Affects Versions: 2.15.2
> Environment: Performance Enviroment
> Reporter: Ayush Dixit
>
> We have implemented a camel route where we are having camel sftp producer to transfer a files to remote SFTP location but on performance testing on client environment and on our local environment we have observed degradation in the time for transferring files to remote SFTP location. Please find the detailed analysis below. The we tried the various test in our local environment. In each test we put around 22 files on camel file consumer and each file took below time to write the file. PFB details
> ========================================================================================
> When target directory having 20,000 files. Camel sftp producer took around 1 minute 43 second to a transfer file
> ========================================================================================
> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt] using exchange: Exchange[ST30PERFORPMAJAR17020726.txt]
> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP directory
> INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ST30PERFORPMAJAR17020726.txt]
> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt (no charset)
> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> About to store file: QST30PERFORPMAJAR17020726.txt using stream: java.io.BufferedInputStream@54a89ff5
> DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP client returned: true
> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QST30PERFORPMAJAR17020726.txt] to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
> DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ST30PERFORPMAJAR17020726.txt
> INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS FTP directory
> =======================================================================================
> When target directory having 40,000 files. Camel sftp producer took around 3 minute 17 second to transfer file
> =======================================================================================
> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/v
> mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] using exchange: Exchange[SX30RPMAJAR1702483756.txt]
> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Picked IOP response file :
> SX30RPMAJAR1702483756.txt from IOP FTP directory
> INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> Sending IOP response file
> : SX30RPMAJAR1702483756.txt to ICOMS FTP directory
> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPer
> Poll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]
> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt (no charset)
> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> About to store file: QSX30RPMAJAR1702483756.txt using stream: java.io.BufferedInputStream@7954cfa8
> DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client returned: true
> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QSX30RPMAJAR1702483756.txt] to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
> DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QSX30RPMAJAR1702483756.txt to: icoms/1/SX30RPMAJAR1702483756.txt
> INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1) com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS FTP directory
> ====================================================================================================
> Similarly when we achieved the files from target directory. It took around 6 sec.It seems like there is a performance issue with camel sftp component. Does it list the files in target directory which is taking time.
> ===================================================================================================
> PFB the producer route which we set up
> sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)