You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by eswar <ea...@gmail.com> on 2014/12/24 01:22:47 UTC
sftp - Exception during concurent execution of sftp endpoint
Hi - I do not get this exception all the time. but happens occasionally.
Two routes are using the same sftp endpoint to send files to same server.
when both the routes are processing concurrently, the first route is
connected and transmitting the file, this first thread is pushed out and the
second route had started to transmit the files through the second route.
First thread came back in to transmit the file, but i got the below DEBUG
log that there is an exception. Though this is not logged as ERROR and this
resulted in continuing the route rather than stopping at this point.
DEBUG RemoteFileProducer: Exception occurred during stopping: Cannot change
directory to: /
is there any way in the camel route to explicitly mention to run the
endpoint to its completion before it gets pushed back for other threads?
any suggestions on this would be really helpful. thank you.
--
View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp - Exception during concurent execution of sftp endpoint
Posted by eswar <ea...@gmail.com>.
Thanks Jens. Yes we have seda queue implemented for certain use cases but for
this issue we would end up processing the files concurrently from two
different routes using the same endpoint. so ideally i would either need to
use sftp component or need to create a custom bean to do sftp.
--
View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049p5761087.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp - Exception during concurent execution of sftp endpoint
Posted by "mailinglist@j-b-s.de" <ma...@j-b-s.de>.
Hi!
What about processing in a serial fashion by sending both messages to a seda queue first and from here to your sftp endpoint? Just to avoid any concurrency issues and I am aware this not exactly an answer for your question but maybe a workaround.
Jens
Von meinem iPhone gesendet
> Am 24.12.2014 um 01:22 schrieb eswar <ea...@gmail.com>:
>
> Hi - I do not get this exception all the time. but happens occasionally.
>
> Two routes are using the same sftp endpoint to send files to same server.
>
> when both the routes are processing concurrently, the first route is
> connected and transmitting the file, this first thread is pushed out and the
> second route had started to transmit the files through the second route.
> First thread came back in to transmit the file, but i got the below DEBUG
> log that there is an exception. Though this is not logged as ERROR and this
> resulted in continuing the route rather than stopping at this point.
>
> DEBUG RemoteFileProducer: Exception occurred during stopping: Cannot change
> directory to: /
>
> is there any way in the camel route to explicitly mention to run the
> endpoint to its completion before it gets pushed back for other threads?
>
> any suggestions on this would be really helpful. thank you.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp - Exception during concurent execution of sftp endpoint
Posted by eswar <ea...@gmail.com>.
Yes I ddnt find more precise logging than the DEBUG statement in case of an
issue. If file gets transmitted properly then I dont get this message
instead I get "Wrote file" from RemoteFileProducer. as you see below
thread#31 is writing the file file1.dat without issues.But with Thread#27 I
see an issue in file transmission and it logs a different message and the
route is not getting stopped though there is an exception as it continues
with informaticaComponent endpoint.
[2014-12-23 02:01:30,837] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #31 - file:///test/filedrop] INFO
RemoteFileProducer: Connected and logged in to:
Endpoint[sftp://account1@19.34.123.34://ftp/inbound/cv1?privateKeyFile=%2FsshF.ssh%2Fid_rsa]
[2014-12-23 02:01:30,845] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #31 - file:///test/filedrop] DEBUG
GenericFileConverter: Read file /test/filedrop/file1.dat (no charset)
[2014-12-23 02:01:30,847] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #27 - file:///real/filedrop] INFO
RemoteFileProducer: Connected and logged in to:
Endpoint[sftp://account1@19.34.123.34://ftp/inbound/cv1?privateKeyFile=%2FsshF.ssh%2Fid_rsa]
[2014-12-23 02:01:30,859] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #27 - file:///real/filedrop] DEBUG
GenericFileConverter: Read file /real/filedrop/file2.dat (no charset)
[2014-12-23 02:01:30,910] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #31 - file:///test/filedrop] DEBUG
RemoteFileProducer: Wrote
[/ftp/inbound/cv1/23-12-14_02-01-29.563_SEP_file1.dat] to
[Endpoint[sftp://account1@19.34.123.34://ftp/inbound/cv1?privateKeyFile=%2FsshF.ssh%2Fid_rsa]]
[2014-12-23 02:01:30,911] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #31 - file:///test/filedrop] DEBUG
RemoteFileProducer: Disconnecting from:
Endpoint[sftp://account1@19.34.123.34://ftp/inbound/cv1?privateKeyFile=%2FsshF.ssh%2Fid_rsa]
[2014-12-23 02:01:30,913] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #31 -
file:///test/filedrop] DEBUG SendProcessor: >>>>
Endpoint[bean://informaticaComponent] Exchange[file1.dat]
[2014-12-23 02:01:30,941] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #27 - file:///real/filedrop] DEBUG
RemoteFileProducer: Exception occurred during stopping: Cannot change
directory to: /
[2014-12-23 02:01:30,941] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #27 - file:///real/filedrop] DEBUG SendProcessor:
>>>> Endpoint[bean://informaticaComponent] Exchange[file2.dat]
[2014-12-23 02:01:30,942] (UserId=?) (ReqId=?) (ReqSrc=?) (ReqDest=?) [Camel
(cavCamelRoutes) thread #27 -
--
View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049p5761105.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp - Exception during concurent execution of sftp endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Dec 25, 2014 at 4:36 AM, eswar <ea...@gmail.com> wrote:
> Thanks Claus. Yes I use same user to login but that doesnt seem to be an
> issue as I have custom JSch class that connects to more than 10 sessions
> concurrently.
> Problem here though is that when an issue happens in file transmission the
> route is not stopped instead it keeps continuing and there is just an debug
> message logged for the issue. Shouldn't this be an exception rather than a
> debug level message?
>
Is it that debug message you refer to about cannot change directory
while stopping?
Is there not more logging that indicate a more precise what the
problem is? And if you can find that can you maybe post more of the
logging? Then we can see if we can find a weakness in the camel-ft p
source code.
Also you can try with stepwise=false which reduces the number of CD
commands in use which can also help.
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049p5761086.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: sftp - Exception during concurent execution of sftp endpoint
Posted by eswar <ea...@gmail.com>.
Thanks Claus. Yes I use same user to login but that doesnt seem to be an
issue as I have custom JSch class that connects to more than 10 sessions
concurrently.
Problem here though is that when an issue happens in file transmission the
route is not stopped instead it keeps continuing and there is just an debug
message logged for the issue. Shouldn't this be an exception rather than a
debug level message?
--
View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049p5761086.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: sftp - Exception during concurent execution of sftp endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Are you using the same user name / password ? Wonder if the server
does not deal well with same user logged in 2 times and concurrently
upload?
If so maybe try to use an unique user.
As the concurrent work is done by java threading you cannot tell it to
complete one thread before another thread etc.
The only way is to not use concurrency, and upload the files in sequence.
On Wed, Dec 24, 2014 at 1:22 AM, eswar <ea...@gmail.com> wrote:
> Hi - I do not get this exception all the time. but happens occasionally.
>
> Two routes are using the same sftp endpoint to send files to same server.
>
> when both the routes are processing concurrently, the first route is
> connected and transmitting the file, this first thread is pushed out and the
> second route had started to transmit the files through the second route.
> First thread came back in to transmit the file, but i got the below DEBUG
> log that there is an exception. Though this is not logged as ERROR and this
> resulted in continuing the route rather than stopping at this point.
>
> DEBUG RemoteFileProducer: Exception occurred during stopping: Cannot change
> directory to: /
>
> is there any way in the camel route to explicitly mention to run the
> endpoint to its completion before it gets pushed back for other threads?
>
> any suggestions on this would be really helpful. thank you.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/sftp-Exception-during-concurent-execution-of-sftp-endpoint-tp5761049.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/