You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by ldam <la...@gmail.com> on 2012/06/22 11:06:51 UTC
consumerTemplate threading worries when having a Batch Consumer
Hi
I had some funny problems when getting exchanges from a consumer template,
the attached code snippet illustrates a minimal example that demonstrates
the problem. My findings are:
When getting a batch of files (in my case via an sftp endpoint) doing a
consumerTemplate.doneUoW(e) afterwards results in the underlying JSCH
library getting corrupted streams and breaks.
My assumption is that the underlying batch retrieval of files is done in
it's own thread, and when the first exchange is processed and i execute
doneUoW on it it accesses the JSCH file streams while the batch retrieval is
going on on the other thread and thus corrupts it.
Accumulating the exchanges in a collection and waiting to do the doneUoW()
on them until after the last batch entry avoids this problem.
So my questions are:
1) Are my assumption correct and is this what actual happens
2) If so, then there might be a problem, in those cases where you might have
a bazingallion files, and the doneUof() execution is not finished before the
batch retrieval starts again
The attached code snippet:
http://camel.465427.n5.nabble.com/file/n5714904/FailDemo.java FailDemo.java
Some of the errors i get from the JSCH library (clearly indicates corrupted
file streams)
2274 [Connect thread localhost session] INFO
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Caught an
exception, leaving main loop due to SSH_MSG_DISCONNECT: 2 Packet corrupt
2506 [Connect thread localhost session] INFO
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Caught an
exception, leaving main loop due to SSH_MSG_DISCONNECT: 2 Invalid ssh2
packet type: 0
2305 [Connect thread localhost session] INFO
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Caught an
exception, leaving main loop due to SSH_MSG_DISCONNECT: 2 Invalid ssh2
packet type: 209
2286 [Connect thread localhost session] INFO
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Received
SSH_MSG_UNIMPLEMENTED for 69
4726 [Connect thread localhost session] INFO
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Received
SSH_MSG_UNIMPLEMENTED for 47
I included these for search engines, in case other gets similar problems.
kr. Lars Dam
--
View this message in context: http://camel.465427.n5.nabble.com/consumerTemplate-threading-worries-when-having-a-Batch-Consumer-tp5714904.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: consumerTemplate threading worries when having a Batch Consumer
Posted by ldam <la...@gmail.com>.
Hi
I found that out, which led me to the reasons for the problem above.
Will go for that, though it would be nice for consumerTemplate to support
batch in the future (in case one forgets that option in a new
configuration).
Thanks.
kr. ldam
--
View this message in context: http://camel.465427.n5.nabble.com/consumerTemplate-threading-worries-when-having-a-Batch-Consumer-tp5714904p5714915.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: consumerTemplate threading worries when having a Batch Consumer
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Jun 22, 2012 at 11:53 AM, ldam <la...@gmail.com> wrote:
> Unfortunately we are not the masters of the filename given :/
>
> The reason we use consumerTemplate is because we use it in an JCA adapter
> and wants to be in control of the synchronization.
>
You can use maxMessagesPerPoll=1 to only poll 1 file at a time.
> kr. ldam
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/consumerTemplate-threading-worries-when-having-a-Batch-Consumer-tp5714904p5714912.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: consumerTemplate threading worries when having a Batch Consumer
Posted by ldam <la...@gmail.com>.
Unfortunately we are not the masters of the filename given :/
The reason we use consumerTemplate is because we use it in an JCA adapter
and wants to be in control of the synchronization.
kr. ldam
--
View this message in context: http://camel.465427.n5.nabble.com/consumerTemplate-threading-worries-when-having-a-Batch-Consumer-tp5714904p5714912.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: consumerTemplate threading worries when having a Batch Consumer
Posted by Claus Ibsen <cl...@gmail.com>.
The consumer template is intended for consuming a *single* message.
So in the case of the ftp endpoint, you should use the fileName option
to specify what file to consume.
For batch consuming then its better to use a Camel route for that.
On Fri, Jun 22, 2012 at 11:06 AM, ldam <la...@gmail.com> wrote:
> Hi
>
> I had some funny problems when getting exchanges from a consumer template,
> the attached code snippet illustrates a minimal example that demonstrates
> the problem. My findings are:
>
> When getting a batch of files (in my case via an sftp endpoint) doing a
> consumerTemplate.doneUoW(e) afterwards results in the underlying JSCH
> library getting corrupted streams and breaks.
>
> My assumption is that the underlying batch retrieval of files is done in
> it's own thread, and when the first exchange is processed and i execute
> doneUoW on it it accesses the JSCH file streams while the batch retrieval is
> going on on the other thread and thus corrupts it.
>
> Accumulating the exchanges in a collection and waiting to do the doneUoW()
> on them until after the last batch entry avoids this problem.
>
> So my questions are:
>
> 1) Are my assumption correct and is this what actual happens
> 2) If so, then there might be a problem, in those cases where you might have
> a bazingallion files, and the doneUof() execution is not finished before the
> batch retrieval starts again
>
>
> The attached code snippet:
>
> http://camel.465427.n5.nabble.com/file/n5714904/FailDemo.java FailDemo.java
>
>
> Some of the errors i get from the JSCH library (clearly indicates corrupted
> file streams)
>
> 2274 [Connect thread localhost session] INFO
> org.apache.camel.component.file.remote.SftpOperations - JSCH -> Caught an
> exception, leaving main loop due to SSH_MSG_DISCONNECT: 2 Packet corrupt
> 2506 [Connect thread localhost session] INFO
> org.apache.camel.component.file.remote.SftpOperations - JSCH -> Caught an
> exception, leaving main loop due to SSH_MSG_DISCONNECT: 2 Invalid ssh2
> packet type: 0
> 2305 [Connect thread localhost session] INFO
> org.apache.camel.component.file.remote.SftpOperations - JSCH -> Caught an
> exception, leaving main loop due to SSH_MSG_DISCONNECT: 2 Invalid ssh2
> packet type: 209
> 2286 [Connect thread localhost session] INFO
> org.apache.camel.component.file.remote.SftpOperations - JSCH -> Received
> SSH_MSG_UNIMPLEMENTED for 69
> 4726 [Connect thread localhost session] INFO
> org.apache.camel.component.file.remote.SftpOperations - JSCH -> Received
> SSH_MSG_UNIMPLEMENTED for 47
>
> I included these for search engines, in case other gets similar problems.
>
>
> kr. Lars Dam
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/consumerTemplate-threading-worries-when-having-a-Batch-Consumer-tp5714904.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: consumerTemplate threading worries when having a Batch Consumer
Posted by ldam <la...@gmail.com>.
I forgot: The test code of course requires an sftp server and bunch of files
for it to work (fail)
--
View this message in context: http://camel.465427.n5.nabble.com/consumerTemplate-threading-worries-when-having-a-Batch-Consumer-tp5714904p5714906.html
Sent from the Camel - Users mailing list archive at Nabble.com.