You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by rsteppac <ra...@derivativepartners.com> on 2011/01/04 09:54:01 UTC
2 Bugs in Splitter (Camel 2.5.0)?
Hello,
I think there are two bugs in the Splitter component.
Setting ‘streaming = "true"’ breaks error handling:
If an exception is thrown in a processor, the exception in the subExchange
is copied to the original exchange in MulticastProcessor line 554. In
Splitter line 140 the original exchange is copied, including the exception
that was thrown while processing the previous exchange. This prevents all
subsequent exchanges from being processed successfully.
Setting ‘streaming="true" parallelProcessing="true"’ consumes large amounts
of heap space for big original messages. E.g. 1024m of heap is not enough to
process an 80Mb with 500'000 lines, splitting it line by line.
The problem seems to be the ArrayList in MulticastProcessor line 224. It
contains a Future<Exchange> object for every token delivered by the
java.util.Scanner. The list is only cleared (going out of scope) after all
Future objects have been completed.
Maybe someone with more Camel experience than me (2 days so far) could
verify this before I open JIRA tickets for these issues?
Ralf
--
View this message in context: http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3326727.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2 Bugs in Splitter (Camel 2.5.0)?
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Jan 6, 2011 at 2:28 AM, Blair <bl...@gmail.com> wrote:
>
> oh, I'll throw in another problem, I'll go confirm it against 2.6.0 first,
> but having a transaction and a splitter doing parallel processing also
> screws things up big time. The things inside the split are NOT in the
> transaction, but only if you are doing parallel processing.
>
That is not a bug. Transactions can only happen in the same thread.
See for example page 304 in the Camel in Action book.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3329852.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: 2 Bugs in Splitter (Camel 2.5.0)?
Posted by Blair <bl...@gmail.com>.
oh, I'll throw in another problem, I'll go confirm it against 2.6.0 first,
but having a transaction and a splitter doing parallel processing also
screws things up big time. The things inside the split are NOT in the
transaction, but only if you are doing parallel processing.
--
View this message in context: http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3329852.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2 Bugs in Splitter (Camel 2.5.0)?
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Jan 5, 2011 at 12:11 PM, rsteppac
<ra...@derivativepartners.com> wrote:
>
> Hello Claus,
>
> thanks for your speedy answer!
>
>
> Claus Ibsen-2 wrote:
>>
>>> Setting ‘streaming = "true"’ breaks error handling: [..]
>>
>> Can you try 2.6-SNAPSHOT as we have fixed some issues in relation to
>> error handling and splitter/multicast.
>>
>
> The problem is still present in the 2.6 snapshot. I have raised CAMEL-3498.
>
As said on this JIRA please attach a sample test case and pin point
what you think is wrong.
>
> Claus Ibsen-2 wrote:
>>
>>> Setting ‘streaming="true" parallelProcessing="true"’ consumes large
>>> amounts
>>> of heap space for big original messages. [..]
>>
>> Hmm I was hoping that a Future instance would be a light weight object
>> in Java, but had my hopes to high :)
>>
>
> I have raised CAMEL-3497.
>
>
> Claus Ibsen-2 wrote:
>>
>> Back in the early days we did have some code using a
>> CompletionService. We most likely need to introduce this back and have
>> logic to aggregate the result as they fly in. Please create a ticket
>> for this issue.
>>
>
> Hm..
> MulticastProcessor::doProcessParallel(Exchange,AtomicExchange,Iterable,boolean,AsyncCallback)
> does use a CompletionService. As I understand it the problem is not the
> Future objects themselves, but the context of the corresponding callable
> (ProcessorExchangePair in line 228 and the Exchange in line 229). The
> context cannot go away until the Callable goes away. I think this is where
> the memory goes. But it is surely a good idea to look at it with a good
> memory profiler. ;-)
>
>
> Ralf
> --
> View this message in context: http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3328590.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: 2 Bugs in Splitter (Camel 2.5.0)?
Posted by rsteppac <ra...@derivativepartners.com>.
Hello Claus,
thanks for your speedy answer!
Claus Ibsen-2 wrote:
>
>> Setting ‘streaming = "true"’ breaks error handling: [..]
>
> Can you try 2.6-SNAPSHOT as we have fixed some issues in relation to
> error handling and splitter/multicast.
>
The problem is still present in the 2.6 snapshot. I have raised CAMEL-3498.
Claus Ibsen-2 wrote:
>
>> Setting ‘streaming="true" parallelProcessing="true"’ consumes large
>> amounts
>> of heap space for big original messages. [..]
>
> Hmm I was hoping that a Future instance would be a light weight object
> in Java, but had my hopes to high :)
>
I have raised CAMEL-3497.
Claus Ibsen-2 wrote:
>
> Back in the early days we did have some code using a
> CompletionService. We most likely need to introduce this back and have
> logic to aggregate the result as they fly in. Please create a ticket
> for this issue.
>
Hm..
MulticastProcessor::doProcessParallel(Exchange,AtomicExchange,Iterable,boolean,AsyncCallback)
does use a CompletionService. As I understand it the problem is not the
Future objects themselves, but the context of the corresponding callable
(ProcessorExchangePair in line 228 and the Exchange in line 229). The
context cannot go away until the Callable goes away. I think this is where
the memory goes. But it is surely a good idea to look at it with a good
memory profiler. ;-)
Ralf
--
View this message in context: http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3328590.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: 2 Bugs in Splitter (Camel 2.5.0)?
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Jan 4, 2011 at 9:54 AM, rsteppac
<ra...@derivativepartners.com> wrote:
>
> Hello,
>
> I think there are two bugs in the Splitter component.
>
> Setting ‘streaming = "true"’ breaks error handling:
> If an exception is thrown in a processor, the exception in the subExchange
> is copied to the original exchange in MulticastProcessor line 554. In
> Splitter line 140 the original exchange is copied, including the exception
> that was thrown while processing the previous exchange. This prevents all
> subsequent exchanges from being processed successfully.
>
Can you try 2.6-SNAPSHOT as we have fixed some issues in relation to
error handling and splitter/multicast.
> Setting ‘streaming="true" parallelProcessing="true"’ consumes large amounts
> of heap space for big original messages. E.g. 1024m of heap is not enough to
> process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It
> contains a Future<Exchange> object for every token delivered by the
> java.util.Scanner. The list is only cleared (going out of scope) after all
> Future objects have been completed.
>
Hmm I was hoping that a Future instance would be a light weight object
in Java, but had my hopes to high :)
Back in the early days we did have some code using a
CompletionService. We most likely need to introduce this back and have
logic to aggregate the result as they fly in. Please create a ticket
for this issue.
> Maybe someone with more Camel experience than me (2 days so far) could
> verify this before I open JIRA tickets for these issues?
>
>
> Ralf
> --
> View this message in context: http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3326727.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/