You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Ralf Steppacher (JIRA)" <ji...@apache.org> on 2011/01/05 10:21:46 UTC
[jira] Created: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
-----------------------------------------------------------------------------------------------------------------------------------------
Key: CAMEL-3497
URL: https://issues.apache.org/jira/browse/CAMEL-3497
Project: Camel
Issue Type: Bug
Components: camel-core
Affects Versions: 2.5.0
Reporter: Ralf Steppacher
Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979113#action_12979113 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
Okay good news. I refactored the logic so Camel now aggregates the parallel tasks on-the-fly.
This makes a tremendous difference. Now I can split a file into 50.000 sub messages and process that in 7 sec, using at most 18mb.
Before I would hit an issue at about 25.000-30.000 message and hit OOME with 130mb.
Since the logic is more complex because there is a separate tasks which aggregates on the fly, while the other task submit new tasks, there is logic to signal between the two tasks. They kinda need to agree when there are no more messages to split, and when it has aggregated all of those.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979521#action_12979521 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
Fixed rare potential deadlock issue with aggregate task not being given time to run due thread pool overloaded when running in parallel mode on multicast/splitter.
trunk: 1057139.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 2.6.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977754#action_12977754 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
Yeah the tasks list is only used for cancelling tasks which isn't needed to be processed anymore due we are done due timeout or stop on exception. So it should be possible to refactor the code to not use a task list for that.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979158#action_12979158 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
I ran a test with 1.000.000 rows in a file
{code}
2011-01-08 18:25:44,216 [read #9 - Split] INFO split - Received: 1000000 messages so far. Last group took: 50 millis which is: 20,000 messages per second. average: 17,775.566
2011-01-08 18:25:44,217 [main ] INFO SplitterParallelBigFileTest - Took 57.423 seconds
2011-01-08 18:25:44,218 [://target/split] INFO route1 - Done splitting bigfile.txt
{code}
And the memory usage was at most 33mb at peak.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977868#action_12977868 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
The issue is the splitter copies the exchange for each splitted message. And the CompletionService keeps reference to all exchanges, which means we end up with a lot of Exchange at once in memory which eats up memory.
Will have to come up with some way of discarding not needed exchanges during processing. Maybe even using something else than the CompletionService if its the culprint.
I added this to the known issues to Camel 2.5 release notes.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 2.6.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978855#action_12978855 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
trunk on Camel 2.6 in rev 1056380:
I have reduced memory consumption used, which should allow it to be a bit better. But the splitter still uses a bit additional memory due the splitting is based on a copy of the input message for each splitted message.
Ralf you are welcome to test again and see if you can process a bit more than previously.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977787#action_12977787 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
The CompletionService holds a reference to the Future so there is no gain really.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978785#action_12978785 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
trunk on Camel 2.6 in rev 1056325:
Cancelling future tasks is now done using a running boolean instead of a keeping a big array list with the future references.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-3497:
-------------------------------
Fix Version/s: 2.6.0
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 2.6.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-3497.
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 3.0.0)
2.6.0
trunk: 1056744.
Now it should run with low memory consumption and you should be able to process very big files.
Ralf fell free to test with latest code on your system.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 2.6.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977725#action_12977725 ]
Claus Ibsen commented on CAMEL-3497:
------------------------------------
See nabble
http://camel.465427.n5.nabble.com/2-Bugs-in-Splitter-Camel-2-5-0-tp3326727p3326727.html
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-3497:
-------------------------------
Fix Version/s: (was: 2.6.0)
3.0.0
Camel 3.0 will have internal optimization which helps reducing memory footprint used during routing.
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
> Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CAMEL-3497) Splitter Component: Setting
'streaming="true" parallelProcessing="true"' consumes large amounts > of
heap space for big original messages
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen reassigned CAMEL-3497:
----------------------------------
Assignee: Claus Ibsen
> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-3497
> URL: https://issues.apache.org/jira/browse/CAMEL-3497
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.5.0
> Reporter: Ralf Steppacher
> Assignee: Claus Ibsen
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000 lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange> object for every token delivered by the java.util.Scanner. The list is only cleared (going out of scope) after all Future objects have been completed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.