You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Christian Schneider (JIRA)" <ji...@apache.org> on 2009/05/27 23:46:51 UTC
[jira] Created: (CAMEL-1652) Memory leak in File component when
using batch processing
Memory leak in File component when using batch processing
---------------------------------------------------------
Key: CAMEL-1652
URL: https://issues.apache.org/activemq/browse/CAMEL-1652
Project: Apache Camel
Issue Type: Bug
Components: camel-core
Affects Versions: 2.0-M1
Reporter: Christian Schneider
Priority: Critical
Fix For: 2.0-M2
See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-1652:
-------------------------------
Priority: Major (was: Critical)
Not critical, as there is a workaround by clearing the exchange body after usage.
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Assignee: Claus Ibsen
> Fix For: 2.0-M2
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Christian Schneider (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Schneider updated CAMEL-1652:
---------------------------------------
Attachment: camel-core.patch
Test and patch for this issue
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Priority: Critical
> Fix For: 2.0-M2
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51951#action_51951 ]
Claus Ibsen commented on CAMEL-1652:
------------------------------------
Running the unit test for 30 sec on my laptop:
*Before patch:* usage = 42mb
*After patch:* usage = 5mb
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Priority: Critical
> Fix For: 2.0-M2
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen reassigned CAMEL-1652:
----------------------------------
Assignee: Claus Ibsen
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Assignee: Claus Ibsen
> Priority: Critical
> Fix For: 2.0-M2
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Christian Schneider (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Schneider closed CAMEL-1652.
--------------------------------------
Thanks for committing the patch
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Assignee: Claus Ibsen
> Fix For: 2.0.0
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-1652.
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 2.0-M2)
2.0.0
trunk: 779434.
Thanks Christian for the patch and unit test.
I have applied the fix for the other BatchConsumer components as well.
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Assignee: Claus Ibsen
> Fix For: 2.0.0
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51950#action_51950 ]
Claus Ibsen commented on CAMEL-1652:
------------------------------------
@willem
The bug has been there before CAMEL-1640, it used a List for quite some time. But glad we discovered it and have a fix for it.
Yes it should be applied to the other listed here:
http://camel.apache.org/batch-consumer.html
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Priority: Critical
> Fix For: 2.0-M2
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-1652) Memory leak in File component when
using batch processing
Posted by "Willem Jiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51949#action_51949 ]
Willem Jiang commented on CAMEL-1652:
-------------------------------------
@claus
I think this issue is introduced by the recent new feature of BatchConsumer CAMEL-1640, you may need to recheck the components of mail, ibatis, jpa.
> Memory leak in File component when using batch processing
> ---------------------------------------------------------
>
> Key: CAMEL-1652
> URL: https://issues.apache.org/activemq/browse/CAMEL-1652
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-core
> Affects Versions: 2.0-M1
> Reporter: Christian Schneider
> Priority: Critical
> Fix For: 2.0-M2
>
> Attachments: camel-core.patch
>
>
> See http://www.nabble.com/convertBodyTo-heap-problem-to23743181.html
> When using a file endpoint with sorting as start of a route then the files are processed in a batch. In the current implementation this means a List of exchanges is created and given to the batch processing. The list is then traversed and each exchange is processed. This means the list references all exchanges during the whole batch processing. So none of the exchanges can be collected by the garbage collector till the whole batch is fully processed.
> The problem really becomes bad when the exchanges are large. For example if you use convertBodyTo(String.class) and the files are large. If you have 100 files with 10 MB each then camel will consume 1 GB of heap. What is even worse is that it seems the processing simply stops when the heap is full.
> I have written a small test that shows the problem and a patch that corrects this problem. I hope this gets into 2.0-M2 as this is a quite serious bug.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.