You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Dr Immo Benjes <im...@locussoftware.com> on 2016/03/30 18:02:20 UTC
Memory leak with enrich("sql:select * from WhatEver")
Hi
I’ve been working with Camel for the last few month.
I have a program that processes several CVS files in order and inserts them into a DB. Once they are all inserted I have to check for duplicate and create new CVS files without duplicates.
That is all working fine but I have one big problem - a memory leak.
I’ve tracked it down to a enrich(“sql”) component.
I have removed everything after that enrich and just added a to(“mock:result”).
Once the CVS files have been processed and inserted into the DB I send an exchange to
from("seda:check_duplicates?concurrentConsumers=4")
.log("Checking for duplicates on ${header.NOTIFY_BY} for event ${header.EVENT_ID} #{header.SFTP_DIRECTORY}")
.enrich("sql:select NOTIFICATION_ID, NOTIFY_BY, NOTIFY_BY_VALUE, LANGUAGE, :#EVENT_ID as EVENT_ID,PDP_SR_PARTICIPANTS_PARTICIPANT_ID,EVENT_DISPLAY_PREMISE_ADDR, CSV_LINE, count(*) as c from PDP_SR_NOTIFICATIONS where NOTIFY_BY = :#NOTIFY_BY and PDP_SR_EVENT_EVENT_ID = :#EVENT_ID group by PDP_SR_PARTICIPANTS_PARTICIPANT_ID, NOTIFY_BY_VALUE, LANGUAGE;")
.to("mock:result");
Below is a screenshot of the memory consumption for two sets of CVS files. You can see the jump in memory when the enrich(“sql”).
I would like to iterate over the result set (not load everything in one go as it can be quite large but the useIterator=true does not seem to be supported (at least I’ve tried) and the memory is never released again.
What am I doing wrong here? It is mainly chars in a HashMap that seems to be taking up the memory which makes sense if the data is that of the sql query.
Regards
Immo
Dr. Immo Benjes
Senior Software Architect
Castle Court
41 London Road
Reigate
Surrey RH2 9RJ
United Kingdom
Tel: +44 (0)1737 735012
Web: www.locussoftware.com
Email: immobenjes@locussoftware.com
Fwd: Memory leak with enrich("sql:select * from WhatEver")
Posted by Dr Immo Benjes <im...@locussoftware.com>.
Bump, anyone that can help me with this one?
Has anyone else experienced memory leaks with enrich(“sql”)? How can I configure it to e.g. only process e.g. 100 results at a time.
If I use maxMessagesPerPoll of e.g. 100 and the result would normally return 1000 results would the enrich(sql) to 10 queries and advance the result pointer automatically?
Regards
Immo
Dr. Immo Benjes
Senior Software Architect
Castle Court
41 London Road
Reigate
Surrey RH2 9RJ
United Kingdom
Tel: +44 (0)1737 735012
Web: www.locussoftware.com
Email: immobenjes@locussoftware.com
> Begin forwarded message:
>
> From: Dr Immo Benjes <im...@locussoftware.com>
> Subject: Memory leak with enrich("sql:select * from WhatEver")
> Date: 30 March 2016 at 18:02:20 GMT+2
> To: users@camel.apache.org
>
>
> Hi
>
> I’ve been working with Camel for the last few month.
>
> I have a program that processes several CVS files in order and inserts them into a DB. Once they are all inserted I have to check for duplicate and create new CVS files without duplicates.
>
> That is all working fine but I have one big problem - a memory leak.
>
> I’ve tracked it down to a enrich(“sql”) component.
>
> I have removed everything after that enrich and just added a to(“mock:result”).
> Once the CVS files have been processed and inserted into the DB I send an exchange to
>
> from("seda:check_duplicates?concurrentConsumers=4")
> .log("Checking for duplicates on ${header.NOTIFY_BY} for event ${header.EVENT_ID} #{header.SFTP_DIRECTORY}")
> .enrich("sql:select NOTIFICATION_ID, NOTIFY_BY, NOTIFY_BY_VALUE, LANGUAGE, :#EVENT_ID as EVENT_ID,PDP_SR_PARTICIPANTS_PARTICIPANT_ID,EVENT_DISPLAY_PREMISE_ADDR, CSV_LINE, count(*) as c from PDP_SR_NOTIFICATIONS where NOTIFY_BY = :#NOTIFY_BY and PDP_SR_EVENT_EVENT_ID = :#EVENT_ID group by PDP_SR_PARTICIPANTS_PARTICIPANT_ID, NOTIFY_BY_VALUE, LANGUAGE;")
> .to("mock:result");
>
> Below is a screenshot of the memory consumption for two sets of CVS files. You can see the jump in memory when the enrich(“sql”).
>
> I would like to iterate over the result set (not load everything in one go as it can be quite large but the useIterator=true does not seem to be supported (at least I’ve tried) and the memory is never released again.
>
> What am I doing wrong here? It is mainly chars in a HashMap that seems to be taking up the memory which makes sense if the data is that of the sql query.
>
>
> Regards
>
> Immo
>
> Dr. Immo Benjes
> Senior Software Architect
>
>
>
> Castle Court
> 41 London Road
> Reigate
> Surrey RH2 9RJ
> United Kingdom
> Tel: +44 (0)1737 735012
> Web: www.locussoftware.com <http://www.locussoftware.com/>
> Email: immobenjes@locussoftware.com <ma...@locussoftware.com>
>
>
>