jdbc producers

[Paul Phillips <dj...@gmail.com>]

Hi 

I have created a route that splits on an arraylist that could be several
thousand items in size.

The route then does a few different things, and every so often sends a
message to a jdbc endpoint

Everything seems ok but if the arraylist is really big (> 1000) I run out of
memory.

I've found adding a .streaming() helps a lot, but eventually I still run out
of mem (even if I allocate 768meg to the jvm) , or the system slows right
down.

I've managed to stop the context and when I do this I see loads of jdbc
producers being unregistered.
If I go into jconsole and to the mbeans tab and look at the producers under
camel I see new jdbc producers being registered all the time when my route
is running. However the other producers on the route like http etc seem
fairly static as if the route is "reusing" them.

Is this expected behaviour or is there a way to stop so many jdbc producers
from being created (if that is even what is happening!)

Let me know if you need more info, I plan to create a test to investigate
the issue, as I oculd be barking up the wrong tree as to if this is even the
issue that is causing my memory problems!

cheers
Paul.
-- 
View this message in context: http://old.nabble.com/jdbc-producers-tp27669119p27669119.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: jdbc producers

[Claus Ibsen <cl...@gmail.com>]

Hi Paul

Thanks for coming back and letting us know you have it resolved now
and sharing the solutions to the problem.


On Thu, Feb 25, 2010 at 11:19 AM, Paul Phillips <dj...@gmail.com> wrote:
>
> I was using camel 2.1
>
> Have upgraded to 2.2 now
>
> I have since discovered that the "extra" jdbc producers were my own fault
> anyway because of a classic mistake.
>
> During my route a was creating a new ProducerTemplate in a processor.
>
> This meant that it had its own ProducerCache that was empty thus making it
> register a new jdbc producer with jmx.
>
> There was also another unrelated problem with my app. I use an embedded hsql
> database and the default for tables is to create them as "memory" tables.
> This means the whole of my 200meg log table was being loaded into the jvm!
>
> Once I sorted these 2 issues the memory usage is looking a hell of a lot
> better....
>
> Thanks for your time anyway!
>
> Paul.
> --
> View this message in context: http://old.nabble.com/jdbc-producers-tp27669119p27714388.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Re: jdbc producers

[Paul Phillips <dj...@gmail.com>]

I was using camel 2.1

Have upgraded to 2.2 now

I have since discovered that the "extra" jdbc producers were my own fault
anyway because of a classic mistake.

During my route a was creating a new ProducerTemplate in a processor.

This meant that it had its own ProducerCache that was empty thus making it
register a new jdbc producer with jmx.

There was also another unrelated problem with my app. I use an embedded hsql
database and the default for tables is to create them as "memory" tables.
This means the whole of my 200meg log table was being loaded into the jvm!

Once I sorted these 2 issues the memory usage is looking a hell of a lot
better....

Thanks for your time anyway!

Paul.
-- 
View this message in context: http://old.nabble.com/jdbc-producers-tp27669119p27714388.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: jdbc producers

[Claus Ibsen <cl...@gmail.com>]

Hi

You didn't write the version number despite it being highlighted on
the how to get help page :)
http://camel.apache.org/support.html

I assume its 2.1 or 2.2.

I think we have been a bit to aggressive on registering Producers in
JMX which we have relaxed in 2.3 which should prevent Camel from
registering all those jdbc producers in JMX as JMX is a memory hog.

Could you try out the latest code?
http://camel.apache.org/download.html


On Sat, Feb 20, 2010 at 7:34 PM, Paul Phillips <dj...@gmail.com> wrote:
>
> Hi
>
> I have created a route that splits on an arraylist that could be several
> thousand items in size.
>
> The route then does a few different things, and every so often sends a
> message to a jdbc endpoint
>
> Everything seems ok but if the arraylist is really big (> 1000) I run out of
> memory.
>
> I've found adding a .streaming() helps a lot, but eventually I still run out
> of mem (even if I allocate 768meg to the jvm) , or the system slows right
> down.
>
> I've managed to stop the context and when I do this I see loads of jdbc
> producers being unregistered.
> If I go into jconsole and to the mbeans tab and look at the producers under
> camel I see new jdbc producers being registered all the time when my route
> is running. However the other producers on the route like http etc seem
> fairly static as if the route is "reusing" them.
>
> Is this expected behaviour or is there a way to stop so many jdbc producers
> from being created (if that is even what is happening!)
>
> Let me know if you need more info, I plan to create a test to investigate
> the issue, as I oculd be barking up the wrong tree as to if this is even the
> issue that is causing my memory problems!
>
> cheers
> Paul.
> --
> View this message in context: http://old.nabble.com/jdbc-producers-tp27669119p27669119.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus