You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Ryan Moquin <fr...@gmail.com> on 2009/08/20 19:00:07 UTC
Servicemix clustering with ActiveMQ
I was noticing today that if I run servicemix for a while (without sending
any requests to it), it would eventually run out of heap space. I hadn't
had that problem in a long time and after looking at jconsole running with
different activemq clustering configurations, it looks like activemq may be
leaking memory if you use this network connector configuration, when neither
of the hosts specified have been started yet:
<amq:networkConnector uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
It seems like it takes around 30 minutes or so... basically, it will just
keep churning through memory (probably 1 mb/sec) and then that memory will
get garbage collected. After a while, not all the memory gets garbage
collected, even if you tell it to garbage collect in jconsole.
If I change back to multicast, it appears as though the problem goes away
and it chews through memory slower (I'm not sure if the issue would still
happen or not). Basically, we don't have multicast enabled on the network
where I test out our software, so I started using the network connector with
the hosts specified. Would I just be better off using an external activemq
instance? If so, what's the best way to make your jms SU's that are
deployed all use the same activemq instance? I know you configure it in
servicemix.conf (I think is the file), but the jms su's get configured
separately for what activemq server they should connect to.
Has anyone ran across this issue or have a good way of handling it? Thanks
for any help!
Ryan
Re: Servicemix clustering with ActiveMQ
Posted by Ryan Moquin <fr...@gmail.com>.
Just an FYI, I had taked off the clustering for our app.. and I've left it
up over night and the servicemix heap usage has stayed completely stable.
So it's definitely something to do with the clustering and the servicemix
hosts not being up.
On Thu, Aug 20, 2009 at 7:02 PM, Ryan Moquin <fr...@gmail.com> wrote:
> I will try setting the initialReconnectDelay.
>
> I mean the smx/amq instance. So the machines that are referenced are
> running, but the smx/amq instance that they are trying to connect to aren't
> running.
>
> Ryan
>
>
> On Thu, Aug 20, 2009 at 2:55 PM, Chris Custine <ch...@gmail.com>wrote:
>
>> 1MB/sec seems exceptionally high for reconnect attempts. I'll try to find
>> some time to hook this up in Yourkit or something to see whats happening,
>> but in the mean time could you try changing initialReconnectDelay or some
>> of
>> the other parameters for the AMQ static transport as shown here:
>> http://activemq.apache.org/static-transport-reference.html
>>
>> Just to clarify, when you say neither of the hosts have been started yet,
>> do
>> you literally mean the host itself, or did you mean the SMX/AMQ instance?
>> I
>> just want to make sure I can reproduce the issue exactly, since those
>> situations may have different behavior.
>>
>> Also, what SMX version?
>>
>> Thanks,
>> Chris
>> --
>> Chris Custine
>> FUSESource :: http://fusesource.com
>> My Blog :: http://blog.organicelement.com
>> Apache ServiceMix :: http://servicemix.apache.org
>> Apache Directory Server :: http://directory.apache.org
>>
>>
>> On Thu, Aug 20, 2009 at 11:00 AM, Ryan Moquin <fragility2.0@gmail.com
>> >wrote:
>>
>> > I was noticing today that if I run servicemix for a while (without
>> sending
>> > any requests to it), it would eventually run out of heap space. I
>> hadn't
>> > had that problem in a long time and after looking at jconsole running
>> with
>> > different activemq clustering configurations, it looks like activemq may
>> be
>> > leaking memory if you use this network connector configuration, when
>> > neither
>> > of the hosts specified have been started yet:
>> >
>> > <amq:networkConnector
>> > uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>> >
>> > It seems like it takes around 30 minutes or so... basically, it will
>> just
>> > keep churning through memory (probably 1 mb/sec) and then that memory
>> will
>> > get garbage collected. After a while, not all the memory gets garbage
>> > collected, even if you tell it to garbage collect in jconsole.
>> >
>> > If I change back to multicast, it appears as though the problem goes
>> away
>> > and it chews through memory slower (I'm not sure if the issue would
>> still
>> > happen or not). Basically, we don't have multicast enabled on the
>> network
>> > where I test out our software, so I started using the network connector
>> > with
>> > the hosts specified. Would I just be better off using an external
>> activemq
>> > instance? If so, what's the best way to make your jms SU's that are
>> > deployed all use the same activemq instance? I know you configure it in
>> > servicemix.conf (I think is the file), but the jms su's get configured
>> > separately for what activemq server they should connect to.
>> >
>> > Has anyone ran across this issue or have a good way of handling it?
>> Thanks
>> > for any help!
>> >
>> > Ryan
>> >
>>
>
>
Re: Servicemix clustering with ActiveMQ
Posted by Ryan Moquin <fr...@gmail.com>.
I will try setting the initialReconnectDelay.
I mean the smx/amq instance. So the machines that are referenced are
running, but the smx/amq instance that they are trying to connect to aren't
running.
Ryan
On Thu, Aug 20, 2009 at 2:55 PM, Chris Custine <ch...@gmail.com>wrote:
> 1MB/sec seems exceptionally high for reconnect attempts. I'll try to find
> some time to hook this up in Yourkit or something to see whats happening,
> but in the mean time could you try changing initialReconnectDelay or some
> of
> the other parameters for the AMQ static transport as shown here:
> http://activemq.apache.org/static-transport-reference.html
>
> Just to clarify, when you say neither of the hosts have been started yet,
> do
> you literally mean the host itself, or did you mean the SMX/AMQ instance?
> I
> just want to make sure I can reproduce the issue exactly, since those
> situations may have different behavior.
>
> Also, what SMX version?
>
> Thanks,
> Chris
> --
> Chris Custine
> FUSESource :: http://fusesource.com
> My Blog :: http://blog.organicelement.com
> Apache ServiceMix :: http://servicemix.apache.org
> Apache Directory Server :: http://directory.apache.org
>
>
> On Thu, Aug 20, 2009 at 11:00 AM, Ryan Moquin <fragility2.0@gmail.com
> >wrote:
>
> > I was noticing today that if I run servicemix for a while (without
> sending
> > any requests to it), it would eventually run out of heap space. I hadn't
> > had that problem in a long time and after looking at jconsole running
> with
> > different activemq clustering configurations, it looks like activemq may
> be
> > leaking memory if you use this network connector configuration, when
> > neither
> > of the hosts specified have been started yet:
> >
> > <amq:networkConnector
> > uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
> >
> > It seems like it takes around 30 minutes or so... basically, it will just
> > keep churning through memory (probably 1 mb/sec) and then that memory
> will
> > get garbage collected. After a while, not all the memory gets garbage
> > collected, even if you tell it to garbage collect in jconsole.
> >
> > If I change back to multicast, it appears as though the problem goes away
> > and it chews through memory slower (I'm not sure if the issue would still
> > happen or not). Basically, we don't have multicast enabled on the
> network
> > where I test out our software, so I started using the network connector
> > with
> > the hosts specified. Would I just be better off using an external
> activemq
> > instance? If so, what's the best way to make your jms SU's that are
> > deployed all use the same activemq instance? I know you configure it in
> > servicemix.conf (I think is the file), but the jms su's get configured
> > separately for what activemq server they should connect to.
> >
> > Has anyone ran across this issue or have a good way of handling it?
> Thanks
> > for any help!
> >
> > Ryan
> >
>
Re: Servicemix clustering with ActiveMQ
Posted by Chris Custine <ch...@gmail.com>.
1MB/sec seems exceptionally high for reconnect attempts. I'll try to find
some time to hook this up in Yourkit or something to see whats happening,
but in the mean time could you try changing initialReconnectDelay or some of
the other parameters for the AMQ static transport as shown here:
http://activemq.apache.org/static-transport-reference.html
Just to clarify, when you say neither of the hosts have been started yet, do
you literally mean the host itself, or did you mean the SMX/AMQ instance? I
just want to make sure I can reproduce the issue exactly, since those
situations may have different behavior.
Also, what SMX version?
Thanks,
Chris
--
Chris Custine
FUSESource :: http://fusesource.com
My Blog :: http://blog.organicelement.com
Apache ServiceMix :: http://servicemix.apache.org
Apache Directory Server :: http://directory.apache.org
On Thu, Aug 20, 2009 at 11:00 AM, Ryan Moquin <fr...@gmail.com>wrote:
> I was noticing today that if I run servicemix for a while (without sending
> any requests to it), it would eventually run out of heap space. I hadn't
> had that problem in a long time and after looking at jconsole running with
> different activemq clustering configurations, it looks like activemq may be
> leaking memory if you use this network connector configuration, when
> neither
> of the hosts specified have been started yet:
>
> <amq:networkConnector
> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>
> It seems like it takes around 30 minutes or so... basically, it will just
> keep churning through memory (probably 1 mb/sec) and then that memory will
> get garbage collected. After a while, not all the memory gets garbage
> collected, even if you tell it to garbage collect in jconsole.
>
> If I change back to multicast, it appears as though the problem goes away
> and it chews through memory slower (I'm not sure if the issue would still
> happen or not). Basically, we don't have multicast enabled on the network
> where I test out our software, so I started using the network connector
> with
> the hosts specified. Would I just be better off using an external activemq
> instance? If so, what's the best way to make your jms SU's that are
> deployed all use the same activemq instance? I know you configure it in
> servicemix.conf (I think is the file), but the jms su's get configured
> separately for what activemq server they should connect to.
>
> Has anyone ran across this issue or have a good way of handling it? Thanks
> for any help!
>
> Ryan
>