You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com> on 2007/04/09 20:44:04 UTC
Load Balancer configuration
Hi,
I am trying to configure multiple AMQ instances with a Load balancer
configuration for High availability. Load balancer I am using wants to
Ping AMQ instances to do the keep-alive check. I tried to set that
keep-alive port to 61617/61616. every time Load Balancer pings, AMQ
throws exception because this is not a real OpenWire/SSL communication;
but this approach works for few Hrs.
But after some time AMQ gets tired of this ping and throws OutofMemory
exception. I guess AMQ is growing in memory every time it throws the
connection exception and not cleaning up properly.
My question is how to manage such a configuration where AMQ's are
configured behind a load balancer.
Thanks!
Vik
RE: Load Balancer configuration
Posted by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com>.
Please ignore my JMX question, I found the details I was looking for in
JavaDocs.
Thanks!
Vik
-----Original Message-----
From: Dhawan, Vikram (LNG-DAY) [mailto:vikram.dhawan@lexisnexis.com]
Sent: Wednesday, April 11, 2007 11:58 PM
To: dev@activemq.apache.org
Subject: RE: Load Balancer configuration
James,
I will run it on Profiler and let you know. I have another question in
JMX, QueueManagementBean has a method exposed called
moveMatchingMessages(p1,p2). I am wondering what this message does. What
type of matching it does?
In our AMQ JMX documentation there is some detail, but I think it will
be great if we can update this page with a little more description of
what each exposed operation does. I can help you with that if you can
provide me some details about the few of the methods I am not aware of.
Thanks!
Vik
-----Original Message-----
From: James Strachan [mailto:james.strachan@gmail.com]
Sent: Wednesday, April 11, 2007 2:17 PM
To: dev@activemq.apache.org
Subject: Re: Load Balancer configuration
Could you run with 4.1.1 in a profiler to see where the memory is
being used? Incidentally I wonder if 4.2-SNAPSHOT has this issue?
On 4/11/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
wrote:
> I see it in AMQ4.0.1 and AMQ4.1.1 both.
>
> Thanks!
>
> Vik
Re: Load Balancer configuration
Posted by James Strachan <ja...@gmail.com>.
On 4/12/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com> wrote:
> James,
>
> I will run it on Profiler and let you know. I have another question in
> JMX, QueueManagementBean has a method exposed called
> moveMatchingMessages(p1,p2). I am wondering what this message does.
It moves messages to another queue :)
> What
> type of matching it does?
A JMS SQL selector as used for subscriptions
> In our AMQ JMX documentation there is some detail, but I think it will
> be great if we can update this page with a little more description of
> what each exposed operation does. I can help you with that if you can
> provide me some details about the few of the methods I am not aware of.
Go for it. FWIW I've updated the javadoc of some of the MBeans
recently; so definitely grab the latest code from SVN and see what
gaps there are. Please ask if there's anything unclear & we can help
patch up the code to make the javadoc better
--
James
-------
http://radio.weblogs.com/0112098/
RE: Load Balancer configuration
Posted by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com>.
James,
I will run it on Profiler and let you know. I have another question in
JMX, QueueManagementBean has a method exposed called
moveMatchingMessages(p1,p2). I am wondering what this message does. What
type of matching it does?
In our AMQ JMX documentation there is some detail, but I think it will
be great if we can update this page with a little more description of
what each exposed operation does. I can help you with that if you can
provide me some details about the few of the methods I am not aware of.
Thanks!
Vik
-----Original Message-----
From: James Strachan [mailto:james.strachan@gmail.com]
Sent: Wednesday, April 11, 2007 2:17 PM
To: dev@activemq.apache.org
Subject: Re: Load Balancer configuration
Could you run with 4.1.1 in a profiler to see where the memory is
being used? Incidentally I wonder if 4.2-SNAPSHOT has this issue?
On 4/11/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
wrote:
> I see it in AMQ4.0.1 and AMQ4.1.1 both.
>
> Thanks!
>
> Vik
Re: Load Balancer configuration
Posted by James Strachan <ja...@gmail.com>.
Could you run with 4.1.1 in a profiler to see where the memory is
being used? Incidentally I wonder if 4.2-SNAPSHOT has this issue?
On 4/11/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com> wrote:
> I see it in AMQ4.0.1 and AMQ4.1.1 both.
>
> Thanks!
>
> Vik
>
> -----Original Message-----
> From: James Strachan [mailto:james.strachan@gmail.com]
> Sent: Wednesday, April 11, 2007 2:13 PM
> To: dev@activemq.apache.org
> Subject: Re: Load Balancer configuration
>
> On 4/11/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
> wrote:
> > Hi James,
> >
> > Thanks for your help and suggestion here. But this brings another
> > concern.
> >
> > There is always a possibility that some one can write a client app
> which
> > connects to AMQ on any of the real AMQ ports (61616/61613/61617 in
> > sample configuration) and disconnects the same way as Load Balancer is
> > doing. And what appears to me is this will cause AMQ to throw
> exceptions
> > like following, and which will result AMQ going out of memory.
>
> That exception looks fine to me.
>
> ActiveMQ should be reclaiming any memory for bad connections -
> exception stack traces don't use up any RAM. If you're seeing a memory
> leak then its a bug. Which version do you see this in?
>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
--
James
-------
http://radio.weblogs.com/0112098/
RE: Load Balancer configuration
Posted by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com>.
I see it in AMQ4.0.1 and AMQ4.1.1 both.
Thanks!
Vik
-----Original Message-----
From: James Strachan [mailto:james.strachan@gmail.com]
Sent: Wednesday, April 11, 2007 2:13 PM
To: dev@activemq.apache.org
Subject: Re: Load Balancer configuration
On 4/11/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
wrote:
> Hi James,
>
> Thanks for your help and suggestion here. But this brings another
> concern.
>
> There is always a possibility that some one can write a client app
which
> connects to AMQ on any of the real AMQ ports (61616/61613/61617 in
> sample configuration) and disconnects the same way as Load Balancer is
> doing. And what appears to me is this will cause AMQ to throw
exceptions
> like following, and which will result AMQ going out of memory.
That exception looks fine to me.
ActiveMQ should be reclaiming any memory for bad connections -
exception stack traces don't use up any RAM. If you're seeing a memory
leak then its a bug. Which version do you see this in?
--
James
-------
http://radio.weblogs.com/0112098/
Re: Load Balancer configuration
Posted by James Strachan <ja...@gmail.com>.
On 4/11/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com> wrote:
> Hi James,
>
> Thanks for your help and suggestion here. But this brings another
> concern.
>
> There is always a possibility that some one can write a client app which
> connects to AMQ on any of the real AMQ ports (61616/61613/61617 in
> sample configuration) and disconnects the same way as Load Balancer is
> doing. And what appears to me is this will cause AMQ to throw exceptions
> like following, and which will result AMQ going out of memory.
That exception looks fine to me.
ActiveMQ should be reclaiming any memory for bad connections -
exception stack traces don't use up any RAM. If you're seeing a memory
leak then its a bug. Which version do you see this in?
--
James
-------
http://radio.weblogs.com/0112098/
RE: Load Balancer configuration
Posted by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com>.
Hi James,
Thanks for your help and suggestion here. But this brings another
concern.
There is always a possibility that some one can write a client app which
connects to AMQ on any of the real AMQ ports (61616/61613/61617 in
sample configuration) and disconnects the same way as Load Balancer is
doing. And what appears to me is this will cause AMQ to throw exceptions
like following, and which will result AMQ going out of memory.
Looking forward to hear your thought.
ERROR TransportConnector - Could not accept connection from
/testserver:50528: javax.net.ssl.SSLHandshakeException: Remote host
closed connection during handshake
javax.net.ssl.SSLHandshakeException: Remote host closed connection
during handshake at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java
:742) at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSo
cketImpl.java:1030)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.jav
a:622)
at
com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:
59)
at
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBuffe
redOutputStream.java:109)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:
119)
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor
.java:141)
at
org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFo
rmatNegotiator.java:155)
at
org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegot
iator.java:73)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java
:54)
at
org.apache.activemq.broker.TransportConnection.start(TransportConnection
.java:816)
at
org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnec
tor.java:148)
at
org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportSer
ver.java:167)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at
com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java
:723)
... 14 more
Vik
-----Original Message-----
From: James Strachan [mailto:james.strachan@gmail.com]
Sent: Monday, April 09, 2007 6:24 PM
To: dev@activemq.apache.org
Subject: Re: Load Balancer configuration
It seems kinda strange to use a ping on a totally separate socket to
decide if another socket can be used. Sounds like a crappy load
balancer to me; can't you just disable the pointless ping and let the
load balancer use a broker/socket until it becomes unresponsive? If
you must have some kinda ping, it should be trivial to write your own
ping POJO and add it to the broker as an interceptor or a Spring bean
(since the ping has nothing at all to do with the broker, its just a
test that the VM is still running)
On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
wrote:
> Hey James,
>
> BTW... I just wanted to let you know that I am not overcrowding the
same
> socket. Clients are connecting on 61616 and 61613 and I am using 61617
> for load balancer ping.
>
> I will appreciate your help, if you can suggest me what is the ideal
way
> to configure AMQ in this type of environment.
>
> Thanks!
>
> Vik
>
> Hi James,
>
> Actually my AMQ clients are connecting to a VIP which is the load
> balancers IP, and load balancer wants to ping and checks the AMQ
> instance status before it routes the client call. That's why Load
> balancer is sending ping to AMQ instances.
>
> Any idea, how should I configure AMQ instances.
>
> Thanks!
>
> Vik
>
> The ActiveMQ client sends ping commands anyway, so why does the load
> balancer need to shove stuff down the same socket?
>
> On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
> wrote:
> > Hi,
> >
> >
> >
> > I am trying to configure multiple AMQ instances with a Load balancer
> > configuration for High availability. Load balancer I am using wants
to
> > Ping AMQ instances to do the keep-alive check. I tried to set that
> > keep-alive port to 61617/61616. every time Load Balancer pings, AMQ
> > throws exception because this is not a real OpenWire/SSL
> communication;
> > but this approach works for few Hrs.
> >
> >
> >
> > But after some time AMQ gets tired of this ping and throws
OutofMemory
> > exception. I guess AMQ is growing in memory every time it throws the
> > connection exception and not cleaning up properly.
> >
> >
> >
> > My question is how to manage such a configuration where AMQ's are
> > configured behind a load balancer.
> >
> >
> >
> > Thanks!
> >
> >
> >
> >
> >
> > Vik
> >
> >
> >
> >
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Load Balancer configuration
Posted by James Strachan <ja...@gmail.com>.
It seems kinda strange to use a ping on a totally separate socket to
decide if another socket can be used. Sounds like a crappy load
balancer to me; can't you just disable the pointless ping and let the
load balancer use a broker/socket until it becomes unresponsive? If
you must have some kinda ping, it should be trivial to write your own
ping POJO and add it to the broker as an interceptor or a Spring bean
(since the ping has nothing at all to do with the broker, its just a
test that the VM is still running)
On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com> wrote:
> Hey James,
>
> BTW... I just wanted to let you know that I am not overcrowding the same
> socket. Clients are connecting on 61616 and 61613 and I am using 61617
> for load balancer ping.
>
> I will appreciate your help, if you can suggest me what is the ideal way
> to configure AMQ in this type of environment.
>
> Thanks!
>
> Vik
>
> Hi James,
>
> Actually my AMQ clients are connecting to a VIP which is the load
> balancers IP, and load balancer wants to ping and checks the AMQ
> instance status before it routes the client call. That's why Load
> balancer is sending ping to AMQ instances.
>
> Any idea, how should I configure AMQ instances.
>
> Thanks!
>
> Vik
>
> The ActiveMQ client sends ping commands anyway, so why does the load
> balancer need to shove stuff down the same socket?
>
> On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
> wrote:
> > Hi,
> >
> >
> >
> > I am trying to configure multiple AMQ instances with a Load balancer
> > configuration for High availability. Load balancer I am using wants to
> > Ping AMQ instances to do the keep-alive check. I tried to set that
> > keep-alive port to 61617/61616. every time Load Balancer pings, AMQ
> > throws exception because this is not a real OpenWire/SSL
> communication;
> > but this approach works for few Hrs.
> >
> >
> >
> > But after some time AMQ gets tired of this ping and throws OutofMemory
> > exception. I guess AMQ is growing in memory every time it throws the
> > connection exception and not cleaning up properly.
> >
> >
> >
> > My question is how to manage such a configuration where AMQ's are
> > configured behind a load balancer.
> >
> >
> >
> > Thanks!
> >
> >
> >
> >
> >
> > Vik
> >
> >
> >
> >
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
--
James
-------
http://radio.weblogs.com/0112098/
RE: Load Balancer configuration
Posted by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com>.
Hey James,
BTW... I just wanted to let you know that I am not overcrowding the same
socket. Clients are connecting on 61616 and 61613 and I am using 61617
for load balancer ping.
I will appreciate your help, if you can suggest me what is the ideal way
to configure AMQ in this type of environment.
Thanks!
Vik
Hi James,
Actually my AMQ clients are connecting to a VIP which is the load
balancers IP, and load balancer wants to ping and checks the AMQ
instance status before it routes the client call. That's why Load
balancer is sending ping to AMQ instances.
Any idea, how should I configure AMQ instances.
Thanks!
Vik
The ActiveMQ client sends ping commands anyway, so why does the load
balancer need to shove stuff down the same socket?
On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
wrote:
> Hi,
>
>
>
> I am trying to configure multiple AMQ instances with a Load balancer
> configuration for High availability. Load balancer I am using wants to
> Ping AMQ instances to do the keep-alive check. I tried to set that
> keep-alive port to 61617/61616. every time Load Balancer pings, AMQ
> throws exception because this is not a real OpenWire/SSL
communication;
> but this approach works for few Hrs.
>
>
>
> But after some time AMQ gets tired of this ping and throws OutofMemory
> exception. I guess AMQ is growing in memory every time it throws the
> connection exception and not cleaning up properly.
>
>
>
> My question is how to manage such a configuration where AMQ's are
> configured behind a load balancer.
>
>
>
> Thanks!
>
>
>
>
>
> Vik
>
>
>
>
--
James
-------
http://radio.weblogs.com/0112098/
RE: Load Balancer configuration
Posted by "Dhawan, Vikram (LNG-DAY)" <vi...@lexisnexis.com>.
Hi James,
Actually my AMQ clients are connecting to a VIP which is the load
balancers IP, and load balancer wants to ping and checks the AMQ
instance status before it routes the client call. That's why Load
balancer is sending ping to AMQ instances.
Any idea, how should I configure AMQ instances.
Thanks!
Vik
-----Original Message-----
From: James Strachan [mailto:james.strachan@gmail.com]
Sent: Monday, April 09, 2007 3:00 PM
To: dev@activemq.apache.org
Subject: Re: Load Balancer configuration
The ActiveMQ client sends ping commands anyway, so why does the load
balancer need to shove stuff down the same socket?
On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com>
wrote:
> Hi,
>
>
>
> I am trying to configure multiple AMQ instances with a Load balancer
> configuration for High availability. Load balancer I am using wants to
> Ping AMQ instances to do the keep-alive check. I tried to set that
> keep-alive port to 61617/61616. every time Load Balancer pings, AMQ
> throws exception because this is not a real OpenWire/SSL
communication;
> but this approach works for few Hrs.
>
>
>
> But after some time AMQ gets tired of this ping and throws OutofMemory
> exception. I guess AMQ is growing in memory every time it throws the
> connection exception and not cleaning up properly.
>
>
>
> My question is how to manage such a configuration where AMQ's are
> configured behind a load balancer.
>
>
>
> Thanks!
>
>
>
>
>
> Vik
>
>
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Load Balancer configuration
Posted by James Strachan <ja...@gmail.com>.
The ActiveMQ client sends ping commands anyway, so why does the load
balancer need to shove stuff down the same socket?
On 4/9/07, Dhawan, Vikram (LNG-DAY) <vi...@lexisnexis.com> wrote:
> Hi,
>
>
>
> I am trying to configure multiple AMQ instances with a Load balancer
> configuration for High availability. Load balancer I am using wants to
> Ping AMQ instances to do the keep-alive check. I tried to set that
> keep-alive port to 61617/61616. every time Load Balancer pings, AMQ
> throws exception because this is not a real OpenWire/SSL communication;
> but this approach works for few Hrs.
>
>
>
> But after some time AMQ gets tired of this ping and throws OutofMemory
> exception. I guess AMQ is growing in memory every time it throws the
> connection exception and not cleaning up properly.
>
>
>
> My question is how to manage such a configuration where AMQ's are
> configured behind a load balancer.
>
>
>
> Thanks!
>
>
>
>
>
> Vik
>
>
>
>
--
James
-------
http://radio.weblogs.com/0112098/