Nms Reconnect

[Bryan Murphy <bm...@gmail.com>]

Hey guys,
I'm still having trouble with the reliability of connections to our message
bus server when using NMS.  Our ActiveMQ server dies with with OutOfMemory
exceptions about once per week.  At least until 5.2 comes out, I'd like to
automate a nightly restart of the ActiveMQ server, and this leads to my
other problem:

I just can't seem to figure out how to get my .NET services to automatically
reconnect to the message bus when it comes back up.  ActiveMQ server goes
down, clients lose connection, ActiveMQ server comes up, clients never
reconnect until I manually restart them.

I don't know what I'm doing wrong, I've tried setting prefetch values,
transport timeouts, connection timeouts, failover, nothing works.  I have a
trunk build that is about a month old, but I can easily make a new build.

What do I need to do to ensure that our .NET clients will reliably
re-connect to the server when I bounce it?

Thanks,
Bryan

Re: Nms Reconnect

[mkeenan <ke...@p2sol.com>]

semog wrote:
> 
> Like you, we can get messages to go for a while, and then the consumer
> just stops getting new messages.  The producer can still send messages,
> though.  It's very odd and frustrating.  A couple of guys on my team are
> working on tracking it down.
> 

It is 100% my fault and I take full responsibility, but as a warning for
others I will openly admit I made the mistake of integrating AMQ/NMS into
our production application and pushing it out the door based on 1) my
limited AMQ/NMS proof-of-concept code and 2) my high opinion of Apache
projects.

Of course, the simple cases of reading and writing messages worked
flawlessly, and the performance was superb. Both PostgreSQL and MySQL worked
without a hitch, along with all the other various broker configurations I
tried.

However, the "stuck message" problem has been a thorn in my side for months
and as a result of my poor decision to rush integration testing I am now
paying the price of babysitting AMQ and our service, restarting as necessary
to push things along.

All is not lost, as this has been a valuable lesson for me. I am hopeful
that the delivery problems will get ironed out eventually and in the
meantime I will continue working around issues as necessary.

For everyone else out there-- here's a reminder from the pages of "Software
Development 101":

Before you choose *any* third party solution (open source or otherwise) to
integrate with your own code, do yourself a favor and don't skimp on the due
diligence phase.  Make sure it works on production hardware, with production
volumes of data, for a sufficient time period.


-- 
View this message in context: http://www.nabble.com/Nms-Reconnect-tp20582834p20605466.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Nms Reconnect

[semog <e....@gmail.com>]

We are also having message flow problems, but our problems are with a 100%
Java solution.  No NMS or .NET stuff in the mix.  Our current suspects are
either ActiveMQ 5.1.0 or JBOSS, or some combination thereof.  Like you, we
can get messages to go for a while, and then the consumer just stops getting
new messages.  The producer can still send messages, though.  It's very odd
and frustrating.  A couple of guys on my team are working on tracking it
down.



mkeenan wrote:
> 
> 
> 
> Bryan Murphy-4 wrote:
>> 
>> What do I need to do to ensure that our .NET clients will reliably
>> re-connect to the server when I bounce it?
>> 
> 
> You might want to take a look at the Spring.NET source code.  I'm using
> message listener containers and they survive an AMQ broker bounce because
> there is reconnect logic in that code. You can set the frequency and
> timeout to attempt reconnect as often as you want for as long as you want. 
> The Spring.Messaging.NMS namespace is relatively small, so hopefully the
> effort to find and extract the pieces you might want would be minimal.
> 
> Having said that, we are having our own significant challenges with
> reliable messaging.  I don't know where the problem is [our code,
> Spring.NET, or NMS] so I can't open any useful JIRA just yet.  The biggest
> issue is that messages stop flowing and we either have to restart AMQ or
> restart our service (which produces and consumes) in order to get the
> messages going again.
> 
> 

-- 
View this message in context: http://www.nabble.com/Nms-Reconnect-tp20582834p20593472.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Nms Reconnect

[mkeenan <ke...@p2sol.com>]

Bryan Murphy-4 wrote:
> 
> What do I need to do to ensure that our .NET clients will reliably
> re-connect to the server when I bounce it?
> 

You might want to take a look at the Spring.NET source code.  I'm using
message listener containers and they survive an AMQ broker bounce because
there is reconnect logic in that code. You can set the frequency and timeout
to attempt reconnect as often as you want for as long as you want.  The
Spring.Messaging.NMS namespace is relatively small, so hopefully the effort
to find and extract the pieces you might want would be minimal.

Having said that, we are having our own significant challenges with reliable
messaging.  I don't know where the problem is [our code, Spring.NET, or NMS]
so I can't open any useful JIRA just yet.  The biggest issue is that
messages stop flowing and we either have to restart AMQ or restart our
service (which produces and consumes) in order to get the messages going
again.

-- 
View this message in context: http://www.nabble.com/Nms-Reconnect-tp20582834p20584909.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Nms Reconnect

[Jim Gomes <e....@gmail.com>]

Hi Bryan,

I would definitely suggest updating to the latest trunk build.  I haven't
started my reliable testing/verification as I have been tasked with other
items.  I would think that the new failover protocol support should help in
this regard.  Prior to that being added, there was no support for automatic
reconnect.  I haven't played with the failover stuff yet, but I would try
listing the same server twice in the failover list so that the code will try
to reconnect to the same server if it gets bounced.  Don't know if that will
work, but it worth a try.

- Jim

On Wed, Nov 19, 2008 at 7:55 AM, Bryan Murphy <bm...@gmail.com> wrote:

> Hey guys,
> I'm still having trouble with the reliability of connections to our message
> bus server when using NMS.  Our ActiveMQ server dies with with OutOfMemory
> exceptions about once per week.  At least until 5.2 comes out, I'd like to
> automate a nightly restart of the ActiveMQ server, and this leads to my
> other problem:
>
> I just can't seem to figure out how to get my .NET services to
> automatically
> reconnect to the message bus when it comes back up.  ActiveMQ server goes
> down, clients lose connection, ActiveMQ server comes up, clients never
> reconnect until I manually restart them.
>
> I don't know what I'm doing wrong, I've tried setting prefetch values,
> transport timeouts, connection timeouts, failover, nothing works.  I have a
> trunk build that is about a month old, but I can easily make a new build.
>
> What do I need to do to ensure that our .NET clients will reliably
> re-connect to the server when I bounce it?
>
> Thanks,
> Bryan
>