static network connection lost and not recreated

[Kai Hackemesser <ka...@gmail.com>]

Hello,

I have set up a static NC between two brokers of Version 5.5.1, and we have
now seen twice within our preproduction test that the connection was lost
and not reestablished automatically. We could not establish it through JMX
using the NC's start operation but had to restart one or both brokers. Is
this a known bug or a  configuration issue?

Cheers,
Kai

Re: static network connection lost and not recreated

[Kai Hackemesser <ka...@gmail.com>]

Hi

The symptoms of our configuration got more and more weird. We use currently
topics where we have up to 5 producers and about the same number of
consumers, mostly spread over the two boxes having a broker each. We did a
run overnight and it ended with one broker falling over with not enough
memory (our fault)) but the other one is in a very inconsistent state. It
doesn't accept incoming connections from the other box (the clients are
configured to try first the local then the other broker) and I can see very
high numbers of subscribers on one of the topics (a value of up to 4 would
make sense but JMX currently shows 314 - sometimes the value goes up to
above 3000 for a second). the clients on the remote box loop in message
like:

INFO [02 Feb 2012 21:33:36,855]: Successfully reconnected to
tcp://otherbox:61616
WARN [02 Feb 2012 21:33:36,855]: Transport (otherbox/10.100.88.17:61616)
failed to tcp://kd-sage3:61616 , attempting to automatically reconnect due
to: java.net.SocketException: Socket closed

On the broker side I got:

2012-02-02 21:33:36,858 | INFO  | Transport failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.100.88.16:36006
2012-02-02 21:33:36,914 | INFO  | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport | Async Exception
Handler

We use a Spring connection factory to connect, which is set to reconnect
automatically if connection is lost:

    @Bean
    public CachingConnectionFactory cachingConnectionFactory() {
        CachingConnectionFactory factory = new CachingConnectionFactory();
        factory.setReconnectOnException(true);
        factory.setTargetConnectionFactory(jmsConnectionFactory());
        return factory;
    }

    @Bean
    public ActiveMQConnectionFactory jmsConnectionFactory() {
        ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();

factory.setBrokerURL("failover:(tcp://localhost:61616,tcp://otherbox:61616)?randomize=false");
        return factory;
    }

The only solution we found was to restart the broker, but as this stuff is
planned to go into production we want to serve them a configuration that
can have the broker running for months without need to interfere with it.

Cheers,
Kai


2012/2/2 Torsten Mielke <to...@fusesource.com>

> Hi,
>
> This should generally work alright.
> You may want to configure for debug logging on org.apache.activemq.network
> and inspect the logging output carefully when the bridge should get
> restored.
>
>
> Torsten Mielke
> torsten@fusesource.com
> tmielke@blogspot.com
>
>
> On Feb 2, 2012, at 1:57 AM, Kai Hackemesser wrote:
>
> > Hello,
> >
> > I have set up a static NC between two brokers of Version 5.5.1, and we
> have
> > now seen twice within our preproduction test that the connection was lost
> > and not reestablished automatically. We could not establish it through
> JMX
> > using the NC's start operation but had to restart one or both brokers. Is
> > this a known bug or a  configuration issue?
> >
> > Cheers,
> > Kai
>
>

Re: static network connection lost and not recreated

[Torsten Mielke <to...@fusesource.com>]

Hi, 

This should generally work alright. 
You may want to configure for debug logging on org.apache.activemq.network 
and inspect the logging output carefully when the bridge should get restored. 


Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com


On Feb 2, 2012, at 1:57 AM, Kai Hackemesser wrote:

> Hello,
> 
> I have set up a static NC between two brokers of Version 5.5.1, and we have
> now seen twice within our preproduction test that the connection was lost
> and not reestablished automatically. We could not establish it through JMX
> using the NC's start operation but had to restart one or both brokers. Is
> this a known bug or a  configuration issue?
> 
> Cheers,
> Kai