You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mserrano <ma...@attivio.com> on 2011/06/03 16:04:23 UTC

messages sometimes lost during pure master/slave failover

Hi,

I have a pure master/slave configuration.  I am using 5.3.2 but have
observed the same problem in testing 5.5.0.

I have an intermittent issue where a message will appear to be successfully
sent (the producer send() method returns without error) but will be lost. 
This occurs sometimes during a master/slave failover.  The master is being
shutdown in a controlled fashion (i.e. the process is not being killed) so
the broker stop() method is being called.

When the problem occurs I see the following error logs (formatting altered
below) on the master:

MasterBroker - Slave encountered an exception processing
MessageDispatchNotification 
  java.net.SocketException - socket closed


At the same time the slave sees:

ERROR MasterConnector - Network connection between vm://jmsBroker#0 and
tcp://localhost/127.0.0.1:17046 shutdown: null
WARN  BrokerService - Master Failed - starting all connectors


I'm not currently running with persistence on but have seen it when it is on
as well.  The thing that confuses me is that I thought the reflection of the
master state to the slave was guaranteed to occur before returning to the
client.  From the documentation:

The master broker will only respond to a client when a message exchange has
been successfully passed to the slave.

Thanks,
Martin

--
View this message in context: http://activemq.2283324.n4.nabble.com/messages-sometimes-lost-during-pure-master-slave-failover-tp3570898p3570898.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: messages sometimes lost during pure master/slave failover

Posted by mserrano <ma...@attivio.com>.
Ok... I figured out what was going on.  I will file a ticket on this shortly.  

When the master broker is undergoing a controlled shutdown, the
BrokerService.stop() method stops in this order:

* services
* connectors
* registered vm transports
* broker

So there is a period where the broker will still process sends after other
(apparently necessary) facilities have been shutdown.  I have not followed
the code paths to understand exactly what goes wrong, but I traced enough to
tell that messages sent in this interval can disappear.  So this appears to
only be an issue during a controlled shutdown.  Process death should not
cause this problem.

I'm currently working around this by having the BrokerService set a stopping
flag and having the MasterBroker check this flag and reject sends (with a
new exception class) if true.  My client code then detects this case and
just retries until the failover is complete.


--
View this message in context: http://activemq.2283324.n4.nabble.com/messages-sometimes-lost-during-pure-master-slave-failover-tp3570898p3572646.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: messages sometimes lost during pure master/slave failover

Posted by mserrano <ma...@attivio.com>.
We are using the ActiveMQ in an embedded mode in our application.  It may be
difficult to reproduce in a test case because of the intermittent nature...

On inspection of the code below (from MasterBroker.java), it seems that an
exception syncing to the slave is logged but otherwise ignored.  Doesn't
this allow the send to return to the client successfully even if the sync
didn't occur?  This seems like a bad thing.


  /**
   * @param context
   * @param message
   * @throws Exception
   */
  @Override
  public void send(ProducerBrokerExchange producerExchange, Message message)
throws Exception {
    /**
     * A message can be dispatched before the super.send() method returns so
-
     * here the order is switched to avoid problems on the slave with
     * receiving acks for messages not received yet
     */
    sendSyncToSlave(message);
    super.send(producerExchange, message);
  }

...

    protected void sendSyncToSlave(Command command) {
        try {
            Response response = (Response)slave.request(command);
            if (response.isException()) {
                ExceptionResponse er = (ExceptionResponse)response;
                LOG.error("Slave Failed", er.getException());
            }
        } catch (Throwable e) {
            LOG.error("Slave Failed", e);
        }
    }




--
View this message in context: http://activemq.2283324.n4.nabble.com/messages-sometimes-lost-during-pure-master-slave-failover-tp3570898p3571086.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: messages sometimes lost during pure master/slave failover

Posted by Dejan Bosanac <de...@nighttale.net>.
Any chance you can create a test case to reproduce this?

Regards
-- 
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net

Connect at CamelOne <http://camelone.com/> May 24-26

The Open Source Integration Conference



On Fri, Jun 3, 2011 at 4:22 PM, mserrano <ma...@attivio.com> wrote:

> Sorry, the master exception log message includes a modification we made to
> the logging on slave failure to distinguish failure conditions (see
> AMQ-2459).
>
> Normally it would say "Slave Failed" with the exception.  Full exception
> trace returned from slave:
>
>
> java.net.SocketException: socket closed
>         at java.net.SocketInputStream.socketRead0(Native Method)
>        at java.net.SocketInputStream.read(SocketInputStream.java:129)
>        at
>
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
>        at
>
> org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
>        at
>
> org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
>        at
>
> org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
>        at java.io.DataInputStream.readInt(DataInputStream.java:370)
>        at
>
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
>        at
>
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
>        at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
>        at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
>        at java.lang.Thread.run(Thread.java:662)
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/messages-sometimes-lost-during-pure-master-slave-failover-tp3570898p3570950.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: messages sometimes lost during pure master/slave failover

Posted by mserrano <ma...@attivio.com>.
Sorry, the master exception log message includes a modification we made to
the logging on slave failure to distinguish failure conditions (see
AMQ-2459).

Normally it would say "Slave Failed" with the exception.  Full exception
trace returned from slave:


java.net.SocketException: socket closed
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
	at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
	at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
	at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
	at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
	at java.lang.Thread.run(Thread.java:662)


--
View this message in context: http://activemq.2283324.n4.nabble.com/messages-sometimes-lost-during-pure-master-slave-failover-tp3570898p3570950.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.