You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Martin Serrano (JIRA)" <ji...@apache.org> on 2011/06/10 16:06:58 UTC
[jira] [Created] (AMQ-3364) Broker can lose messages during
master/slave failover when master undergoes a controlled shutdown
Broker can lose messages during master/slave failover when master undergoes a controlled shutdown
-------------------------------------------------------------------------------------------------
Key: AMQ-3364
URL: https://issues.apache.org/jira/browse/AMQ-3364
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.5.0, 5.4.2
Reporter: Martin Serrano
Priority: Critical
I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
* services
* connectors
* registered vm transports
* broker
So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (AMQ-3364) Broker can lose messages during pure
master/slave failover when master undergoes a controlled shutdown
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish closed AMQ-3364.
-----------------------------
Resolution: Won't Fix
Pure master/slave removed in upcoming v5.8.0
> Broker can lose messages during pure master/slave failover when master undergoes a controlled shutdown
> ------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3364
> URL: https://issues.apache.org/jira/browse/AMQ-3364
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2, 5.5.0
> Reporter: Martin Serrano
> Priority: Critical
>
> I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
> * services
> * connectors
> * registered vm transports
> * broker
> So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
> This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
> I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete. It seems like there should be a better and more integrated solution that does not require the client code to handle this but is handled inside of the FailoverTransport code on the client's behalf.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3364) Broker can lose messages during pure
master/slave failover when master undergoes a controlled shutdown
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully updated AMQ-3364:
----------------------------
Summary: Broker can lose messages during pure master/slave failover when master undergoes a controlled shutdown (was: Broker can lose messages during master/slave failover when master undergoes a controlled shutdown)
> Broker can lose messages during pure master/slave failover when master undergoes a controlled shutdown
> ------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3364
> URL: https://issues.apache.org/jira/browse/AMQ-3364
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2, 5.5.0
> Reporter: Martin Serrano
> Priority: Critical
>
> I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
> * services
> * connectors
> * registered vm transports
> * broker
> So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
> This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
> I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete. It seems like there should be a better and more integrated solution that does not require the client code to handle this but is handled inside of the FailoverTransport code on the client's behalf.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3364) Broker can lose messages during
master/slave failover when master undergoes a controlled shutdown
Posted by "Martin Serrano (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047203#comment-13047203 ]
Martin Serrano commented on AMQ-3364:
-------------------------------------
Some discussion of this during my initial investigation is [here|http://activemq.2283324.n4.nabble.com/messages-sometimes-lost-during-pure-master-slave-failover-td3570898.html]
> Broker can lose messages during master/slave failover when master undergoes a controlled shutdown
> -------------------------------------------------------------------------------------------------
>
> Key: AMQ-3364
> URL: https://issues.apache.org/jira/browse/AMQ-3364
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2, 5.5.0
> Reporter: Martin Serrano
> Priority: Critical
>
> I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
> * services
> * connectors
> * registered vm transports
> * broker
> So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
> This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
> I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete. It seems like there should be a better and more integrated solution that does not require the client code to handle this but is handled inside of the FailoverTransport code on the client's behalf.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3364) Broker can lose messages during
master/slave failover when master undergoes a controlled shutdown
Posted by "Martin Serrano (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Serrano updated AMQ-3364:
--------------------------------
Description:
I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
* services
* connectors
* registered vm transports
* broker
So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete. It seems like there should be a better and more integrated solution that does not require the client code to handle this but is handled inside of the FailoverTransport code on the client's behalf.
was:
I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
* services
* connectors
* registered vm transports
* broker
So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete.
> Broker can lose messages during master/slave failover when master undergoes a controlled shutdown
> -------------------------------------------------------------------------------------------------
>
> Key: AMQ-3364
> URL: https://issues.apache.org/jira/browse/AMQ-3364
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2, 5.5.0
> Reporter: Martin Serrano
> Priority: Critical
>
> I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
> * services
> * connectors
> * registered vm transports
> * broker
> So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
> This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
> I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete. It seems like there should be a better and more integrated solution that does not require the client code to handle this but is handled inside of the FailoverTransport code on the client's behalf.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3364) Broker can lose messages during pure
master/slave failover when master undergoes a controlled shutdown
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067597#comment-13067597 ]
Gary Tully commented on AMQ-3364:
---------------------------------
one thought, there is a shutdownOnSlaveFailure attribute on a broker such that a controlled shutdown of the master can be triggered by a slave failure or shutdown. I wonder if shutting down the slave can provide a workaround.
But the shutdown order does seem to be problematic in this case. The connectors and active connections should be terminated first so that the clients can see the failure early and failover kicks in.
> Broker can lose messages during pure master/slave failover when master undergoes a controlled shutdown
> ------------------------------------------------------------------------------------------------------
>
> Key: AMQ-3364
> URL: https://issues.apache.org/jira/browse/AMQ-3364
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2, 5.5.0
> Reporter: Martin Serrano
> Priority: Critical
>
> I see this problem consistently when a producer is continuously sending messages and the master is shutdown in a controlled fashion. When the master broker is undergoing a controlled shutdown, the BrokerService.stop() method stops things in this order:
> * services
> * connectors
> * registered vm transports
> * broker
> So there is a period where the broker will still process sends after other (apparently necessary) facilities have been shutdown. I have not followed the code paths to understand exactly what goes wrong, but I traced enough to tell that messages sent in this interval can disappear. That is, the client send call will return without error but after failover the slave will not replay the message.
> This appears to only be an issue during a controlled shutdown. Process death should not cause this problem.
> I'm currently working around this by having the BrokerService set a stopping flag and having the MasterBroker check this flag and reject sends (with a new exception class) if true. My client code then detects this case and just retries until the failover is complete. It seems like there should be a better and more integrated solution that does not require the client code to handle this but is handled inside of the FailoverTransport code on the client's behalf.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira