You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Tom (Created) (JIRA)" <ji...@apache.org> on 2012/04/11 16:57:19 UTC
[jira] [Created] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Consumers frequently fail to reconnect after broker outage/failover.
--------------------------------------------------------------------
Key: AMQNET-379
URL: https://issues.apache.org/jira/browse/AMQNET-379
Project: ActiveMQ .Net
Issue Type: Bug
Components: ActiveMQ
Affects Versions: 1.5.3
Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
Apache.NMS.ActiveMQ v1.5.3
Windows7 64bit, .Net 4
Java 64bit 1.6.0_31, -server VM
Reporter: Tom
Assignee: Jim Gomes
When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
Broker config is attached.
Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
Test code available at https://github.com/chillitom/NmsFailoverTest
To run:
- compile with VS2010
- edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
- configure Broker2Address to point to a second broker or a non-existent broker.
- run the NmsFailoverTest project
- producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
- repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Timothy Bish (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish resolved AMQNET-379.
---------------------------------
Resolution: Fixed
Fixed on trunk and the 1.5.x branch
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Timothy Bish
> Fix For: 1.5.5, 1.6.0
>
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Tom (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom updated AMQNET-379:
-----------------------
Attachment: activemq.xml
attached broker configuration
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Jim Gomes
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Timothy Bish (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252459#comment-13252459 ]
Timothy Bish commented on AMQNET-379:
-------------------------------------
An initial observation here is that the configuration has the client
connections disabling the inactivity monitor. This could be the cause
of the consumer count decreasing over time with a remote broker.
Without the inactivity monitor checking connection liveness a pure
consumer could miss the fact that the broker connection died since its
just listening for messages from the broker and its not guaranteed that
it will notice the socket connection die. If they want reliable
consumer failover they need to enable the inactivity monitor.
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Timothy Bish
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Timothy Bish (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish updated AMQNET-379:
--------------------------------
Fix Version/s: 1.6.0
1.5.5
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Timothy Bish
> Fix For: 1.5.5, 1.6.0
>
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Timothy Bish (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253657#comment-13253657 ]
Timothy Bish commented on AMQNET-379:
-------------------------------------
A v1.5.5 SNAPSHOT build with the latest fixes is here: http://people.apache.org/~tabish/nms-1.5.0 for you to test with.
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Timothy Bish
> Fix For: 1.5.5, 1.6.0
>
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Timothy Bish (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish reassigned AMQNET-379:
-----------------------------------
Assignee: Timothy Bish (was: Jim Gomes)
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Timothy Bish
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQNET-379) Consumers frequently fail to
reconnect after broker outage/failover.
Posted by "Tom (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQNET-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252488#comment-13252488 ]
Tom commented on AMQNET-379:
----------------------------
Thanks Tim,
I was unsure what effect that would have and it does sound like a likely culprit, however, after removing the setting we're still seeing the failures at the same rate as before.
The default for inactivityMonitorDuration is 30,000ms so we'd expect them to kick back in pretty swiftly but we're just not seeing it.
> Consumers frequently fail to reconnect after broker outage/failover.
> --------------------------------------------------------------------
>
> Key: AMQNET-379
> URL: https://issues.apache.org/jira/browse/AMQNET-379
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: ActiveMQ
> Affects Versions: 1.5.3
> Environment: brokers 5.4.2, 5.6-snapshot and 5.5.1-fuse-03-06
> Apache.NMS.ActiveMQ v1.5.3
> Windows7 64bit, .Net 4
> Java 64bit 1.6.0_31, -server VM
> Reporter: Tom
> Assignee: Timothy Bish
> Attachments: activemq.xml
>
>
> When using the failover transport we frequently see consumers fail to reconnect after failover between brokers or a broker outage. This behaviour is something we have been able to easily replicate in a test environment.
> The failure seems exagerated when connecting to a remote broker, we've tried to replicate it running the producers and consumers on the same local host but with no joy.
> Processes/connections that work purely as producers don't experience the same problem. In our tests frequently failing can cause all consumers to disconnect. The failure doesn't occurr unless broker is under load i.e. when producers must be active for failure to occur.
> The NMS client's connection threads and failover threads appear to have died/ended after a consumer fails to failover.
> Broker config is attached.
> Client connections use AsyncSend = true and have WatchTopicAdvisories = false (to avoid AMQNET-371). We've also tested this with and without DispatchAsync = true .
> Error messages - nothing obvious in NMS Trace or broker logs though we have on occasion whilst performing this test seen the messages detailed in AMQNET-370.
> Test code available at https://github.com/chillitom/NmsFailoverTest
> To run:
> - compile with VS2010
> - edit App.config in NmsFailoverTest project to point Broker1Address to a valid broker address on a different box.
> - configure Broker2Address to point to a second broker or a non-existent broker.
> - run the NmsFailoverTest project
> - producer stats will appear on console and periodically the broker will be queried for the number of consumer it sees, this will be printed to the console (assumes broker with <statisticsBrokerPlugin/> plugin)
> - repeatedly stop and start Broker1 and witness the decline in consumers after some failovers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira