You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Chris Robison (JIRA)" <ji...@apache.org> on 2012/05/18 22:59:12 UTC
[jira] [Created] (AMQ-3851) Infinite TemqQueue advisory messages
Chris Robison created AMQ-3851:
----------------------------------
Summary: Infinite TemqQueue advisory messages
Key: AMQ-3851
URL: https://issues.apache.org/jira/browse/AMQ-3851
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.6.0
Reporter: Chris Robison
I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467734#comment-13467734 ]
Dejan Bosanac commented on AMQ-3851:
------------------------------------
Hi,
I see you're using some kind of a custom <solidInstanceCachedLDAPAuthorizationMap/> authorization plugin. Are you sure the problem isn't somewhere in there.
Can you give Cached LDAP Authorization Module a try http://activemq.apache.org/cached-ldap-authorization-module.html and see if it helps.
Also, can you share more info on your further testings and how do you reproduce the problem
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468440#comment-13468440 ]
Dejan Bosanac commented on AMQ-3851:
------------------------------------
Thanks for confirming. I see one potential problem that can cause it. I'll try to reproduce it now.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467783#comment-13467783 ]
Chris Robison commented on AMQ-3851:
------------------------------------
No, I just made some changes so that the regular cachedLDAPAuthorizationMap would work with Active Directory. Those changes have now been put into trunk so I could still go back to using the regular one and still see the problem. In a previous comment I said that I wrote an extremely simple plugin that always returned the same ACLs and was still able to reproduce the problem. To reproduce the issue, I just setup a simple request/reply system using text messages. I believe I've attached the code I used. There was never a set number of messages I had to send before the flooding started. At first I was only able to reproduce the problem going over VPN, but eventually I was able to reproduce the problem between VMs.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: mstmip103-activemq.xml
mssd5d101-activemq.xml
ActiveMQ server configuration files
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286695#comment-13286695 ]
Chris Robison commented on AMQ-3851:
------------------------------------
After more testing, I think there is a race condition in AuthorizationMap. Instead of going to LDAP, I defined the authorization entries in configuration and was able to reproduce the issue.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dejan Bosanac resolved AMQ-3851.
--------------------------------
Resolution: Fixed
Fix Version/s: 5.8.0
It should be fixed now with svn revision 1393988.
I did some refactoring of AdvisoryBroker to handle addDestinationInfo case properly.
Can you test the latest snapshot
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.8-SNAPSHOT/apache-activemq-5.8-20121004.112554-4-bin.tar.gz
and report back if you still see the problem.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Fix For: 5.8.0
>
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Mark Brightwell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468391#comment-13468391 ]
Mark Brightwell commented on AMQ-3851:
--------------------------------------
Forgot to add: we are using the SimpleAuthenticationPlugin.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: ActiveMqRequestReceiver.cs
ActiveMqMessagingProvider.cs
This is the code I use to wrap the ActiveMQ library that all my other software uses.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: (was: mstmip103-activemq.xml)
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Mark Brightwell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468386#comment-13468386 ]
Mark Brightwell commented on AMQ-3851:
--------------------------------------
Hi.
I think we are seeing the same issue with fuse version 5.5.1-fuse-08-15 (was already the case for previous versions). We are running 2 brokers is a cluster with dynamic temporary queue forwarding. After some time, the 2 brokers get into some "vicious circle", whereby the same temporary queues are repeatedly added then removed from each of the brokers. As Chris mentions, this seems to use up a single thread on each process, presumably the thread removing & creating this temporary queues. It looks like the temporary queues are simply passed back and forth between the 2 brokers, and never get removed correctly. Only a restart of the brokers fixes the problem.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: web-portal-code-snippet.cs
Snippet of code used in the web portal to send messages and receive responses.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Dejan Bosanac (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dejan Bosanac reassigned AMQ-3851:
----------------------------------
Assignee: Dejan Bosanac
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: (was: mssd5d101-activemq.xml)
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: run-2012-05-30.snapshot
I profiled the server instance using YourKit while the flood was taking place. I've included the trace to hopefully help in the debugging process. I've found that when I disable any kind of authorization that things seems to work fine. Looking at the trace, it looks like there might be a problem related to the AuthorizationBroker, but I'm not sure.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: Log.zip
Logs
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: Log.zip
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Robison updated AMQ-3851:
-------------------------------
Attachment: mstmip103-activemq.xml
mssd5d101-activemq.xml
ActiveMQ configuration files
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3851) Infinite TemqQueue advisory messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286904#comment-13286904 ]
Chris Robison commented on AMQ-3851:
------------------------------------
I've done further testing, I now think the problem is higher up than the DefaultAuthorizationMap in the AuthorizationBroker. I created a very simple class that implements AuthorizationMap that always returns a static set of ACLs and I was able to reproduce the flooding.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (AMQ-3851) Infinite TemqQueue advisory
messages
Posted by "Chris Robison (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286695#comment-13286695 ]
Chris Robison edited comment on AMQ-3851 at 5/31/12 4:23 PM:
-------------------------------------------------------------
After more testing, I think there is a race condition in DefaultAuthorizationMap. Instead of going to LDAP, I defined the authorization entries in configuration and was able to reproduce the issue.
was (Author: chrisdrobison):
After more testing, I think there is a race condition in AuthorizationMap. Instead of going to LDAP, I defined the authorization entries in configuration and was able to reproduce the issue.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (AMQ-3851) Infinite TemqQueue advisory
messages
Posted by "Mark Brightwell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468391#comment-13468391 ]
Mark Brightwell edited comment on AMQ-3851 at 10/3/12 6:41 PM:
---------------------------------------------------------------
Forgot to add: we are using the SimpleAuthenticationPlugin with a tempDestinationAuthorizationEntry in the AuthorizationMap.
was (Author: mbright):
Forgot to add: we are using the SimpleAuthenticationPlugin.
> Infinite TemqQueue advisory messages
> ------------------------------------
>
> Key: AMQ-3851
> URL: https://issues.apache.org/jira/browse/AMQ-3851
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0
> Reporter: Chris Robison
> Assignee: Dejan Bosanac
> Attachments: ActiveMqMessagingProvider.cs, ActiveMqRequestReceiver.cs, Log.zip, mssd5d101-activemq.xml, mstmip103-activemq.xml, run-2012-05-30.snapshot, web-portal-code-snippet.cs
>
>
> I'm seeing a behavior that looks like a race condition of some kind. I have two brokers (MSSD5D101, MSTMIP103). MSSD5D101 has a duplex network connector connecting it to MSTMIP103. The two machines are in different US states so there is a VPN in between. I have two consumer services that reply to queued messages. One consumer (WISD5P101) connects to MSSD5D101 and the other consumer (POSD5P101) connects to MSTMIP103. I also have a web portal that sends messages out to different queues expecting a response. The web portal, and two consumers are all .NET based.
> What is happening is that after some number of messages, something goes crazy and floods the network of brokers with advisory messages pertaining to temp queues being added or removed. By restarting one or the other of the ActiveMQ servers, things return to normal. But, while the flood is happening, one of the machines will be at 100% CPU and become worthless. I will be attaching logs and some code that I'm using to interact with ActiveMQ.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira