You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Loc Truong (JIRA)" <ji...@apache.org> on 2009/09/03 05:52:12 UTC
[jira] Created: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Number of pending messages are greater than 0 but no actual messages on queues
------------------------------------------------------------------------------
Key: AMQ-2372
URL: https://issues.apache.org/activemq/browse/AMQ-2372
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.1.0
Environment: CentOS 5
Reporter: Loc Truong
- Start up AMQ normally
- Running 2m to 3m messages daily
- After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (AMQ-2372) Number of pending messages are
greater than 0 but no actual messages on queues
Posted by Gary Tully <ga...@gmail.com>.
With a 5.3 snapshot, try the configuration from:
http://svn.apache.org/viewvc/activemq/trunk/assembly/src/sample-conf/activemq-scalability.xml?view=markup
2009/9/15 Loc Truong (JIRA) <ji...@apache.org>
>
> [
> https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54261#action_54261]
>
> Loc Truong commented on AMQ-2372:
> ---------------------------------
>
> Nothing works so far, every changes I made still result the same problem at
> 4m messages.
>
> On restarting AMQ I notice:
>
> INFO KahaStore - Kaha Store using data directory
> /usr/local/gtweb/activemq/data/kr-store/state
> INFO AMQPersistenceAdapter - Active data files: []
> WARN AMQPersistenceAdapter - The ReferenceStore is not valid -
> recovering ...
> INFO KahaStore - Kaha Store successfully deleted data
> directory /usr/local/gtweb/activemq/data/kr-store/data
>
>
> Is that something we should look at ?
>
> I'm getting tired of this because it's like being middle of nowhere.
>
> > Number of pending messages are greater than 0 but no actual messages on
> queues
> >
> ------------------------------------------------------------------------------
> >
> > Key: AMQ-2372
> > URL: https://issues.apache.org/activemq/browse/AMQ-2372
> > Project: ActiveMQ
> > Issue Type: Bug
> > Affects Versions: 5.1.0
> > Environment: CentOS 5
> > Reporter: Loc Truong
> > Attachments: Abstract.php, activemq.xml, IPBonus.php
> >
> >
> > - Start up AMQ normally
> > - Running 2m to 3m messages daily
> > - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of
> Pending Messages" showing a positive number (messages available) but nothing
> can get delivered. If I view the messages of a queue there are no messages,
> the "Number Of Pending Messages" stays positive and keeps increasing
> (messages are being sent continuously). The consumers also receives no
> messages (as what happens when I view the queues).
> > At the moment, we have to restart the AMQ manually as it's the only way
> to fix. It's kind of painful tho when it happens on weekend, holidays or
> midnight.
> > FYI, we use Stomp connector.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54151#action_54151 ]
Loc Truong commented on AMQ-2372:
---------------------------------
I'm load testing with the new config of indexBinSize and see how it goes. Anyway, I'm running AMQ5.3 Snapshot but it's still saying " ActiveMQ 5.2.0 JMS Message Broker (localhost) is starting". Is that normal ?
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Loc Truong updated AMQ-2372:
----------------------------
Attachment: Abstract.php
IPBonus.php
activemq.xml
Sure, check the files.
Note that ever since upgrading to 5.3 Snapshot, it's happening more frequent whenever it reaches 2m messages.
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54142#action_54142 ]
Gary Tully commented on AMQ-2372:
---------------------------------
resizing the index persistent hash does lock the store and can take some time. To prevent a resize at run time you can configure the initial size of the hash bins using the "indexBinSize" attribute of the kaha persistence adapter.
You may also want to consider the kahaDB store in the 5.3 snapshot which has more consistent enqueue rate profile. An example configuration can be found at http://svn.apache.org/viewvc/activemq/trunk/assembly/src/sample-conf/activemq-scalability.xml?view=markup
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Bruce Snyder (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bruce Snyder updated AMQ-2372:
------------------------------
Fix Version/s: 5.5.0
(was: 5.4.1)
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Fix For: 5.5.0
>
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=53986#action_53986 ]
Gary Tully commented on AMQ-2372:
---------------------------------
think this is resolved on trunk.
Can you try out the current 5.3-SNAPSHOT?
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQ-2372) Number of pending messages
are greater than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54426#action_54426 ]
Loc Truong edited comment on AMQ-2372 at 9/22/09 5:23 AM:
----------------------------------------------------------
Thought it would be good to update with you that I switched to use kahaPersistenceAdapter and the enqueued and dequeued messages are 6,5m now. Consider that's a big different with usual 2m-3m.
I'll keep you posted.
was (Author: deathemperor):
Thought it would be good to update with you that I switched to use kahaPersistenceAdapter and the enqueued and dequeued messages are 6,5m now. Consider that's a big different with usual 2m-3m.
I'll update more about this.
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=53987#action_53987 ]
Loc Truong commented on AMQ-2372:
---------------------------------
I'll absolutely try that but could you tell me what's happening ?
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54054#action_54054 ]
Gary Tully commented on AMQ-2372:
---------------------------------
can you post your actiivemq.xml and any test code - or an entire test case, this will help reproduce?
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54426#action_54426 ]
Loc Truong commented on AMQ-2372:
---------------------------------
Thought it would be good to update with you that I switched to use kahaPersistenceAdapter and the enqueued and dequeued messages are 6,5m now. Consider that's a big different with usual 2m-3m.
I'll update more about this.
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54040#action_54040 ]
Loc Truong commented on AMQ-2372:
---------------------------------
Confirming the snapshot doesn't fix this. It still happens after 5 million of messages.
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54134#action_54134 ]
Loc Truong commented on AMQ-2372:
---------------------------------
My assumption on this problem may be because of the resize hash bin problem. Every time this happens I check the AMQ log and found no error but the resizing logs:
Resize hash bins queue-data_queue#3a#2f#2fipbonus from 1024 to 2048
Can you point me to where I can find additional log if there is ?
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2372) Number of pending messages are greater
than 0 but no actual messages on queues
Posted by "Loc Truong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=54261#action_54261 ]
Loc Truong commented on AMQ-2372:
---------------------------------
Nothing works so far, every changes I made still result the same problem at 4m messages.
On restarting AMQ I notice:
INFO KahaStore - Kaha Store using data directory /usr/local/gtweb/activemq/data/kr-store/state
INFO AMQPersistenceAdapter - Active data files: []
WARN AMQPersistenceAdapter - The ReferenceStore is not valid - recovering ...
INFO KahaStore - Kaha Store successfully deleted data directory /usr/local/gtweb/activemq/data/kr-store/data
Is that something we should look at ?
I'm getting tired of this because it's like being middle of nowhere.
> Number of pending messages are greater than 0 but no actual messages on queues
> ------------------------------------------------------------------------------
>
> Key: AMQ-2372
> URL: https://issues.apache.org/activemq/browse/AMQ-2372
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: CentOS 5
> Reporter: Loc Truong
> Attachments: Abstract.php, activemq.xml, IPBonus.php
>
>
> - Start up AMQ normally
> - Running 2m to 3m messages daily
> - After 36 to 72 hours, AMQ starts to act abnormally with "Number Of Pending Messages" showing a positive number (messages available) but nothing can get delivered. If I view the messages of a queue there are no messages, the "Number Of Pending Messages" stays positive and keeps increasing (messages are being sent continuously). The consumers also receives no messages (as what happens when I view the queues).
> At the moment, we have to restart the AMQ manually as it's the only way to fix. It's kind of painful tho when it happens on weekend, holidays or midnight.
> FYI, we use Stomp connector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.