You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org> on 2012/08/09 10:39:18 UTC
[jira] [Created] (AMQ-3966) Messages get stuck in queue
Robin Kåveland Hansen created AMQ-3966:
------------------------------------------
Summary: Messages get stuck in queue
Key: AMQ-3966
URL: https://issues.apache.org/jira/browse/AMQ-3966
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.6.0, 5.5.1, 5.5.0
Environment: java version "1.6.0_32"
Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 5.8 (Final)
Kernel \r on an \m
Reporter: Robin Kåveland Hansen
Priority: Critical
The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a `SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'` results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (AMQ-3966) Messages get stuck in queue
Posted by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431743#comment-13431743 ]
Robin Kåveland Hansen edited comment on AMQ-3966 at 8/9/12 11:50 AM:
---------------------------------------------------------------------
Moving the retrievable messages to a different queue seemed to resolve the problem in this case - once the retrievable messages were moved, the missing messages became visible on the queue. Could this be related to caching in some way?
was (Author: kaaveland):
Moving the retrievable messages to a different queue seemed to resolve the problem in this case - once they were moved, the missing messages became visible on the queue. Could this be related to caching in some way?
> Messages get stuck in queue
> ---------------------------
>
> Key: AMQ-3966
> URL: https://issues.apache.org/jira/browse/AMQ-3966
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0, 5.5.1, 5.6.0
> Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
> Reporter: Robin Kåveland Hansen
> Priority: Critical
> Attachments: brokerlog.zip
>
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3966) Messages get stuck in queue
Posted by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431743#comment-13431743 ]
Robin Kåveland Hansen commented on AMQ-3966:
--------------------------------------------
Moving the retrievable messages to a different queue seemed to resolve the problem in this case - once they were moved, the missing became visible on the queue. Could this be related to caching in some way?
> Messages get stuck in queue
> ---------------------------
>
> Key: AMQ-3966
> URL: https://issues.apache.org/jira/browse/AMQ-3966
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0, 5.5.1, 5.6.0
> Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
> Reporter: Robin Kåveland Hansen
> Priority: Critical
> Attachments: brokerlog.zip
>
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3966) Messages get stuck in queue
Posted by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robin Kåveland Hansen updated AMQ-3966:
---------------------------------------
Description:
The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
was:
The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a `SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'` results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
> Messages get stuck in queue
> ---------------------------
>
> Key: AMQ-3966
> URL: https://issues.apache.org/jira/browse/AMQ-3966
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0, 5.5.1, 5.6.0
> Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
> Reporter: Robin Kåveland Hansen
> Priority: Critical
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3966) Messages get stuck in queue
Posted by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robin Kåveland Hansen updated AMQ-3966:
---------------------------------------
Attachment: brokerlog.zip
This is the brokers log with level="DEBUG" for 10:57 to 11:13 today. We managed to reproduce the bug in error.queue at approximately 11:12 when we moved 29 messages from the queue foo.bar to error.queue. At the time, error.queue had 2 messages. These are still reachable. The other 29 are in the database, but not reachable. Neither foo.bar or error.queue has consumers, both are viewed only through the use of QueueViewMBeans.
> Messages get stuck in queue
> ---------------------------
>
> Key: AMQ-3966
> URL: https://issues.apache.org/jira/browse/AMQ-3966
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0, 5.5.1, 5.6.0
> Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
> Reporter: Robin Kåveland Hansen
> Priority: Critical
> Attachments: brokerlog.zip
>
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3966) Messages get stuck in queue
Posted by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433059#comment-13433059 ]
Robin Kåveland Hansen commented on AMQ-3966:
--------------------------------------------
This seems to be manifesting in two different ways. I was able to create a situtation in which a queue reported to having -219 messages. I did this by starting 3 jobs simultaneously that tried to move the same 400 messages from one queue to another by calling QueueViewMBean.moveMessageTo. All 400 messages were moved, but afterwards the queue had very different enqueuecount and dequeuecount. Neither queue had consumers aside from these jobs. Sending messages to the broken queue resulted in the amount of messages in the queue increasing, but they were not visible through the use of QueueViewMBean.browse. Using QueueViewMBean.getMessage, I was able to obtain the messages by JMSMessageID. After sending 13 messages to the queue, I found a log entry saying toPageIn=13 for this queue.
Thinking I had a workaround for the issue by fetching the message ids from the broker database, I did a quick experiment with another broken queue. I was not able to extract messages from this one, using the same method. In this case, both dequeuecount and enqueuecount were 0. toPageIn=200 for this queue.
> Messages get stuck in queue
> ---------------------------
>
> Key: AMQ-3966
> URL: https://issues.apache.org/jira/browse/AMQ-3966
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0, 5.5.1, 5.6.0
> Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
> Reporter: Robin Kåveland Hansen
> Priority: Critical
> Attachments: brokerlog.zip
>
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (AMQ-3966) Messages get stuck in queue
Posted by "Robin Kåveland Hansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431743#comment-13431743 ]
Robin Kåveland Hansen edited comment on AMQ-3966 at 8/9/12 11:24 AM:
---------------------------------------------------------------------
Moving the retrievable messages to a different queue seemed to resolve the problem in this case - once they were moved, the missing messages became visible on the queue. Could this be related to caching in some way?
was (Author: kaaveland):
Moving the retrievable messages to a different queue seemed to resolve the problem in this case - once they were moved, the missing became visible on the queue. Could this be related to caching in some way?
> Messages get stuck in queue
> ---------------------------
>
> Key: AMQ-3966
> URL: https://issues.apache.org/jira/browse/AMQ-3966
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0, 5.5.1, 5.6.0
> Environment: java version "1.6.0_32"
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
> Linux ****** 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> CentOS release 5.8 (Final)
> Kernel \r on an \m
> Reporter: Robin Kåveland Hansen
> Priority: Critical
> Attachments: brokerlog.zip
>
>
> The broker is using the jdbc-persistence-adapter and persists to an oracle database. We have so far been unable to reproduce this issue reliably, but it appears both in our testing and production environments with some regularity.
> The issue we are having is superficially very similar to [this description|http://activemq.2283324.n4.nabble.com/Messages-stuck-in-queue-td3244342.html]. Retrieving the queue message count with jmx or inspecting the activemq database with a {code}SELECT COUNT(*) FROM ACTIVEMQ_MSGS WHERE CONTAINER LIKE '%DeadLetterQueue%'{code} results in a number that is larger than the amount of messages which can actually be retrieved using a QueueBrowser or a QueueViewMBean. This difference grows over time, as no messages that are sent to the queue after this problem occurs are retrievable. We have also observed this problem happening on queues with normal JMS consumers, where the queue will grow without the consumer receiving any messages.
> A broker restart can sometimes resolve this issue. Our current workaround is to use a program that fetches the rows for the queue from the database, unmarshals them and sends them to a temporary queue. The queue with the stuck messages does not work (does not deliver any messages to consumers or browsers) until after it has been purged.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira