Mysterious exactly one message missing in the queue

[QiS <qi...@gmail.com>]

ActiveMQ: 5.3 Queue + Persistent Messaging(AMQ Message Store)
OS: Linux version 2.6.18-128.el5 (gcc version 4.1.2 20080704 (Red Hat
4.1.2-44)) 
JVM: 1.5
Application Server: Tomcat 6.0

We've been running an application using ActiveMQ 5.3 for quite a while and
we've been puzzled by one sporadic problem. 

Our application is consist of a "server" and a number of "workers". Server
sends messages to a RequestQueue and workers will pick up the messages,
process them and then send the responses back to the server via a
ResponseQueue. 

We've been experiencing a sporadic issue where the server sends x number of
messages to the RequestQueue, the worker would only receive x-1 messages -
there seems to be one message missing. When that happens, we use Hermes to
check the RequestQueue, and it would show the RequestQueue is empty. So we
don't know how that one message gets lost. 

What's interesting is, if the server starts to send another x number of
messages, it would somehow "push" the previously lost message into the
RequestQueue again, and the workers would also pick it up and process it.
But again, there will be exactly one missing message for the new group of
messages, which would be pushed into the queue again when the server sends
one more group of messages again. And the problem will keep repeating like
this.

The mystery seems to be that the missing message is "hiding" somewhere until
new messages come in to push it into the RequestQueue again. 

We really need to resolve this issue urgently. Your help is highly
appreciated!

Thanks,
QiS




-- 
View this message in context: http://old.nabble.com/Mysterious-exactly-one-message-missing-in-the-queue-tp27173813p27173813.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Mysterious exactly one message missing in the queue

[Rob Davies <ra...@gmail.com>]

Hi QiS,

can you provide a Junit test case - it would help us help you a lot  
quicker

thanks,

Rob

On 15 Jan 2010, at 10:27, QiS wrote:

>
> ActiveMQ: 5.3 Queue + Persistent Messaging(AMQ Message Store)
> OS: Linux version 2.6.18-128.el5 (gcc version 4.1.2 20080704 (Red Hat
> 4.1.2-44))
> JVM: 1.5
> Application Server: Tomcat 6.0
>
> We've been running an application using ActiveMQ 5.3 for quite a  
> while and
> we've been puzzled by one sporadic problem.
>
> Our application is consist of a "server" and a number of "workers".  
> Server
> sends messages to a RequestQueue and workers will pick up the  
> messages,
> process them and then send the responses back to the server via a
> ResponseQueue.
>
> We've been experiencing a sporadic issue where the server sends x  
> number of
> messages to the RequestQueue, the worker would only receive x-1  
> messages -
> there seems to be one message missing. When that happens, we use  
> Hermes to
> check the RequestQueue, and it would show the RequestQueue is empty.  
> So we
> don't know how that one message gets lost.
>
> What's interesting is, if the server starts to send another x number  
> of
> messages, it would somehow "push" the previously lost message into the
> RequestQueue again, and the workers would also pick it up and  
> process it.
> But again, there will be exactly one missing message for the new  
> group of
> messages, which would be pushed into the queue again when the server  
> sends
> one more group of messages again. And the problem will keep  
> repeating like
> this.
>
> The mystery seems to be that the missing message is "hiding"  
> somewhere until
> new messages come in to push it into the RequestQueue again.
>
> We really need to resolve this issue urgently. Your help is highly
> appreciated!
>
> Thanks,
> QiS
>
>
>
>
> -- 
> View this message in context: http://old.nabble.com/Mysterious-exactly-one-message-missing-in-the-queue-tp27173813p27173813.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/