You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Wladimir Safonov <wl...@itemis.de> on 2008/02/29 23:08:25 UTC

Producer locks while awaiting broker response

Hello,

as I found in the older mailings some people experienced the same 
problem as I do in the moment. Seems like problem persisted already in 
the earlier versions of the broker (< 4.0), but no solution approach was 
found so far. So in my situation, I have a couple of topics, some of 
which stay under relatively high message load (>10 messages pro sec) 
from the system start. After 1-2 hours a random producer hangs up with 
the following stack trace:

sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:76)
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1172)
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1640)
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:226)
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:240)

This can also happen to several producers at the same time. As I read in 
the preceding posts, such behavior is possible if the consumer fails. As 
I use failover, I can't say for sure if the consumer failed just before 
the problem occurred. Anyway after the consumer has recovered, the 
publisher should definitely receive some response, which does not 
happen. Generally speaking, such broker strategy is kind of odd, because 
possible listener crash & recover situations should not influence the 
work of the producers - they should be really decoupled.

I checked already this page: 
http://activemq.apache.org/my-producer-blocks.html. But in my case the 
problem also occurs on the empty topics which process a couple of 
messages pro minute, whose consumer also fails now and then. So current 
topic usage cannot be the reason for this behavior.
I'm currently using AMQ 5.0, but I had the same problem with AMQ 4.1.1.

Does anybody know how such broker behavior can be influenced or if not, 
will there be any fixes regarding this bug in the future releases?
Appreciate any help!

Thanks a lot in advance!

Best regards,
Wladimir

Re: Producer locks while awaiting broker response

Posted by Wladimir Safonov <wl...@itemis.de>.
On 3 Mar 2008, at 14:31, Rob Davies wrote:
> On 29 Feb 2008, at 22:08, Wladimir Safonov wrote:
>
>> Hello,
>>
>> as I found in the older mailings some people experienced the same 
>> problem as I do in the moment. Seems like problem persisted already 
>> in the earlier versions of the broker (< 4.0), but no solution 
>> approach was found so far. So in my situation, I have a couple of 
>> topics, some of which stay under relatively high message load (>10 
>> messages pro sec) from the system start. After 1-2 hours a random 
>> producer hangs up with the following stack trace:
>>
>> sun.misc.Unsafe.park(Native Method)
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) 
>>
>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341) 
>>
>> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40) 
>>
>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:76) 
>>
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1172) 
>>
>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1640)
>> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:226) 
>>
>> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:240) 
>>
>>
>> This can also happen to several producers at the same time. As I read 
>> in the preceding posts, such behavior is possible if the consumer 
>> fails. As I use failover, I can't say for sure if the consumer failed 
>> just before the problem occurred. Anyway after the consumer has 
>> recovered, the publisher should definitely receive some response, 
>> which does not happen. Generally speaking, such broker strategy is 
>> kind of odd, because possible listener crash & recover situations 
>> should not influence the work of the producers - they should be 
>> really decoupled.
>>
>> I checked already this page: 
>> http://activemq.apache.org/my-producer-blocks.html. But in my case 
>> the problem also occurs on the empty topics which process a couple of 
>> messages pro minute, whose consumer also fails now and then. So 
>> current topic usage cannot be the reason for this behavior.
>> I'm currently using AMQ 5.0, but I had the same problem with AMQ 4.1.1.
>>
>> Does anybody know how such broker behavior can be influenced or if 
>> not, will there be any fixes regarding this bug in the future releases?
>> Appreciate any help!
>>
>> Thanks a lot in advance!
>>
>> Best regards,
>> Wladimir
>>
>
> Looks like your producer is waiting for resources (memory) to be freed 
> up on the broker. Could you try a 5.1-SNAPSHOT ?
Thanks for your advice! I've tried it with latest SNAPSHOT version, but 
unfortunately producer still hangs up with the same stack trace after 
1-2 hours of work. It doesn't occur to an overloaded topic, but rather 
to a infrequently used one with a throughput 1 msg/min. What I also 
noticed, from the two consumers on that topic only one crashes and 
recovers, whereas the other one stays connected all the time. So merely 
the recover process of a single consumer makes the broker block the 
corresponding producer.

Do you have any other ideas how to fix this issue? Any workaround perhaps?

Thanks for any attempt to help in advance!

Re: Producer locks while awaiting broker response

Posted by Rob Davies <ra...@gmail.com>.
On 29 Feb 2008, at 22:08, Wladimir Safonov wrote:

> Hello,
>
> as I found in the older mailings some people experienced the same  
> problem as I do in the moment. Seems like problem persisted already  
> in the earlier versions of the broker (< 4.0), but no solution  
> approach was found so far. So in my situation, I have a couple of  
> topics, some of which stay under relatively high message load (>10  
> messages pro sec) from the system start. After 1-2 hours a random  
> producer hangs up with the following stack trace:
>
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
> java.util.concurrent.locks.AbstractQueuedSynchronizer 
> $ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java: 
> 341)
> org 
> .apache 
> .activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
> org 
> .apache 
> .activemq 
> .transport.ResponseCorrelator.request(ResponseCorrelator.java:76)
> org 
> .apache 
> .activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java: 
> 1172)
> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1640)
> org 
> .apache 
> .activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java: 
> 226)
> org 
> .apache 
> .activemq 
> .ActiveMQMessageProducerSupport 
> .send(ActiveMQMessageProducerSupport.java:240)
>
> This can also happen to several producers at the same time. As I  
> read in the preceding posts, such behavior is possible if the  
> consumer fails. As I use failover, I can't say for sure if the  
> consumer failed just before the problem occurred. Anyway after the  
> consumer has recovered, the publisher should definitely receive some  
> response, which does not happen. Generally speaking, such broker  
> strategy is kind of odd, because possible listener crash & recover  
> situations should not influence the work of the producers - they  
> should be really decoupled.
>
> I checked already this page: http://activemq.apache.org/my-producer-blocks.html 
> . But in my case the problem also occurs on the empty topics which  
> process a couple of messages pro minute, whose consumer also fails  
> now and then. So current topic usage cannot be the reason for this  
> behavior.
> I'm currently using AMQ 5.0, but I had the same problem with AMQ  
> 4.1.1.
>
> Does anybody know how such broker behavior can be influenced or if  
> not, will there be any fixes regarding this bug in the future  
> releases?
> Appreciate any help!
>
> Thanks a lot in advance!
>
> Best regards,
> Wladimir
>

Looks like your producer is waiting for resources (memory) to be freed  
up on the broker. Could you try a 5.1-SNAPSHOT ?



cheers,

Rob

http://open.iona.com/ -Enterprise Open Integration
http://rajdavies.blogspot.com/