Re: Question about activemq queue message events

[Brian Wright <br...@marketo.com>]

Hi Matt or whomever can help,

We finally rolled out SMX 4.4.1 with AMQ 5.5.x to our production 
environment a few weeks ago.  Yeah, it takes a while to get through testing.

Anyway, we're still seeing the stuck queue condition even with the AMQ 
5.5.x version.  Matt suggested it could be a problem of an exhausted 
connection pool or leaked connections.  Our engineer has some questions 
involving AMQ:

Tianchi Writes:

Yes, we are using AMQ 5.5 now. I'm not sure how could we tell that it is 
a result of exhausting connection pools or leaking connections, but I 
checked the log file and found the traces for the stuck queue, the 
consumer for that queue actually consumed the messages and tried to send 
the lead db file to the server [via sftp]. But after it was connected 
and logged in, there were no further trace logged for that 
queue(normally, a disconnect trace should be logged). I suspect that the 
process got hanged during this sftp process and when we failed over, we 
killed that process and message was requeued and then consumed 
successfully. Since we are using transactions, if we set a timeout in 
the transaction manager, would that solve this problem?

--- End Question ---

Any ideas on how we could go about debugging this condition and would 
the suggestion of setting a transaction timeout help?

Thanks.

On 2/10/12 2:10 PM, Matt Pavlovich wrote:
> No problem, let us know how it goes.
>
> Thanks!
> Matt Pavlovich
>
> On 2/10/12 3:50 PM, Brian Wright wrote:
>> Hi Matt,
>>
>> Thanks for the information.  I will pass this response onto the 
>> developer to look into testing (and moving to) 5.5.x in our 
>> environment.  I will also forward your FuseSource build suggestions 
>> onto our developer.
>>
>> As for the back-end datastore, we are using MySQL for the AMQ and 
>> probably also the transactions.  I can confirm this if there's any 
>> issues involving the use of MySQL.
>>
>> Thanks.
>>
>> On 2/7/12 9:31 AM, Matt Pavlovich wrote:
>>> Hi Brian-
>>>
>>> This list is fine, as it sounds like an ActiveMQ-related issue.  
>>> There are some known issues with AMQ 5.4.2 that could lead to that 
>>> problem.  Also, client-side issues such as exhausting connection 
>>> pools, or leaking connections could cause "stuck message" behavior.
>>>
>>> I suggest doing some testing against new 5.5.x ActiveMQ releases.  
>>> Specifically, I suggest using the FuseSource builds, since they are 
>>> more frequent than the Apache releases at this time.   They are free 
>>> to use.
>>>
>>> Are you using KahaDB as a back-end data store for AMQ? Transactions?
>>>
>>> Hope this helps,
>>> Matt Pavlovich
>>>
>>> On 2/6/12 9:40 PM, Brian Wright wrote:
>>>> Hello,
>>>>
>>>> I'm actually asking this question on behalf of one of our 
>>>> developers.  As I understand ActiveMQ (and Servicemix), when a 
>>>> message is placed into a queue, an event is fired off telling the 
>>>> consumer a new message is now present.  It is then the consumer's 
>>>> job to pick up the new message and process it.
>>>>
>>>> We had a recent issue where it appeared that new message arrival 
>>>> events were either not firing or firing off very slowly to the 
>>>> consumer as messages were placed into the queue.  So, the message 
>>>> queue was backing up and messages were not being processed.  Note 
>>>> that we have about 8 different queues on this Servicemix server, 
>>>> but only one of the queues exhibited this behavior.  As a result, 
>>>> the processing ground to a halt on this specific queue and queue 
>>>> messages backed up.  Once we restarted everything involving 
>>>> servicemix, activemx and the consumer of the queue, everything went 
>>>> back to normal processing speed.
>>>>
>>>> I'm trying to determine if there are any known issues that could 
>>>> cause this behavior within ActiveMQ.  We are running Apache 
>>>> Servicemix version 4.3.1-fuse-01-09 using ActiveMQ version 5.4.2 
>>>> (looks like the ActiveMQ packages were included with the Servicemix 
>>>> package).
>>>>
>>>> As another question, since it appears ActiveMQ was included in the 
>>>> Servicemix package, should I ask this question on the Servicemix 
>>>> forum instead?
>>>>
>>>> Any help would be appreciated.
>>>>
>>>> Thanks.
>>>>
>>>> -- 
>>>> Signature
>>>>
>>>> *Brian Wright*
>>>> *Sr. UNIX Systems Administrator*
>>>> 901 Mariners Island Blvd Suite 200
>>>> San Mateo, CA 94404 USA
>>>> *Email *brianw@marketo.com <ma...@marketo.com>
>>>> *Phone *+1.650.539.3530**
>>>> *****www.marketo.com <http://www.marketo.com/>*
>>>>
>>>> 	
>>>>
>>>>
>>
>> -- 
>> Signature
>>
>> *Brian Wright*
>> *Sr. UNIX Systems Administrator*
>> 901 Mariners Island Blvd Suite 200
>> San Mateo, CA 94404 USA
>> *Email *brianw@marketo.com <ma...@marketo.com>
>> *Phone *+1.650.539.3530**
>> *****www.marketo.com <http://www.marketo.com/>*
>>
>> 	
>>
>>

-- 
Signature

*Brian Wright*
*Sr. UNIX Systems Administrator*
901 Mariners Island Blvd Suite 200
San Mateo, CA 94404 USA
*Email *brianw@marketo.com <ma...@marketo.com>
*Phone *+1.650.539.3530**
*****www.marketo.com <http://www.marketo.com/>*

	Marketo Logo