You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Brian Wright <br...@marketo.com> on 2012/02/07 04:40:39 UTC
Question about activemq queue message events
Hello,
I'm actually asking this question on behalf of one of our developers.
As I understand ActiveMQ (and Servicemix), when a message is placed into
a queue, an event is fired off telling the consumer a new message is now
present. It is then the consumer's job to pick up the new message and
process it.
We had a recent issue where it appeared that new message arrival events
were either not firing or firing off very slowly to the consumer as
messages were placed into the queue. So, the message queue was backing
up and messages were not being processed. Note that we have about 8
different queues on this Servicemix server, but only one of the queues
exhibited this behavior. As a result, the processing ground to a halt
on this specific queue and queue messages backed up. Once we restarted
everything involving servicemix, activemx and the consumer of the queue,
everything went back to normal processing speed.
I'm trying to determine if there are any known issues that could cause
this behavior within ActiveMQ. We are running Apache Servicemix version
4.3.1-fuse-01-09 using ActiveMQ version 5.4.2 (looks like the ActiveMQ
packages were included with the Servicemix package).
As another question, since it appears ActiveMQ was included in the
Servicemix package, should I ask this question on the Servicemix forum
instead?
Any help would be appreciated.
Thanks.
--
*Brian Wright*
*Sr. UNIX Systems Administrator*
901 Mariners Island Blvd Suite 200
San Mateo, CA 94404 USA
*Email *brianw@marketo.com <ma...@marketo.com>
*Phone *+1.650.539.3530**
*****www.marketo.com <http://www.marketo.com/>*
Marketo Logo
Re: Question about activemq queue message events
Posted by Brian Wright <br...@marketo.com>.
Hi Matt or whomever can help,
We finally rolled out SMX 4.4.1 with AMQ 5.5.x to our production
environment a few weeks ago. Yeah, it takes a while to get through testing.
Anyway, we're still seeing the stuck queue condition even with the AMQ
5.5.x version. Matt suggested it could be a problem of an exhausted
connection pool or leaked connections. Our engineer has some questions
involving AMQ:
Tianchi Writes:
Yes, we are using AMQ 5.5 now. I'm not sure how could we tell that it is
a result of exhausting connection pools or leaking connections, but I
checked the log file and found the traces for the stuck queue, the
consumer for that queue actually consumed the messages and tried to send
the lead db file to the server [via sftp]. But after it was connected
and logged in, there were no further trace logged for that
queue(normally, a disconnect trace should be logged). I suspect that the
process got hanged during this sftp process and when we failed over, we
killed that process and message was requeued and then consumed
successfully. Since we are using transactions, if we set a timeout in
the transaction manager, would that solve this problem?
--- End Question ---
Any ideas on how we could go about debugging this condition and would
the suggestion of setting a transaction timeout help?
Thanks.
On 2/10/12 2:10 PM, Matt Pavlovich wrote:
> No problem, let us know how it goes.
>
> Thanks!
> Matt Pavlovich
>
> On 2/10/12 3:50 PM, Brian Wright wrote:
>> Hi Matt,
>>
>> Thanks for the information. I will pass this response onto the
>> developer to look into testing (and moving to) 5.5.x in our
>> environment. I will also forward your FuseSource build suggestions
>> onto our developer.
>>
>> As for the back-end datastore, we are using MySQL for the AMQ and
>> probably also the transactions. I can confirm this if there's any
>> issues involving the use of MySQL.
>>
>> Thanks.
>>
>> On 2/7/12 9:31 AM, Matt Pavlovich wrote:
>>> Hi Brian-
>>>
>>> This list is fine, as it sounds like an ActiveMQ-related issue.
>>> There are some known issues with AMQ 5.4.2 that could lead to that
>>> problem. Also, client-side issues such as exhausting connection
>>> pools, or leaking connections could cause "stuck message" behavior.
>>>
>>> I suggest doing some testing against new 5.5.x ActiveMQ releases.
>>> Specifically, I suggest using the FuseSource builds, since they are
>>> more frequent than the Apache releases at this time. They are free
>>> to use.
>>>
>>> Are you using KahaDB as a back-end data store for AMQ? Transactions?
>>>
>>> Hope this helps,
>>> Matt Pavlovich
>>>
>>> On 2/6/12 9:40 PM, Brian Wright wrote:
>>>> Hello,
>>>>
>>>> I'm actually asking this question on behalf of one of our
>>>> developers. As I understand ActiveMQ (and Servicemix), when a
>>>> message is placed into a queue, an event is fired off telling the
>>>> consumer a new message is now present. It is then the consumer's
>>>> job to pick up the new message and process it.
>>>>
>>>> We had a recent issue where it appeared that new message arrival
>>>> events were either not firing or firing off very slowly to the
>>>> consumer as messages were placed into the queue. So, the message
>>>> queue was backing up and messages were not being processed. Note
>>>> that we have about 8 different queues on this Servicemix server,
>>>> but only one of the queues exhibited this behavior. As a result,
>>>> the processing ground to a halt on this specific queue and queue
>>>> messages backed up. Once we restarted everything involving
>>>> servicemix, activemx and the consumer of the queue, everything went
>>>> back to normal processing speed.
>>>>
>>>> I'm trying to determine if there are any known issues that could
>>>> cause this behavior within ActiveMQ. We are running Apache
>>>> Servicemix version 4.3.1-fuse-01-09 using ActiveMQ version 5.4.2
>>>> (looks like the ActiveMQ packages were included with the Servicemix
>>>> package).
>>>>
>>>> As another question, since it appears ActiveMQ was included in the
>>>> Servicemix package, should I ask this question on the Servicemix
>>>> forum instead?
>>>>
>>>> Any help would be appreciated.
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Signature
>>>>
>>>> *Brian Wright*
>>>> *Sr. UNIX Systems Administrator*
>>>> 901 Mariners Island Blvd Suite 200
>>>> San Mateo, CA 94404 USA
>>>> *Email *brianw@marketo.com <ma...@marketo.com>
>>>> *Phone *+1.650.539.3530**
>>>> *****www.marketo.com <http://www.marketo.com/>*
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Signature
>>
>> *Brian Wright*
>> *Sr. UNIX Systems Administrator*
>> 901 Mariners Island Blvd Suite 200
>> San Mateo, CA 94404 USA
>> *Email *brianw@marketo.com <ma...@marketo.com>
>> *Phone *+1.650.539.3530**
>> *****www.marketo.com <http://www.marketo.com/>*
>>
>>
>>
>>
--
Signature
*Brian Wright*
*Sr. UNIX Systems Administrator*
901 Mariners Island Blvd Suite 200
San Mateo, CA 94404 USA
*Email *brianw@marketo.com <ma...@marketo.com>
*Phone *+1.650.539.3530**
*****www.marketo.com <http://www.marketo.com/>*
Marketo Logo
Re: Question about activemq queue message events
Posted by Matt Pavlovich <ma...@gmail.com>.
No problem, let us know how it goes.
Thanks!
Matt Pavlovich
On 2/10/12 3:50 PM, Brian Wright wrote:
> Hi Matt,
>
> Thanks for the information. I will pass this response onto the
> developer to look into testing (and moving to) 5.5.x in our
> environment. I will also forward your FuseSource build suggestions
> onto our developer.
>
> As for the back-end datastore, we are using MySQL for the AMQ and
> probably also the transactions. I can confirm this if there's any
> issues involving the use of MySQL.
>
> Thanks.
>
> On 2/7/12 9:31 AM, Matt Pavlovich wrote:
>> Hi Brian-
>>
>> This list is fine, as it sounds like an ActiveMQ-related issue.
>> There are some known issues with AMQ 5.4.2 that could lead to that
>> problem. Also, client-side issues such as exhausting connection
>> pools, or leaking connections could cause "stuck message" behavior.
>>
>> I suggest doing some testing against new 5.5.x ActiveMQ releases.
>> Specifically, I suggest using the FuseSource builds, since they are
>> more frequent than the Apache releases at this time. They are free
>> to use.
>>
>> Are you using KahaDB as a back-end data store for AMQ? Transactions?
>>
>> Hope this helps,
>> Matt Pavlovich
>>
>> On 2/6/12 9:40 PM, Brian Wright wrote:
>>> Hello,
>>>
>>> I'm actually asking this question on behalf of one of our
>>> developers. As I understand ActiveMQ (and Servicemix), when a
>>> message is placed into a queue, an event is fired off telling the
>>> consumer a new message is now present. It is then the consumer's
>>> job to pick up the new message and process it.
>>>
>>> We had a recent issue where it appeared that new message arrival
>>> events were either not firing or firing off very slowly to the
>>> consumer as messages were placed into the queue. So, the message
>>> queue was backing up and messages were not being processed. Note
>>> that we have about 8 different queues on this Servicemix server, but
>>> only one of the queues exhibited this behavior. As a result, the
>>> processing ground to a halt on this specific queue and queue
>>> messages backed up. Once we restarted everything involving
>>> servicemix, activemx and the consumer of the queue, everything went
>>> back to normal processing speed.
>>>
>>> I'm trying to determine if there are any known issues that could
>>> cause this behavior within ActiveMQ. We are running Apache
>>> Servicemix version 4.3.1-fuse-01-09 using ActiveMQ version 5.4.2
>>> (looks like the ActiveMQ packages were included with the Servicemix
>>> package).
>>>
>>> As another question, since it appears ActiveMQ was included in the
>>> Servicemix package, should I ask this question on the Servicemix
>>> forum instead?
>>>
>>> Any help would be appreciated.
>>>
>>> Thanks.
>>>
>>> --
>>>
>>> *Brian Wright*
>>> *Sr. UNIX Systems Administrator*
>>> 901 Mariners Island Blvd Suite 200
>>> San Mateo, CA 94404 USA
>>> *Email *brianw@marketo.com <ma...@marketo.com>
>>> *Phone *+1.650.539.3530**
>>> *****www.marketo.com <http://www.marketo.com/>*
>>>
>>> Marketo Logo
>>>
>>>
>
> --
>
> *Brian Wright*
> *Sr. UNIX Systems Administrator*
> 901 Mariners Island Blvd Suite 200
> San Mateo, CA 94404 USA
> *Email *brianw@marketo.com <ma...@marketo.com>
> *Phone *+1.650.539.3530**
> *****www.marketo.com <http://www.marketo.com/>*
>
> Marketo Logo
>
>
Re: Question about activemq queue message events
Posted by Brian Wright <br...@marketo.com>.
Hi Matt,
Thanks for the information. I will pass this response onto the
developer to look into testing (and moving to) 5.5.x in our
environment. I will also forward your FuseSource build suggestions onto
our developer.
As for the back-end datastore, we are using MySQL for the AMQ and
probably also the transactions. I can confirm this if there's any
issues involving the use of MySQL.
Thanks.
On 2/7/12 9:31 AM, Matt Pavlovich wrote:
> Hi Brian-
>
> This list is fine, as it sounds like an ActiveMQ-related issue. There
> are some known issues with AMQ 5.4.2 that could lead to that problem.
> Also, client-side issues such as exhausting connection pools, or
> leaking connections could cause "stuck message" behavior.
>
> I suggest doing some testing against new 5.5.x ActiveMQ releases.
> Specifically, I suggest using the FuseSource builds, since they are
> more frequent than the Apache releases at this time. They are free
> to use.
>
> Are you using KahaDB as a back-end data store for AMQ? Transactions?
>
> Hope this helps,
> Matt Pavlovich
>
> On 2/6/12 9:40 PM, Brian Wright wrote:
>> Hello,
>>
>> I'm actually asking this question on behalf of one of our
>> developers. As I understand ActiveMQ (and Servicemix), when a
>> message is placed into a queue, an event is fired off telling the
>> consumer a new message is now present. It is then the consumer's job
>> to pick up the new message and process it.
>>
>> We had a recent issue where it appeared that new message arrival
>> events were either not firing or firing off very slowly to the
>> consumer as messages were placed into the queue. So, the message
>> queue was backing up and messages were not being processed. Note
>> that we have about 8 different queues on this Servicemix server, but
>> only one of the queues exhibited this behavior. As a result, the
>> processing ground to a halt on this specific queue and queue messages
>> backed up. Once we restarted everything involving servicemix,
>> activemx and the consumer of the queue, everything went back to
>> normal processing speed.
>>
>> I'm trying to determine if there are any known issues that could
>> cause this behavior within ActiveMQ. We are running Apache
>> Servicemix version 4.3.1-fuse-01-09 using ActiveMQ version 5.4.2
>> (looks like the ActiveMQ packages were included with the Servicemix
>> package).
>>
>> As another question, since it appears ActiveMQ was included in the
>> Servicemix package, should I ask this question on the Servicemix
>> forum instead?
>>
>> Any help would be appreciated.
>>
>> Thanks.
>>
>> --
>>
>> *Brian Wright*
>> *Sr. UNIX Systems Administrator*
>> 901 Mariners Island Blvd Suite 200
>> San Mateo, CA 94404 USA
>> *Email *brianw@marketo.com <ma...@marketo.com>
>> *Phone *+1.650.539.3530**
>> *****www.marketo.com <http://www.marketo.com/>*
>>
>> Marketo Logo
>>
>>
--
*Brian Wright*
*Sr. UNIX Systems Administrator*
901 Mariners Island Blvd Suite 200
San Mateo, CA 94404 USA
*Email *brianw@marketo.com <ma...@marketo.com>
*Phone *+1.650.539.3530**
*****www.marketo.com <http://www.marketo.com/>*
Marketo Logo
Re: Question about activemq queue message events
Posted by Matt Pavlovich <ma...@gmail.com>.
Hi Brian-
This list is fine, as it sounds like an ActiveMQ-related issue. There
are some known issues with AMQ 5.4.2 that could lead to that problem.
Also, client-side issues such as exhausting connection pools, or leaking
connections could cause "stuck message" behavior.
I suggest doing some testing against new 5.5.x ActiveMQ releases.
Specifically, I suggest using the FuseSource builds, since they are more
frequent than the Apache releases at this time. They are free to use.
Are you using KahaDB as a back-end data store for AMQ? Transactions?
Hope this helps,
Matt Pavlovich
On 2/6/12 9:40 PM, Brian Wright wrote:
> Hello,
>
> I'm actually asking this question on behalf of one of our developers.
> As I understand ActiveMQ (and Servicemix), when a message is placed
> into a queue, an event is fired off telling the consumer a new message
> is now present. It is then the consumer's job to pick up the new
> message and process it.
>
> We had a recent issue where it appeared that new message arrival
> events were either not firing or firing off very slowly to the
> consumer as messages were placed into the queue. So, the message
> queue was backing up and messages were not being processed. Note that
> we have about 8 different queues on this Servicemix server, but only
> one of the queues exhibited this behavior. As a result, the
> processing ground to a halt on this specific queue and queue messages
> backed up. Once we restarted everything involving servicemix,
> activemx and the consumer of the queue, everything went back to normal
> processing speed.
>
> I'm trying to determine if there are any known issues that could cause
> this behavior within ActiveMQ. We are running Apache Servicemix
> version 4.3.1-fuse-01-09 using ActiveMQ version 5.4.2 (looks like the
> ActiveMQ packages were included with the Servicemix package).
>
> As another question, since it appears ActiveMQ was included in the
> Servicemix package, should I ask this question on the Servicemix forum
> instead?
>
> Any help would be appreciated.
>
> Thanks.
>
> --
>
> *Brian Wright*
> *Sr. UNIX Systems Administrator*
> 901 Mariners Island Blvd Suite 200
> San Mateo, CA 94404 USA
> *Email *brianw@marketo.com <ma...@marketo.com>
> *Phone *+1.650.539.3530**
> *****www.marketo.com <http://www.marketo.com/>*
>
> Marketo Logo
>
>