You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Colleen Velo <cm...@gmail.com> on 2010/09/01 14:52:07 UTC
Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s in
the "Pending Msgs" column
After only running my 3 ActiveMQ brokers for short time, I am seeing
negative numbers in
the "Number of Pending Messages" column. I started seeing this at 5.3.2
(after upgrading
from 5.2.0) and opened and issue on it:
• https://issues.apache.org/activemq/browse/AMQ-2887
It was recommended that I upgrade to ActiveMQ 5.4.0, which I did, however, I
am still
seeing this issue.
Current Environment:
• CentOS release 5.2 (both consumers and brokers)
• 3 ActiveMQ brokers
• 7 consumers (across different queues)
Thanks,
--
Colleen Velo
email: cmvelo@gmail.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary,
Forgot to add that I do see the same thing in JConsole, that is the negative
counts.
On Fri, Sep 3, 2010 at 1:37 PM, Colleen Velo <cm...@gmail.com> wrote:
> Gary,
>
> Thanks for the clarification - still working on a JUnit test.
> Currently right now the steps that I use to reproduce it is
> to have the 3 producers each send *60,000* messages each to
> the ActiveMQ cluster at the same time - more often than
> not, when the queue is trending down towards 0 on the
> broker(s), sometimes a negative # will appear in the "Pending Msgs"
> column (it has been frustrating because it seems to happen
> intermittently).
>
> *Current Setup:*
>
> • 3 producers
> • 3 brokers (all part of the same ActiveMQ cluster)
> • 3 consumers
> • 1 queue
>
>
> *Broker definition*:
> • <networkConnector name="ActiveMQ Cluster"
> uri="static:(tcp://brokersvr1:61616,tcp://brokersvr2:61616,tcp://brokersvr3:61616)"
> conduitSubscriptions="false" dynamicOnly="true"
> decreaseNetworkConsumerPriority="true"
> suppressDuplicateQueueSubscriptions="true" duplex="false" networkTTL="5"/>
>
> *Client definition:*
> •
> amq.publisher.broker.url=failover\:tcp\://brokersvr1\:61616?wireFormat.maxInactivityDuration=0&daemon=true,tcp\://brokersvr2\:61616?wireFormat.maxInactivityDuration=0&daemon=true,tcp\://brokersvr3\:61616?wireFormat.maxInactivityDuration=0&daemon=true
>
>
> Let me know if you have any more questions and I'll pass along the JUnit
> test as soon
> as I can get one to fail consistently.
>
> Thanks,
>
> --
> Colleen Velo
> email: cmvelo@gmail.com
>
>
> On Thu, Sep 2, 2010 at 2:25 AM, Gary Tully <ga...@gmail.com> wrote:
>
>> Some series of steps that can reproduce or in the ideal case a junit
>> test we can drop into the source.
>>
>> On Wednesday, September 1, 2010, Colleen Velo <cm...@gmail.com> wrote:
>> > Gary
>> > re: JConsole - don't know yet. I will try to hook JConsole up and look
>> > through that as well -
>> > currently using the bundled web admin console.
>> >
>> > As far as test case, I don't have one - just the standard message
>> traffic
>> > from
>> > our producers to our consumers (slightly less than the traffic that is
>> > running
>> > currently in our Product environment with ActiveMQ server version 5.2.0.
>> >
>> > What would you need exactly for a "test" case?
>> >
>> > Thanks,
>> >
>> > --
>> > Colleen Velo
>> > *email: cmvelo@gmail.com*
>> >
>> >
>> > On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com>
>> wrote:
>> >
>> >> does jconsole show the same negative number? Do you have a test case?
>> >>
>> >> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
>> >> > After only running my 3 ActiveMQ brokers for short time, I am seeing
>> >> > negative numbers in
>> >> > the "Number of Pending Messages" column. I started seeing this at
>> 5.3.2
>> >> > (after upgrading
>> >> > from 5.2.0) and opened and issue on it:
>> >> > • https://issues.apache.org/activemq/browse/AMQ-2887
>> >> >
>> >> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
>> >> however, I
>> >> > am still
>> >> > seeing this issue.
>> >> >
>> >> > Current Environment:
>> >> > • CentOS release 5.2 (both consumers and brokers)
>> >> > • 3 ActiveMQ brokers
>> >> > • 7 consumers (across different queues)
>> >> >
>> >> >
>> >> > Thanks,
>> >> >
>> >> > --
>> >> > Colleen Velo
>> >> > email: cmvelo@gmail.com
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> http://blog.garytully.com
>> >>
>> >> Open Source Integration
>> >> http://fusesource.com
>> >>
>> >
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source Integration
>> http://fusesource.com
>>
>
>
--
Colleen Velo
email: cmvelo@gmail.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary,
Thanks for the clarification - still working on a JUnit test.
Currently right now the steps that I use to reproduce it is
to have the 3 producers each send *60,000* messages each to
the ActiveMQ cluster at the same time - more often than
not, when the queue is trending down towards 0 on the
broker(s), sometimes a negative # will appear in the "Pending Msgs"
column (it has been frustrating because it seems to happen
intermittently).
*Current Setup:*
• 3 producers
• 3 brokers (all part of the same ActiveMQ cluster)
• 3 consumers
• 1 queue
*Broker definition*:
• <networkConnector name="ActiveMQ Cluster"
uri="static:(tcp://brokersvr1:61616,tcp://brokersvr2:61616,tcp://brokersvr3:61616)"
conduitSubscriptions="false" dynamicOnly="true"
decreaseNetworkConsumerPriority="true"
suppressDuplicateQueueSubscriptions="true" duplex="false" networkTTL="5"/>
*Client definition:*
•
amq.publisher.broker.url=failover\:tcp\://brokersvr1\:61616?wireFormat.maxInactivityDuration=0&daemon=true,tcp\://brokersvr2\:61616?wireFormat.maxInactivityDuration=0&daemon=true,tcp\://brokersvr3\:61616?wireFormat.maxInactivityDuration=0&daemon=true
Let me know if you have any more questions and I'll pass along the JUnit
test as soon
as I can get one to fail consistently.
Thanks,
--
Colleen Velo
email: cmvelo@gmail.com
On Thu, Sep 2, 2010 at 2:25 AM, Gary Tully <ga...@gmail.com> wrote:
> Some series of steps that can reproduce or in the ideal case a junit
> test we can drop into the source.
>
> On Wednesday, September 1, 2010, Colleen Velo <cm...@gmail.com> wrote:
> > Gary
> > re: JConsole - don't know yet. I will try to hook JConsole up and look
> > through that as well -
> > currently using the bundled web admin console.
> >
> > As far as test case, I don't have one - just the standard message traffic
> > from
> > our producers to our consumers (slightly less than the traffic that is
> > running
> > currently in our Product environment with ActiveMQ server version 5.2.0.
> >
> > What would you need exactly for a "test" case?
> >
> > Thanks,
> >
> > --
> > Colleen Velo
> > *email: cmvelo@gmail.com*
> >
> >
> > On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com> wrote:
> >
> >> does jconsole show the same negative number? Do you have a test case?
> >>
> >> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
> >> > After only running my 3 ActiveMQ brokers for short time, I am seeing
> >> > negative numbers in
> >> > the "Number of Pending Messages" column. I started seeing this at
> 5.3.2
> >> > (after upgrading
> >> > from 5.2.0) and opened and issue on it:
> >> > • https://issues.apache.org/activemq/browse/AMQ-2887
> >> >
> >> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
> >> however, I
> >> > am still
> >> > seeing this issue.
> >> >
> >> > Current Environment:
> >> > • CentOS release 5.2 (both consumers and brokers)
> >> > • 3 ActiveMQ brokers
> >> > • 7 consumers (across different queues)
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> > Colleen Velo
> >> > email: cmvelo@gmail.com
> >> >
> >>
> >>
> >>
> >> --
> >> http://blog.garytully.com
> >>
> >> Open Source Integration
> >> http://fusesource.com
> >>
> >
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Gary Tully <ga...@gmail.com>.
Some series of steps that can reproduce or in the ideal case a junit
test we can drop into the source.
On Wednesday, September 1, 2010, Colleen Velo <cm...@gmail.com> wrote:
> Gary
> re: JConsole - don't know yet. I will try to hook JConsole up and look
> through that as well -
> currently using the bundled web admin console.
>
> As far as test case, I don't have one - just the standard message traffic
> from
> our producers to our consumers (slightly less than the traffic that is
> running
> currently in our Product environment with ActiveMQ server version 5.2.0.
>
> What would you need exactly for a "test" case?
>
> Thanks,
>
> --
> Colleen Velo
> *email: cmvelo@gmail.com*
>
>
> On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com> wrote:
>
>> does jconsole show the same negative number? Do you have a test case?
>>
>> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
>> > After only running my 3 ActiveMQ brokers for short time, I am seeing
>> > negative numbers in
>> > the "Number of Pending Messages" column. I started seeing this at 5.3.2
>> > (after upgrading
>> > from 5.2.0) and opened and issue on it:
>> > • https://issues.apache.org/activemq/browse/AMQ-2887
>> >
>> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
>> however, I
>> > am still
>> > seeing this issue.
>> >
>> > Current Environment:
>> > • CentOS release 5.2 (both consumers and brokers)
>> > • 3 ActiveMQ brokers
>> > • 7 consumers (across different queues)
>> >
>> >
>> > Thanks,
>> >
>> > --
>> > Colleen Velo
>> > email: cmvelo@gmail.com
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source Integration
>> http://fusesource.com
>>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary
re: JConsole - don't know yet. I will try to hook JConsole up and look
through that as well -
currently using the bundled web admin console.
As far as test case, I don't have one - just the standard message traffic
from
our producers to our consumers (slightly less than the traffic that is
running
currently in our Product environment with ActiveMQ server version 5.2.0.
What would you need exactly for a "test" case?
Thanks,
--
Colleen Velo
*email: cmvelo@gmail.com*
On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com> wrote:
> does jconsole show the same negative number? Do you have a test case?
>
> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
> > After only running my 3 ActiveMQ brokers for short time, I am seeing
> > negative numbers in
> > the "Number of Pending Messages" column. I started seeing this at 5.3.2
> > (after upgrading
> > from 5.2.0) and opened and issue on it:
> > • https://issues.apache.org/activemq/browse/AMQ-2887
> >
> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
> however, I
> > am still
> > seeing this issue.
> >
> > Current Environment:
> > • CentOS release 5.2 (both consumers and brokers)
> > • 3 ActiveMQ brokers
> > • 7 consumers (across different queues)
> >
> >
> > Thanks,
> >
> > --
> > Colleen Velo
> > email: cmvelo@gmail.com
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary,
It does appear that it is just the stats that are wrong in this case. The
messages
are getting delivered as expected (as long as there is an application
consumer on
that broker).
A couple of other questions:
• Since I only seem to see this issue when there are 2 or more brokers, is
there
anything in our networkConnector string that could cause this (or any other
part
of the server configuration that would be advisable to re-check)?
<networkConnector name="ActiveMQ Cluster"
uri="static:(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)"
conduitSubscriptions="true" dynamicOnly="true"
decreaseNetworkConsumerPriority="false"
suppressDuplicateQueueSubscriptions="true" duplex="false" networkTTL="5"/>
• Also, currently we are running ActiveMQ server *5.4.0*, but our clients
(consumers)
are still running version 5.2.0 - is there any possibility that this could
be causing
the issue (I am in the process of rebuilding the consumers with 5.4.0
libraries to
confirm this).
Thanks,
On Thu, Sep 9, 2010 at 8:49 AM, Gary Tully <ga...@gmail.com> wrote:
> ok, I guess this is going to be very hard to track down without a
> reproducible test case. Can you confirm that it is just the stats that
> are wrong in this case, and that all expected messages are getting
> delivered as expected.
>
> On 9 September 2010 14:28, Colleen Velo <cm...@gmail.com> wrote:
> > Gary,
> >
> > That doesn't seem to be the case here - namely, the JConsole queue count
> is
> > accurately
> > reflecting what the ActiveMQ web console is showing. I have rechecked
> our
> > application
> > code as well as the ActiveMQ server settings and in both cases we are
> using
> > the default
> > *dispatchAsync* value of false which, according to your blog post from
> Jan
> > 2010, should
> > take care of the "inflight" counts issue.
> >
> > It seems to be more a case where the queue "counters" seem to get "thrown
> > off"
> > and never get back in-sync until the broker is restarted.
> > For example, I can see a number "-10" in the "Pending Messages" on the
> > ActiveMQ
> > web admin console (and in JConsole), but clicking on the queue name and
> > showing
> > the actual contents of the queue might reveal 0 items or 5 items, etc.
> > (Note: in the above scenario, the producers were not putting any new
> items
> > onto the
> > queue, so the # of "Pending Messages" wasn't changing).
> >
> >
> > On Thu, Sep 9, 2010 at 5:44 AM, Gary Tully <ga...@gmail.com> wrote:
> >
> >> Is it by any chance the inflight count in jconsole that is negative?
> >> There is a good explanation for that,
> >>
> http://blog.garytully.com/2010/01/activemq-prefetch-and-asyncdispatch.html
> >>
> >> On 8 September 2010 21:42, Colleen Velo <cm...@gmail.com> wrote:
> >> > Gary,
> >> > * *
> >> > *Sorry - last message was incomplete - technical difficulties*
> >> >
> >> > Haven't had any luck reproducing this issue with a JMeter test,
> however,
> >> I
> >> > can reproduce
> >> > it on a fairly regular basis using our in-house application script,
> which
> >> > puts tons of application
> >> > messages on the queues.
> >> >
> >> > Also, I don't seem to see this issue when there is only a single
> broker
> >> > running, only when
> >> > there is two or more running in a network of brokers configuration.
> >> >
> >> > When I do see this issue, I •do• see it on JConsole as well as in the
> >> > ActiveMQ WebAdmin console.
> >> >
> >> > *Possible Test Case:*
> >> >
> >> > To reproduce, start up multiple Producers, both of which are throwing
> >> > 80,000+ messages
> >> > to the two brokers (which are configured as a network of brokers ) and
> >> have
> >> > two consumers running on
> >> > separate servers pulling items off of the application queue(s).
> >> >
> >> >
> >> > On Wed, Sep 8, 2010 at 3:39 PM, Colleen Velo <cm...@gmail.com>
> wrote:
> >> >
> >> >> Gary,
> >> >>
> >> >> Haven't had any luck reproducing this issue with a JMeter test,
> however,
> >> I
> >> >> can reproduce
> >> >> it on a fairly regular basis using our in-house application script,
> >> which
> >> >> puts
> >> >>
> >> >> Also, I don't seem to see this issue when there is only a single
> broker
> >> >> running, only when
> >> >> there is two or more running in a network of brokers configuration.
> >> >>
> >> >> When I do see this issue, I •do• see it on JConsole as well as in the
> >> >> ActiveMQ WebAdmin console.
> >> >>
> >> >>
> >> >
> >> >> To reproduce, start up multiple Producers, both of which are throwing
> >> >> 80,000+ messages
> >> >> at two brokers (which are networked together) and have the two
> consumers
> >> >> running on
> >> >> separate servers pulling items off of the application queue.
> >> >>
> >> >> On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com>
> >> wrote:
> >> >>
> >> >>> does jconsole show the same negative number? Do you have a test
> case?
> >> >>>
> >> >>> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
> >> >>> > After only running my 3 ActiveMQ brokers for short time, I am
> seeing
> >> >>> > negative numbers in
> >> >>> > the "Number of Pending Messages" column. I started seeing this at
> >> 5.3.2
> >> >>> > (after upgrading
> >> >>> > from 5.2.0) and opened and issue on it:
> >> >>> > • https://issues.apache.org/activemq/browse/AMQ-2887
> >> >>> >
> >> >>> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
> >> >>> however, I
> >> >>> > am still
> >> >>> > seeing this issue.
> >> >>> >
> >> >>> > Current Environment:
> >> >>> > • CentOS release 5.2 (both consumers and brokers)
> >> >>> > • 3 ActiveMQ brokers
> >> >>> > • 7 consumers (across different queues)
> >> >>> >
> >> >>> >
> >> >>> > Thanks,
> >> >>> >
> >> >>> > --
> >> >>> > Colleen Velo
> >> >>> > email: cmvelo@gmail.com
> >> >>> >
> >> >>> --
> >> >>> http://blog.garytully.com
> >> >>>
> >> >>> Open Source Integration
> >> >>> http://fusesource.com
> >> >>>
> >>
> >> >
> >>
> >>
> >>
> >> --
> >> http://blog.garytully.com
> >>
> >> Open Source Integration
> >> http://fusesource.com
> >>
> >
> >
> >
> > --
> > Colleen Velo
> > email: cmvelo@gmail.com
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
--
Colleen Velo
email: cmvelo@gmail.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Gary Tully <ga...@gmail.com>.
ok, I guess this is going to be very hard to track down without a
reproducible test case. Can you confirm that it is just the stats that
are wrong in this case, and that all expected messages are getting
delivered as expected.
On 9 September 2010 14:28, Colleen Velo <cm...@gmail.com> wrote:
> Gary,
>
> That doesn't seem to be the case here - namely, the JConsole queue count is
> accurately
> reflecting what the ActiveMQ web console is showing. I have rechecked our
> application
> code as well as the ActiveMQ server settings and in both cases we are using
> the default
> *dispatchAsync* value of false which, according to your blog post from Jan
> 2010, should
> take care of the "inflight" counts issue.
>
> It seems to be more a case where the queue "counters" seem to get "thrown
> off"
> and never get back in-sync until the broker is restarted.
> For example, I can see a number "-10" in the "Pending Messages" on the
> ActiveMQ
> web admin console (and in JConsole), but clicking on the queue name and
> showing
> the actual contents of the queue might reveal 0 items or 5 items, etc.
> (Note: in the above scenario, the producers were not putting any new items
> onto the
> queue, so the # of "Pending Messages" wasn't changing).
>
>
> On Thu, Sep 9, 2010 at 5:44 AM, Gary Tully <ga...@gmail.com> wrote:
>
>> Is it by any chance the inflight count in jconsole that is negative?
>> There is a good explanation for that,
>> http://blog.garytully.com/2010/01/activemq-prefetch-and-asyncdispatch.html
>>
>> On 8 September 2010 21:42, Colleen Velo <cm...@gmail.com> wrote:
>> > Gary,
>> > * *
>> > *Sorry - last message was incomplete - technical difficulties*
>> >
>> > Haven't had any luck reproducing this issue with a JMeter test, however,
>> I
>> > can reproduce
>> > it on a fairly regular basis using our in-house application script, which
>> > puts tons of application
>> > messages on the queues.
>> >
>> > Also, I don't seem to see this issue when there is only a single broker
>> > running, only when
>> > there is two or more running in a network of brokers configuration.
>> >
>> > When I do see this issue, I •do• see it on JConsole as well as in the
>> > ActiveMQ WebAdmin console.
>> >
>> > *Possible Test Case:*
>> >
>> > To reproduce, start up multiple Producers, both of which are throwing
>> > 80,000+ messages
>> > to the two brokers (which are configured as a network of brokers ) and
>> have
>> > two consumers running on
>> > separate servers pulling items off of the application queue(s).
>> >
>> >
>> > On Wed, Sep 8, 2010 at 3:39 PM, Colleen Velo <cm...@gmail.com> wrote:
>> >
>> >> Gary,
>> >>
>> >> Haven't had any luck reproducing this issue with a JMeter test, however,
>> I
>> >> can reproduce
>> >> it on a fairly regular basis using our in-house application script,
>> which
>> >> puts
>> >>
>> >> Also, I don't seem to see this issue when there is only a single broker
>> >> running, only when
>> >> there is two or more running in a network of brokers configuration.
>> >>
>> >> When I do see this issue, I •do• see it on JConsole as well as in the
>> >> ActiveMQ WebAdmin console.
>> >>
>> >>
>> >
>> >> To reproduce, start up multiple Producers, both of which are throwing
>> >> 80,000+ messages
>> >> at two brokers (which are networked together) and have the two consumers
>> >> running on
>> >> separate servers pulling items off of the application queue.
>> >>
>> >> On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com>
>> wrote:
>> >>
>> >>> does jconsole show the same negative number? Do you have a test case?
>> >>>
>> >>> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
>> >>> > After only running my 3 ActiveMQ brokers for short time, I am seeing
>> >>> > negative numbers in
>> >>> > the "Number of Pending Messages" column. I started seeing this at
>> 5.3.2
>> >>> > (after upgrading
>> >>> > from 5.2.0) and opened and issue on it:
>> >>> > • https://issues.apache.org/activemq/browse/AMQ-2887
>> >>> >
>> >>> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
>> >>> however, I
>> >>> > am still
>> >>> > seeing this issue.
>> >>> >
>> >>> > Current Environment:
>> >>> > • CentOS release 5.2 (both consumers and brokers)
>> >>> > • 3 ActiveMQ brokers
>> >>> > • 7 consumers (across different queues)
>> >>> >
>> >>> >
>> >>> > Thanks,
>> >>> >
>> >>> > --
>> >>> > Colleen Velo
>> >>> > email: cmvelo@gmail.com
>> >>> >
>> >>> --
>> >>> http://blog.garytully.com
>> >>>
>> >>> Open Source Integration
>> >>> http://fusesource.com
>> >>>
>>
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source Integration
>> http://fusesource.com
>>
>
>
>
> --
> Colleen Velo
> email: cmvelo@gmail.com
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary,
That doesn't seem to be the case here - namely, the JConsole queue count is
accurately
reflecting what the ActiveMQ web console is showing. I have rechecked our
application
code as well as the ActiveMQ server settings and in both cases we are using
the default
*dispatchAsync* value of false which, according to your blog post from Jan
2010, should
take care of the "inflight" counts issue.
It seems to be more a case where the queue "counters" seem to get "thrown
off"
and never get back in-sync until the broker is restarted.
For example, I can see a number "-10" in the "Pending Messages" on the
ActiveMQ
web admin console (and in JConsole), but clicking on the queue name and
showing
the actual contents of the queue might reveal 0 items or 5 items, etc.
(Note: in the above scenario, the producers were not putting any new items
onto the
queue, so the # of "Pending Messages" wasn't changing).
On Thu, Sep 9, 2010 at 5:44 AM, Gary Tully <ga...@gmail.com> wrote:
> Is it by any chance the inflight count in jconsole that is negative?
> There is a good explanation for that,
> http://blog.garytully.com/2010/01/activemq-prefetch-and-asyncdispatch.html
>
> On 8 September 2010 21:42, Colleen Velo <cm...@gmail.com> wrote:
> > Gary,
> > * *
> > *Sorry - last message was incomplete - technical difficulties*
> >
> > Haven't had any luck reproducing this issue with a JMeter test, however,
> I
> > can reproduce
> > it on a fairly regular basis using our in-house application script, which
> > puts tons of application
> > messages on the queues.
> >
> > Also, I don't seem to see this issue when there is only a single broker
> > running, only when
> > there is two or more running in a network of brokers configuration.
> >
> > When I do see this issue, I •do• see it on JConsole as well as in the
> > ActiveMQ WebAdmin console.
> >
> > *Possible Test Case:*
> >
> > To reproduce, start up multiple Producers, both of which are throwing
> > 80,000+ messages
> > to the two brokers (which are configured as a network of brokers ) and
> have
> > two consumers running on
> > separate servers pulling items off of the application queue(s).
> >
> >
> > On Wed, Sep 8, 2010 at 3:39 PM, Colleen Velo <cm...@gmail.com> wrote:
> >
> >> Gary,
> >>
> >> Haven't had any luck reproducing this issue with a JMeter test, however,
> I
> >> can reproduce
> >> it on a fairly regular basis using our in-house application script,
> which
> >> puts
> >>
> >> Also, I don't seem to see this issue when there is only a single broker
> >> running, only when
> >> there is two or more running in a network of brokers configuration.
> >>
> >> When I do see this issue, I •do• see it on JConsole as well as in the
> >> ActiveMQ WebAdmin console.
> >>
> >>
> >
> >> To reproduce, start up multiple Producers, both of which are throwing
> >> 80,000+ messages
> >> at two brokers (which are networked together) and have the two consumers
> >> running on
> >> separate servers pulling items off of the application queue.
> >>
> >> On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com>
> wrote:
> >>
> >>> does jconsole show the same negative number? Do you have a test case?
> >>>
> >>> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
> >>> > After only running my 3 ActiveMQ brokers for short time, I am seeing
> >>> > negative numbers in
> >>> > the "Number of Pending Messages" column. I started seeing this at
> 5.3.2
> >>> > (after upgrading
> >>> > from 5.2.0) and opened and issue on it:
> >>> > • https://issues.apache.org/activemq/browse/AMQ-2887
> >>> >
> >>> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
> >>> however, I
> >>> > am still
> >>> > seeing this issue.
> >>> >
> >>> > Current Environment:
> >>> > • CentOS release 5.2 (both consumers and brokers)
> >>> > • 3 ActiveMQ brokers
> >>> > • 7 consumers (across different queues)
> >>> >
> >>> >
> >>> > Thanks,
> >>> >
> >>> > --
> >>> > Colleen Velo
> >>> > email: cmvelo@gmail.com
> >>> >
> >>> --
> >>> http://blog.garytully.com
> >>>
> >>> Open Source Integration
> >>> http://fusesource.com
> >>>
>
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
--
Colleen Velo
email: cmvelo@gmail.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Gary Tully <ga...@gmail.com>.
Is it by any chance the inflight count in jconsole that is negative?
There is a good explanation for that,
http://blog.garytully.com/2010/01/activemq-prefetch-and-asyncdispatch.html
On 8 September 2010 21:42, Colleen Velo <cm...@gmail.com> wrote:
> Gary,
> * *
> *Sorry - last message was incomplete - technical difficulties*
>
> Haven't had any luck reproducing this issue with a JMeter test, however, I
> can reproduce
> it on a fairly regular basis using our in-house application script, which
> puts tons of application
> messages on the queues.
>
> Also, I don't seem to see this issue when there is only a single broker
> running, only when
> there is two or more running in a network of brokers configuration.
>
> When I do see this issue, I •do• see it on JConsole as well as in the
> ActiveMQ WebAdmin console.
>
> *Possible Test Case:*
>
> To reproduce, start up multiple Producers, both of which are throwing
> 80,000+ messages
> to the two brokers (which are configured as a network of brokers ) and have
> two consumers running on
> separate servers pulling items off of the application queue(s).
>
>
> On Wed, Sep 8, 2010 at 3:39 PM, Colleen Velo <cm...@gmail.com> wrote:
>
>> Gary,
>>
>> Haven't had any luck reproducing this issue with a JMeter test, however, I
>> can reproduce
>> it on a fairly regular basis using our in-house application script, which
>> puts
>>
>> Also, I don't seem to see this issue when there is only a single broker
>> running, only when
>> there is two or more running in a network of brokers configuration.
>>
>> When I do see this issue, I •do• see it on JConsole as well as in the
>> ActiveMQ WebAdmin console.
>>
>>
>
>> To reproduce, start up multiple Producers, both of which are throwing
>> 80,000+ messages
>> at two brokers (which are networked together) and have the two consumers
>> running on
>> separate servers pulling items off of the application queue.
>>
>> On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com> wrote:
>>
>>> does jconsole show the same negative number? Do you have a test case?
>>>
>>> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
>>> > After only running my 3 ActiveMQ brokers for short time, I am seeing
>>> > negative numbers in
>>> > the "Number of Pending Messages" column. I started seeing this at 5.3.2
>>> > (after upgrading
>>> > from 5.2.0) and opened and issue on it:
>>> > • https://issues.apache.org/activemq/browse/AMQ-2887
>>> >
>>> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
>>> however, I
>>> > am still
>>> > seeing this issue.
>>> >
>>> > Current Environment:
>>> > • CentOS release 5.2 (both consumers and brokers)
>>> > • 3 ActiveMQ brokers
>>> > • 7 consumers (across different queues)
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > --
>>> > Colleen Velo
>>> > email: cmvelo@gmail.com
>>> >
>>>
>>>
>>>
>>> --
>>> http://blog.garytully.com
>>>
>>> Open Source Integration
>>> http://fusesource.com
>>>
>>
>>
>>
>> --
>> Colleen Velo
>> email: cmvelo@gmail.com
>>
>
>
>
> --
> Colleen Velo
> email: cmvelo@gmail.com
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary,
* *
*Sorry - last message was incomplete - technical difficulties*
Haven't had any luck reproducing this issue with a JMeter test, however, I
can reproduce
it on a fairly regular basis using our in-house application script, which
puts tons of application
messages on the queues.
Also, I don't seem to see this issue when there is only a single broker
running, only when
there is two or more running in a network of brokers configuration.
When I do see this issue, I •do• see it on JConsole as well as in the
ActiveMQ WebAdmin console.
*Possible Test Case:*
To reproduce, start up multiple Producers, both of which are throwing
80,000+ messages
to the two brokers (which are configured as a network of brokers ) and have
two consumers running on
separate servers pulling items off of the application queue(s).
On Wed, Sep 8, 2010 at 3:39 PM, Colleen Velo <cm...@gmail.com> wrote:
> Gary,
>
> Haven't had any luck reproducing this issue with a JMeter test, however, I
> can reproduce
> it on a fairly regular basis using our in-house application script, which
> puts
>
> Also, I don't seem to see this issue when there is only a single broker
> running, only when
> there is two or more running in a network of brokers configuration.
>
> When I do see this issue, I •do• see it on JConsole as well as in the
> ActiveMQ WebAdmin console.
>
>
> To reproduce, start up multiple Producers, both of which are throwing
> 80,000+ messages
> at two brokers (which are networked together) and have the two consumers
> running on
> separate servers pulling items off of the application queue.
>
> On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com> wrote:
>
>> does jconsole show the same negative number? Do you have a test case?
>>
>> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
>> > After only running my 3 ActiveMQ brokers for short time, I am seeing
>> > negative numbers in
>> > the "Number of Pending Messages" column. I started seeing this at 5.3.2
>> > (after upgrading
>> > from 5.2.0) and opened and issue on it:
>> > • https://issues.apache.org/activemq/browse/AMQ-2887
>> >
>> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
>> however, I
>> > am still
>> > seeing this issue.
>> >
>> > Current Environment:
>> > • CentOS release 5.2 (both consumers and brokers)
>> > • 3 ActiveMQ brokers
>> > • 7 consumers (across different queues)
>> >
>> >
>> > Thanks,
>> >
>> > --
>> > Colleen Velo
>> > email: cmvelo@gmail.com
>> >
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source Integration
>> http://fusesource.com
>>
>
>
>
> --
> Colleen Velo
> email: cmvelo@gmail.com
>
--
Colleen Velo
email: cmvelo@gmail.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Colleen Velo <cm...@gmail.com>.
Gary,
Haven't had any luck reproducing this issue with a JMeter test, however, I
can reproduce
it on a fairly regular basis using our in-house application script, which
puts
Also, I don't seem to see this issue when there is only a single broker
running, only when
there is two or more running in a network of brokers configuration.
When I do see this issue, I •do• see it on JConsole as well as in the
ActiveMQ WebAdmin console.
To reproduce, start up multiple Producers, both of which are throwing
80,000+ messages
at two brokers (which are networked together) and have the two consumers
running on
separate servers pulling items off of the application queue.
On Wed, Sep 1, 2010 at 8:52 AM, Gary Tully <ga...@gmail.com> wrote:
> does jconsole show the same negative number? Do you have a test case?
>
> On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
> > After only running my 3 ActiveMQ brokers for short time, I am seeing
> > negative numbers in
> > the "Number of Pending Messages" column. I started seeing this at 5.3.2
> > (after upgrading
> > from 5.2.0) and opened and issue on it:
> > • https://issues.apache.org/activemq/browse/AMQ-2887
> >
> > It was recommended that I upgrade to ActiveMQ 5.4.0, which I did,
> however, I
> > am still
> > seeing this issue.
> >
> > Current Environment:
> > • CentOS release 5.2 (both consumers and brokers)
> > • 3 ActiveMQ brokers
> > • 7 consumers (across different queues)
> >
> >
> > Thanks,
> >
> > --
> > Colleen Velo
> > email: cmvelo@gmail.com
> >
>
>
>
> --
> http://blog.garytully.com
>
> Open Source Integration
> http://fusesource.com
>
--
Colleen Velo
email: cmvelo@gmail.com
Re: Even after upgrade to ActiveMQ 5.4.0, still seeing negative #'s
in the "Pending Msgs" column
Posted by Gary Tully <ga...@gmail.com>.
does jconsole show the same negative number? Do you have a test case?
On 1 September 2010 13:52, Colleen Velo <cm...@gmail.com> wrote:
> After only running my 3 ActiveMQ brokers for short time, I am seeing
> negative numbers in
> the "Number of Pending Messages" column. I started seeing this at 5.3.2
> (after upgrading
> from 5.2.0) and opened and issue on it:
> • https://issues.apache.org/activemq/browse/AMQ-2887
>
> It was recommended that I upgrade to ActiveMQ 5.4.0, which I did, however, I
> am still
> seeing this issue.
>
> Current Environment:
> • CentOS release 5.2 (both consumers and brokers)
> • 3 ActiveMQ brokers
> • 7 consumers (across different queues)
>
>
> Thanks,
>
> --
> Colleen Velo
> email: cmvelo@gmail.com
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com