You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by fenbers <Ma...@noaa.gov> on 2010/12/09 19:28:33 UTC
ActiveMQ slowness every 30 days
Hi,
My AMQ clients get very slow or freeze up altogether after about 30 days of
ActiveMQ running. I solve the problem (until about 30 days later) by
deleting the $ACTIVEMQ_HOME/data/kahadb directory and restarting ActiveMQ.
In looking for clues to this problem, I notice (before I delete the
directory and its contents) that the db-*.log files (where * is an
incrementing number) are all about 33MB in size but have a timestamp about
every 3 or 4 days in the early part of the 30 days, but gradually become
more frequent such that by nearly 30 days since an AMQ restart, there are
about 3 or 4 files per DAY. I really have no idea if this is a cause to the
problem or a result of it. Nor do I know what the purpose of these files
are as they are binary, not text files.
Here's another possible cause: My client apps (subscribers) do not
acknowledge "topic" messages. This is because I want messages to be keep
being redelivered any time the subscriber is restarted (so long as the
messages haven't expired). But perhaps this is the cause of the exponential
frequency of log files and the slowdown after 30 days.
What can I try (without acknowledging messages) to keep AMQ from bogging
down my clients after 30 days and consuming an exponential amount of disk
space?
Mark
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slowness-every-30-days-tp3080715p3080715.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slowness every 30 days
Posted by James Green <ja...@gmail.com>.
I would call this a mis-use of the technology. What you are trying to do is
to restore state across application restarts.
By all means have a client component listening for messages ("progress
updates" if you like). However, it should update a database that your GUI
should reflect.
I can't comment on the rest as I don't have the necessary experience -
though what you are seeing is likely the result of your perhaps
unanticipated use of AMQ.
James
On 10 December 2010 18:34, fenbers <Ma...@noaa.gov> wrote:
>
>
> James Green-3 [via ActiveMQ] wrote:
> Why do you need to redeliver topic messages? To me you
> are bending a message
>
> queue to react like a database.
>
> I appreciate the reply. Here's a little bit of background
> info... A
> hydrologist takes 10 different actions on a particular river to issue a
> river forecast. Several hydrologists work concurrently on different
> rivers. The goal is to show a status of where each of 30 rivers are
> in
> the forecast process at any one time. This prevents one hydrologist
> from beginning work on a river forecast that another hydrologist is
> already working on. A JMS message is sent to AMQ each time one of the
> actions is taken on a river. So my AMQ client app consumes these
> messages and displays the status of these actions in a GUI.
>
> If my client app (consumer) is restarted or is started late n the
> process, then it needs to correctly redisplay the status of all 30
> rivers -- even those that occurred prior to the launch of the latest
> instance, thus redelivering the message causes my consumer to correctly
> redisplay message that were sent prior to the launch of the consumer.
> Each hydrologist runs an instance of my consumer to see the progress of
> all the other hydrologists working on the river forecasts. The
> subscribers remain alive in ActiveMQ. This is why my consumer can be
> relaunched and be able to get all of the back messages.
>
> This approach works very well (except for the bogging down of this
> consumer and producers after about 30 days). If there is a smarter
> way
> to have my consumer redisplay actions taken prior to its launch, then
> please share it with me because I'm eager to learn of better ways to do
> things!
>
> Does the log show anything? I'm wondering if the kahadb cleanup is
> kicking
>
> in and is eventually going over some threshold causing the slowdown.
>
>
> The log shows lots of stuff (33MB in about 6 hours) but most of it
> comprises of this error:
> YYYY-MM-DD HH:MM:SS,ddd | WARN | Duplicate message add attempt
> rejected. Message id: somehostname-59695-1291947282050-0:1:1:1:513 |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport:
> tcp:///165.92.xx.xx:34201
>
> This message means very little to me, although the 513 increments with
> each log entry. Is the 59695 number the process ID? Or what is
> the
> 34201 number? What's the 0:1:1:1: mean? The text would seem to
> imply
> that I'm trying to submit the same message repeatedly to the broker,
> but I haven't found where I could be doing this... I could learn more
> if the log reported what the repeated message was, but it doesn't,
> unfortunately. Any ideas?
>
> A grep of "cleanup" on the log files shows a handful of the following
> messages:
>
> INFO | Slow KahaDB access: cleanup took xxx
>
> where xxx ranges anywhere from 500 to 9000. Are these
> seconds, or milliseconds? And are these numbers typical or
> astronomical?
>
> I should also mention that messages sent from the producers are set to
> expire at less than 12 from the current time, so it's not like message
> are kept for long periods of time.
>
> Mark
>
>
>
>
> begin:vcard
> fn:Mark Fenbers
> n:Fenbers;Mark
> org:National Weather Service;Ohio River Forecast Center
> adr:;;1901 South OH-134;Wilmington;OH;45177-1908;USA
> email;internet:Mark.Fenbers@noaa.gov<em...@noaa.gov>
> title:Sr. HAS Meteorologist
> tel;work:937-383-0430
> tel;fax:937-383-0033
> note:"HAS" is an acronym for Hydrometeorological Analysis and Support
> x-mozilla-html:TRUE
> url:http://weather.gov/ohrfc
> version:2.1
> end:vcard
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-slowness-every-30-days-tp3080715p3082497.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ slowness every 30 days
Posted by fenbers <Ma...@noaa.gov>.
James Green-3 [via ActiveMQ] wrote:
Why do you need to redeliver topic messages? To me you
are bending a message
queue to react like a database.
I appreciate the reply. Here's a little bit of background info... A
hydrologist takes 10 different actions on a particular river to issue a
river forecast. Several hydrologists work concurrently on different
rivers. The goal is to show a status of where each of 30 rivers are in
the forecast process at any one time. This prevents one hydrologist
from beginning work on a river forecast that another hydrologist is
already working on. A JMS message is sent to AMQ each time one of the
actions is taken on a river. So my AMQ client app consumes these
messages and displays the status of these actions in a GUI.
If my client app (consumer) is restarted or is started late n the
process, then it needs to correctly redisplay the status of all 30
rivers -- even those that occurred prior to the launch of the latest
instance, thus redelivering the message causes my consumer to correctly
redisplay message that were sent prior to the launch of the consumer.
Each hydrologist runs an instance of my consumer to see the progress of
all the other hydrologists working on the river forecasts. The
subscribers remain alive in ActiveMQ. This is why my consumer can be
relaunched and be able to get all of the back messages.
This approach works very well (except for the bogging down of this
consumer and producers after about 30 days). If there is a smarter way
to have my consumer redisplay actions taken prior to its launch, then
please share it with me because I'm eager to learn of better ways to do
things!
Does the log show anything? I'm wondering if the kahadb cleanup is
kicking
in and is eventually going over some threshold causing the slowdown.
The log shows lots of stuff (33MB in about 6 hours) but most of it
comprises of this error:
YYYY-MM-DD HH:MM:SS,ddd | WARN | Duplicate message add attempt
rejected. Message id: somehostname-59695-1291947282050-0:1:1:1:513 |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport:
tcp:///165.92.xx.xx:34201
This message means very little to me, although the 513 increments with
each log entry. Is the 59695 number the process ID? Or what is the
34201 number? What's the 0:1:1:1: mean? The text would seem to imply
that I'm trying to submit the same message repeatedly to the broker,
but I haven't found where I could be doing this... I could learn more
if the log reported what the repeated message was, but it doesn't,
unfortunately. Any ideas?
A grep of "cleanup" on the log files shows a handful of the following
messages:
INFO | Slow KahaDB access: cleanup took xxx
where xxx ranges anywhere from 500 to 9000. Are these
seconds, or milliseconds? And are these numbers typical or
astronomical?
I should also mention that messages sent from the producers are set to
expire at less than 12 from the current time, so it's not like message
are kept for long periods of time.
Mark
begin:vcard
fn:Mark Fenbers
n:Fenbers;Mark
org:National Weather Service;Ohio River Forecast Center
adr:;;1901 South OH-134;Wilmington;OH;45177-1908;USA
email;internet:Mark.Fenbers@noaa.gov
title:Sr. HAS Meteorologist
tel;work:937-383-0430
tel;fax:937-383-0033
note:"HAS" is an acronym for Hydrometeorological Analysis and Support
x-mozilla-html:TRUE
url:http://weather.gov/ohrfc
version:2.1
end:vcard
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slowness-every-30-days-tp3080715p3082497.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ slowness every 30 days
Posted by James Green <ja...@gmail.com>.
Why do you need to redeliver topic messages? To me you are bending a message
queue to react like a database.
Does the log show anything? I'm wondering if the kahadb cleanup is kicking
in and is eventually going over some threshold causing the slowdown.
James
On 9 December 2010 18:28, fenbers <Ma...@noaa.gov> wrote:
>
> Hi,
>
> My AMQ clients get very slow or freeze up altogether after about 30 days of
> ActiveMQ running. I solve the problem (until about 30 days later) by
> deleting the $ACTIVEMQ_HOME/data/kahadb directory and restarting ActiveMQ.
>
> In looking for clues to this problem, I notice (before I delete the
> directory and its contents) that the db-*.log files (where * is an
> incrementing number) are all about 33MB in size but have a timestamp about
> every 3 or 4 days in the early part of the 30 days, but gradually become
> more frequent such that by nearly 30 days since an AMQ restart, there are
> about 3 or 4 files per DAY. I really have no idea if this is a cause to
> the
> problem or a result of it. Nor do I know what the purpose of these files
> are as they are binary, not text files.
>
> Here's another possible cause: My client apps (subscribers) do not
> acknowledge "topic" messages. This is because I want messages to be keep
> being redelivered any time the subscriber is restarted (so long as the
> messages haven't expired). But perhaps this is the cause of the
> exponential
> frequency of log files and the slowdown after 30 days.
>
> What can I try (without acknowledging messages) to keep AMQ from bogging
> down my clients after 30 days and consuming an exponential amount of disk
> space?
>
> Mark
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-slowness-every-30-days-tp3080715p3080715.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ slowness every 30 days
Posted by fenbers <Ma...@noaa.gov>.
artnaseef [via ActiveMQ] wrote:
> Can you take a look at the web interface to see the destinations and
> numbers of messages on each?
Queues:
ActiveMQ.DLQ: # pending 141990; # consumers 0; enqueued 21798;
dequeued 0
Topics:
(listing only the ones not beginning with ActiveMQ.Advisory...)
GIFTS.fromSnooper: # consumers 30; # enqueued -1181; # dequeued 0
GIFTS.icon: # consumers 30; # enqueued -17341; # dequeued 0
GIFTS.toSnooper: # consumers 1; # enqueued 424; # dequeued 0
What does it mean to have these negative numbers (-1181 and -17341)??
Mark
begin:vcard
fn:Mark Fenbers
n:Fenbers;Mark
org:National Weather Service;Ohio River Forecast Center
adr:;;1901 South OH-134;Wilmington;OH;45177-1908;USA
email;internet:Mark.Fenbers@noaa.gov
title:Sr. HAS Meteorologist
tel;work:937-383-0430
tel;fax:937-383-0033
note:"HAS" is an acronym for Hydrometeorological Analysis and Support
x-mozilla-html:TRUE
url:http://weather.gov/ohrfc
version:2.1
end:vcard
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-slowness-every-30-days-tp3080715p3082521.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.