You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Scammell <ma...@meridianenergy.co.nz> on 2015/07/27 22:47:22 UTC
Store percent used
Hello,
I'm using ActiveMQ version 5.10.1.
Since we started using ActiveMQ a few weeks ago, the store percentage for
the broker has been steadily increasing - from 10% at the start of the month
to 29% currently.
In that period, we have processed (sent to queue and subsequently removed)
approximately 5 million records.
We currently hold on to a ever-rotating amount of ~20,000 messages for a
maximum of 7 days, after which we then delete them.
We don't seem to be seeing any slow-down in the increase of store percent
utilised even though the system has reached a point where the amount of
messages we are processing each day is not increasing. Could anyone advise
on a strategy for reducing this figure?
Thanks in advance for any assistance.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store percent used
Posted by Tim Bain <tb...@alumni.duke.edu>.
That's good to know. I (wrongly) assumed that what counted towards
storeUsage was undeleted messages, not undeleted files. Now that I think
about it, I can see why that's the decision that was made, though it's a
little confusing to see that we're "using" X bytes when in reality almost
all of them are deleted. But it's the lesser of the two evils, I suppose.
I'm frankly surprised I've never heard anyone suggest having KahaDB do
periodic compaction of data files in lieu of periodic deletion of entirely
unused files, to avoid all the nonsense surrounding this issue. So let me
be the first (that I've heard):
https://issues.apache.org/jira/browse/AMQ-5905
Tim
On Jul 28, 2015 4:56 AM, "Christopher Shannon" <
christopher.l.shannon@gmail.com> wrote:
> Tim, the problem of KahaDB files not being deleted is visible in the
> broker. All of the KahaDB data files are counted against store usage and
> I've seen it occur several times where the store usage steadily increases
> as reported by the SystemUsage metric the point where the store fills up
> and producers get blocked because of scattered messages across data files.
>
> Now, this may not be the actual issue in this case because the original
> post mentions that messages shouldn't be held onto for more than 7 days but
> I figured it was something worth looking at as the symptom of a steadily
> increasing store usage as the month goes on is definitely a symptom of what
> Gary describes in his post.
>
> On Tue, Jul 28, 2015 at 1:00 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > If you examine the broker via JMX, do you see an increasing number of
> > messages on the various destinations as the percent usage increases
> (which
> > would indicate that the deletions are not succeeding)?
> >
> > Have you confirmed that messages are not building up in the DLQ (dead
> > letter queue)? You either need to configure a custom process to go
> through
> > any messages routed there and do whatever you want with them (delete
> them,
> > log them for a developer to look at later, whatever) or you need to
> > configure the broker to not put them in the DLQ to begin with (via the
> > discarding DQL plugin).
> >
> > Chris, the problem Gary's post solves is one of KahaDB data files not
> being
> > deleted rather than increasing storeUsage; I don't think that problem
> would
> > manifest as something visible from within the broker like what Mark has
> > described, only from the outside when examining the file system.
> >
> > Tim
> >
> > On Mon, Jul 27, 2015 at 3:06 PM, Christopher Shannon <
> > christopher.l.shannon@gmail.com> wrote:
> >
> > > If you are using KahaDB you might be running into a limitation of
> mixing
> > > destinations with fast message rates vs slow message rates. Gary
> wrote a
> > > good blog post about it here:
> > >
> >
> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
> > > Take a look at that and see if it helps you.
> > >
> > >
> > > On Mon, Jul 27, 2015 at 4:47 PM, Scammell <
> > > mark.harris@meridianenergy.co.nz>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > I'm using ActiveMQ version 5.10.1.
> > > >
> > > > Since we started using ActiveMQ a few weeks ago, the store percentage
> > for
> > > > the broker has been steadily increasing - from 10% at the start of
> the
> > > > month
> > > > to 29% currently.
> > > >
> > > > In that period, we have processed (sent to queue and subsequently
> > > removed)
> > > > approximately 5 million records.
> > > >
> > > > We currently hold on to a ever-rotating amount of ~20,000 messages
> for
> > a
> > > > maximum of 7 days, after which we then delete them.
> > > >
> > > > We don't seem to be seeing any slow-down in the increase of store
> > percent
> > > > utilised even though the system has reached a point where the amount
> of
> > > > messages we are processing each day is not increasing. Could anyone
> > > advise
> > > > on a strategy for reducing this figure?
> > > >
> > > > Thanks in advance for any assistance.
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > >
> > >
> >
>
Re: Store percent used
Posted by Christopher Shannon <ch...@gmail.com>.
Tim, the problem of KahaDB files not being deleted is visible in the
broker. All of the KahaDB data files are counted against store usage and
I've seen it occur several times where the store usage steadily increases
as reported by the SystemUsage metric the point where the store fills up
and producers get blocked because of scattered messages across data files.
Now, this may not be the actual issue in this case because the original
post mentions that messages shouldn't be held onto for more than 7 days but
I figured it was something worth looking at as the symptom of a steadily
increasing store usage as the month goes on is definitely a symptom of what
Gary describes in his post.
On Tue, Jul 28, 2015 at 1:00 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> If you examine the broker via JMX, do you see an increasing number of
> messages on the various destinations as the percent usage increases (which
> would indicate that the deletions are not succeeding)?
>
> Have you confirmed that messages are not building up in the DLQ (dead
> letter queue)? You either need to configure a custom process to go through
> any messages routed there and do whatever you want with them (delete them,
> log them for a developer to look at later, whatever) or you need to
> configure the broker to not put them in the DLQ to begin with (via the
> discarding DQL plugin).
>
> Chris, the problem Gary's post solves is one of KahaDB data files not being
> deleted rather than increasing storeUsage; I don't think that problem would
> manifest as something visible from within the broker like what Mark has
> described, only from the outside when examining the file system.
>
> Tim
>
> On Mon, Jul 27, 2015 at 3:06 PM, Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
> > If you are using KahaDB you might be running into a limitation of mixing
> > destinations with fast message rates vs slow message rates. Gary wrote a
> > good blog post about it here:
> >
> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
> > Take a look at that and see if it helps you.
> >
> >
> > On Mon, Jul 27, 2015 at 4:47 PM, Scammell <
> > mark.harris@meridianenergy.co.nz>
> > wrote:
> >
> > > Hello,
> > >
> > > I'm using ActiveMQ version 5.10.1.
> > >
> > > Since we started using ActiveMQ a few weeks ago, the store percentage
> for
> > > the broker has been steadily increasing - from 10% at the start of the
> > > month
> > > to 29% currently.
> > >
> > > In that period, we have processed (sent to queue and subsequently
> > removed)
> > > approximately 5 million records.
> > >
> > > We currently hold on to a ever-rotating amount of ~20,000 messages for
> a
> > > maximum of 7 days, after which we then delete them.
> > >
> > > We don't seem to be seeing any slow-down in the increase of store
> percent
> > > utilised even though the system has reached a point where the amount of
> > > messages we are processing each day is not increasing. Could anyone
> > advise
> > > on a strategy for reducing this figure?
> > >
> > > Thanks in advance for any assistance.
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>
Re: Store percent used
Posted by Tim Bain <tb...@alumni.duke.edu>.
Hmm, that doesn't sound like the magnitude we were expecting. How old was
oldest DLQ message compared to the oldest "live" message? Is there any
chance there are old messages in destinations other than the DLQ? (Either
queues or durable topic subscriptions.) Your initial description sounded
like that wasn't the case (messages were processed in a relatively timely
manner, unless they got DLQed), but I may have read something into it that
wasn't actually there.
On Tue, Jul 28, 2015 at 8:48 PM, Scammell <ma...@meridianenergy.co.nz>
wrote:
> Thanks for this.
>
> I tried this with a test server which was showing 25% store used. I deleted
> each of the 140 or so DLQ entries manually (the "Purge" option for th
> ewhole
> queue didn't seem to do anything) and the store went down to 21%.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945p4700024.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Store percent used
Posted by Scammell <ma...@meridianenergy.co.nz>.
Thanks for this.
I tried this with a test server which was showing 25% store used. I deleted
each of the 140 or so DLQ entries manually (the "Purge" option for th ewhole
queue didn't seem to do anything) and the store went down to 21%.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945p4700024.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store percent used
Posted by Tim Bain <tb...@alumni.duke.edu>.
One old message in the DLQ is all it takes to keep every single KahaDB data
file after it from being deleted. Not thousands, not hundreds, not tens.
One. As currently implemented, KahaDB requires you to choose between a DLQ
policy that involves keeping DLQ messages around and a KahaDB storage
profile that is proportional to the volume of live messages.
mKahaDB might allow you to have both if you use a separate KahaDB instance
for the DLQ, though I'm not sure I've yet heard anyone report successfully
configuring that setup so I can't say for sure that it would work properly.
It would be easy to test whether the DLQ messages are the root cause of
this behavior: delete them all and see whether your store usage drops to
the level you expect shortly thereafter. (No more than 30 seconds, unless
you've overridden the cleanupInterval property.)
Tim
On Tue, Jul 28, 2015 at 2:39 PM, Scammell <ma...@meridianenergy.co.nz>
wrote:
> Thanks for your thoughts.
>
> The web console shows that the messages are being dequeued at the same rate
> as they are being queued so I was presuming that they were being deleted
> successfully.
>
> I will investigate how to use JMX to view the broker and report back.
>
> There are a few messages in the DLQ but they number tens, not hundreds of
> thousands. These are placed in the DLQ for a valid error scenario which we
> are aware of.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945p4700008.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Store percent used
Posted by Scammell <ma...@meridianenergy.co.nz>.
Thanks for your thoughts.
The web console shows that the messages are being dequeued at the same rate
as they are being queued so I was presuming that they were being deleted
successfully.
I will investigate how to use JMX to view the broker and report back.
There are a few messages in the DLQ but they number tens, not hundreds of
thousands. These are placed in the DLQ for a valid error scenario which we
are aware of.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945p4700008.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store percent used
Posted by Tim Bain <tb...@alumni.duke.edu>.
If you examine the broker via JMX, do you see an increasing number of
messages on the various destinations as the percent usage increases (which
would indicate that the deletions are not succeeding)?
Have you confirmed that messages are not building up in the DLQ (dead
letter queue)? You either need to configure a custom process to go through
any messages routed there and do whatever you want with them (delete them,
log them for a developer to look at later, whatever) or you need to
configure the broker to not put them in the DLQ to begin with (via the
discarding DQL plugin).
Chris, the problem Gary's post solves is one of KahaDB data files not being
deleted rather than increasing storeUsage; I don't think that problem would
manifest as something visible from within the broker like what Mark has
described, only from the outside when examining the file system.
Tim
On Mon, Jul 27, 2015 at 3:06 PM, Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> If you are using KahaDB you might be running into a limitation of mixing
> destinations with fast message rates vs slow message rates. Gary wrote a
> good blog post about it here:
> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
> Take a look at that and see if it helps you.
>
>
> On Mon, Jul 27, 2015 at 4:47 PM, Scammell <
> mark.harris@meridianenergy.co.nz>
> wrote:
>
> > Hello,
> >
> > I'm using ActiveMQ version 5.10.1.
> >
> > Since we started using ActiveMQ a few weeks ago, the store percentage for
> > the broker has been steadily increasing - from 10% at the start of the
> > month
> > to 29% currently.
> >
> > In that period, we have processed (sent to queue and subsequently
> removed)
> > approximately 5 million records.
> >
> > We currently hold on to a ever-rotating amount of ~20,000 messages for a
> > maximum of 7 days, after which we then delete them.
> >
> > We don't seem to be seeing any slow-down in the increase of store percent
> > utilised even though the system has reached a point where the amount of
> > messages we are processing each day is not increasing. Could anyone
> advise
> > on a strategy for reducing this figure?
> >
> > Thanks in advance for any assistance.
> >
> >
> >
> > --
> > View this message in context:
> > http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Re: Store percent used
Posted by Christopher Shannon <ch...@gmail.com>.
If you are using KahaDB you might be running into a limitation of mixing
destinations with fast message rates vs slow message rates. Gary wrote a
good blog post about it here:
http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
Take a look at that and see if it helps you.
On Mon, Jul 27, 2015 at 4:47 PM, Scammell <ma...@meridianenergy.co.nz>
wrote:
> Hello,
>
> I'm using ActiveMQ version 5.10.1.
>
> Since we started using ActiveMQ a few weeks ago, the store percentage for
> the broker has been steadily increasing - from 10% at the start of the
> month
> to 29% currently.
>
> In that period, we have processed (sent to queue and subsequently removed)
> approximately 5 million records.
>
> We currently hold on to a ever-rotating amount of ~20,000 messages for a
> maximum of 7 days, after which we then delete them.
>
> We don't seem to be seeing any slow-down in the increase of store percent
> utilised even though the system has reached a point where the amount of
> messages we are processing each day is not increasing. Could anyone advise
> on a strategy for reducing this figure?
>
> Thanks in advance for any assistance.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Store-percent-used-tp4699945.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>