You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by patrikwettergren <pa...@tr.com> on 2015/11/18 10:38:21 UTC
ActiveMQ Brokers starts the reindex of the KahaDB on every startup
Hello we are running on a Master/Slave Setup of two ActiveMQ Brokers with
version 5.9.
We have taken a copy of our ActiveMQ Production KahaDB and placed it in the
Stage Environment. This to resolve an issue with DLQ placed Messages.
The issue we are seeing now is that the Stage ActiveMQ Brokers are running a
full re-index every time they are restarted and when the re-indexing are
done the Broker states that there is not enough space on the disk and it
then caps the usage of the disk to what it reads is available.
The problem is here that the used disk space is from the copied KahaDB that
the AMQ Broker just re-indexed.
So in other word it reads the KahaDB files fine to re-index the data in
them, but does not take into account of what diskspace it is already using
when reading space available when it is finished the re-indexing.
The Size of the KahaDB Set is 226GB it is placed on a 500GB Mount on and the
ActiveMQ Brokers are runninng on Oracle Linux Server release 6.5 OL-6.5.1-64
servers with JDK 6.
In activemq.xml we have set the flow control so that anyhting above 75% of
storage usage stops messages handling on specific queues and the storage
usage is set :
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="75" />
</memoryUsage>
<storeUsage>
<storeUsage limit="400 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
But even so the AMQ Brokers only sees the availble Disk space in '500G 226G
275G 46%' on the Mount and activates the flow control since the 275Gb
available space is smaller then the 400GB stated in activemq.xml.
So the question is now why is the Broker doing this even after it have read
through the KahaDB data and reindexed it and updated the db.data file and is
using the 226GB but does not take this into account when reading the
'storageUsage' limit in activemq.xml
I might be missing something here ot is it just as simple that I need to get
the Mountpoint of 500GB larger. So any help would be very appriciated.
Kind regards
Patrik Wettergren
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-reindex-of-the-KahaDB-on-every-startup-tp4704035.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Brokers starts the reindex of the KahaDB on every
startup
Posted by patrikwettergren <pa...@tr.com>.
Thank you Tim.
Now I have get it ready for the upgrade run.
Kind regards
Patrik
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-reindex-of-the-KahaDB-on-every-startup-tp4704035p4704054.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Brokers starts the reindex of the KahaDB on every startup
Posted by Tim Bain <tb...@alumni.duke.edu>.
If only. The worst part is that the mobile version of the webapp doesn't
show Fixed Version, which can make it completely useless if you're trying
to figure out whether a given bug fix is already in the version you're
using or you need to upgrade.
On Wed, Nov 18, 2015 at 8:32 AM, Basmajian, Raffi <rb...@ofiglobal.com>
wrote:
> Tim, we need an app for that :-)
>
> Raffi
>
> -----Original Message-----
> From: tbain98@gmail.com [mailto:tbain98@gmail.com] On Behalf Of Tim Bain
> Sent: Wednesday, November 18, 2015 10:27 AM
> To: ActiveMQ Users
> Subject: Re: ActiveMQ Brokers starts the reindex of the KahaDB on every
> startup [ EXTERNAL ]
>
> Yep, that's the one. Thanks for finding it; JIRA's not the easiest to use
> from a phone.
>
> On Wed, Nov 18, 2015 at 8:18 AM, Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
> > Tim, I think you are referring to this fix here:
> > https://issues.apache.org/jira/browse/AMQ-4832
> >
> > Looks like it was done in 5.9.1 and later.
> >
> > There's also been some other fixes and improvements done in this area
> > recently so I would definitely recommend upgrading to a more recent
> > version and giving that a shot.
> >
> > On Wed, Nov 18, 2015 at 8:55 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> > > I have a vague memory of seeing a fix for exactly this bug in one of
> > > the most recent (5.11.x or later) versions, but I couldn't find it
> > > in a quick search of JIRA. Can anyone confirm my memory?
> > >
> > > If I'm remembering correctly, then an upgrade to 5.12.1 should fix
> > > your problem. If your organization will let you, you could upgrade
> > > your
> > staging
> > > environment to see if the observed behavior changes.
> > >
> > > Tim
> > > On Nov 18, 2015 2:53 AM, "patrikwettergren"
> > > <pa...@tr.com>
> > > wrote:
> > >
> > > > Hello we are running on a Master/Slave Setup of two ActiveMQ
> > > > Brokers
> > with
> > > > version 5.9.
> > > >
> > > > We have taken a copy of our ActiveMQ Production KahaDB and placed
> > > > it in
> > > the
> > > > Stage Environment. This to resolve an issue with DLQ placed Messages.
> > > >
> > > > The issue we are seeing now is that the Stage ActiveMQ Brokers are
> > > running
> > > > a
> > > > full re-index every time they are restarted and when the
> > > > re-indexing
> > are
> > > > done the Broker states that there is not enough space on the disk
> > > > and
> > it
> > > > then caps the usage of the disk to what it reads is available.
> > > >
> > > > The problem is here that the used disk space is from the copied
> > > > KahaDB
> > > that
> > > > the AMQ Broker just re-indexed.
> > > >
> > > > So in other word it reads the KahaDB files fine to re-index the
> > > > data in them, but does not take into account of what diskspace it
> > > > is already
> > > using
> > > > when reading space available when it is finished the re-indexing.
> > > >
> > > > The Size of the KahaDB Set is 226GB it is placed on a 500GB Mount
> > > > on
> > and
> > > > the
> > > > ActiveMQ Brokers are runninng on Oracle Linux Server release 6.5
> > > > OL-6.5.1-64
> > > > servers with JDK 6.
> > > >
> > > >
> > > > In activemq.xml we have set the flow control so that anyhting
> > > > above 75%
> > > of
> > > > storage usage stops messages handling on specific queues and the
> > storage
> > > > usage is set :
> > > > <systemUsage>
> > > > <systemUsage>
> > > > <memoryUsage>
> > > > <memoryUsage percentOfJvmHeap="75" />
> > > > </memoryUsage>
> > > > <storeUsage>
> > > > <storeUsage limit="400 gb"/>
> > > > </storeUsage>
> > > > <tempUsage>
> > > > <tempUsage limit="50 gb"/>
> > > > </tempUsage>
> > > > </systemUsage>
> > > > </systemUsage>
> > > >
> > > > But even so the AMQ Brokers only sees the availble Disk space in
> > > > '500G 226G 275G 46%' on the Mount and activates the flow control
> > > > since the 275Gb available space is smaller then the 400GB stated
> > > > in activemq.xml.
> > > >
> > > > So the question is now why is the Broker doing this even after it
> > > > have
> > > read
> > > > through the KahaDB data and reindexed it and updated the db.data
> > > > file
> > and
> > > > is
> > > > using the 226GB but does not take this into account when reading
> > > > the 'storageUsage' limit in activemq.xml
> > > >
> > > >
> > > > I might be missing something here ot is it just as simple that I
> > > > need
> > to
> > > > get
> > > > the Mountpoint of 500GB larger. So any help would be very
> appriciated.
> > > >
> > > >
> > > > Kind regards
> > > > Patrik Wettergren
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-rein
> > dex-of-the-KahaDB-on-every-startup-tp4704035.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > >
> > >
> >
>
> This e-mail transmission may contain information that is proprietary,
> privileged and/or confidential and is intended exclusively for the
> person(s) to whom it is addressed. Any use, copying, retention or
> disclosure by any person other than the intended recipient or the intended
> recipient's designees is strictly prohibited. If you are not the intended
> recipient or their designee, please notify the sender immediately by return
> e-mail and delete all copies. OppenheimerFunds may, at its sole discretion,
> monitor, review, retain and/or disclose the content of all email
> communications.
>
RE: ActiveMQ Brokers starts the reindex of the KahaDB on every startup
Posted by "Basmajian, Raffi" <rb...@ofiglobal.com>.
Tim, we need an app for that :-)
Raffi
-----Original Message-----
From: tbain98@gmail.com [mailto:tbain98@gmail.com] On Behalf Of Tim Bain
Sent: Wednesday, November 18, 2015 10:27 AM
To: ActiveMQ Users
Subject: Re: ActiveMQ Brokers starts the reindex of the KahaDB on every startup [ EXTERNAL ]
Yep, that's the one. Thanks for finding it; JIRA's not the easiest to use from a phone.
On Wed, Nov 18, 2015 at 8:18 AM, Christopher Shannon < christopher.l.shannon@gmail.com> wrote:
> Tim, I think you are referring to this fix here:
> https://issues.apache.org/jira/browse/AMQ-4832
>
> Looks like it was done in 5.9.1 and later.
>
> There's also been some other fixes and improvements done in this area
> recently so I would definitely recommend upgrading to a more recent
> version and giving that a shot.
>
> On Wed, Nov 18, 2015 at 8:55 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > I have a vague memory of seeing a fix for exactly this bug in one of
> > the most recent (5.11.x or later) versions, but I couldn't find it
> > in a quick search of JIRA. Can anyone confirm my memory?
> >
> > If I'm remembering correctly, then an upgrade to 5.12.1 should fix
> > your problem. If your organization will let you, you could upgrade
> > your
> staging
> > environment to see if the observed behavior changes.
> >
> > Tim
> > On Nov 18, 2015 2:53 AM, "patrikwettergren"
> > <pa...@tr.com>
> > wrote:
> >
> > > Hello we are running on a Master/Slave Setup of two ActiveMQ
> > > Brokers
> with
> > > version 5.9.
> > >
> > > We have taken a copy of our ActiveMQ Production KahaDB and placed
> > > it in
> > the
> > > Stage Environment. This to resolve an issue with DLQ placed Messages.
> > >
> > > The issue we are seeing now is that the Stage ActiveMQ Brokers are
> > running
> > > a
> > > full re-index every time they are restarted and when the
> > > re-indexing
> are
> > > done the Broker states that there is not enough space on the disk
> > > and
> it
> > > then caps the usage of the disk to what it reads is available.
> > >
> > > The problem is here that the used disk space is from the copied
> > > KahaDB
> > that
> > > the AMQ Broker just re-indexed.
> > >
> > > So in other word it reads the KahaDB files fine to re-index the
> > > data in them, but does not take into account of what diskspace it
> > > is already
> > using
> > > when reading space available when it is finished the re-indexing.
> > >
> > > The Size of the KahaDB Set is 226GB it is placed on a 500GB Mount
> > > on
> and
> > > the
> > > ActiveMQ Brokers are runninng on Oracle Linux Server release 6.5
> > > OL-6.5.1-64
> > > servers with JDK 6.
> > >
> > >
> > > In activemq.xml we have set the flow control so that anyhting
> > > above 75%
> > of
> > > storage usage stops messages handling on specific queues and the
> storage
> > > usage is set :
> > > <systemUsage>
> > > <systemUsage>
> > > <memoryUsage>
> > > <memoryUsage percentOfJvmHeap="75" />
> > > </memoryUsage>
> > > <storeUsage>
> > > <storeUsage limit="400 gb"/>
> > > </storeUsage>
> > > <tempUsage>
> > > <tempUsage limit="50 gb"/>
> > > </tempUsage>
> > > </systemUsage>
> > > </systemUsage>
> > >
> > > But even so the AMQ Brokers only sees the availble Disk space in
> > > '500G 226G 275G 46%' on the Mount and activates the flow control
> > > since the 275Gb available space is smaller then the 400GB stated
> > > in activemq.xml.
> > >
> > > So the question is now why is the Broker doing this even after it
> > > have
> > read
> > > through the KahaDB data and reindexed it and updated the db.data
> > > file
> and
> > > is
> > > using the 226GB but does not take this into account when reading
> > > the 'storageUsage' limit in activemq.xml
> > >
> > >
> > > I might be missing something here ot is it just as simple that I
> > > need
> to
> > > get
> > > the Mountpoint of 500GB larger. So any help would be very appriciated.
> > >
> > >
> > > Kind regards
> > > Patrik Wettergren
> > >
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-rein
> dex-of-the-KahaDB-on-every-startup-tp4704035.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>
This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies. OppenheimerFunds may, at its sole discretion, monitor, review, retain and/or disclose the content of all email communications.
Re: ActiveMQ Brokers starts the reindex of the KahaDB on every startup
Posted by Tim Bain <tb...@alumni.duke.edu>.
Yep, that's the one. Thanks for finding it; JIRA's not the easiest to use
from a phone.
On Wed, Nov 18, 2015 at 8:18 AM, Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Tim, I think you are referring to this fix here:
> https://issues.apache.org/jira/browse/AMQ-4832
>
> Looks like it was done in 5.9.1 and later.
>
> There's also been some other fixes and improvements done in this area
> recently so I would definitely recommend upgrading to a more recent version
> and giving that a shot.
>
> On Wed, Nov 18, 2015 at 8:55 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > I have a vague memory of seeing a fix for exactly this bug in one of the
> > most recent (5.11.x or later) versions, but I couldn't find it in a quick
> > search of JIRA. Can anyone confirm my memory?
> >
> > If I'm remembering correctly, then an upgrade to 5.12.1 should fix your
> > problem. If your organization will let you, you could upgrade your
> staging
> > environment to see if the observed behavior changes.
> >
> > Tim
> > On Nov 18, 2015 2:53 AM, "patrikwettergren" <pa...@tr.com>
> > wrote:
> >
> > > Hello we are running on a Master/Slave Setup of two ActiveMQ Brokers
> with
> > > version 5.9.
> > >
> > > We have taken a copy of our ActiveMQ Production KahaDB and placed it in
> > the
> > > Stage Environment. This to resolve an issue with DLQ placed Messages.
> > >
> > > The issue we are seeing now is that the Stage ActiveMQ Brokers are
> > running
> > > a
> > > full re-index every time they are restarted and when the re-indexing
> are
> > > done the Broker states that there is not enough space on the disk and
> it
> > > then caps the usage of the disk to what it reads is available.
> > >
> > > The problem is here that the used disk space is from the copied KahaDB
> > that
> > > the AMQ Broker just re-indexed.
> > >
> > > So in other word it reads the KahaDB files fine to re-index the data in
> > > them, but does not take into account of what diskspace it is already
> > using
> > > when reading space available when it is finished the re-indexing.
> > >
> > > The Size of the KahaDB Set is 226GB it is placed on a 500GB Mount on
> and
> > > the
> > > ActiveMQ Brokers are runninng on Oracle Linux Server release 6.5
> > > OL-6.5.1-64
> > > servers with JDK 6.
> > >
> > >
> > > In activemq.xml we have set the flow control so that anyhting above 75%
> > of
> > > storage usage stops messages handling on specific queues and the
> storage
> > > usage is set :
> > > <systemUsage>
> > > <systemUsage>
> > > <memoryUsage>
> > > <memoryUsage percentOfJvmHeap="75" />
> > > </memoryUsage>
> > > <storeUsage>
> > > <storeUsage limit="400 gb"/>
> > > </storeUsage>
> > > <tempUsage>
> > > <tempUsage limit="50 gb"/>
> > > </tempUsage>
> > > </systemUsage>
> > > </systemUsage>
> > >
> > > But even so the AMQ Brokers only sees the availble Disk space in '500G
> > > 226G
> > > 275G 46%' on the Mount and activates the flow control since the 275Gb
> > > available space is smaller then the 400GB stated in activemq.xml.
> > >
> > > So the question is now why is the Broker doing this even after it have
> > read
> > > through the KahaDB data and reindexed it and updated the db.data file
> and
> > > is
> > > using the 226GB but does not take this into account when reading the
> > > 'storageUsage' limit in activemq.xml
> > >
> > >
> > > I might be missing something here ot is it just as simple that I need
> to
> > > get
> > > the Mountpoint of 500GB larger. So any help would be very appriciated.
> > >
> > >
> > > Kind regards
> > > Patrik Wettergren
> > >
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-reindex-of-the-KahaDB-on-every-startup-tp4704035.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>
Re: ActiveMQ Brokers starts the reindex of the KahaDB on every startup
Posted by Christopher Shannon <ch...@gmail.com>.
Tim, I think you are referring to this fix here:
https://issues.apache.org/jira/browse/AMQ-4832
Looks like it was done in 5.9.1 and later.
There's also been some other fixes and improvements done in this area
recently so I would definitely recommend upgrading to a more recent version
and giving that a shot.
On Wed, Nov 18, 2015 at 8:55 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> I have a vague memory of seeing a fix for exactly this bug in one of the
> most recent (5.11.x or later) versions, but I couldn't find it in a quick
> search of JIRA. Can anyone confirm my memory?
>
> If I'm remembering correctly, then an upgrade to 5.12.1 should fix your
> problem. If your organization will let you, you could upgrade your staging
> environment to see if the observed behavior changes.
>
> Tim
> On Nov 18, 2015 2:53 AM, "patrikwettergren" <pa...@tr.com>
> wrote:
>
> > Hello we are running on a Master/Slave Setup of two ActiveMQ Brokers with
> > version 5.9.
> >
> > We have taken a copy of our ActiveMQ Production KahaDB and placed it in
> the
> > Stage Environment. This to resolve an issue with DLQ placed Messages.
> >
> > The issue we are seeing now is that the Stage ActiveMQ Brokers are
> running
> > a
> > full re-index every time they are restarted and when the re-indexing are
> > done the Broker states that there is not enough space on the disk and it
> > then caps the usage of the disk to what it reads is available.
> >
> > The problem is here that the used disk space is from the copied KahaDB
> that
> > the AMQ Broker just re-indexed.
> >
> > So in other word it reads the KahaDB files fine to re-index the data in
> > them, but does not take into account of what diskspace it is already
> using
> > when reading space available when it is finished the re-indexing.
> >
> > The Size of the KahaDB Set is 226GB it is placed on a 500GB Mount on and
> > the
> > ActiveMQ Brokers are runninng on Oracle Linux Server release 6.5
> > OL-6.5.1-64
> > servers with JDK 6.
> >
> >
> > In activemq.xml we have set the flow control so that anyhting above 75%
> of
> > storage usage stops messages handling on specific queues and the storage
> > usage is set :
> > <systemUsage>
> > <systemUsage>
> > <memoryUsage>
> > <memoryUsage percentOfJvmHeap="75" />
> > </memoryUsage>
> > <storeUsage>
> > <storeUsage limit="400 gb"/>
> > </storeUsage>
> > <tempUsage>
> > <tempUsage limit="50 gb"/>
> > </tempUsage>
> > </systemUsage>
> > </systemUsage>
> >
> > But even so the AMQ Brokers only sees the availble Disk space in '500G
> > 226G
> > 275G 46%' on the Mount and activates the flow control since the 275Gb
> > available space is smaller then the 400GB stated in activemq.xml.
> >
> > So the question is now why is the Broker doing this even after it have
> read
> > through the KahaDB data and reindexed it and updated the db.data file and
> > is
> > using the 226GB but does not take this into account when reading the
> > 'storageUsage' limit in activemq.xml
> >
> >
> > I might be missing something here ot is it just as simple that I need to
> > get
> > the Mountpoint of 500GB larger. So any help would be very appriciated.
> >
> >
> > Kind regards
> > Patrik Wettergren
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-reindex-of-the-KahaDB-on-every-startup-tp4704035.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Re: ActiveMQ Brokers starts the reindex of the KahaDB on every startup
Posted by Tim Bain <tb...@alumni.duke.edu>.
I have a vague memory of seeing a fix for exactly this bug in one of the
most recent (5.11.x or later) versions, but I couldn't find it in a quick
search of JIRA. Can anyone confirm my memory?
If I'm remembering correctly, then an upgrade to 5.12.1 should fix your
problem. If your organization will let you, you could upgrade your staging
environment to see if the observed behavior changes.
Tim
On Nov 18, 2015 2:53 AM, "patrikwettergren" <pa...@tr.com>
wrote:
> Hello we are running on a Master/Slave Setup of two ActiveMQ Brokers with
> version 5.9.
>
> We have taken a copy of our ActiveMQ Production KahaDB and placed it in the
> Stage Environment. This to resolve an issue with DLQ placed Messages.
>
> The issue we are seeing now is that the Stage ActiveMQ Brokers are running
> a
> full re-index every time they are restarted and when the re-indexing are
> done the Broker states that there is not enough space on the disk and it
> then caps the usage of the disk to what it reads is available.
>
> The problem is here that the used disk space is from the copied KahaDB that
> the AMQ Broker just re-indexed.
>
> So in other word it reads the KahaDB files fine to re-index the data in
> them, but does not take into account of what diskspace it is already using
> when reading space available when it is finished the re-indexing.
>
> The Size of the KahaDB Set is 226GB it is placed on a 500GB Mount on and
> the
> ActiveMQ Brokers are runninng on Oracle Linux Server release 6.5
> OL-6.5.1-64
> servers with JDK 6.
>
>
> In activemq.xml we have set the flow control so that anyhting above 75% of
> storage usage stops messages handling on specific queues and the storage
> usage is set :
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="75" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="400 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> But even so the AMQ Brokers only sees the availble Disk space in '500G
> 226G
> 275G 46%' on the Mount and activates the flow control since the 275Gb
> available space is smaller then the 400GB stated in activemq.xml.
>
> So the question is now why is the Broker doing this even after it have read
> through the KahaDB data and reindexed it and updated the db.data file and
> is
> using the 226GB but does not take this into account when reading the
> 'storageUsage' limit in activemq.xml
>
>
> I might be missing something here ot is it just as simple that I need to
> get
> the Mountpoint of 500GB larger. So any help would be very appriciated.
>
>
> Kind regards
> Patrik Wettergren
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Brokers-starts-the-reindex-of-the-KahaDB-on-every-startup-tp4704035.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>