You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Charan Reddy G <re...@gmail.com> on 2016/08/04 21:12:33 UTC
Question regarding exit strategy when we reach threshold space usage (0.95)
Hi,
in LedgerDirsListener of InterleavedLedgerStorage, if the configured value
of isForceGCAllowWhenNoSpace is false, then when all disks are full, we are
suspending both Major and Minor GC. I'm wondering what is the exit strategy
from here? how will we come out of Readonly mode? Since we suspended both
Major and Minor GC even deleting Ledgers wouldn't help
@Override
*public* *void* allDisksFull() {
*if* (gcThread.isForceGCAllowWhenNoSpace) {
gcThread.enableForceGC();
} *else* {
gcThread.suspendMajorGC();
gcThread.suspendMinorGC();
}
}
Thanks,
Charan
Re: Question regarding exit strategy when we reach threshold space
usage (0.95)
Posted by Venkateswara Rao Jujjuri <ju...@gmail.com>.
Another thing to add
- Avoid bookie becoming RW and R in quick successions. We have observed
that if we keep 95% threshold,
when it goes to 95% it becomes Ronly, and as soon as it comes to 94% it
becomes RW it kept toggling between
RW and R .. which was not good.
On Tue, Aug 9, 2016 at 12:44 PM, Sijie Guo <si...@apache.org> wrote:
> On Fri, Aug 5, 2016 at 9:05 AM, Venkateswara Rao Jujjuri <
> jujjuri@gmail.com>
> wrote:
>
> > Hi Enrico,
> > All are valid points, and wondering if we need to have
> > more offered from BookKeeper itself. May be we need to open a Jira for
> > this, and also wondering how
> > production deployments are dealing with this real issue.
> >
> >
> > - Right now I believe our GC strategy is very basic: Major compaction,
> > Minor Compaction with interval and %of reclamation.
> > - User defined % of disk space max before going into readOnly; and no
> real
> > exit strategy once we are there.
> > - How do we keep the ZK cookie valid even when we "add" ledger storage,
> > which can be an exit strategy.
> >
> > - Compaction: Add Low Water Mark(LWM) and High Water Mark(HWM) to the
> disk
> > space. Have different compaction frequency and reclaim %s when the disk
> > space is < low water mark , > LWM < HWM, > HWM.
> > - Add more intelligence to the major and minor compaction when it can
> run.
> > Based on system load, and time of the day etc.
> > - Have an option to keep 1 Ledger File size disk 'reserved' for exist
> > strategy.
> > - Make cookie computation so that cookie won't change when storage is
> > added.
> >
> > Anything more? If we agree we can open one or more Jira tickets and work
> on
> > these.
> >
>
> +1 Look great to me.
>
>
> >
> > Thanks,
> > JV
> >
> > On Thu, Aug 4, 2016 at 11:40 PM, Enrico Olivelli - Diennea <
> > enrico.olivelli@diennea.com> wrote:
> >
> > > Hi Charan,
> > > I think that the main problem is that GC need some disk space and
> cannot
> > > be performed if disks are full.
> > >
> > > Usually it is better to keep disk space under monitoring using external
> > > tools (Zabbix...) and prevent the system to be in that situation.
> > >
> > > You can also keep on your disks some "parachute" to have the ability to
> > > free up some space during emergency or at least remove some logs
> > >
> > > Maybe we can think to implement a way to configure a set of
> > > emergency-directories to be used for GC operations as tmp space
> > >
> > >
> > > Il giorno gio, 04/08/2016 alle 14.12 -0700, Charan Reddy G ha scritto:
> > >
> > > Hi,
> > >
> > > in LedgerDirsListener of InterleavedLedgerStorage, if the configured
> > value
> > > of isForceGCAllowWhenNoSpace is false, then when all disks are full, we
> > are
> > > suspending both Major and Minor GC. I'm wondering what is the exit
> > strategy
> > > from here? how will we come out of Readonly mode? Since we suspended
> both
> > > Major and Minor GC even deleting Ledgers wouldn't help
> > >
> > > @Override
> > >
> > > *public* *void* allDisksFull() {
> > >
> > > *if* (gcThread.isForceGCAllowWhenNoSpace) {
> > >
> > > gcThread.enableForceGC();
> > >
> > > } *else* {
> > >
> > > gcThread.suspendMajorGC();
> > >
> > > gcThread.suspendMinorGC();
> > >
> > > }
> > >
> > > }
> > > Thanks,
> > > Charan
> > >
> > >
> > > --
> > > Enrico Olivelli
> > > Software Development Manager @Diennea
> > > Tel.: (+39) 0546 066100 - Int. 925
> > > Viale G.Marconi 30/14 - 48018 Faenza (RA)
> > >
> > > MagNews - E-mail Marketing Solutions
> > > http://www.magnews.it
> > > Diennea - Digital Marketing Solutions
> > > http://www.diennea.com
> > >
> > >
> > > ________________________________
> > >
> > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> > > email marketing! http://www.magnews.it/newsletter/
> > >
> > > The information in this email is confidential and may be legally
> > > privileged. If you are not the intended recipient please notify the
> > sender
> > > immediately and destroy this email. Any unauthorized, direct or
> indirect,
> > > disclosure, copying, storage, distribution or other use is strictly
> > > forbidden.
> > >
> >
> >
> >
> > --
> > Jvrao
> > ---
> > First they ignore you, then they laugh at you, then they fight you, then
> > you win. - Mahatma Gandhi
> >
>
--
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi
Re: Question regarding exit strategy when we reach threshold space
usage (0.95)
Posted by Sijie Guo <si...@apache.org>.
On Fri, Aug 5, 2016 at 9:05 AM, Venkateswara Rao Jujjuri <ju...@gmail.com>
wrote:
> Hi Enrico,
> All are valid points, and wondering if we need to have
> more offered from BookKeeper itself. May be we need to open a Jira for
> this, and also wondering how
> production deployments are dealing with this real issue.
>
>
> - Right now I believe our GC strategy is very basic: Major compaction,
> Minor Compaction with interval and %of reclamation.
> - User defined % of disk space max before going into readOnly; and no real
> exit strategy once we are there.
> - How do we keep the ZK cookie valid even when we "add" ledger storage,
> which can be an exit strategy.
>
> - Compaction: Add Low Water Mark(LWM) and High Water Mark(HWM) to the disk
> space. Have different compaction frequency and reclaim %s when the disk
> space is < low water mark , > LWM < HWM, > HWM.
> - Add more intelligence to the major and minor compaction when it can run.
> Based on system load, and time of the day etc.
> - Have an option to keep 1 Ledger File size disk 'reserved' for exist
> strategy.
> - Make cookie computation so that cookie won't change when storage is
> added.
>
> Anything more? If we agree we can open one or more Jira tickets and work on
> these.
>
+1 Look great to me.
>
> Thanks,
> JV
>
> On Thu, Aug 4, 2016 at 11:40 PM, Enrico Olivelli - Diennea <
> enrico.olivelli@diennea.com> wrote:
>
> > Hi Charan,
> > I think that the main problem is that GC need some disk space and cannot
> > be performed if disks are full.
> >
> > Usually it is better to keep disk space under monitoring using external
> > tools (Zabbix...) and prevent the system to be in that situation.
> >
> > You can also keep on your disks some "parachute" to have the ability to
> > free up some space during emergency or at least remove some logs
> >
> > Maybe we can think to implement a way to configure a set of
> > emergency-directories to be used for GC operations as tmp space
> >
> >
> > Il giorno gio, 04/08/2016 alle 14.12 -0700, Charan Reddy G ha scritto:
> >
> > Hi,
> >
> > in LedgerDirsListener of InterleavedLedgerStorage, if the configured
> value
> > of isForceGCAllowWhenNoSpace is false, then when all disks are full, we
> are
> > suspending both Major and Minor GC. I'm wondering what is the exit
> strategy
> > from here? how will we come out of Readonly mode? Since we suspended both
> > Major and Minor GC even deleting Ledgers wouldn't help
> >
> > @Override
> >
> > *public* *void* allDisksFull() {
> >
> > *if* (gcThread.isForceGCAllowWhenNoSpace) {
> >
> > gcThread.enableForceGC();
> >
> > } *else* {
> >
> > gcThread.suspendMajorGC();
> >
> > gcThread.suspendMinorGC();
> >
> > }
> >
> > }
> > Thanks,
> > Charan
> >
> >
> > --
> > Enrico Olivelli
> > Software Development Manager @Diennea
> > Tel.: (+39) 0546 066100 - Int. 925
> > Viale G.Marconi 30/14 - 48018 Faenza (RA)
> >
> > MagNews - E-mail Marketing Solutions
> > http://www.magnews.it
> > Diennea - Digital Marketing Solutions
> > http://www.diennea.com
> >
> >
> > ________________________________
> >
> > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> > email marketing! http://www.magnews.it/newsletter/
> >
> > The information in this email is confidential and may be legally
> > privileged. If you are not the intended recipient please notify the
> sender
> > immediately and destroy this email. Any unauthorized, direct or indirect,
> > disclosure, copying, storage, distribution or other use is strictly
> > forbidden.
> >
>
>
>
> --
> Jvrao
> ---
> First they ignore you, then they laugh at you, then they fight you, then
> you win. - Mahatma Gandhi
>
Re: Question regarding exit strategy when we reach threshold space
usage (0.95)
Posted by Venkateswara Rao Jujjuri <ju...@gmail.com>.
Hi Enrico,
All are valid points, and wondering if we need to have
more offered from BookKeeper itself. May be we need to open a Jira for
this, and also wondering how
production deployments are dealing with this real issue.
- Right now I believe our GC strategy is very basic: Major compaction,
Minor Compaction with interval and %of reclamation.
- User defined % of disk space max before going into readOnly; and no real
exit strategy once we are there.
- How do we keep the ZK cookie valid even when we "add" ledger storage,
which can be an exit strategy.
- Compaction: Add Low Water Mark(LWM) and High Water Mark(HWM) to the disk
space. Have different compaction frequency and reclaim %s when the disk
space is < low water mark , > LWM < HWM, > HWM.
- Add more intelligence to the major and minor compaction when it can run.
Based on system load, and time of the day etc.
- Have an option to keep 1 Ledger File size disk 'reserved' for exist
strategy.
- Make cookie computation so that cookie won't change when storage is added.
Anything more? If we agree we can open one or more Jira tickets and work on
these.
Thanks,
JV
On Thu, Aug 4, 2016 at 11:40 PM, Enrico Olivelli - Diennea <
enrico.olivelli@diennea.com> wrote:
> Hi Charan,
> I think that the main problem is that GC need some disk space and cannot
> be performed if disks are full.
>
> Usually it is better to keep disk space under monitoring using external
> tools (Zabbix...) and prevent the system to be in that situation.
>
> You can also keep on your disks some "parachute" to have the ability to
> free up some space during emergency or at least remove some logs
>
> Maybe we can think to implement a way to configure a set of
> emergency-directories to be used for GC operations as tmp space
>
>
> Il giorno gio, 04/08/2016 alle 14.12 -0700, Charan Reddy G ha scritto:
>
> Hi,
>
> in LedgerDirsListener of InterleavedLedgerStorage, if the configured value
> of isForceGCAllowWhenNoSpace is false, then when all disks are full, we are
> suspending both Major and Minor GC. I'm wondering what is the exit strategy
> from here? how will we come out of Readonly mode? Since we suspended both
> Major and Minor GC even deleting Ledgers wouldn't help
>
> @Override
>
> *public* *void* allDisksFull() {
>
> *if* (gcThread.isForceGCAllowWhenNoSpace) {
>
> gcThread.enableForceGC();
>
> } *else* {
>
> gcThread.suspendMajorGC();
>
> gcThread.suspendMinorGC();
>
> }
>
> }
> Thanks,
> Charan
>
>
> --
> Enrico Olivelli
> Software Development Manager @Diennea
> Tel.: (+39) 0546 066100 - Int. 925
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it
> Diennea - Digital Marketing Solutions
> http://www.diennea.com
>
>
> ________________________________
>
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> email marketing! http://www.magnews.it/newsletter/
>
> The information in this email is confidential and may be legally
> privileged. If you are not the intended recipient please notify the sender
> immediately and destroy this email. Any unauthorized, direct or indirect,
> disclosure, copying, storage, distribution or other use is strictly
> forbidden.
>
--
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi
Re: Question regarding exit strategy when we reach threshold space
usage (0.95)
Posted by Enrico Olivelli - Diennea <en...@diennea.com>.
Hi Charan,
I think that the main problem is that GC need some disk space and cannot be performed if disks are full.
Usually it is better to keep disk space under monitoring using external tools (Zabbix...) and prevent the system to be in that situation.
You can also keep on your disks some "parachute" to have the ability to free up some space during emergency or at least remove some logs
Maybe we can think to implement a way to configure a set of emergency-directories to be used for GC operations as tmp space
Il giorno gio, 04/08/2016 alle 14.12 -0700, Charan Reddy G ha scritto:
Hi,
in LedgerDirsListener of InterleavedLedgerStorage, if the configured value
of isForceGCAllowWhenNoSpace is false, then when all disks are full, we are
suspending both Major and Minor GC. I'm wondering what is the exit strategy
from here? how will we come out of Readonly mode? Since we suspended both
Major and Minor GC even deleting Ledgers wouldn't help
@Override
*public* *void* allDisksFull() {
*if* (gcThread.isForceGCAllowWhenNoSpace) {
gcThread.enableForceGC();
} *else* {
gcThread.suspendMajorGC();
gcThread.suspendMinorGC();
}
}
Thanks,
Charan
--
Enrico Olivelli
Software Development Manager @Diennea
Tel.: (+39) 0546 066100 - Int. 925
Viale G.Marconi 30/14 - 48018 Faenza (RA)
MagNews - E-mail Marketing Solutions
http://www.magnews.it
Diennea - Digital Marketing Solutions
http://www.diennea.com
________________________________
Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing! http://www.magnews.it/newsletter/
The information in this email is confidential and may be legally privileged. If you are not the intended recipient please notify the sender immediately and destroy this email. Any unauthorized, direct or indirect, disclosure, copying, storage, distribution or other use is strictly forbidden.