You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sheng Chen <ch...@gmail.com> on 2011/07/07 07:55:29 UTC
Is there a way to increase the compaction queue size?
Hi, all,
I have a hbase region server of 10 regions and find the compaction queue
size is always 10 at maximum.
After I finished inserting 1 billion records, there are still hundreds of
HFiles in each region and no further minor compaction is queuing.
My settings,
hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
hbase.hstore.compactionThreshold=3
hbase.hstore.max.filesize=100G
Is it the expected behavior?
Can I make the compaction queue larger in order to keep doing minor
compaction in the background?
Thank you.
Sean
RE: Is there a way to increase the compaction queue size?
Posted by Zhoushuaifeng <zh...@huawei.com>.
O, I understand your requirement.
Because hbase put op is write into memory, insert speed will be very high. Files in the store will aways more than 3, then compact is aways needed, so your compaction queen size is aways 10. This means the minor compaction is already aways working in back ground. But insert is to quick, compact cann't follow it.(There is only 1 compact thread in 1 regionserver, 0.90.x).
If the region doesn't split in time, the compact file size may be larger and larger, and may take too long to finish a compact.
Change this two settings may help:
hbase.hstore.compaction.max: this makes one compact job can compact more files.
hbase.hregion.memstore.flush.size: this will make your files to compact bigger, and decrease total file number in the store.
Zhou Shuaifeng(Frank)
-----Original Message-----
From: Sheng Chen [mailto:chensheng2010@gmail.com]
Sent: Thursday, July 07, 2011 6:49 PM
To: user@hbase.apache.org
Subject: Re: Is there a way to increase the compaction queue size?
Thank you very much.
Actually I was trying to manage splitting manually, and to leave the
compaction automatical.
I really do not want to block my inserts by waiting for compactions. Is it
possible to keep minor compaction working in the background, like starting
a periodic job to do 'compact mytable' in the shell?
Regards,
Sean
2011/7/7 Zhoushuaifeng <zh...@huawei.com>
> hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> It's better not setting this to max_value. Setting this to a limited value
> will help hbase to control the insert speed, and have enough time to finish
> compact. Suggest to setting this to less than 20.
>
> hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ?
> This may be too large, if a store file too large, it will take very long
> time to compact. I suggest to set this less than 8G.
>
> And the compaction queue size means how many regions need compaction, not
> the number of files to compact. So, you have 10 regions, all need
> compaction, and the compaction queue is 10.
>
>
> Zhou Shuaifeng(Frank)
>
> -----Original Message-----
> From: Sheng Chen [mailto:chensheng2010@gmail.com]
> Sent: Thursday, July 07, 2011 3:36 PM
> To: user@hbase.apache.org
> Subject: Re: Is there a way to increase the compaction queue size?
>
> Thank you for your reply.
>
> No more minor compaction is going on, while there are still about 100
> HStoreFiles in each region.
> I think it might because the compactionQueueSize has a maximum of 10 for a
> region server (only one per region?), so after I stopped inserting and the
> 10 compactions in the queue were finished, no more compaction is triggered.
>
> I think that "hbase.hstore.compaction.max" sets the max number of files for
> one minor compaction, does it has anything to do with my problem?
>
> Thanks,
> Sean
>
> 2011/7/7 Stack <st...@duboce.net>
>
> > Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> > default.
> >
> > What do you mean by 'no further minor compaction is queuing'? There
> > is no more compaction going on? Are there exceptions in the log?
> >
> > St.Ack
> >
> > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <ch...@gmail.com>
> > wrote:
> > > Hi, all,
> > > I have a hbase region server of 10 regions and find the compaction
> queue
> > > size is always 10 at maximum.
> > > After I finished inserting 1 billion records, there are still hundreds
> of
> > > HFiles in each region and no further minor compaction is queuing.
> > >
> > > My settings,
> > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > > hbase.hstore.compactionThreshold=3
> > > hbase.hstore.max.filesize=100G
> > >
> > > Is it the expected behavior?
> > > Can I make the compaction queue larger in order to keep doing minor
> > > compaction in the background?
> > >
> > > Thank you.
> > >
> > > Sean
> > >
> >
>
Re: Is there a way to increase the compaction queue size?
Posted by Sheng Chen <ch...@gmail.com>.
Thank you very much.
Actually I was trying to manage splitting manually, and to leave the
compaction automatical.
I really do not want to block my inserts by waiting for compactions. Is it
possible to keep minor compaction working in the background, like starting
a periodic job to do 'compact mytable' in the shell?
Regards,
Sean
2011/7/7 Zhoushuaifeng <zh...@huawei.com>
> hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> It's better not setting this to max_value. Setting this to a limited value
> will help hbase to control the insert speed, and have enough time to finish
> compact. Suggest to setting this to less than 20.
>
> hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ?
> This may be too large, if a store file too large, it will take very long
> time to compact. I suggest to set this less than 8G.
>
> And the compaction queue size means how many regions need compaction, not
> the number of files to compact. So, you have 10 regions, all need
> compaction, and the compaction queue is 10.
>
>
> Zhou Shuaifeng(Frank)
>
> -----Original Message-----
> From: Sheng Chen [mailto:chensheng2010@gmail.com]
> Sent: Thursday, July 07, 2011 3:36 PM
> To: user@hbase.apache.org
> Subject: Re: Is there a way to increase the compaction queue size?
>
> Thank you for your reply.
>
> No more minor compaction is going on, while there are still about 100
> HStoreFiles in each region.
> I think it might because the compactionQueueSize has a maximum of 10 for a
> region server (only one per region?), so after I stopped inserting and the
> 10 compactions in the queue were finished, no more compaction is triggered.
>
> I think that "hbase.hstore.compaction.max" sets the max number of files for
> one minor compaction, does it has anything to do with my problem?
>
> Thanks,
> Sean
>
> 2011/7/7 Stack <st...@duboce.net>
>
> > Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> > default.
> >
> > What do you mean by 'no further minor compaction is queuing'? There
> > is no more compaction going on? Are there exceptions in the log?
> >
> > St.Ack
> >
> > On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <ch...@gmail.com>
> > wrote:
> > > Hi, all,
> > > I have a hbase region server of 10 regions and find the compaction
> queue
> > > size is always 10 at maximum.
> > > After I finished inserting 1 billion records, there are still hundreds
> of
> > > HFiles in each region and no further minor compaction is queuing.
> > >
> > > My settings,
> > > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > > hbase.hstore.compactionThreshold=3
> > > hbase.hstore.max.filesize=100G
> > >
> > > Is it the expected behavior?
> > > Can I make the compaction queue larger in order to keep doing minor
> > > compaction in the background?
> > >
> > > Thank you.
> > >
> > > Sean
> > >
> >
>
RE: Is there a way to increase the compaction queue size?
Posted by Zhoushuaifeng <zh...@huawei.com>.
hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
It's better not setting this to max_value. Setting this to a limited value will help hbase to control the insert speed, and have enough time to finish compact. Suggest to setting this to less than 20.
hbase.hstore.max.filesize = 100G, do you mean hbase.hregion.max.filesize ?
This may be too large, if a store file too large, it will take very long time to compact. I suggest to set this less than 8G.
And the compaction queue size means how many regions need compaction, not the number of files to compact. So, you have 10 regions, all need compaction, and the compaction queue is 10.
Zhou Shuaifeng(Frank)
-----Original Message-----
From: Sheng Chen [mailto:chensheng2010@gmail.com]
Sent: Thursday, July 07, 2011 3:36 PM
To: user@hbase.apache.org
Subject: Re: Is there a way to increase the compaction queue size?
Thank you for your reply.
No more minor compaction is going on, while there are still about 100
HStoreFiles in each region.
I think it might because the compactionQueueSize has a maximum of 10 for a
region server (only one per region?), so after I stopped inserting and the
10 compactions in the queue were finished, no more compaction is triggered.
I think that "hbase.hstore.compaction.max" sets the max number of files for
one minor compaction, does it has anything to do with my problem?
Thanks,
Sean
2011/7/7 Stack <st...@duboce.net>
> Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> default.
>
> What do you mean by 'no further minor compaction is queuing'? There
> is no more compaction going on? Are there exceptions in the log?
>
> St.Ack
>
> On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <ch...@gmail.com>
> wrote:
> > Hi, all,
> > I have a hbase region server of 10 regions and find the compaction queue
> > size is always 10 at maximum.
> > After I finished inserting 1 billion records, there are still hundreds of
> > HFiles in each region and no further minor compaction is queuing.
> >
> > My settings,
> > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > hbase.hstore.compactionThreshold=3
> > hbase.hstore.max.filesize=100G
> >
> > Is it the expected behavior?
> > Can I make the compaction queue larger in order to keep doing minor
> > compaction in the background?
> >
> > Thank you.
> >
> > Sean
> >
>
Re: Is there a way to increase the compaction queue size?
Posted by Sheng Chen <ch...@gmail.com>.
Thank you for your reply.
No more minor compaction is going on, while there are still about 100
HStoreFiles in each region.
I think it might because the compactionQueueSize has a maximum of 10 for a
region server (only one per region?), so after I stopped inserting and the
10 compactions in the queue were finished, no more compaction is triggered.
I think that "hbase.hstore.compaction.max" sets the max number of files for
one minor compaction, does it has anything to do with my problem?
Thanks,
Sean
2011/7/7 Stack <st...@duboce.net>
> Try setting "hbase.hstore.compaction.max" to a value other than 10, its
> default.
>
> What do you mean by 'no further minor compaction is queuing'? There
> is no more compaction going on? Are there exceptions in the log?
>
> St.Ack
>
> On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <ch...@gmail.com>
> wrote:
> > Hi, all,
> > I have a hbase region server of 10 regions and find the compaction queue
> > size is always 10 at maximum.
> > After I finished inserting 1 billion records, there are still hundreds of
> > HFiles in each region and no further minor compaction is queuing.
> >
> > My settings,
> > hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> > hbase.hstore.compactionThreshold=3
> > hbase.hstore.max.filesize=100G
> >
> > Is it the expected behavior?
> > Can I make the compaction queue larger in order to keep doing minor
> > compaction in the background?
> >
> > Thank you.
> >
> > Sean
> >
>
Re: Is there a way to increase the compaction queue size?
Posted by Stack <st...@duboce.net>.
Try setting "hbase.hstore.compaction.max" to a value other than 10, its default.
What do you mean by 'no further minor compaction is queuing'? There
is no more compaction going on? Are there exceptions in the log?
St.Ack
On Wed, Jul 6, 2011 at 10:55 PM, Sheng Chen <ch...@gmail.com> wrote:
> Hi, all,
> I have a hbase region server of 10 regions and find the compaction queue
> size is always 10 at maximum.
> After I finished inserting 1 billion records, there are still hundreds of
> HFiles in each region and no further minor compaction is queuing.
>
> My settings,
> hbase.hstore.blockingStoreFiles=Integer.MAX_VALUE
> hbase.hstore.compactionThreshold=3
> hbase.hstore.max.filesize=100G
>
> Is it the expected behavior?
> Can I make the compaction queue larger in order to keep doing minor
> compaction in the background?
>
> Thank you.
>
> Sean
>