You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Schubert Zhang <zs...@gmail.com> on 2011/04/27 03:32:54 UTC
Compaction problems
I have a busy region, and there are 43 StoreFiles (>
compactionThreshold=8) in this region.
Now, I stopped the client and stopped put new data into it. I expect
these StoreFiles to be compacted later.
But, one day after, these 43 StoreFiles are still there.
It seems the compaction does not work automatically.
And I checked the code, it seems the compaction would only start after
split....
Re: Compaction problems
Posted by Schubert Zhang <zs...@gmail.com>.
Thanks J-D, I have opened a jira here:
https://issues.apache.org/jira/browse/HBASE-3826
On Thu, Apr 28, 2011 at 12:55 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> That make sense, would you mind opening a jira?
>
> Thx,
>
> J-D
>
> On Tue, Apr 26, 2011 at 8:52 PM, Schubert Zhang <zs...@gmail.com> wrote:
> > After more test, a obvious issue/problem is, the complete of a minor
> > compaction does not check if current storefiles need more minor
> compaction.
> >
> > I think this may be a bug or leak.
> >
> > Try this test:
> >
> > 1. Put many data to a region, then there is 30 storefiles accumulated,
> > because the backend compaction cannot catch up with the fast put.
> > (hbase.hstore.compactionThreshold=8, hbase.hstore.compaction.max=12)
> >
> > 2. Then stop put.
> >
> > 3. Then, these many 30 storefiles are still there for a long time, (no
> > automatic minor compaction)
> >
> > 4. Submit a compaction on this region, then, only 12 files are
> compaction,
> > now, we have 19 storefiles. The minor compaction stopped.
> >
> > I think, when a minor compaction complete, it should check if the number
> of
> > storefiles still many, if so, another minor compaction should start
> > continuiously.
> > Schubert Zhang
> > On Wed, Apr 27, 2011 at 9:32 AM, Schubert Zhang <zs...@gmail.com>
> wrote:
> >
> >> I have a busy region, and there are 43 StoreFiles (>
> >> compactionThreshold=8) in this region.
> >> Now, I stopped the client and stopped put new data into it. I expect
> >> these StoreFiles to be compacted later.
> >>
> >> But, one day after, these 43 StoreFiles are still there.
> >>
> >> It seems the compaction does not work automatically.
> >> And I checked the code, it seems the compaction would only start after
> >> split....
> >>
> >>
> >>
> >
>
Re: Compaction problems
Posted by Nicolas Spiegelberg <ns...@fb.com>.
Note that we have a compaction recursive enqueue patch on our internal
branches, but we wanted to give it some run time before contributing back
to make sure it was safe. I'll port that to trunk.
On 4/27/11 9:55 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>That make sense, would you mind opening a jira?
>
>Thx,
>
>J-D
>
>On Tue, Apr 26, 2011 at 8:52 PM, Schubert Zhang <zs...@gmail.com> wrote:
>> After more test, a obvious issue/problem is, the complete of a minor
>> compaction does not check if current storefiles need more minor
>>compaction.
>>
>> I think this may be a bug or leak.
>>
>> Try this test:
>>
>> 1. Put many data to a region, then there is 30 storefiles accumulated,
>> because the backend compaction cannot catch up with the fast put.
>> (hbase.hstore.compactionThreshold=8, hbase.hstore.compaction.max=12)
>>
>> 2. Then stop put.
>>
>> 3. Then, these many 30 storefiles are still there for a long time, (no
>> automatic minor compaction)
>>
>> 4. Submit a compaction on this region, then, only 12 files are
>>compaction,
>> now, we have 19 storefiles. The minor compaction stopped.
>>
>> I think, when a minor compaction complete, it should check if the
>>number of
>> storefiles still many, if so, another minor compaction should start
>> continuiously.
>> Schubert Zhang
>> On Wed, Apr 27, 2011 at 9:32 AM, Schubert Zhang <zs...@gmail.com>
>>wrote:
>>
>>> I have a busy region, and there are 43 StoreFiles (>
>>> compactionThreshold=8) in this region.
>>> Now, I stopped the client and stopped put new data into it. I expect
>>> these StoreFiles to be compacted later.
>>>
>>> But, one day after, these 43 StoreFiles are still there.
>>>
>>> It seems the compaction does not work automatically.
>>> And I checked the code, it seems the compaction would only start after
>>> split....
>>>
>>>
>>>
>>
Re: Compaction problems
Posted by Jean-Daniel Cryans <jd...@apache.org>.
That make sense, would you mind opening a jira?
Thx,
J-D
On Tue, Apr 26, 2011 at 8:52 PM, Schubert Zhang <zs...@gmail.com> wrote:
> After more test, a obvious issue/problem is, the complete of a minor
> compaction does not check if current storefiles need more minor compaction.
>
> I think this may be a bug or leak.
>
> Try this test:
>
> 1. Put many data to a region, then there is 30 storefiles accumulated,
> because the backend compaction cannot catch up with the fast put.
> (hbase.hstore.compactionThreshold=8, hbase.hstore.compaction.max=12)
>
> 2. Then stop put.
>
> 3. Then, these many 30 storefiles are still there for a long time, (no
> automatic minor compaction)
>
> 4. Submit a compaction on this region, then, only 12 files are compaction,
> now, we have 19 storefiles. The minor compaction stopped.
>
> I think, when a minor compaction complete, it should check if the number of
> storefiles still many, if so, another minor compaction should start
> continuiously.
> Schubert Zhang
> On Wed, Apr 27, 2011 at 9:32 AM, Schubert Zhang <zs...@gmail.com> wrote:
>
>> I have a busy region, and there are 43 StoreFiles (>
>> compactionThreshold=8) in this region.
>> Now, I stopped the client and stopped put new data into it. I expect
>> these StoreFiles to be compacted later.
>>
>> But, one day after, these 43 StoreFiles are still there.
>>
>> It seems the compaction does not work automatically.
>> And I checked the code, it seems the compaction would only start after
>> split....
>>
>>
>>
>
Re: Compaction problems
Posted by Schubert Zhang <zs...@gmail.com>.
After more test, a obvious issue/problem is, the complete of a minor
compaction does not check if current storefiles need more minor compaction.
I think this may be a bug or leak.
Try this test:
1. Put many data to a region, then there is 30 storefiles accumulated,
because the backend compaction cannot catch up with the fast put.
(hbase.hstore.compactionThreshold=8, hbase.hstore.compaction.max=12)
2. Then stop put.
3. Then, these many 30 storefiles are still there for a long time, (no
automatic minor compaction)
4. Submit a compaction on this region, then, only 12 files are compaction,
now, we have 19 storefiles. The minor compaction stopped.
I think, when a minor compaction complete, it should check if the number of
storefiles still many, if so, another minor compaction should start
continuiously.
Schubert Zhang
On Wed, Apr 27, 2011 at 9:32 AM, Schubert Zhang <zs...@gmail.com> wrote:
> I have a busy region, and there are 43 StoreFiles (>
> compactionThreshold=8) in this region.
> Now, I stopped the client and stopped put new data into it. I expect
> these StoreFiles to be compacted later.
>
> But, one day after, these 43 StoreFiles are still there.
>
> It seems the compaction does not work automatically.
> And I checked the code, it seems the compaction would only start after
> split....
>
>
>