You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by jeff saremi <je...@hotmail.com> on 2017/03/22 19:29:02 UTC
Need guidance on Custom Compaction Policy
I mentioned some of this in another thread. We have a readonly database which get bulk loaded using HFiles.
We want to keep only two versions/generations of data. Since the size of data is massive we need to delete the older generation.
Since we write one single HBase for each region for each CF for each generation, could we just yank the older files using a separate standalone process? (sounds a little scary)
If not, could we write a custom compactor? what's involved (some pointers please)? thanks
Jeff
Re: Need guidance on Custom Compaction Policy
Posted by jeff saremi <je...@hotmail.com>.
@Ted
No i had not looked at DateTiered. I will. Thanks
@Vladimir
I will look at the example code you gave me. thanks
________________________________
From: Ted Yu <yu...@gmail.com>
Sent: Wednesday, March 22, 2017 2:12:00 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on Custom Compaction Policy
Have you taken look at http://hbase.apache.org/book.html#ops.date.tiered ?
Cheers
On Wed, Mar 22, 2017 at 12:29 PM, jeff saremi <je...@hotmail.com>
wrote:
> I mentioned some of this in another thread. We have a readonly database
> which get bulk loaded using HFiles.
> We want to keep only two versions/generations of data. Since the size of
> data is massive we need to delete the older generation.
>
> Since we write one single HBase for each region for each CF for each
> generation, could we just yank the older files using a separate standalone
> process? (sounds a little scary)
>
> If not, could we write a custom compactor? what's involved (some pointers
> please)? thanks
>
> Jeff
>
Re: Need guidance on Custom Compaction Policy
Posted by Ted Yu <yu...@gmail.com>.
Have you taken look at http://hbase.apache.org/book.html#ops.date.tiered ?
Cheers
On Wed, Mar 22, 2017 at 12:29 PM, jeff saremi <je...@hotmail.com>
wrote:
> I mentioned some of this in another thread. We have a readonly database
> which get bulk loaded using HFiles.
> We want to keep only two versions/generations of data. Since the size of
> data is massive we need to delete the older generation.
>
> Since we write one single HBase for each region for each CF for each
> generation, could we just yank the older files using a separate standalone
> process? (sounds a little scary)
>
> If not, could we write a custom compactor? what's involved (some pointers
> please)? thanks
>
> Jeff
>
Re: Need guidance on Custom Compaction Policy
Posted by Vladimir Rodionov <vl...@gmail.com>.
Older files will be purged by default HBase compactor if all data inside
expired (you have TTL for data?)
As for custom compaction policy you can refer to FIFOCompactionPolicy class
to get the idea how custom compaction works.
-Vlad
On Wed, Mar 22, 2017 at 12:29 PM, jeff saremi <je...@hotmail.com>
wrote:
> I mentioned some of this in another thread. We have a readonly database
> which get bulk loaded using HFiles.
> We want to keep only two versions/generations of data. Since the size of
> data is massive we need to delete the older generation.
>
> Since we write one single HBase for each region for each CF for each
> generation, could we just yank the older files using a separate standalone
> process? (sounds a little scary)
>
> If not, could we write a custom compactor? what's involved (some pointers
> please)? thanks
>
> Jeff
>