[jira] [Updated] (HBASE-7885) bloom filter compaction is too aggressive for Hfile which only contains small count of records

["Lars Hofhansl (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Hofhansl updated HBASE-7885:
---------------------------------

    Fix Version/s:     (was: 0.94.6)
                   0.94.7

Moving into 0.94.7 (just a few weeks out, really)
Personally I do not know much about bloom filters, would like to know why the estimated error rate is incorrect.

                
> bloom filter compaction is too aggressive for Hfile which only contains small count of records
> ----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-7885
>                 URL: https://issues.apache.org/jira/browse/HBASE-7885
>             Project: HBase
>          Issue Type: Bug
>          Components: Performance, Scanners
>    Affects Versions: 0.94.5
>            Reporter: clockfly
>            Assignee: clockfly
>            Priority: Minor
>             Fix For: 0.94.7
>
>         Attachments: hbase-7885.patch, hbase_bloom_shrink_fix.patch
>
>
> For HFile V2, the bloom filter will take a initial size, 128KB. 
> When there are not that much records inserted into the bloom filter, the bloom fitler will start to shrink itself to do compaction. 
> For example, for 128K, it will compact to 64K ->32K->16K->8K->4K->2K->1K->512->256->128->64->32, as long as it think that it can be bounded by the estimate error rate. 
> If we puts only a few records in the HFile, the bloom filter will be compacted to too small, then it will break the assumption that shrinking will still be bounded by the estimated error rate. The False positive rate will becomes un-acceptable high. 
> For example, if we set the expected error rate is 0.00001, for 10 records, after compaction, The size of the bloom filter will be 64 bytes. The real effective false positive rate will be 50%.
> The use case is like this, if we are using HBase to store big record like images, and binaries, each record will take megabytes. Then for a 128M file, it will only contains dozens of records.
> The suggested fix is to set a lower limit for the bloom filter compaction process. I suggest to use 1000 bytes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira