[jira] [Created] (HBASE-27786) CompoundBloomFilters break with an error rate that is too high

["Ray Mattingly (Jira)" <ji...@apache.org>]

Ray Mattingly created HBASE-27786:
-------------------------------------

             Summary: CompoundBloomFilters break with an error rate that is too high
                 Key: HBASE-27786
                 URL: https://issues.apache.org/jira/browse/HBASE-27786
             Project: HBase
          Issue Type: Bug
    Affects Versions: 2.5.2
            Reporter: Ray Mattingly


At my company we're beginning to more heavily utilize the bloom error rate configuration. This is because bloom filters are a nice optimization, but for well distributed workloads with relatively dense data (many rows:host), we've found that they can cause lots of memory/GC pressure unless they can entirely fit in the block cache (and consequently not churn memory that's subject to GC).

Because it's easier to estimate the memory requirements of changes in existing bloom filters, rather than net new bloom filters, we wanted to begin with very high bloom error rates (and consequently small bloom filters), and then ratchet down as memory availability allowed.

This led to us discovering that bloom filters appear to become corrupt at a relatively arbitrary error rate threshold. Blooms with an error rate of 0.61 work as expected, but produce nonsensical results with an error rate of 0.62. I've pushed this branch with test updates to demonstrate the deficit: [https://github.com/apache/hbase/compare/master...HubSpot:hbase:rmattingly/bloom-error-rate-bug]

The test changes confirm that the BloomFilterUtil works as expected, at least with respect to its error rate : size relationship. You can see this in the output of {{{}TestBloomFilterChunk#testBloomErrorRateSizeRelationship{}}}:

 
{noformat}
previousErrorRate=0.01, previousSize=1048568
currentErrorRate=0.05, currentSize=682109
previousErrorRate=0.05, previousSize=682109
currentErrorRate=0.1, currentSize=524284
previousErrorRate=0.1, previousSize=524284
currentErrorRate=0.2, currentSize=366459
previousErrorRate=0.2, previousSize=366459
currentErrorRate=0.4, currentSize=208634
previousErrorRate=0.4, previousSize=208634
currentErrorRate=0.5, currentSize=157826
previousErrorRate=0.5, previousSize=157826
currentErrorRate=0.75, currentSize=65504
previousErrorRate=0.75, previousSize=65504
currentErrorRate=0.99, currentSize=2289
{noformat}
 

With this in mind, the updates to {{TestCompoundBloomFilter}} tell us that the bug must live somewhere in the {{CompoundBloomFilter}} logic. The output indicates this:

 
{noformat}
2023-04-10T15:07:50,925 INFO  [Time-limited test] regionserver.TestCompoundBloomFilter(245): Functional bloom has error rate 0.01 and size 1kb
...
2023-04-10T15:07:56,657 INFO  [Time-limited test] regionserver.TestCompoundBloomFilter(245): Functional bloom has error rate 0.61 and size 1kb
...
java.lang.AssertionError: False positive is too high: 0.9998533333333334 (greater than 0.65), fake lookup is enabled. Bloom size is 4687kb
    at org.junit.Assert.fail(Assert.java:89)
    at org.junit.Assert.assertTrue(Assert.java:42)
    at org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter.readStoreFile(TestCompoundBloomFilter.java:243)
{noformat}
 

The bloom size change from ~1kb -> 4687kb and total lack of precision is clearly not as intended, and totally inline with what we saw in our HBase clusters that attempted to use high bloom error rates.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)