[jira] [Commented] (KYLIN-5640) Support to automatically adjust the Bloom Filter based on data distribution

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744951#comment-17744951 ] 

ASF subversion and git services commented on KYLIN-5640:
--------------------------------------------------------

Commit a48b85314bc545b95ab310d1494784a0f8d5d42a in kylin's branch refs/heads/kylin5_beta from ChenliangLu
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a48b85314b ]

KYLIN-5640 Support building dynamic bloom filter that adapts to data


> Support to automatically adjust the Bloom Filter based on data distribution
> ---------------------------------------------------------------------------
>
>                 Key: KYLIN-5640
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5640
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Query Engine
>    Affects Versions: 5.0-alpha
>            Reporter: Zhiting Guo
>            Assignee: Zhiting Guo
>            Priority: Major
>             Fix For: 5.0-beta
>
>
> h3. Why are the changes needed?
> Now the usage of bloom filter is to specify the NDV(number of distinct values), and then build BloomFilter. In general scenarios, it is actually not sure how much the distinct value is.
> If BloomFilter can be automatically generated according to the data, the file size can be reduced and the reading efficiency can also be improved.
> h3. What changes were proposed in this pull request?
> {{DynamicBlockBloomFilter}} contains multiple {{BlockSplitBloomFilter}} as candidates and inserts values in the candidates at the same time. Use the largest bloom filter as an approximate deduplication counter, and then remove incapable bloom filter candidates during data insertion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)