[jira] [Updated] (HUDI-4738) [MOR] Bloom Index missing new records inserted into Log files

["Prashant Wason (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prashant Wason updated HUDI-4738:
---------------------------------
    Fix Version/s: 0.14.1
                       (was: 0.14.0)

> [MOR] Bloom Index missing new records inserted into Log files
> -------------------------------------------------------------
>
>                 Key: HUDI-4738
>                 URL: https://issues.apache.org/jira/browse/HUDI-4738
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: writer-core
>            Reporter: Alexey Kudinkin
>            Priority: Blocker
>             Fix For: 0.14.1
>
>
> Currently, Bloom Index is implemented under following assumption that _file-group (once written), has fixed set of records that could not be expanded_ (this is encoded t/h assumption that at least one version of every record w/in the file group is stored w/in its base file).
> This is relied upon when we tag incoming records w/ the locations of the file-groups they could potentially belong to (in case, when such records are updates), by fetching the Bloom Index info from either a) base-file or b) record in MT Bloom Index associated w/ particular file-group id.
>  
> However this assumption is not always true, since it's possible for _new_ records to be inserted into the log-files, which would mean that the records key-set of a single file-group could expand. This could lead to potentially some records that were previously written to log-files to be duplicated.
>  
> We need to reconcile these 2 aspects and do either of:
>  # Disallow expansion of the file-group records' set (by not allowing inserts into log-files)
>  # Fix Bloom Index implementation to also check log-files during tagging.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)