[jira] [Updated] (FLINK-27557) Submit compaction when prepareCommit is triggered

["Jane Chan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jane Chan updated FLINK-27557:
------------------------------
        Parent:     (was: FLINK-27515)
    Issue Type: New Feature  (was: Sub-task)

> Submit compaction when prepareCommit is triggered
> -------------------------------------------------
>
>                 Key: FLINK-27557
>                 URL: https://issues.apache.org/jira/browse/FLINK-27557
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table Store
>    Affects Versions: table-store-0.2.0
>            Reporter: Jane Chan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: table-store-0.2.0
>
>         Attachments: normal-compaction.png, optimized-normal-compaction.png, rescale-bucket-compaction.png
>
>
> Currently, FileStoreWrite will scan and plan files when creating a non-empty writer. We should also create a non-empty writer for non-rescale compaction cases but instead use the pre-planned manifest entries.
> The sink records should be dumped, and when prepareCommit is invoked, it should trigger submitCompaction. This will generate two snapshots, the first commit kind is "ADD" with empty new files, and the second commit kind is "COMPACT" with specified manifest entries marked as deleted and compacted files marked as added.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)