[jira] [Comment Edited] (FLINK-27557) Let FileStoreWrite accept pre-planned manifest entries

["Jane Chan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539352#comment-17539352 ] 

Jane Chan edited comment on FLINK-27557 at 5/20/22 3:18 AM:
------------------------------------------------------------

{quote}Can we just use overwrite?
 * overwrite specific manifest entries.
 * overwrite whole table or some whole partition. (eg: when rescale in compaction){quote}
Overwrite means the writer cannot accept the specified manifest entries as restored files, so how to pass them to FileStoreCommit as compact before(mark as delete) is a new question. Beyond that, overwrite corresponds to the commit kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this situation.

I rethink it, and maybe we don't need to generate the new files for the non-rescale compaction.
 * The simplest way is to build the levels from the restored files, don't sink records to MemTable, and submit compaction when precommit is invoked.
 * In the future, maybe we can introduce a new dedicated compaction source to restore LSM, perform compaction and commit, which avoids the shuffle cost.

 

 

 


was (Author: qingyue):
{quote}Can we just use overwrite?
 * overwrite specific manifest entries.
 * overwrite whole table or some whole partition. (eg: when rescale in compaction){quote}
Overwrite means the writer cannot accept the specified manifest entries as restored files, so how to pass them to FileStoreCommit as compact before(mark as delete) is a new question. Beyond that, overwrite corresponds to the commit kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this situation.

I rethink it and maybe we don't need to generate the new files for the normal compaction.
 * The simplest way is to build the levels from the restored files, and don't sink record to MemTable, and submit compaction when precommit is invoked.
 * In the future, maybe we can introduce a new dedicated compaction source to restore LSM, perform compaction and commit, which avoids the shuffle cost.

 

 

 

> Let FileStoreWrite accept pre-planned manifest entries
> ------------------------------------------------------
>
>                 Key: FLINK-27557
>                 URL: https://issues.apache.org/jira/browse/FLINK-27557
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table Store
>    Affects Versions: table-store-0.2.0
>            Reporter: Jane Chan
>            Priority: Major
>             Fix For: table-store-0.2.0
>
>         Attachments: normal-compaction.png, optimized-normal-compaction.png, rescale-bucket-compaction.png
>
>
> Currently, FileStoreWrite will scan and plan files when creating a non-empty writer. We should also create a non-empty writer for non-rescale compaction cases but use the pre-planned manifest entries instead.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)