[jira] [Comment Edited] (FLINK-27557) Create the empty writer for 'ALTER TABLE ... COMPACT'

["Jane Chan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539352#comment-17539352 ] 

Jane Chan edited comment on FLINK-27557 at 5/20/22 2:59 AM:
------------------------------------------------------------

{quote}Can we just use overwrite?
 * overwrite specific manifest entries.
 * overwrite whole table or some whole partition. (eg: when rescale in compaction){quote}
Overwrite means the writer cannot accept the specified manifest entries as restored files, so how to pass them to FileStoreCommit as compact before(mark as delete) is a new question. Beyond that, overwrite corresponds to the commit kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this situation.

I rethink it and maybe we don't need to generate the new files for the normal compaction.
 * The simplest way is to build the levels from the restored files, and don't sink record to MemTable, and submit compaction when precommit is invoked.
 * In the future, maybe we can introduce a new dedicated compaction source to restore LSM, perform compaction and commit, which avoids the shuffle cost.

 

 

 


was (Author: qingyue):
{quote}Can we just use overwrite?
 * overwrite specific manifest entries.
 * overwrite whole table or some whole partition. (eg: when rescale in compaction){quote}

Overwrite means the writer cannot accept the specified manifest entries as restored files, so how to pass them to FileStoreCommit as compact before(mark as delete) is a new question. Beyond that, overwrite corresponds to the commit kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this situation.

I rethink it and maybe we don't need to generate the new files for the normal compaction. 

* The simplest way is to build the levels from the restored files, and don't sink record to MemTable, and submit compaction when precommit is invoked.
* In the future, maybe we can introduce a new dedicated compaction source to restore LSM, perform compaction and commit, which avoids the shuffle cost.

 !normal-compaction.png! 

 !rescale-bucket-compaction.png! 

 !optimized-normal-compaction.png! 

> Create the empty writer for 'ALTER TABLE ... COMPACT'
> -----------------------------------------------------
>
>                 Key: FLINK-27557
>                 URL: https://issues.apache.org/jira/browse/FLINK-27557
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table Store
>    Affects Versions: table-store-0.2.0
>            Reporter: Jane Chan
>            Priority: Major
>             Fix For: table-store-0.2.0
>
>         Attachments: normal-compaction.png, optimized-normal-compaction.png, rescale-bucket-compaction.png
>
>
> Currently, FileStoreWrite only creates an empty writer for the \{{INSERT OVERWRITE}} clause. We should also create the empty writer for manual compaction.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)