[jira] [Closed] (FLINK-30204) Table Store support separated compact jobs

["Caizhi Weng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Caizhi Weng closed FLINK-30204.
-------------------------------
    Resolution: Fixed

> Table Store support separated compact jobs
> ------------------------------------------
>
>                 Key: FLINK-30204
>                 URL: https://issues.apache.org/jira/browse/FLINK-30204
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table Store
>    Affects Versions: table-store-0.3.0
>            Reporter: Caizhi Weng
>            Assignee: Caizhi Weng
>            Priority: Major
>             Fix For: table-store-0.3.0
>
>
> Currently table store sinks will write and compact data files from the same job. While this implementation is enough and more economical for most users, some user may expect higher or more steady write throughput.
> We decided to support creating separated compact jobs for Table Store. This will bring us the following advantages:
> * Write jobs can concentrate only on writing files. Their throughput will be higher and more steady.
> * By creating only one compact job for each table, no commit conflicts will occur.
> The structure of a separated compact job is sketched out as follows:
> * There should be three vertices in a compact job. One source vertex, one sink (compactor) vertex and one commit vertex.
> * The source vertex is responsible for generating records containing partitions and buckets to be compacted.
> * The sink vertex accepts records containing partitions and buckets, and compact these buckets.
> * The commit vertex commit the changes from the sink vertex. It is possible that the user mistakenly creates other compact jobs so commit conflicts may still occur. However as compact changes are optional, this commit vertex will commit changes in an at-most-once style.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)