You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zhuxiaoshang (Jira)" <ji...@apache.org> on 2020/12/07 12:45:00 UTC
[jira] [Commented] (FLINK-19345) In Table File Sink, introduce
streaming sink compaction
[ https://issues.apache.org/jira/browse/FLINK-19345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245183#comment-17245183 ]
zhuxiaoshang commented on FLINK-19345:
--------------------------------------
HI [~lzljs3620320],when the `auto-compaction` is open,the bucket should not be committed before the compaction is done.Otherwise,may cause compacting failure.Correct me if I'm wrong.
When i dig into the code,i found that the bucket is committed in `AbstractStreamingWriter#notifyCheckpointComplete` no matter whether the `auto-compaction` is open.
> In Table File Sink, introduce streaming sink compaction
> -------------------------------------------------------
>
> Key: FLINK-19345
> URL: https://issues.apache.org/jira/browse/FLINK-19345
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Priority: Major
> Fix For: 1.12.0
>
>
> Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.
> Target:
> * Compact all files generated by this job in a single checkpoint.
> * With compaction, Users can have smaller checkpoint interval, even to seconds.
> Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)