You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Kostas Kloudas (Jira)" <ji...@apache.org> on 2020/09/24 07:05:00 UTC

[jira] [Comment Edited] (FLINK-19345) Introduce File streaming sink compaction

    [ https://issues.apache.org/jira/browse/FLINK-19345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201317#comment-17201317 ] 

Kostas Kloudas edited comment on FLINK-19345 at 9/24/20, 7:04 AM:
------------------------------------------------------------------

Hi [~lzljs3620320], as you may have noticed there is an ongoing discussion about the new sink interface going on in the dev mailing list and the target is to have it ready for the 1.12.

Instead of having parallel efforts, why not contributing to the discussion there so that our efforts are aligned? Also opening a new discussion although there is already an open one for similar problems, I think it slows down the process of reaching a consensus as potentially parts of the already done discussion may have to be repeated.
 
I will have a look at the document attached here, but I would think that participating in the discussion on the FLIP about the unified sinks would be a lot more helpful also for the community.
 


was (Author: kkl0u):
Hi [~lzljs3620320], as you may have noticed there is an ongoing discussion about the new sink interface going on in the dev mailing list and the target is to have it ready for the 1.12.

Instead of having parallel efforts, why not contributing to the discussion there so that our efforts are aligned? Also opening a new discussion although there is already an open one for similar problems, I think it slows down the process of reaching a consensus. 
I will have a look at the document attached here, but I would think that participating in the discussion on the FLIP about the unified sinks would be a lot more helpful also for the community.
 

> Introduce File streaming sink compaction
> ----------------------------------------
>
>                 Key: FLINK-19345
>                 URL: https://issues.apache.org/jira/browse/FLINK-19345
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table SQL / Runtime
>            Reporter: Jingsong Lee
>            Assignee: Jingsong Lee
>            Priority: Major
>             Fix For: 1.12.0
>
>
> Users often complain that many small files are written out. Small files will affect the performance of file reading and the DFS system, and even the stability of the DFS system.
> Target: 
>  * Compact all files generated by this job in a single checkpoint.
>  * With compaction, Users can have smaller checkpoint interval, even to seconds.
> Document: https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing



--
This message was sent by Atlassian Jira
(v8.3.4#803005)