[jira] [Comment Edited] (FLINK-11499) Extend StreamingFileSink BulkFormats to support arbitrary roll policies

["Piotr Nowojski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070963#comment-17070963 ] 

Piotr Nowojski edited comment on FLINK-11499 at 3/30/20, 4:03 PM:
------------------------------------------------------------------

[~zenfenan] no, sorry for not being clear. By
> we need to have a point in the past were both of the streams align, so probably if we want to roll main output stream at most once per hour, we still need to roll it on checkpoint.

I meant that once we decide to roll the main output stream (after one hour for example), we still need to roll together with the WAL stream and both of those things will need to happen on checkpoint. In other words, we probably can not roll main stream in the middle of checkpoint. Or am I wrong?

Re the latency, slow recovery and double writes. I get those points. In the end I think I'm +1 for any more specialised implementations for some particular file formats, like Parquet and ORC that could avoid those problems. I think they wouldn't contradict the WAL approach. We might have WAL approach as a general solution for any bulk format, and we can also independently work on more specialised solutions.

Also if you think that we can easily provide specialised support for both Parquet and ORC, we could down prioritise WAL approach.

edit: Also I don't think that your current work on the FLINK-10114 and the WAL proposal contradicts one another. I think it might be quite opposite, as if we go on with the WAL, it would probably re-use your ORC bulk format support for the main data output stream. 


was (Author: pnowojski):
[~zenfenan] no, sorry for not being clear. By
> we need to have a point in the past were both of the streams align, so probably if we want to roll main output stream at most once per hour, we still need to roll it on checkpoint.

I meant that once we decide to roll the main output stream (after one hour for example), we still need to roll together with the WAL stream and both of those things will need to happen on checkpoint. In other words, we probably can not roll main stream in the middle of checkpoint. Or am I wrong?

Re the latency, slow recovery and double writes. I get those points. In the end I think I'm +1 for any more specialised implementations for some particular file formats, like Parquet and ORC that could avoid those problems. I think they wouldn't contradict the WAL approach. We might have WAL approach as a general solution for any bulk format, and we can also independently work on more specialised solutions.

Also if you think that we can easily provide specialised support for both Parquet and ORC, we could down prioritise WAL approach.

> Extend StreamingFileSink BulkFormats to support arbitrary roll policies
> -----------------------------------------------------------------------
>
>                 Key: FLINK-11499
>                 URL: https://issues.apache.org/jira/browse/FLINK-11499
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / FileSystem
>            Reporter: Seth Wiesman
>            Priority: Major
>              Labels: usability
>             Fix For: 1.11.0
>
>
> Currently when using the StreamingFilleSink Bulk-encoding formats can only be combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress part file on every checkpoint.
> However, many bulk formats such as parquet are most efficient when written as large files; this is not possible when frequent checkpointing is enabled. Currently the only work-around is to have long checkpoint intervals which is not ideal.
>  
> The StreamingFileSink should be enhanced to support arbitrary roll policy's so users may write large bulk files while retaining frequent checkpoints.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)