You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Yun Gao (Jira)" <ji...@apache.org> on 2020/05/09 08:01:00 UTC

[jira] [Created] (FLINK-17593) Support arbitrary recovery mechanism for PartFileWriter

Yun Gao created FLINK-17593:
-------------------------------

             Summary: Support arbitrary recovery mechanism for PartFileWriter
                 Key: FLINK-17593
                 URL: https://issues.apache.org/jira/browse/FLINK-17593
             Project: Flink
          Issue Type: New Feature
          Components: Connectors / FileSystem
            Reporter: Yun Gao
             Fix For: 1.11.0


Currently Bucket relies directly on _RecoverableOutputStream_ provided by FileSystem to achieve snapshotting and recovery the in-progress part file for all the PartFileWriter implementations. This would require that the PartFileWriter must be based on the OutputStream.

To support the path-based PartFileWriter required by the Hive Sink, we will first need to abstract the snapshotting mechanism of the PartFileWriter and make RecoverableOutputStream to be one type of implementation, thus we could decouple PartFileWriter with the output streams. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)