You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yun Gao (Jira)" <ji...@apache.org> on 2020/05/09 08:01:00 UTC
[jira] [Created] (FLINK-17593) Support arbitrary recovery mechanism
for PartFileWriter
Yun Gao created FLINK-17593:
-------------------------------
Summary: Support arbitrary recovery mechanism for PartFileWriter
Key: FLINK-17593
URL: https://issues.apache.org/jira/browse/FLINK-17593
Project: Flink
Issue Type: New Feature
Components: Connectors / FileSystem
Reporter: Yun Gao
Fix For: 1.11.0
Currently Bucket relies directly on _RecoverableOutputStream_ provided by FileSystem to achieve snapshotting and recovery the in-progress part file for all the PartFileWriter implementations. This would require that the PartFileWriter must be based on the OutputStream.
To support the path-based PartFileWriter required by the Hive Sink, we will first need to abstract the snapshotting mechanism of the PartFileWriter and make RecoverableOutputStream to be one type of implementation, thus we could decouple PartFileWriter with the output streams.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)