You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Stephan Ewen (JIRA)" <ji...@apache.org> on 2018/09/18 08:46:00 UTC
[jira] [Commented] (FLINK-8939) Provide better support for saving
streaming data to s3
[ https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618710#comment-16618710 ]
Stephan Ewen commented on FLINK-8939:
-------------------------------------
The new StreamingFileSink will be extended with S3 support in the next release (1.7). See issue [FLINK-9752].
> Provide better support for saving streaming data to s3
> ------------------------------------------------------
>
> Key: FLINK-8939
> URL: https://issues.apache.org/jira/browse/FLINK-8939
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API, Streaming Connectors
> Reporter: chris snow
> Priority: Major
> Attachments: 18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png
>
>
> Flink seems to struggle saving data to s3 due to the lack of a truncate method, and in my test this resulted in lots of files with a .valid-length suffix
> I’m using a bucketing sink:
> {code:java}
> return new BucketingSink<Tuple2<String, Object>>(path)
> .setWriter(writer)
> .setBucketer(new DateTimeBucketer<Tuple2<String, Object>>(formatString));{code}
>
> !18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png!
> See also, the discussion in the comments here: https://issues.apache.org/jira/browse/FLINK-8543?filter=-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)