You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yanxiaobin (JIRA)" <ji...@apache.org> on 2018/03/14 08:08:00 UTC
[jira] [Comment Edited] (FLINK-8939) Provide better support for
saving streaming data to s3
[ https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398211#comment-16398211 ]
yanxiaobin edited comment on FLINK-8939 at 3/14/18 8:07 AM:
------------------------------------------------------------
I am also using BucketingSink API for saving streaming data to s3. Have you ever met this problem？see : [FLINK-8794|https://issues.apache.org/jira/browse/FLINK-8794]
was (Author: backlight):
I am also using BucketingSink API for saving streaming data to s3. Have you ever met this problem？see : https://issues.apache.org/jira/browse/FLINK-8794
> Provide better support for saving streaming data to s3
> ------------------------------------------------------
>
> Key: FLINK-8939
> URL: https://issues.apache.org/jira/browse/FLINK-8939
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: chris snow
> Priority: Major
> Attachments: 18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png
>
>
> Flink seems to struggle saving data to s3 due to the lack of a truncate method, and in my test this resulted in lots of files with a .valid-length suffix
> I’m using a bucketing sink:
> {code:java}
> return new BucketingSink<Tuple2<String, Object>>(path)
> .setWriter(writer)
> .setBucketer(new DateTimeBucketer<Tuple2<String, Object>>(formatString));{code}
>
> !18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png!
> See also, the discussion in the comments here: https://issues.apache.org/jira/browse/FLINK-8543?filter=-2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)