You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "chris snow (JIRA)" <ji...@apache.org> on 2018/03/14 07:52:00 UTC

[jira] [Created] (FLINK-8939) Provide better support for saving streaming data to s3

chris snow created FLINK-8939:
---------------------------------

             Summary: Provide better support for saving streaming data to s3
                 Key: FLINK-8939
                 URL: https://issues.apache.org/jira/browse/FLINK-8939
             Project: Flink
          Issue Type: Improvement
          Components: DataStream API
            Reporter: chris snow
         Attachments: 18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png

Flink seems to struggle saving data to s3 due to the lack of a truncate method, and in my test this resulted in lots of files with a .valid-length suffix

I’m using a bucketing sink:
{code:java}
return new BucketingSink<Tuple2<String, Object>>(path)
.setWriter(writer)
.setBucketer(new DateTimeBucketer<Tuple2<String, Object>>(formatString));{code}
 

!18652BC0-DD67-42C3-9A33-12F7BC10F9F3.png!

See also, the discussion in the comments here: https://issues.apache.org/jira/browse/FLINK-8543?filter=-2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)