You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "zhangxinyu (JIRA)" <ji...@apache.org> on 2018/05/15 11:22:00 UTC

[jira] [Created] (FLINK-9367) Truncate() in BucketingSink is only allowed after hadoop2.7

zhangxinyu created FLINK-9367:
---------------------------------

             Summary: Truncate() in BucketingSink is only allowed after hadoop2.7
                 Key: FLINK-9367
                 URL: https://issues.apache.org/jira/browse/FLINK-9367
             Project: Flink
          Issue Type: Improvement
          Components: Streaming Connectors
    Affects Versions: 1.5.0
            Reporter: zhangxinyu
             Fix For: 1.5.0


When output to HDFS using BucketingSink, truncate() is only allowed after hadoop2.7.
If some tasks failed, the ".valid-length" file is created for the lower version hadoop.

The problem is, if other people want to use the data in HDFS, they must know how to deal with the ".valid-length" file, otherwise, the data may be not exactly-once.

I think it's not convenient for other people to use the data. Why not just read the in-progress file and write a new file when restoring instead of writing a ".valid-length" file.

In this way, others who use the data in HDFS don't need to know how to deal with the ".valid-length" file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)