You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zhangxinyu (JIRA)" <ji...@apache.org> on 2018/06/19 12:04:00 UTC
[jira] [Closed] (FLINK-9367) Truncate() in BucketingSink is only
allowed after hadoop2.7
[ https://issues.apache.org/jira/browse/FLINK-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhangxinyu closed FLINK-9367.
-----------------------------
Resolution: Won't Do
BucketingSink is going to be rewritten, and hadoop below 2.7 won't be supported. Therefore, this issue is unnecessary.
> Truncate() in BucketingSink is only allowed after hadoop2.7
> -----------------------------------------------------------
>
> Key: FLINK-9367
> URL: https://issues.apache.org/jira/browse/FLINK-9367
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
> Affects Versions: 1.5.0
> Reporter: zhangxinyu
> Priority: Major
>
> When output to HDFS using BucketingSink, truncate() is only allowed after hadoop2.7.
> If some tasks failed, the ".valid-length" file is created for the lower version hadoop.
> The problem is, if other people want to use the data in HDFS, they must know how to deal with the ".valid-length" file, otherwise, the data may be not exactly-once.
> I think it's not convenient for other people to use the data. Why not just read the in-progress file and write a new file when restoring instead of writing a ".valid-length" file.
> In this way, others who use the data in HDFS don't need to know how to deal with the ".valid-length" file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)