You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Kostas Kloudas (Jira)" <ji...@apache.org> on 2019/12/18 11:41:00 UTC
[jira] [Assigned] (FLINK-14170) Support hadoop < 2.7 with
StreamingFileSink.BulkFormatBuilder
[ https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kostas Kloudas reassigned FLINK-14170:
--------------------------------------
Assignee: Bhagavan
> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -------------------------------------------------------------
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
> Reporter: Bhagavan
> Assignee: Bhagavan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7 irrespective of Row/bulk format builder. This restriction is due to truncate is not supported in Hadoop < 2.7
> However, BulkFormatBuilder does not use truncate method to restore the file. So the restricting StreamingFileSink.BulkFormatBuilder to be used only with Hadoop >= 2.7 is not necessary.
> So requested improvement is to remove the precondition on HadoopRecoverableWriter and allow BulkFormatBuilder (Parquet) to be used in Hadoop 2.6 ( Most of the enterprises still on CDH 5.x)
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)