[jira] [Commented] (FLINK-9367) Truncate() in BucketingSink is only allowed after hadoop2.7

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496524#comment-16496524 ] 

ASF GitHub Bot commented on FLINK-9367:
---------------------------------------

GitHub user zhangxinyu1 opened a pull request:

    https://github.com/apache/flink/pull/6108

    [FLINK-9367] [Streaming Connectors] Allow to do truncate() in calss BucketingSink when hadoop version is lower than 2.7

    ## What is the purpose of the change
    
    In the current implementation of class BucketingSink, we cannot use truncate() function if the hadoop version is lower than 2.7. Instead, it use a valid-length file to mark how much data is valid.
    However, users which reads data from HDFS may not or should not know how deal with this valid-length file. 
    Hence, we need a configuration to decide whether use the valid-length file. If not, we should rewrite the valid file.
    
    ## Brief change log
    
    Add a function `enableForceTruncateInProgressFile()` for BucketingSink to decide whether use the valid-length file. If it's true, the valid-length file wouldn't be produced. Instead, the valid in-progress file will be rewritten. 
    
    ## Verifying this change
    
    This change is a trivial work without any test coverage.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zhangxinyu1/flink force-recovery-file-in-bucketingsink

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/6108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #6108
    
----
commit 7c5ba6d54658916e65c40fbbed646efce2c40645
Author: unknown <zh...@...>
Date:   2018-05-31T12:52:09Z

    allow to do truncate() when hadoop version is lower than 2.7

----


> Truncate() in BucketingSink is only allowed after hadoop2.7
> -----------------------------------------------------------
>
>                 Key: FLINK-9367
>                 URL: https://issues.apache.org/jira/browse/FLINK-9367
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming Connectors
>    Affects Versions: 1.5.0
>            Reporter: zhangxinyu
>            Priority: Major
>
> When output to HDFS using BucketingSink, truncate() is only allowed after hadoop2.7.
> If some tasks failed, the ".valid-length" file is created for the lower version hadoop.
> The problem is, if other people want to use the data in HDFS, they must know how to deal with the ".valid-length" file, otherwise, the data may be not exactly-once.
> I think it's not convenient for other people to use the data. Why not just read the in-progress file and write a new file when restoring instead of writing a ".valid-length" file.
> In this way, others who use the data in HDFS don't need to know how to deal with the ".valid-length" file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)