[jira] [Commented] (FLINK-19069) finalizeOnMaster takes too much time and client timeouts

["Kenneth William Krugler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186231#comment-17186231 ] 

Kenneth William Krugler commented on FLINK-19069:
-------------------------------------------------

I'd recently posted about a related issue to the dev mailing list, where I asked:

{quote}[...] the default behavior of Hadoop’s FileOutputCommitter (with algorithm == 1) is to put files in task-specific sub-dirs.

It’s depending on a post-completion “merge paths” action to be taken by what is (for Hadoop) the Application Master.

I assume that when running on a real cluster, the HadoopOutputFormat.finalizeGlobal() method’s call to commitJob() would do this, but it doesn’t seem to be happening when I run locally.

If I set the algorithm version to 2, then “merge paths” is handled by FileOutputCommitter immediately, and the HadoopOutputFormat code finds files in the expected location.

Wondering if Flink should always be using version 2 of the algorithm, as that’s more performant when there are a lot of results (which is why it was added).

{quote}

> finalizeOnMaster takes too much time and client timeouts
> --------------------------------------------------------
>
>                 Key: FLINK-19069
>                 URL: https://issues.apache.org/jira/browse/FLINK-19069
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Task
>    Affects Versions: 1.9.0
>            Reporter: Jiayi Liao
>            Priority: Major
>
> Currently we execute {{finalizeOnMaster}} in JM's main thread, which may stuck the JM for a very long time and client timeouts eventually. 
> For example, we'd like to write data to HDFS  and commit files on JM, which takes more than ten minutes to commit tens of thousands files.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)