[jira] [Commented] (FLINK-24757) Yarn application is not terminated after the job finishes when submitting a yarn-per-job insert job with SQL client

["Caizhi Weng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-24757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440151#comment-17440151 ] 

Caizhi Weng commented on FLINK-24757:
-------------------------------------

This issue is caused by the mismatch of {{table.dml-sync}} (whose default value is false) and {{execution.attached}} (whose default value is true).

According to the offline discussion with [~twalthr], these two options are orthogonal. {{execution.attached}} determines if cluster needs to wait for client to fetch the result or just leaves by itself; {{table.dml-sync}}  determines if client will wait for and fetch that result.

However they are somewhat related. If client doesn’t want that result but the cluster is willing to wait then it will have to wait forever; On the other hand if client needs that result but cluster leaves by itself there will be an exception.

Currently we don't have a good solution to the case when user specify the two options at the same time, so we're just fixing it in SQL client submission and improve the documentation for {{table.dml-sync}}. Let's come back to this issue when a {{flink run}} + SQL user complains about this.

> Yarn application is not terminated after the job finishes when submitting a yarn-per-job insert job with SQL client
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-24757
>                 URL: https://issues.apache.org/jira/browse/FLINK-24757
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Client
>    Affects Versions: 1.14.0
>            Reporter: Caizhi Weng
>            Priority: Major
>             Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> I've seen this problem for about three times in the user mailing thread (previously I suspect that the users are specifying the wrong {{execution.target}}) until I myself also bumped into this problem. I've submitted a yarn-per-job batch insert SQL with Flink SQL client and after the job finishes Yarn application is not terminated.
> This is because yarn job cluster is using {{MiniDispatcher}} and it will directly terminate only in detached execution mode. This execution mode is (through some function calls) related to {{DeploymentOptions#ATTACHED}} which is true by default.
> When submitting an insert job, SQL client will not wait for the job to finish. Instead it only reports the job id. So I think it is reasonable to set detached mode for every insert job.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)