[jira] [Closed] (SPARK-3596) Support changing the yarn client monitor interval

["Andrew Or (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Or closed SPARK-3596.
----------------------------
          Resolution: Fixed
       Fix Version/s: 1.4.0
            Assignee: Weizhong
    Target Version/s: 1.4.0

> Support changing the yarn client monitor interval 
> --------------------------------------------------
>
>                 Key: SPARK-3596
>                 URL: https://issues.apache.org/jira/browse/SPARK-3596
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 1.2.0
>            Reporter: Thomas Graves
>            Assignee: Weizhong
>             Fix For: 1.4.0
>
>
> Right now spark on yarn has a monitor interval that can be configured by spark.yarn.report.interval.  This is how often the client checks with the RM to get status on the running application in cluster mode.   We should allow users to set this interval as some may not need to check so often.   There is another jira filed to make it so the client doesn't have to stay around for cluster mode.
> With the changes in https://github.com/apache/spark/pull/2350, it further extends that to affect client mode. 
> We may want to add in specific configs for that since the monitorApplication function is now used in multiple different scenarios it actually might make sense for it to take the timeout as a parameter. You could want different timeout for different situations.
> for instance how quickly we poll on client side and print information (cluster mode) vs how quickly we recognize the application quit and we want to terminate (client mode), I want the latter to happen quickly where as in cluster mode I might not care as much about how often it is printing updated info to the screen. I guess its private so we could leave it as is and change if we add support for that later.
> my suggestion for name would be something like spark.yarn.client.progress.pollinterval. If we were to add separate ones in the future then they could be something like spark.yarn.app.ready.pollinterval and spark.yarn.app.completion.pollinterval 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org