[jira] [Work started] (AIRFLOW-6359) spark_submit_hook.py polling too frequent

["t oo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-6359 started by t oo.
-------------------------------------
> spark_submit_hook.py polling too frequent
> -----------------------------------------
>
>                 Key: AIRFLOW-6359
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: hooks
>    Affects Versions: 1.10.6
>            Reporter: t oo
>            Assignee: t oo
>            Priority: Major
>
> spark polling every second is too frequent, use config to change to desired interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval instead of every second
>  
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>  
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)