[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022316#comment-17022316 ] 

ASF subversion and git services commented on AIRFLOW-6359:
----------------------------------------------------------

Commit 7423a6ec8eb36b9f17188b115deca7b0ddd49047 in airflow's branch refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7423a6e ]

Revert "[AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978)"

This reverts commit edb4f62f518cc28adb54542bb39ad808c29ad2c4.


> spark_submit_hook.py polling too frequent
> -----------------------------------------
>
>                 Key: AIRFLOW-6359
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: hooks
>    Affects Versions: 1.10.6
>            Reporter: t oo
>            Assignee: t oo
>            Priority: Major
>             Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval instead of every second
>  
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>  
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)