[jira] [Resolved] (AIRFLOW-1718) Increase num_retries polling value on Dataproc hook

["Chris Riccomini (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Riccomini resolved AIRFLOW-1718.
--------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.10.0

> Increase num_retries polling value on Dataproc hook
> ---------------------------------------------------
>
>                 Key: AIRFLOW-1718
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1718
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Crystal Qian
>            Assignee: Crystal Qian
>            Priority: Minor
>             Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, `num_retries = 0` when execute() is called (https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.HttpRequest-class.html#execute), which causes intermittent 500 errors (https://stackoverflow.com/questions/46522261/deadline-exceeded-when-airflow-runs-spark-jobs). We should increase this to allow retries for internal Dataproc queries to other services in the short-term; also seeing if the `num_retries` count can be increased at the _google-api-python-client_ level in the long-term.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)