[jira] [Commented] (AIRFLOW-2769) Increase num_retries polling value on Dataflow hook

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551635#comment-16551635 ] 

ASF subversion and git services commented on AIRFLOW-2769:
----------------------------------------------------------

Commit 97d5176c06823e344a12494222156b48108588f2 in incubator-airflow's branch refs/heads/master from [~pwoods25443]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=97d5176 ]

[AIRFLOW-2769] Increase num_retries polling value on Dataflow hook

Closes #3617 from pwoods25443/2769-dataflow-num-
retries


> Increase num_retries polling value on Dataflow hook
> ---------------------------------------------------
>
>                 Key: AIRFLOW-2769
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2769
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, Dataflow
>    Affects Versions: 1.10
>            Reporter: Paul Woods
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> *Problem Description*
> When airflow launches a Job in Dataflow, it polls the GCP api for job status until the job is complete or fails.  The GCP API occasionally returns 500 and 429  errors on these API requests, which causes the airflow task to fail intermittently, particularly for long-running tasks, while the dataflow job itself does not terminate.
> The recommended action is to retry the request with exponential backoff ([https://developers.google.com/drive/api/v3/handle-errors)].   The gcp api provides this service via the `num_retries` parameter on execute(), but that parameter is not used in
> {code:java}
> airflow.contrib.hooks.gcp_dataflow_hook{code}
> *Proposed Solution*
> Add num_retries to the execute() calls in 
> {code:java}
> _DataflowJob._get_job_id_from_name{code}
> and _
> {code:java}
> _DataflowJob._get_job{code}
>  
> *NOTE:*  the same problem was addressed for Dataproc in ([https://issues.apache.org/jira/browse/AIRFLOW-1718)]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)