You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pmcquighan-camus (via GitHub)" <gi...@apache.org> on 2026/02/04 17:29:31 UTC

Re: [I] Google Dataflow provider does not retry on service 503 errors [airflow]

pmcquighan-camus commented on issue #57359:
URL: https://github.com/apache/airflow/issues/57359#issuecomment-3848772245

   Hi - we have been using `apache-airflow-providers-google==19.3.0` for the past several weeks and the issue recurred:
   
   ```
   [2026-02-02 04:42:52] ERROR - Exception occurred while checking for job completion. source=airflow.providers.google.cloud.triggers.dataflow.TemplateJobStartTrigger loc=dataflow.py:149
   ServiceUnavailable: 503 Visibility check was unavailable. Please retry the request and contact support if the problem persists
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/triggers/dataflow.py", line 113 in run
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/dataflow.py", line 1480 in get_job_status
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/dataflow.py", line 1457 in get_job
   File "/home/airflow/.local/lib/python3.12/site-packages/google/cloud/dataflow_v1beta3/services/jobs_v1_beta3/async_client.py", line 478 in get_job
   File "/home/airflow/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 88 in __await__
   AioRpcError: <AioRpcError of RPC that terminated with:
   	status = StatusCode.UNAVAILABLE
   	details = "Visibility check was unavailable. Please retry the request and contact support if the problem persists"
   	debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.126.95:443 {created_time:"2026-02-02T12:42:52.166480427+00:00", grpc_status:14, grpc_message:"Visibility check was unavailable. Please retry the request and contact support if the problem persists"}"
   >
   File "/home/airflow/.local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 85 in __await__
   File "/home/airflow/.local/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 472 in __await__
   File "/home/airflow/.local/lib/python3.12/site-packages/grpc/aio/_call.py", line 327 in __await__
   ```
   
   The task retry error from the initial bug report `ValueError: dictionary update sequence element #0 has length 1; 2 is required` no longer occurs but as expected the task is still just considered failed since the triggerer marked the task as failed, and the dataflow job is *not* actually retried (nor does the triggerer or any component attempt to get the status a second time)
   ```
   [2026-02-02 04:45:10] ERROR - Task failed with exception source=task loc=task_runner.py:972
   AirflowException: 503 Visibility check was unavailable. Please retry the request and contact support if the problem persists
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 920 in run
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1307 in _execute_task
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 1632 in resume_execution
   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/operators/dataflow.py", line 650 in execute_complete
   ```
   
   >  manage to find how to reproduce it, please comment (without clear reproduction steps there's not too much that we can do).
   
   it is dependent on the GCP dataflow api returning a 503 - the only way to reliably reproduce it would be with a mock http server that the gcp client connects to and having it return a 503 (or by mocking out the `JobsV1Beta3AsyncClient.get_job` method to throw the AioRpcError in the stacktrace above).  i don't have permissions to reopen this issue but it is definitely a still a bug with the retry logic in the provider 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org