You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Olivier Van Goethem (JIRA)" <ji...@apache.org> on 2019/04/23 13:59:00 UTC
[jira] [Updated] (AIRFLOW-4393) Add retry logic when fetching pod
status and/or logs in KubernetesPodOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivier Van Goethem updated AIRFLOW-4393:
-----------------------------------------
Description:
Over the last weeks we have observed 2 occasions where:
* KubernetesPodOperator successfully launches the pod
* KubernetesPodOperator then fails when attempting to check the pod status/logs due to:
{code:java}
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1329dc0cf8>: Failed to establish a new connection: [Errno 111] Connection refused',)
{code}
* KubernetesPodOperator launches another pod
* We now have 2 pods running in parallel
The cause for the 'Connection refused' is due to a transient network error. As such, the KubernetesPodOperator should attempt to retry checking the pod status/logs rather than failing the task.
Airflow is being run through GCP's Cloud Composer.
was:
Over the last weeks we have observed 2 occasions where:
* KubernetesPodOperator successfully launches the pod
* KubernetesPodOperator then fails when attempting to check the pod status due to:
{code:java}
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1329dc0cf8>: Failed to establish a new connection: [Errno 111] Connection refused',)
{code}
* KubernetesPodOperator launches another pod
* We know have 2 pods running in parallel
The cause for the 'Connection refused' is due to a transient network error. As such, the KubernetesPodOperator should attempt to retry checking the pod status/logs rather than failing the task.
Airflow is being run through GCP's Cloud Composer.
> Add retry logic when fetching pod status and/or logs in KubernetesPodOperator
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-4393
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4393
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Olivier Van Goethem
> Priority: Minor
>
> Over the last weeks we have observed 2 occasions where:
> * KubernetesPodOperator successfully launches the pod
> * KubernetesPodOperator then fails when attempting to check the pod status/logs due to:
> {code:java}
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1329dc0cf8>: Failed to establish a new connection: [Errno 111] Connection refused',)
> {code}
> * KubernetesPodOperator launches another pod
> * We now have 2 pods running in parallel
> The cause for the 'Connection refused' is due to a transient network error. As such, the KubernetesPodOperator should attempt to retry checking the pod status/logs rather than failing the task.
> Airflow is being run through GCP's Cloud Composer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)