You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/19 11:27:00 UTC
[GitHub] [airflow] TPapajCin opened a new issue #19704: Pods that end in CreateContainerConfigError are not deleted by Airflow after DAG ends
TPapajCin opened a new issue #19704:
URL: https://github.com/apache/airflow/issues/19704
### Apache Airflow version
2.1.3
### Operating System
Debian 10
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-cassandra==2.0.1 \
apache-airflow-providers-apache-hive==2.0.2 \
apache-airflow-providers-celery==2.1.0 \
apache-airflow-providers-cncf-kubernetes==2.0.2 \
apache-airflow-providers-ftp==2.0.1 \
apache-airflow-providers-http==2.0.1 \
apache-airflow-providers-imap==2.0.1 \
apache-airflow-providers-jdbc==2.0.1 \
apache-airflow-providers-mysql==2.1.1 \
apache-airflow-providers-papermill==2.0.1 \
apache-airflow-providers-postgres==2.2.0 \
apache-airflow-providers-sftp==2.1.1 \
apache-airflow-providers-sqlite==2.0.1 \
apache-airflow-providers-ssh==2.1.1 \
apache-airflow-providers-google==5.1.0 \
apache-airflow-providers-apache-beam==3.1.0 \
### Deployment
Other Docker-based deployment
### Deployment details
Kubernetes - GKE
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS = True
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE = True
KubernetesCeleryExecutor
### What happened
I set the `AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG` parameter to the wrong, nonexisting version.
Before realizing it, Airflow created few pods that ended with `ImagePullBackOff` and are stuck in restarting state `Back-off pulling image "xxx"` and status of pods is `CreateContainerConfigError`.
I see no logs in scheduler about these pods, it seems that they were abandoned by Airflow when DAGs entered failed state because pods were not ready.
### What you expected to happen
_No response_
### How to reproduce
- Set these configuration options:
- AIRFLOW__KUBERNETES__DELETE_WORKER_PODS = True
- AIRFLOW__KUBERNETES__DELETE_WORKER_PODS_ON_FAILURE = True
- Use KubernetesCeleryExecutor
- Set AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG to nonexisting tag
- Run some Airflow tasks
- Check that pods are failing and are in CreateContainerConfigError state.
- Mark the task as finished/failed or wait for tasks to fail.
- Pods are still there and will remain until deleted manually
### Anything else
It happens always
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #19704: Pods that end in CreateContainerConfigError are not deleted by Airflow after DAG ends
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #19704:
URL: https://github.com/apache/airflow/issues/19704#issuecomment-999951215
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19704: Pods that end in CreateContainerConfigError are not deleted by Airflow after DAG ends
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19704:
URL: https://github.com/apache/airflow/issues/19704#issuecomment-975619055
Can you try in 2.2.2? There's a pending timeout and the pods are deleted if they fail to start after a pending timeout. See https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-pods-pending-timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #19704: Pods that end in CreateContainerConfigError are not deleted by Airflow after DAG ends
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #19704:
URL: https://github.com/apache/airflow/issues/19704#issuecomment-1002819768
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #19704: Pods that end in CreateContainerConfigError are not deleted by Airflow after DAG ends
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #19704:
URL: https://github.com/apache/airflow/issues/19704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org