You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/19 12:06:44 UTC
[GitHub] [airflow] TPapajCin opened a new issue #19707: Airflow is not using most recent pod template for new tasks in 2.2.2
TPapajCin opened a new issue #19707:
URL: https://github.com/apache/airflow/issues/19707
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Debian 10
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-cassandra==2.0.1 \
apache-airflow-providers-apache-hive==2.0.2 \
apache-airflow-providers-celery==2.1.0 \
apache-airflow-providers-cncf-kubernetes==2.0.2 \
apache-airflow-providers-ftp==2.0.1 \
apache-airflow-providers-http==2.0.1 \
apache-airflow-providers-imap==2.0.1 \
apache-airflow-providers-jdbc==2.0.1 \
apache-airflow-providers-mysql==2.1.1 \
apache-airflow-providers-papermill==2.0.1 \
apache-airflow-providers-postgres==2.2.0 \
apache-airflow-providers-sftp==2.1.1 \
apache-airflow-providers-sqlite==2.0.1 \
apache-airflow-providers-ssh==2.1.1 \
apache-airflow-providers-google==5.1.0 \
apache-airflow-providers-apache-beam==3.1.0 \
### Deployment
Other Docker-based deployment
### Deployment details
Kubernetes - GKE
KubernetesCeleryExecutor
Airflow configuration stored in configmap and secrets
### What happened
I bumped into extremely frustrating bug that breaks my Airflow deployment.
I have a CI/CD K8S pipeline that:
- Creates new configmaps and secrets
- Deploys new version of deoployment file with new configmaps and secrets
- After deployment, deletes old configmaps and secrets.
Pod Template contains also contains references to configmaps and secrets and here lies the problem.
After upgrade from 2.1.3 to 2.2.2 very small portion of DAGs is using old configmap names, even from deployments that are ~2-3 deployems old. Out of 29 DAGs it happens to ~2-4 of them and often next deployments are fixing one but breaking other DAGs.
"K8s Pod Spec" on Airflow shows correct configmap names but once they are deployed on k8s, the wrong one is used and this prevents pod from starting because these old configmaps and secrets do not exist anymore.
I can't tell what could be the reason but I guess that Airflow must be storing configmap and secret names in database somehow. It did not happen in 2.1.3 version or any other versions before.
### What you expected to happen
K8s pods deployed by Airflow should have correct configmap and secret names used.
### How to reproduce
- Deploy airflow with kubernetes executor.
- Create a pod template that contains a secret or configmap.
- Get lots of DAGs - the more the better because it doesn't happen to every DAG. Let them run.
- Update the pod template with new configmap or secret.
- Restart Airflow instance with that new pod template.
- Immediately after previous step, delete old configmaps or secrets that are no longer used(but were before) in pod template.
- Wait for new tasks of DAGs to run.
- Check that small amount of new pods is trying to still use old configmap/secret and they fail because of that.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TPapajCin commented on issue #19707: Airflow is not using most recent pod template for new tasks in 2.2.2
Posted by GitBox <gi...@apache.org>.
TPapajCin commented on issue #19707:
URL: https://github.com/apache/airflow/issues/19707#issuecomment-974025728
Now I can't downgrade to pre-2.2-x because it shows `alembic.util.exc.CommandError: Can't locate revision identified by '7b2661a43ba3'`
Can anybody help? because only few DAGs are running in Dev and Test environments it went into Production where there are many more DAGs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TPapajCin commented on issue #19707: Airflow is not using most recent pod template for new tasks in 2.2.2
Posted by GitBox <gi...@apache.org>.
TPapajCin commented on issue #19707:
URL: https://github.com/apache/airflow/issues/19707#issuecomment-974031398
False alarm, it seems that someone broke some script, it's not related to airflow :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TPapajCin edited a comment on issue #19707: Airflow is not using most recent pod template for new tasks in 2.2.2
Posted by GitBox <gi...@apache.org>.
TPapajCin edited a comment on issue #19707:
URL: https://github.com/apache/airflow/issues/19707#issuecomment-974031398
False alarm, it seems that someone broke some scripts, it's not related to airflow :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TPapajCin closed issue #19707: Airflow is not using most recent pod template for new tasks in 2.2.2
Posted by GitBox <gi...@apache.org>.
TPapajCin closed issue #19707:
URL: https://github.com/apache/airflow/issues/19707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org