You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/01 20:35:48 UTC

[GitHub] [airflow] gabrielboehme opened a new issue, #28043: Scheduler keeps dying - Killing DAGFileProcessorProcess

gabrielboehme opened a new issue, #28043:
URL: https://github.com/apache/airflow/issues/28043

   ### Apache Airflow version
   
   2.4.3
   
   ### What happened
   
   I'm running airflow on Docker (inside Windows WSL2) with Celery Executor. Suddenly, from one day to another, my airflow scheduler started to die. I have this specific dag file that it creates dynamically 53 dags from a db. Even if I comment out this file and leave the remaining ones, airflow will kill the scheduler. 
   
   This behavior occurs when all containers are healthy after start up.
   
   Error:
   ` {processor.py:233} WARNING - Killing DAGFileProcessorProcess (PID=<PID>)` 
   
   I checked and that process exists in the container before it's killed.
   
   ### What you think should happen instead
   
   My airflow scheduler should be running as usual, without errors.
   
   ### How to reproduce
   
   You can see this dynamic dag that I think its breaking everything here in this Gist: https://gist.github.com/gabrielboehme/1e62cb5e4b26633130882544d59ad058
   
   Also, the dockerfile, docker-compose and .env file are in the gist above.
   
   ### Operating System
   
   Ubuntu 20.04 LTS
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==4.1.0
   apache-airflow-providers-celery==3.0.0
   apache-airflow-providers-cncf-kubernetes==4.3.0
   apache-airflow-providers-common-sql==1.2.0
   apache-airflow-providers-docker==3.1.0
   apache-airflow-providers-elasticsearch==4.2.0
   apache-airflow-providers-ftp==3.1.0
   apache-airflow-providers-google==8.3.0
   apache-airflow-providers-grpc==3.0.0
   apache-airflow-providers-hashicorp==3.1.0
   apache-airflow-providers-http==2.0.0
   apache-airflow-providers-imap==3.0.0
   apache-airflow-providers-microsoft-azure==4.2.0
   apache-airflow-providers-mysql==3.2.0
   apache-airflow-providers-odbc==3.1.1
   apache-airflow-providers-postgres==5.2.1
   apache-airflow-providers-redis==3.0.0
   apache-airflow-providers-sendgrid==3.0.0
   apache-airflow-providers-sftp==4.0.0
   apache-airflow-providers-slack==5.1.0
   apache-airflow-providers-sqlite==3.2.1
   apache-airflow-providers-ssh==3.1.0
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   Docker-compose, Dockerfile, and .env can be found here:
   https://gist.github.com/gabrielboehme/32ada28b251d758ef5ce334444e0d433
   
   ### Anything else
   
   Seems that every 40 or 60 seconds the airflow-scheduler gets killed.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28043: Scheduler keeps dying - Killing DAGFileProcessorProcess

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28043:
URL: https://github.com/apache/airflow/issues/28043#issuecomment-1334415036

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #28043: Scheduler keeps dying - Killing DAGFileProcessorProcess

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28043: Scheduler keeps dying - Killing DAGFileProcessorProcess
URL: https://github.com/apache/airflow/issues/28043


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org