You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/17 07:07:34 UTC
[GitHub] [airflow] vengi57 opened a new issue #17646: Scheduler is not working
vengi57 opened a new issue #17646:
URL: https://github.com/apache/airflow/issues/17646
**Apache Airflow version**:
Version - 2.1.1
**OS**:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
**Apache Airflow Provider versions**:
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-sqlite==2.0.0
**Deployment**:
VM
**What happened**:
Getting this warning message
The scheduler does not appear to be running. The last heartbeat was received 2 days ago. The DAGs list may not update, and new tasks will not be scheduled
So that when I trigger my DAG it was stuck in the in-progress state for a long time and none of the tasks of DAGs are running
If I restart the scheduler it was working fine for some hours but again getting the same above mentioned issue
**What you expected to happen**:
The scheduler should work properly in order to execute the dags which will take around 3 to 5 hrs to complete
<!-- What do you think went wrong? -->
Even though no DAGs are running we getting "The scheduler does not appear to be running. The last heartbeat was received 2 days ago. The DAGs list may not update, and new tasks will not be scheduled" error
**How to reproduce it**:
1.Install Airflow
2. Change the executor to Local Executor
3. Change sql_alchemy_conn to Postgres DB
4. Run airflow webserver &
5. Run airflow scheduler &
<!--
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images/screen-casts etc. by drag-dropping the image here.
-->
**Anything else we need to know**:
This problem occurs when not dags are running for a while and getting The scheduler does not appear to be running error
Attached screenshot of the issue
If I checked the scheduler status with /heath API getting unhealthy for the scheduler
Attached airflow.cfg file for the reference
[airflow.txt](https://github.com/apache/airflow/files/6997420/airflow.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #17646: Scheduler is not working
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #17646:
URL: https://github.com/apache/airflow/issues/17646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17646: Scheduler is not working
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17646:
URL: https://github.com/apache/airflow/issues/17646#issuecomment-900229324
Yep. For me it looks like the scheduler could simply crash and you have not restarted it (or for whatever reason it stopped running because of some deadlock or similar). But we need to know logs. Also - what is the way you implemented automated restarting of scheduler if it crashed and monitoring? I think if you do not have automated restart nor monitoring implemented, you should do it as soon as you can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17646: Scheduler is not working
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17646:
URL: https://github.com/apache/airflow/issues/17646#issuecomment-900229324
Yep. For me it looks like the scheduler could simply crash and you have not restarted it (or for whatever reason it stopped running because of some deadlock or similar). But we need to see the logs. Also - what is the way you implemented automated restarting of scheduler if it crashed and monitoring? I think if you do not have automated restart nor monitoring implemented, you should do it as soon as you can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17646: Scheduler is not working
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17646:
URL: https://github.com/apache/airflow/issues/17646#issuecomment-900283451
It's up to you - but our suggested (and community sypported) way of deploying Airflow is through Kubernetes and Official Helm Chart https://airflow.apache.org/docs/helm-chart/stable/index.html. When it comes to "home-grown" way of doing it when you do not want or can afford learning K8S, I have no suggestions - it's whatever you are used to for anything else you already manage. Airflow has nothing "specific" that would make it difficult to integrate with any system, it's a standard tool as many others and you can adapt it to whatever deployment you have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vengi57 commented on issue #17646: Scheduler is not working
Posted by GitBox <gi...@apache.org>.
vengi57 commented on issue #17646:
URL: https://github.com/apache/airflow/issues/17646#issuecomment-900245875
For now, no automation is implemented for the webserver and scheduler when any issue happens
Can you please suggest to me a way to implement the auto-restart of the scheduler if any crash or deadlock occurs
Can I use pm2 for starting of scheduler and webserver will it works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17646: Scheduler is not working
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17646:
URL: https://github.com/apache/airflow/issues/17646#issuecomment-900131990
Can you include the scheduler log covering the period that this happened?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org