You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/08 12:36:02 UTC
[GitHub] [airflow] maijh opened a new issue #12180: scheduler create many scheduler process
maijh opened a new issue #12180:
URL: https://github.com/apache/airflow/issues/12180
Apache Airflow version: v1109
executor: CeleryExecutor
some time the scheduler create many sub scheduler process. When this happens, the CPU usage of these sub schduler process is at 99%, and then i can see web ui appear:**The scheduler does not appear to be running. Last heartbeat was received xxx ago.The DAGs list may not update, and new tasks will not be scheduled.**
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
run_duration = -1
num_runs = -1
processor_poll_interval = 1
min_file_process_interval = 30
here is some log i mark,but the scheduler log had no error
etl 23905 12016 0 Oct17 ? 00:00:00 /bin/bash -c source /usr/soft/anaconda3/bin/activate etl_airflow_v1109 && airflow scheduler
etl 23908 23905 1 Oct17 ? 03:13:04 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 23947 23908 1 Oct17 ? 02:40:24 airflow scheduler -- DagFileProcessorManager
etl 30570 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30573 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30574 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30575 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30576 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30577 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30578 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30579 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30580 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30581 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
etl 30583 23908 99 03:07 ? 00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
Please help. I would be happy to provide any other information needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12180: scheduler create many scheduler process
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723571600
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maijh commented on issue #12180: scheduler create many scheduler process
Posted by GitBox <gi...@apache.org>.
maijh commented on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723806587
> Duplicate: #7935
https://github.com/apache/airflow/pull/11278/files
can the url fix my problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maijh edited a comment on issue #12180: scheduler create many scheduler process
Posted by GitBox <gi...@apache.org>.
maijh edited a comment on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723806587
> Duplicate: #7935
https://github.com/apache/airflow/pull/11278/files
can the url fix my problem? @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12180: scheduler create many scheduler process
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723603386
Duplicate: https://github.com/apache/airflow/issues/7935
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #12180: scheduler create many scheduler process
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #12180:
URL: https://github.com/apache/airflow/issues/12180
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org