You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/08 12:36:02 UTC

[GitHub] [airflow] maijh opened a new issue #12180: scheduler create many scheduler process

maijh opened a new issue #12180:
URL: https://github.com/apache/airflow/issues/12180


   Apache Airflow version: v1109
   executor: CeleryExecutor
   
   some time the scheduler create many sub scheduler process. When this happens, the CPU usage of these sub schduler process is at 99%, and then i can see web ui appear:**The scheduler does not appear to be running. Last heartbeat was received xxx ago.The DAGs list may not update, and new tasks will not be scheduled.**
   
   job_heartbeat_sec = 5
   scheduler_heartbeat_sec = 5
   run_duration = -1
   num_runs = -1
   processor_poll_interval = 1
   min_file_process_interval = 30
   
   here is some log i mark,but  the scheduler log had no error
   
   etl   23905 12016  0 Oct17 ?        00:00:00 /bin/bash -c source /usr/soft/anaconda3/bin/activate etl_airflow_v1109 && airflow scheduler
   etl   23908 23905  1 Oct17 ?        03:13:04 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   23947 23908  1 Oct17 ?        02:40:24 airflow scheduler -- DagFileProcessorManager
   etl   30570 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30573 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30574 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30575 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30576 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30577 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30578 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30579 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30580 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30581 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   etl   30583 23908 99 03:07 ?        00:07:22 /root/etl_airflow_v1109/bin/python /root/etl_airflow_v1109/bin/airflow scheduler
   
   
   Please help. I would be happy to provide any other information needed


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #12180: scheduler create many scheduler process

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723571600


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] maijh commented on issue #12180: scheduler create many scheduler process

Posted by GitBox <gi...@apache.org>.
maijh commented on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723806587


   > Duplicate: #7935
   
   https://github.com/apache/airflow/pull/11278/files
   can the url  fix my problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] maijh edited a comment on issue #12180: scheduler create many scheduler process

Posted by GitBox <gi...@apache.org>.
maijh edited a comment on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723806587


   > Duplicate: #7935
   
   https://github.com/apache/airflow/pull/11278/files
   can the url  fix my problem?  @mik-laj 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #12180: scheduler create many scheduler process

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12180:
URL: https://github.com/apache/airflow/issues/12180#issuecomment-723603386


   Duplicate: https://github.com/apache/airflow/issues/7935


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj closed issue #12180: scheduler create many scheduler process

Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #12180:
URL: https://github.com/apache/airflow/issues/12180


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org