You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "krisgeus (via GitHub)" <gi...@apache.org> on 2025/04/24 07:17:30 UTC
[I] Dag processor gets SIGKILL signal and all DAGs are removed from UI [airflow]
krisgeus opened a new issue, #49689:
URL: https://github.com/apache/airflow/issues/49689
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Dag processor got killed. DAGs disappeared, The UI shows the dag-processor healthy still, Restarting the dag-processor service solves this but I do not think this will be the way to go.
### What you think should happen instead?
DAG processor should not be killed or recover correctly from this if this is correct behaviour
### How to reproduce
Use the provided docker compose with a couple of dag in the mounted dag directory. After a while (computer went to sleep sometimes) the dags disapear
### Operating System
Docker compose on OSX
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
Log extract:
```
[2025-04-23T15:21:19.334+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:21:24.409+0000] {manager.py:748} INFO -
================================================================================
DAG File Processing Stats
Bundle File Path PID Current Duration # DAGs # Errors Last Duration Last Run At
----------- --------------------------------- ----- ------------------ -------- ---------- --------------- -------------------
dags-folder 04_PythonOperator_get_pictures.py 1 0 0.74s 2025-04-23T15:20:55
dags-folder 03_BashOperator.py 1 0 0.74s 2025-04-23T15:20:55
dags-folder 02_download_rocket_launches.py 1 0 0.74s 2025-04-23T15:20:57
dags-folder 01_download_rocket_launches.py 1 0 0.76s 2025-04-23T15:20:54
dags-folder 05_download_rocket_launches.py 1 0 0.76s 2025-04-23T15:20:54
================================================================================
[2025-04-23T15:21:24.411+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:21:24.413+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:21:24.420+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:21:25.478+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:25.481+0000] {dag.py:2517} INFO - Setting next_dagrun for 01_download_rocket_launches to None, run_after=None
[2025-04-23T15:21:25.487+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:25.491+0000] {dag.py:2517} INFO - Setting next_dagrun for 05_download_rocket_launches to 2025-04-24 00:00:00+00:00, run_after=2025-04-24 00:00:00+00:00
[2025-04-23T15:21:26.558+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:26.561+0000] {dag.py:2517} INFO - Setting next_dagrun for 03_BashOperator to None, run_after=None
[2025-04-23T15:21:26.569+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:26.571+0000] {dag.py:2517} INFO - Setting next_dagrun for 04_PythonOperator_get_pictures to None, run_after=None
[2025-04-23T15:21:27.626+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:27.629+0000] {dag.py:2517} INFO - Setting next_dagrun for 02_download_rocket_launches to None, run_after=None
[2025-04-23T15:21:29.661+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:21:34.713+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:24:45.683+0000] {job.py:232} INFO - Heartbeat recovered after 186.91 seconds
[2025-04-23T15:24:45.689+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:24:45.690+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:24:45.696+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:24:45.721+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 794 started 186 ago killing it.
2025-04-23 15:24:45 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=794 signal_sent=SIGKILL
[2025-04-23T15:24:45.725+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 795 started 186 ago killing it.
2025-04-23 15:24:45 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=795 signal_sent=SIGKILL
[2025-04-23T15:24:45.763+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 796 started 186 ago killing it.
2025-04-23 15:24:45 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=796 signal_sent=SIGKILL
[2025-04-23T15:24:45.767+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 797 started 186 ago killing it.
2025-04-23 15:24:45 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=797 signal_sent=SIGKILL
[2025-04-23T15:24:45.796+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 798 started 186 ago killing it.
2025-04-23 15:24:45 [info ] Process exited [supervisor] exit_code=0 pid=798 signal_sent=SIGKILL
[2025-04-23T15:24:50.836+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:24:55.899+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:00.974+0000] {manager.py:748} INFO -
================================================================================
DAG File Processing Stats
Bundle File Path PID Current Duration # DAGs # Errors Last Duration Last Run At
----------- --------------------------------- ----- ------------------ -------- ---------- --------------- -------------------
dags-folder 04_PythonOperator_get_pictures.py 0 1 186.62s 2025-04-23T15:24:45
dags-folder 03_BashOperator.py 0 1 186.62s 2025-04-23T15:24:45
dags-folder 02_download_rocket_launches.py 0 1 186.62s 2025-04-23T15:24:45
dags-folder 01_download_rocket_launches.py 0 1 186.61s 2025-04-23T15:24:45
dags-folder 05_download_rocket_launches.py 0 1 186.60s 2025-04-23T15:24:45
================================================================================
[2025-04-23T15:25:00.983+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:06.048+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:11.141+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:12.155+0000] {manager.py:303} INFO - DAG 01_download_rocket_launches is missing and will be deactivated.
[2025-04-23T15:25:12.155+0000] {manager.py:303} INFO - DAG 05_download_rocket_launches is missing and will be deactivated.
[2025-04-23T15:25:12.156+0000] {manager.py:303} INFO - DAG 03_BashOperator is missing and will be deactivated.
[2025-04-23T15:25:12.156+0000] {manager.py:303} INFO - DAG 04_PythonOperator_get_pictures is missing and will be deactivated.
[2025-04-23T15:25:12.156+0000] {manager.py:303} INFO - DAG 02_download_rocket_launches is missing and will be deactivated.
[2025-04-23T15:25:12.160+0000] {manager.py:315} INFO - Deactivated 5 DAGs which are no longer present in file.
[2025-04-23T15:26:54.257+0000] {job.py:232} INFO - Heartbeat recovered after 103.12 seconds
[2025-04-23T15:26:54.277+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 807 started 285 ago killing it.
2025-04-23 15:26:54 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=807 signal_sent=SIGKILL
[2025-04-23T15:26:54.280+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 808 started 285 ago killing it.
2025-04-23 15:26:54 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=808 signal_sent=SIGKILL
[2025-04-23T15:26:54.314+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 809 started 285 ago killing it.
2025-04-23 15:26:54 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=809 signal_sent=SIGKILL
[2025-04-23T15:26:54.317+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 810 started 285 ago killing it.
2025-04-23 15:26:54 [info ] Process exited [supervisor] exit_code=0 pid=810 signal_sent=SIGKILL
[2025-04-23T15:26:54.346+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 811 started 285 ago killing it.
2025-04-23 15:26:54 [info ] Process exited [supervisor] exit_code=0 pid=811 signal_sent=SIGKILL
[2025-04-23T15:26:55.329+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:26:55.332+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:26:55.340+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:27:00.378+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:05.447+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:10.535+0000] {manager.py:748} INFO -
================================================================================
DAG File Processing Stats
Bundle File Path PID Current Duration # DAGs # Errors Last Duration Last Run At
----------- --------------------------------- ----- ------------------ -------- ---------- --------------- -------------------
dags-folder 04_PythonOperator_get_pictures.py 0 1 285.64s 2025-04-23T15:26:54
dags-folder 03_BashOperator.py 0 1 285.64s 2025-04-23T15:26:54
dags-folder 02_download_rocket_launches.py 0 1 285.65s 2025-04-23T15:26:54
dags-folder 01_download_rocket_launches.py 0 1 285.63s 2025-04-23T15:26:54
dags-folder 05_download_rocket_launches.py 0 1 285.63s 2025-04-23T15:26:54
================================================================================
[2025-04-23T15:27:10.538+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:15.621+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:20.697+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:24.791+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 820 started 285 ago killing it.
2025-04-23 15:27:24 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=820 signal_sent=SIGKILL
[2025-04-23T15:27:24.793+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 821 started 285 ago killing it.
2025-04-23 15:27:24 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=821 signal_sent=SIGKILL
[2025-04-23T15:27:24.831+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 822 started 285 ago killing it.
2025-04-23 15:27:24 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=822 signal_sent=SIGKILL
[2025-04-23T15:27:24.834+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 823 started 285 ago killing it.
2025-04-23 15:27:24 [info ] Process exited [supervisor] exit_code=0 pid=823 signal_sent=SIGKILL
[2025-04-23T15:27:24.863+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 824 started 285 ago killing it.
2025-04-23 15:27:24 [info ] Process exited [supervisor] exit_code=0 pid=824 signal_sent=SIGKILL
[2025-04-23T15:27:25.849+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:30.897+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:35.937+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:41.018+0000] {manager.py:748} INFO -
================================================================================
DAG File Processing Stats
Bundle File Path PID Current Duration # DAGs # Errors Last Duration Last Run At
----------- --------------------------------- ----- ------------------ -------- ---------- --------------- -------------------
dags-folder 04_PythonOperator_get_pictures.py 0 1 285.65s 2025-04-23T15:27:24
dags-folder 03_BashOperator.py 0 1 285.65s 2025-04-23T15:27:24
dags-folder 02_download_rocket_launches.py 0 1 285.64s 2025-04-23T15:27:24
dags-folder 01_download_rocket_launches.py 0 1 285.63s 2025-04-23T15:27:24
dags-folder 05_download_rocket_launches.py 0 1 285.63s 2025-04-23T15:27:24
================================================================================
[2025-04-23T15:27:41.020+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:46.093+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:51.184+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:55.373+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 833 started 285 ago killing it.
2025-04-23 15:27:55 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=833 signal_sent=SIGKILL
[2025-04-23T15:27:55.377+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 834 started 285 ago killing it.
2025-04-23 15:27:55 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=834 signal_sent=SIGKILL
[2025-04-23T15:27:55.417+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 835 started 285 ago killing it.
2025-04-23 15:27:55 [info ] Process exited [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=835 signal_sent=SIGKILL
[2025-04-23T15:27:55.420+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 836 started 285 ago killing it.
2025-04-23 15:27:55 [info ] Process exited [supervisor] exit_code=0 pid=836 signal_sent=SIGKILL
[2025-04-23T15:27:55.449+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 837 started 285 ago killing it.
2025-04-23 15:27:55 [info ] Process exited [supervisor] exit_code=0 pid=837 signal_sent=SIGKILL
[2025-04-23T15:27:56.433+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:27:56.435+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:27:56.444+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:28:01.501+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:28:06.571+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:28:11.648+0000] {manager.py:748} INFO -
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag processor gets SIGKILL signal and all DAGs are removed from UI [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed issue #49689: Dag processor gets SIGKILL signal and all DAGs are removed from UI
URL: https://github.com/apache/airflow/issues/49689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag processor gets SIGKILL signal and all DAGs are removed from UI [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #49689:
URL: https://github.com/apache/airflow/issues/49689#issuecomment-2837466465
@krisgeus I merged a fix for this, please try with the fix and let me know if it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag processor gets SIGKILL signal and all DAGs are removed from UI [airflow]
Posted by "AmeyKu1Karni (via GitHub)" <gi...@apache.org>.
AmeyKu1Karni commented on issue #49689:
URL: https://github.com/apache/airflow/issues/49689#issuecomment-2844170417
@amoghrajesh I am also using Airflow Version 3.0.0. Can you guide me what to do exactly. I always have to restart the Airflow UI to get the DAGs back after making some changes in the DAG python file. Nothing happens even if i refresh the Airflow UI page. It just says No DAGs found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag processor gets SIGKILL signal and all DAGs are removed from UI [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #49689:
URL: https://github.com/apache/airflow/issues/49689#issuecomment-2834557629
Marking as high priority as this is an issue related to user experience and from my analysis, it will mostly happen only in dev environments, in any case, worth fixing for 3.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dag processor gets SIGKILL signal and all DAGs are removed from UI [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #49689:
URL: https://github.com/apache/airflow/issues/49689#issuecomment-2830727292
Got a fix coming for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org