You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shorrocka (via GitHub)" <gi...@apache.org> on 2025/06/05 14:55:44 UTC

[D] Airflow Segmentation Faults after DagProcessor Crash [airflow]

GitHub user shorrocka created a discussion: Airflow Segmentation Faults after DagProcessor Crash

### Apache Airflow version

Other Airflow 2 version (please specify below)

### If "Other Airflow 2 version" selected, which one?

2.10.5

### What happened?

Periodically, about every other day our airflow scheduler will crash after a dag run with the following error:
`Process ForkProcess-35:
Traceback (most recent call last):
  File "/usr/lib64/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib64/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/data/apache-airflow/lib64/python3.12/site-packages/airflow/dag_processing/manager.py", line 247, in _run_processor_manager
    processor_manager.start()
  File "/data/apache-airflow/lib64/python3.12/site-packages/airflow/dag_processing/manager.py", line 489, in start
    return self._run_parsing_loop()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/apache-airflow/lib64/python3.12/site-packages/airflow/dag_processing/manager.py", line 616, in _run_parsing_loop
    self._collect_results_from_processor(processor)
  File "/data/apache-airflow/lib64/python3.12/site-packages/airflow/dag_processing/manager.py", line 1143, in _collect_results_from_processor
    if processor.result is not None:
       ^^^^^^^^^^^^^^^^
  File "/data/apache-airflow/lib64/python3.12/site-packages/airflow/dag_processing/processor.py", line 379, in result
    raise AirflowException("Tried to get the result before it's done!")
airflow.exceptions.AirflowException: Tried to get the result before it's done!`

This happens directly after seemingly normal scheduler events and a dag finishes ie. here are the preceding log events:

`[2025-06-05T10:01:04.202-0400] {scheduler_job_runner.py:813} INFO - TaskInstance Finished: dag_id=restore_study_schema, task_id=create_schema, run_id=manual__2025-06-05T14:00:39.494709+00:00, map_index=-1, run_start_date=2025-06-05 14:01:03.157949+00:00, run_end_date=2025-06-05 14:01:03.614029+00:00, run_duration=0.45608, state=up_for_retry, executor=LocalExecutor(parallelism=32), executor_state=success, try_number=3, max_tries=5, job_id=924122, pool=default_pool, queue=default, priority_weight=4, operator=MySQLExecuteQueryOperator, queued_dttm=2025-06-05 14:01:02.534776+00:00, queued_by_job_id=922565, pid=558903
[2025-06-05T10:01:04.202-0400] {scheduler_job_runner.py:813} INFO - TaskInstance Finished: dag_id=restore_study_schema, task_id=create_schema, run_id=manual__2025-06-05T14:00:39.495018+00:00, map_index=-1, run_start_date=2025-06-05 14:01:03.157949+00:00, run_end_date=2025-06-05 14:01:03.613053+00:00, run_duration=0.455104, state=up_for_retry, executor=LocalExecutor(parallelism=32), executor_state=success, try_number=3, max_tries=5, job_id=924124, pool=default_pool, queue=default, priority_weight=4, operator=MySQLExecuteQueryOperator, queued_dttm=2025-06-05 14:01:02.534776+00:00, queued_by_job_id=922565, pid=558902`

Then we get a whole bunch of the following:

`[2025-06-05T10:01:16.606-0400] {scheduler_job_runner.py:922} ERROR - Executor LocalExecutor(parallelism=32) reported that the task instance <TaskInstance: restore_study_schema.create_schema manual__2025-06-05T14:00:39.493053+00:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
[2025-06-05T10:01:16.621-0400] {taskinstance.py:3315} ERROR - Executor LocalExecutor(parallelism=32) reported that the task instance <TaskInstance: restore_study_schema.create_schema manual__2025-06-05T14:00:39.493053+00:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally`

This is then followed by repeating instances of the dagfileprocessormanager exiting restarting in a loop:

`[2025-06-05T10:02:06.106-0400] {manager.py:280} WARNING - DagFileProcessorManager (PID=560642) exited with exit code -11 - re-launching
[2025-06-05T10:02:06.110-0400] {manager.py:174} INFO - Launched DagFileProcessorManager with pid: 560675
[2025-06-05T10:02:06.118-0400] {settings.py:63} INFO - Configured default timezone UTC
[2025-06-05T10:02:09.262-0400] {manager.py:280} WARNING - DagFileProcessorManager (PID=560675) exited with exit code -11 - re-launching
[2025-06-05T10:02:09.266-0400] {manager.py:174} INFO - Launched DagFileProcessorManager with pid: 560715`

Before we finally get a 
Segmentation fault (core dumped)

I've tried to inspect the core but it's always cut off and doesn't give me any kind of actually useful information. 


### What you think should happen instead?

_No response_

### How to reproduce

We install airflow from py-pi using python 3.12 in a virtual environment. The scheduler and web-server have both been running in a tmux session or using nohup, either way the crash occurs. 

### Operating System

Red Hat Enterprise Linux 9.5 (Plow)

### Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.8.0
apache-airflow-providers-apache-spark==5.0.0
apache-airflow-providers-celery==3.10.0
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.5.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-docker==4.2.0
apache-airflow-providers-elasticsearch==6.2.0
apache-airflow-providers-fab==1.5.3
apache-airflow-providers-ftp==3.12.2
apache-airflow-providers-http==5.2.0
apache-airflow-providers-imap==3.8.2
apache-airflow-providers-postgres==6.1.0
apache-airflow-providers-sftp==5.1.0
apache-airflow-providers-smtp==2.0.0
apache-airflow-providers-snowflake==6.1.0
apache-airflow-providers-sqlite==4.0.0
apache-airflow-providers-ssh==4.0.0
apache-airflow-providers-trino==6.0.1

### Deployment

Virtualenv installation

### Deployment details

Python 3.12.5


### Anything else?

This occurs almost every other day when we are running with only a few dags. 
I have tried a whole manner of changing configuration settings increasing timeouts for zombie jobs, the timeout for dag parsing, increased postgres connections. 

I am really not sure what is the underlying cause and couldn't find another instance of this issue. 

Any help or guidance would be hugely appreciated. 

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)


GitHub link: https://github.com/apache/airflow/discussions/51451

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] Airflow Segmentation Faults after DagProcessor Crash [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow Segmentation Faults after DagProcessor Crash

SEGFAULT is almost always the result of some library you have locally that has bugs. Airlfow is Python-only so there is really no place where segfault could be generated from within. The thing is that you should find the core file (aprently core is dumped) and look at it with gdb or similar to find out what caused it - that's the nature of SEGFAULT. You can look for AI or some tutorial help on how to do it.

Also look at other discussions and issues here that mention SIGSEGV - you can see a number of workarounds for some typical libraries - and issues - especially on MacOS there are poorly written libraries that mix threading and forking and it might cause problem - there is something about PROXY. But other than that - you need to find out which native library is causing it.

Converted to a discussion

GitHub link: https://github.com/apache/airflow/discussions/51451#discussioncomment-13380983

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] Airflow Segmentation Faults after DagProcessor Crash [airflow]

Posted by "shorrocka (via GitHub)" <gi...@apache.org>.
GitHub user shorrocka added a comment to the discussion: Airflow Segmentation Faults after DagProcessor Crash

Thank you that gives a good place to start digging again from! 

GitHub link: https://github.com/apache/airflow/discussions/51451#discussioncomment-13383747

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org


Re: [D] Airflow Segmentation Faults after DagProcessor Crash [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: Airflow Segmentation Faults after DagProcessor Crash

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.


GitHub link: https://github.com/apache/airflow/discussions/51451#discussioncomment-13380935

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org