You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "awesomescot (via GitHub)" <gi...@apache.org> on 2024/08/27 20:24:05 UTC
[I] standalone dag processor gets stuck when over 1k dag files [airflow]
awesomescot opened a new issue, #41806:
URL: https://github.com/apache/airflow/issues/41806
### Apache Airflow version
2.10.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When I have over about 1000 dag files the standalone processor seems to stop functioning properly. I see CPU drop to almost zero. Parsing processes is also around 0. The dagbag never fills up. Logs are unhelpful. I can't seem to figure out what the dag processor is doing, seems as though it's silently crashing.
### What you think should happen instead?
I think the standalone dag processor should process in the same or less time than the scheduler dag processor.
### How to reproduce
I'm not sure I can share our dag files, but I will post my values file and would love to see if others can reproduce.
### Operating System
kubernetes helm chart
### Versions of Apache Airflow Providers
The ones in the helm chart.
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
We are connecting to an RDS postgres instance(also very low cpu usage).
### Anything else?
I've been trying to play around with settings to see if I can figure out what is happening, but no luck so far. I'm happy to post any logs that would be helpful.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] standalone dag processor gets stuck when over 1k dag files [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #41806:
URL: https://github.com/apache/airflow/issues/41806#issuecomment-2313443344
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] standalone dag processor gets stuck when over 1k dag files [airflow]
Posted by "awesomescot (via GitHub)" <gi...@apache.org>.
awesomescot closed issue #41806: standalone dag processor gets stuck when over 1k dag files
URL: https://github.com/apache/airflow/issues/41806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] standalone dag processor gets stuck when over 1k dag files [airflow]
Posted by "seifrajhi (via GitHub)" <gi...@apache.org>.
seifrajhi commented on issue #41806:
URL: https://github.com/apache/airflow/issues/41806#issuecomment-2518739812
> Just in case anyone else stumbles on this - we hit a very similar issue, and worked around it by wrapping the dag processor command with `timeout`. So in our case, where we have a dag-processor per subdir:
>
> ```shell
> timeout --kill-after=10 600 airflow dag-processor --subdir $AIRFLOW_HOME/dags/$SUB_FOLDER -n 1
> ```
>
> (And then surrounding code/infrastructure will ensure that another dag-processor is spun up)
Hello @michalc, I am facing the same issue and I am trying to solve it..
using the helm chart, this is snippet of my values file, what should I change ?
```
dagProcessor:
enabled: true
replicas: 2
revisionHistoryLimit: 5
resources:
requests:
cpu: 2500m
ephemeral-storage: 200Mi
memory: 2500Mi
limits:
ephemeral-storage: 200Mi
memory: 2500Mi
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] standalone dag processor gets stuck when over 1k dag files [airflow]
Posted by "michalc (via GitHub)" <gi...@apache.org>.
michalc commented on issue #41806:
URL: https://github.com/apache/airflow/issues/41806#issuecomment-2379084737
Just in case anyone else stumbles on this - we hit a very similar issue, and worked around it by wrapping the dag processor command with `timeout`. So in our case, where we have a dag-processor per subdir:
```bash
timeout --kill-after=10 600 airflow dag-processor --subdir $AIRFLOW_HOME/dags/$SUB_FOLDER -n 1
```
(And then surrounding code/infrastructure will ensure that another dag-processor is spun up)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] standalone dag processor gets stuck when over 1k dag files [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #41806:
URL: https://github.com/apache/airflow/issues/41806#issuecomment-2321038804
I think the standalone DAG processor should be the same like integrated.
Can you please check: Is it running stable when DAG processing is integrated and not separated?
Is the failure happening on first run already, so never completing? Or is it a in-stability that sometimes hot your environment?
Can you bi-sect and in increments cut the amount of DAG files by half? Is there a specific expensive DAG file which takes long to parse? Or can you create an artificial file set which makes it re-producible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] standalone dag processor gets stuck when over 1k dag files [airflow]
Posted by "awesomescot (via GitHub)" <gi...@apache.org>.
awesomescot commented on issue #41806:
URL: https://github.com/apache/airflow/issues/41806#issuecomment-2321564386
Thanks for the reply. I have been testing it on the scheduler and it seems to also be struggling, so this should probably be a discussion and not an issue. I'll raise something over there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org