You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vidyapanchalZS (via GitHub)" <gi...@apache.org> on 2023/06/20 08:51:15 UTC
[GitHub] [airflow] vidyapanchalZS opened a new issue, #32020: Airflow tasks run -m cli command giving 504 response
vidyapanchalZS opened a new issue, #32020:
URL: https://github.com/apache/airflow/issues/32020
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Hello ,
we are facing "504 Gateway Time-out" error when running "tasks run -m" CLI command. We are trying to create Complex DAG and run tasks from cli command. When we are trying to run "tasks run -m" then we received gateway timeout error.
We also observed high resources spike in web server when we tried to execute this cli command. After looking into further, when we run "task run -m" Airflow CLI command, what it does is it parses the list of DAGs and the parses through the task list. Because this , we observed high resources of webserver and received gateway timeout error.
### What you think should happen instead
We are expecting that when to execute "tasks run" cli command, it should only parse DAG name and task name provided in the command and not parse DAG lists followed by task list.
### How to reproduce
please follow below steps to reproduce this issue.
1. we have 900 DAG in airflow environment.
2. we have created web login token to access web server.
3. after that we tried to run "tasks run" using python script
### Operating System
Amazon linux
### Versions of Apache Airflow Providers
Airflow version 2.2.2
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32020:
URL: https://github.com/apache/airflow/issues/32020#issuecomment-1600638990
@ephraimbuddy It was a mistake on my part. I didn't realize that it was assigned before creating the PR, as the issue had the `needs-triage` label.
My apologies @Lee-W for this oversight!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #32020: Airflow tasks run -m cli command giving 504 response
URL: https://github.com/apache/airflow/issues/32020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #32020:
URL: https://github.com/apache/airflow/issues/32020#issuecomment-1602167429
Ah, I forgot to remove the `needs-triage` tag 🤦♂️. @hussein-awala Should I assign it to you as you've already worked on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on issue #32020:
URL: https://github.com/apache/airflow/issues/32020#issuecomment-1599212873
👍 I agree that `tasks run xxx` shouldn't re-parse all the dags if the dag & task IDs are "known".
There is a tiny risk that we'd use out of date code by doing that, but it'd be the same as in the UI, when you click "run dag" it doesn't do a parsing pass on all the dags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #32020:
URL: https://github.com/apache/airflow/issues/32020#issuecomment-1598375373
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #32020:
URL: https://github.com/apache/airflow/issues/32020#issuecomment-1599647914
@vidyapanchalZS Personally, I use something similar to #32038, and it works flawlessly. Could you try it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #32020: Airflow tasks run -m cli command giving 504 response
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #32020:
URL: https://github.com/apache/airflow/issues/32020#issuecomment-1600626848
@hussein-awala , I think we should always inquire if the assignee is working on an issue before we start working on it. Saves everybody's time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org