You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/13 07:29:26 UTC
[GitHub] [airflow] sun2everyone opened a new issue, #28326: Viewing logs of running task in grid view hangs airflow-webserver
sun2everyone opened a new issue, #28326:
URL: https://github.com/apache/airflow/issues/28326
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
**Airflow version:** Airflow 2.4.2 with Elasticsearch log provider (ELK 7.17.0)
**Python version:** 3.8.13
**Short description:**
Airflow webserver stops responding due to infinity log loading loop.
**Details:**
If several users open grid view page and try to view logs of running task, they see no logs, but gunicorn workers became busy processing log loading requests. After some time all gunicorn workers became busy waiting for the end of log, and webserver stops responding to further client requests.
**What you can see in webserver logs:**
constantly appearing requests to elastic
```
{base.py:270} INFO - POST http://127.0.0.1:9202/_count [status:200 request:0.006s]
{base.py:270} INFO - POST http://127.0.0.1:9202/_search [status:200 request:0.015s]
{base.py:270} INFO - POST http://127.0.0.1:9202/_search [status:200 request:0.015s]
{base.py:270} INFO - POST http://127.0.0.1:9202/_count [status:200 request:0.010s]
{base.py:270} INFO - POST http://127.0.0.1:9202/_search [status:200 request:0.014s]
{base.py:270} INFO - POST http://127.0.0.1:9202/_count [status:200 request:0.011s]
```
### What you think should happen instead
When trying to view logs of running task in grid view, you should receive one of the following:
1. Message that running task logs couldn't be shown
2. See partially loaded logs, that exist at the moment you load the page
3. See continuously appearing logs, as it is done on the main task log page
### How to reproduce
1. Start airflow webserver with ELK logging and sync gunicorn workers
2. Create dag with a task that is running for a long time and constantly writing it's logs
3. Run that dag
4. Open that dag in a grid view, try to view logs of that running task in a grid view
5. Open more pages viewing logs of that running task in a grid view (5-6 or more)
6. Webserver starts to work slowly, skips healthchecks and does not respond client requests
### Operating System
Ubuntu 20.04.2 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-elasticsearch 4.2.1
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
I guess the problem is in this parts:
- https://github.com/apache/airflow/blob/febf35500d5de172e25280e5f5492257f898fdf5/airflow/api_connexion/endpoints/log_endpoint.py#L111
- https://github.com/apache/airflow/blob/fa2bec042995004f45b914dd1d66b466ccced410/airflow/utils/log/log_reader.py#L80
Log reading stream keeps gunicorn worker busy all the time the task is in the Running state.
Using async workers like tornado makes a little better, but doesn't solve the problem. For now, I use this workaround:
```
--- log_endpoint_py.default 2022-12-12 19:16:30.280526903 +0300
+++ log_endpoint.py 2022-12-12 19:03:29.128970011 +0300
@@ -33,7 +33,7 @@
from airflow.utils.airflow_flask_app import get_airflow_app
from airflow.utils.log.log_reader import TaskLogReader
from airflow.utils.session import NEW_SESSION, provide_session
-
+from airflow.utils.state import State
@security.requires_access(
[
@@ -108,6 +108,8 @@
token = URLSafeSerializer(key).dumps(metadata) # type: ignore[assignment]
return logs_schema.dump(LogResponseObject(continuation_token=token, content=logs))
# text/plain. Stream
- logs = task_log_reader.read_log_stream(ti, task_try_number, metadata)
-
+ if ti.state not in State.running:
+ logs = task_log_reader.read_log_stream(ti, task_try_number, metadata)
+ else:
+ logs = {"Task is still running, logs in grid view not available."}
return Response(logs, headers={"Content-Type": return_type})
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2020745269
@rightx2 happy to reopen it, but it would help if you could provide more information that can help someone who might look at the issue (or maybe even look at it yourself and attempt to make PR to fix it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "sun2everyone (via GitHub)" <gi...@apache.org>.
sun2everyone commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2022366729
Hello. I think ideal solution to this issue is to rewrite code for displaying logs on grid view page as it is done on logs page (continious batch loading). This is more simple solution I implemented (Airflow 2.4.2):
```
diff --git a/lib/python3.8/site-packages/airflow/api_connexion/endpoints/log_endpoint.py b/lib/python3.8/site-packages/airflow/api_connexion/endpoints/log_endpoint.py
index bda0af3..364a48f 100644
--- a/lib/python3.8/site-packages/airflow/api_connexion/endpoints/log_endpoint.py
+++ b/lib/python3.8/site-packages/airflow/api_connexion/endpoints/log_endpoint.py
@@ -33,7 +33,7 @@
from airflow.utils.airflow_flask_app import get_airflow_app
from airflow.utils.log.log_reader import TaskLogReader
from airflow.utils.session import NEW_SESSION, provide_session
-
+from airflow.utils.state import State
@security.requires_access(
[
@@ -108,6 +108,8 @@
token = URLSafeSerializer(key).dumps(metadata) # type: ignore[assignment]
return logs_schema.dump(LogResponseObject(continuation_token=token, content=logs))
# text/plain. Stream
- logs = task_log_reader.read_log_stream(ti, task_try_number, metadata)
-
+ if ti.state not in State.running:
+ logs = task_log_reader.read_log_stream(ti, task_try_number, metadata)
+ else:
+ logs = {"Task is still running, logs in grid view not available."}
return Response(logs, headers={"Content-Type": return_type})
```
It just disables displaying logs in grid view if task state is 'Running'. Works in production for more than a year, without any problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-1931404656
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28326: Viewing logs of running task in grid view hangs airflow-webserver
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-1347858933
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2021707638
Simply speaking - the more likely and easier you make it for somoene to diagnose and fix it, the more likely it will be diagnoe and fix it. So if you want it to get it fixed, spending time on making it easier (which mostly means gathering evidences, spotting the patterns, and doing more diagnosis on your side) is a good way to increase chances for it to happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "rightx2 (via GitHub)" <gi...@apache.org>.
rightx2 commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2019856231
I have same problem, exactly. This issue should not be as stale.. @sun2everyone Did you solve this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28326: Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch]
URL: https://github.com/apache/airflow/issues/28326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "rightx2 (via GitHub)" <gi...@apache.org>.
rightx2 commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2021707640
Sounds reasonable. I'll do some experiments to reproduce them a lot easier and update it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-1879976301
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "rightx2 (via GitHub)" <gi...@apache.org>.
rightx2 commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2021670409
@potiuk I think what sun2everyone wrote is enough to describe the problem... Do you want think it needs some other info ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Viewing logs of running task in grid view hangs airflow-webserver [elasticsearch] [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #28326:
URL: https://github.com/apache/airflow/issues/28326#issuecomment-2021705124
> @potiuk I think what sun2everyone wrote is enough to describe the problem... Do you want think it needs some other info ?
No idea. But by providing more info, specific to your case you increase chances that someone will have easier time of diagnosing it and maybe fixing. This is Open-source project. Maybe someone will take a look at the issue, maybe not. Maybe they will decide to spend their time on trying to reproduce it, diagnose and fix, or maybe not. Maybe if you gather some evidences you will find more informaation and show differences vs what the original report that will make it easier to diagnose. Or maybe by gathering evidences you will find it yourself, or maybe you will find that this is a different issue. Finally - maybe no-one will look at it. Seems like no-one was looking at it and diagnosing it since December 2022.
All these are maybes. All at scale of proability. And by providing more information you likely move the probability to be in favour of problem being diagnosed and fixed.
That's all I am saying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org