You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/05 12:59:42 UTC
[GitHub] [airflow] hterik opened a new issue, #28116: Present task-errors caused by scheduler in UI
hterik opened a new issue, #28116:
URL: https://github.com/apache/airflow/issues/28116
### Description
Some times tasks are marked as failed by the scheduler or other internal airflow components, with the reason being known by the scheduler, but not surfaced to users. Instead the task is just marked as failed, without any indication why in the Logs-tab nor under Task instance details. Only way to figure out is for admins to go through the scheduler-logs.
Example: `{scheduler_job.py:1526} ERROR - Detected zombie job: .....`, marks the task as failed.
Here the UI looks like this, with the logs-tab being only showing an empty attempt of fetching log files from worker:
If there is a need to hide potential scheduler-secrets from users, there should at least be some generic error message saying that the task was not run to completion because of internal scheduler error. So that users know if they should debug task-logic themselves, or involve admins in debugging the infrastructure.
Looking at `taskinstance.handle_failure`-function it seem like most the code for handling such errors is already very centralized. Is there a way to attach the error there onto the taskinstance itself, for use in the UI? Or is it possible to merge the relevant log streams to show in the Logs-tab?
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Present task-errors caused by scheduler in UI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #28116:
URL: https://github.com/apache/airflow/issues/28116#issuecomment-3986503051
@Ashish0253 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Present task-errors caused by scheduler in UI [airflow]
Posted by "Ashish0253 (via GitHub)" <gi...@apache.org>.
Ashish0253 commented on issue #28116:
URL: https://github.com/apache/airflow/issues/28116#issuecomment-3734479444
I would like to work on this issue. I have successfully reproduced the 'Zombie'/Heartbeat timeout scenario in a local Airflow 3.0 Breeze environment.
My findings: I've identified that when the SchedulerJobRunner detects a stale heartbeat (around line 2732), it logs the error to the console but does not persist this reason to the database. Consequently, the UI logs for the Task Instance remain empty or uninformative because the failure reason isn't captured in the metadata.
**Proposed Approach:**
Model Update: Add a state_reason (or similar) string column to the TaskInstance model to store system-detected failure reasons.
Scheduler Update: Modify the zombie/heartbeat reaper logic in scheduler_job_runner.py to populate this field when marking a TI as failed.
API/UI: Ensure the Internal API includes this field so the UI can display a 'System Error' banner or note to the user.
If I am approaching this from the wrong angle or if there's an existing mechanism for storing system-level failure reasons I should be using instead, please point me in the right direction!
Could you please assign this to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28116: Present task-errors caused by scheduler in UI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28116:
URL: https://github.com/apache/airflow/issues/28116#issuecomment-1338382467
this is not as straightforward as you think - you should take a look at airflow's architecture - scheduler logs are the logs from scheduler, not from task. They are stored on a different instance where UI has no access (UI only accesses task logs from the worker where workers are involved and not from scheduler. Scheduler is really a control plane for the tasks.
Implementing such a feature would lkely require somethign similar to "import errors" - where errors are stored in a DB and then displayed in the UI - but those errors would have to be stored in task instance model.
But yeah. might be an interesting feature. Maybe someone will pick it up - marked it as good first issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Present task-errors caused by scheduler in UI [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #28116:
URL: https://github.com/apache/airflow/issues/28116#issuecomment-3755442984
You can try to attempt it, and prepare a PR - and I guess comments will happen when draft PR implementing what you proposed will be ready to show. This touches core part of Airflow and if you are unsure how to do it or need a lot of handholding here, there might be not much guidance and time spent from maintainers you will have to figure out the PR mostly on your own (and ideally not rely solely on AI generated code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28116: Present task-errors caused by scheduler in UI
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28116:
URL: https://github.com/apache/airflow/issues/28116#issuecomment-1342540552
yeah this is also why cluster policy overrides don't appear in logs nor in task instance.
I think this is part of a greater challange that we need to figure out how to handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org