You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "omkar-foss (via GitHub)" <gi...@apache.org> on 2024/10/18 20:41:03 UTC
[I] Increase Airflow logging coverage, review existing logs and add where missing [airflow]
omkar-foss opened a new issue, #43180:
URL: https://github.com/apache/airflow/issues/43180
### Description
As per users' feedback in the Airflow Debugging Survey 2024, about 42% of respondents mentioned that logs are missing or non-existent in some cases.
### Use case/motivation
Goals for this issue is to review the existing Airflow logs coverage and enhance it by adding logs wherever appropriately required.
### Related issues
Parent Issue: https://github.com/apache/airflow/issues/40975
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-3538812730
I noticed that when a task is configured to use a pool that doesn't exist - tasks get stuck indefinitely with no indication of why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-2443547804
A good example of an issue with logging is when (fatal) errors from workers are not being forwarded to the scheduler, and it's very hard to trace what's actually wrong. In cases when a worker is exhibiting an early enough error (being unable to load the full Airflow machinery that records logs locally and forwards them to the scheduler/webserver), displaying a stack trace captured on the worker would help immensely.
Ref: https://github.com/apache/airflow/issues/42136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-3431431945
One thing I came across recently:
When we have code such as below in an operator, and unless one adds an explicit logging call, the task ends up being skipped silently with no indication of the reason (not even a "`AirflowSkipException` encountered - skipping"). I would expect a textual rendering of the "upstream" (custom) exception at the very least.
```python
raise AirflowSkipException from CustomExceptionWithParameter(param1)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-3399226221
I guess that one is quite substantial and it requires quite an extensive knowledge of various components and airflow and logging approach. @KoviAnusha -> If that's one of your first tasks, then I would not encourage this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-3399474644
@potiuk, yes, this will be my first task. As suggested, I will take up other tasks in parallel improve my knowledge to take up such tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "KoviAnusha (via GitHub)" <gi...@apache.org>.
KoviAnusha commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-3392995938
@potiuk @amoghrajesh @kaxil, can I work on this task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Increase Airflow logging coverage, review existing logs and add missing essential logs [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43180:
URL: https://github.com/apache/airflow/issues/43180#issuecomment-3538842937
> I noticed that when a task is configured to use a pool that doesn't exist - tasks get stuck indefinitely with no indication of why.
I don't think we even have place for logs for that - because essentially what happens is scheduler runs a join with `pool` table and this is how it selects tasks to run - so if you want to fix this one @Dev-iL , this is a new feature to add rather than logging - figure out a place somewhere where pool_id for tasks is checked against pool table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org