You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/30 06:49:44 UTC
[GitHub] [airflow] thesuperzapper opened a new issue #22614: if deleted, the `dag_processor_manager.log` file is never recreated
thesuperzapper opened a new issue #22614:
URL: https://github.com/apache/airflow/issues/22614
### Apache Airflow version
2.2.4 (latest released)
### What happened
If a user deletes the `dag_processor_manager.log` file (stored at `AIRFLOW__CORE__DAG_PROCESSOR_MANAGER_LOG_LOCATION`), then the file is only re-created if the scheduler is restarted, which means that all logs until the scheduler next restarts are lost.
It probably has something to do with our logging configuration, which can be found at [`airflow/config_templates/airflow_local_settings.py#L131-L149`](https://github.com/apache/airflow/blob/c58b5e583bcad83048fdee72bc9ddf6c3a7b4229/airflow/config_templates/airflow_local_settings.py#L131-L149).
### What you think should happen instead
_No response_
### How to reproduce
_No response_
### Operating System
N/A
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22614: if deleted, the `dag_processor_manager.log` file is never recreated
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22614:
URL: https://github.com/apache/airflow/issues/22614#issuecomment-1082838303
this is expected behaviour on POSIX systems. When Rotating file Handler is used, the file it writes to should not be removed because that removes only link to the file from teh directory but the actual file is not removed until there is any process writing to it (and all processes that opened the file for wrting are unaware that the file has been removed from a folder). The file can be hard-linked somewhere else and in this case it wont even be deleted.
The solution is to only remove the "rotated" files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #22614: if deleted, the `dag_processor_manager.log` file is never recreated
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #22614:
URL: https://github.com/apache/airflow/issues/22614#issuecomment-1082856409
BTW. It would be possible to write a handler to "open" and "close" the file for every log line (that would not have that "property")- but this would be TERRIBLY slow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22614: if deleted, the `dag_processor_manager.log` file is never recreated
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22614:
URL: https://github.com/apache/airflow/issues/22614#issuecomment-1082856409
BTW. It would be possible to write a handler to "open" and "close" the file for every log - but this would be TERRIBLY slow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #22614: if deleted, the `dag_processor_manager.log` file is never recreated
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #22614:
URL: https://github.com/apache/airflow/issues/22614#issuecomment-1082838303
this is expected behaviour on POSIX systems. When Rotating file Handler is used, the file it writes to should not be removed because that removes only link to the file from the directory but the actual file is not removed until there is any process writing to it (and all processes that opened the file for wrting are unaware that the file has been removed from a folder). The file can be hard-linked somewhere else and in this case it wont even be deleted.
The solution is to only remove the "rotated" files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #22614: if deleted, the `dag_processor_manager.log` file is never recreated
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #22614:
URL: https://github.com/apache/airflow/issues/22614
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org