You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/22 21:42:38 UTC
[GitHub] [airflow] jsnb-devoted opened a new issue #19757: Change the default for `dag_processor_manager_log_location`
jsnb-devoted opened a new issue #19757:
URL: https://github.com/apache/airflow/issues/19757
### Description
Should the default for the [dag_processor_manager_log_location](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-processor-manager-log-location) be `{BASE_LOG_FOLDER}/dag_processor_manager/dag_processor_manager.log` instead of `{AIRFLOW_HOME}/logs/dag_processor_manager/dag_processor_manager.log`?
### Use case/motivation
I'm running the k8s executor and we are changing our security profile on the pods such that the filesystem is readonly except for `/tmp`. I started out by changing `base_log_folder` and I spent a while trying to debug parts of my logging config that were still trying to write to `{AIRFLOW_HOME}/logs`
I found that the processor config was the issue because the default location was `{AIRFLOW_HOME}/logs/dag_processor_manager/dag_processor_manager.log` ([here](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-processor-manager-log-location))
Maybe it is fine as is but I found it hard to debug
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19757: Change the default for `dag_processor_manager_log_location`
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19757:
URL: https://github.com/apache/airflow/issues/19757#issuecomment-975979665
This is nontrivial because in it's general solution it will depend on the sequence of how the entties are specified and used and some cases are unsolvable.
I.e. what should happens if you specify `base_log_folder={DAG_PROCESSOR_MANAGER_LOG_LOCATION}` and `dag_processor_manger_log_location={BASE_LOG_FOLDER}` ?
How to avoid infiinite recursion?
The current approach is that we can specify ENV_VARIABLES in the config but we can't specify other config variables (which is what you basically proposed). As this could cuase infinite recursion.
Maybe a solution is to add some comments to the descriptions of the variables to mention that those configuration variables are independent from each other.
Maybe you'd lilke to make PR with such documentation change that will make future debugging much easier for other users? As a person who struggled with it, maybe you can find the riight wording?
It's basically updating this file: https://github.com/apache/airflow/blob/968952bb8d0b410e5cf013f3d0d074fc5783c6f1/airflow/config_templates/config.yml#L449 - and you can even open PR and edit it directly from the Github UI - just make a fork and edit the file and make a PR there.
Shall I assign you this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19757: Change the default for `dag_processor_manager_log_location`
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19757:
URL: https://github.com/apache/airflow/issues/19757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19757: Change the default for `dag_processor_manager_log_location`
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19757:
URL: https://github.com/apache/airflow/issues/19757#issuecomment-975979665
This is nontrivial because in it's general solution it will depend on the sequence of how the entties are specified and used and some cases are unsolvable.
I.e. what should happens if you specify `base_log_folder={DAG_PROCESSOR_MANAGER_LOG_LOCATION}` and `dag_processor_manger_log_location={BASE_LOG_FOLDER}` ?
How to avoid infiinite recursion?
The current approach is that we can specify ENV_VARIABLES in the config but we can't specify other config variables (which is what you basically proposed) as this could cuase infinite recursion.
Maybe a solution is to add some comments to the descriptions of the variables to mention that those configuration variables are independent from each other.
Maybe you'd lilke to make PR with such documentation change that will make future debugging much easier for other users? As a person who struggled with it, maybe you are the best one to find the right wording?
It's basically updating this file: https://github.com/apache/airflow/blob/968952bb8d0b410e5cf013f3d0d074fc5783c6f1/airflow/config_templates/config.yml#L449 - and you can even open PR and edit it directly from the Github UI - just make a fork and edit the file and make a PR there.
Shall I assign you this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jsnb-devoted commented on issue #19757: Change the default for `dag_processor_manager_log_location`
Posted by GitBox <gi...@apache.org>.
jsnb-devoted commented on issue #19757:
URL: https://github.com/apache/airflow/issues/19757#issuecomment-975983190
That totally checks out. Not sure how many others will run into the issue I had but I think a documentation change could have helped in my case.
I can take a stab at the wording -- assign it to me and I'll try to get to it tomorrow.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19757: Change the default for `dag_processor_manager_log_location`
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19757:
URL: https://github.com/apache/airflow/issues/19757#issuecomment-975979665
This is nontrivial because in it's general solution it will depend on the sequence of how the entties are specified and used and some cases are unsolvable.
I.e. what should happens if you specify `base_log_folder={DAG_PROCESSOR_MANAGER_LOG_LOCATION}` and `dag_processor_manger_log_location={BASE_LOG_FOLDER}` ?
How to avoid infiinite recursion?
The current approach is that we can specify ENV_VARIABLES in the config but we can't specify other config variables (which is what you basically proposed) as this could cuase infinite recursion.
Maybe a solution is to add some comments to the descriptions of the variables to mention that those configuration variables are independent from each other.
Maybe you'd lilke to make PR with such documentation change that will make future debugging much easier for other users? As a person who struggled with it, maybe you can find the riight wording?
It's basically updating this file: https://github.com/apache/airflow/blob/968952bb8d0b410e5cf013f3d0d074fc5783c6f1/airflow/config_templates/config.yml#L449 - and you can even open PR and edit it directly from the Github UI - just make a fork and edit the file and make a PR there.
Shall I assign you this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19757: Change the default for `dag_processor_manager_log_location`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19757:
URL: https://github.com/apache/airflow/issues/19757#issuecomment-975979665
This is nontrivial because in it's general solution it will depend on the sequence of how the entties are specified and used and some cases are unsolvable.
I.e. what should happens if you specify `base_log_folder={DAG_PROCESSOR_MANAGER_LOG_LOCATION}` and `dag_processor_manger_log_location={BASE_LOG_FOLDER}' ?
How to avoid infiinite recursion?
The current approach is that we can specify ENV_VARIABLES in the config but we can't specify other config variables (which is what you basically proposed). As this could cuase infinite recursion.
Maybe a solution is to add some comments to the descriptions of the variables to mention that those configuration variables are independent from each other.
Maybe you'd lilke to make PR with such documentation change that will make future debugging much easier for other users? As a person who struggled with it, maybe you can find the riight wording?
It's basically updating this file: https://github.com/apache/airflow/blob/968952bb8d0b410e5cf013f3d0d074fc5783c6f1/airflow/config_templates/config.yml#L449 - and you can even open PR and edit it directly from the Github UI - just make a fork and edit the file and make a PR there.
Shall I assign you this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org