You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/02 09:27:52 UTC
[GitHub] [airflow] xelita opened a new issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
xelita opened a new issue #19962:
URL: https://github.com/apache/airflow/issues/19962
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Ubuntu 18.04.4 LTS
### Versions of Apache Airflow Providers
Providers info
apache-airflow-providers-amazon | 2.4.0
apache-airflow-providers-celery | 2.1.0
apache-airflow-providers-cncf-kubernetes | 2.1.0
apache-airflow-providers-docker | 2.3.0
apache-airflow-providers-elasticsearch | 2.1.0
apache-airflow-providers-ftp | 2.0.1
apache-airflow-providers-google | 6.1.0
apache-airflow-providers-grpc | 2.0.1
apache-airflow-providers-hashicorp | 2.1.1
apache-airflow-providers-http | 2.0.1
apache-airflow-providers-imap | 2.0.1
apache-airflow-providers-microsoft-azure | 3.3.0
apache-airflow-providers-mysql | 2.1.1
apache-airflow-providers-odbc | 2.0.1
apache-airflow-providers-postgres | 2.3.0
apache-airflow-providers-redis | 2.0.1
apache-airflow-providers-sendgrid | 2.0.1
apache-airflow-providers-sftp | 2.2.0
apache-airflow-providers-slack | 4.1.0
apache-airflow-providers-sqlite | 2.0.1
apache-airflow-providers-ssh | 2.3.0
### Deployment
Docker-Compose
### Deployment details
### structure of my local directory:
<pre>
.
├── airflow.sh
├── config
│ └── log_config.py
├── dags
│ └── test.py
├── docker-compose.yaml
├── logs
├── Makefile
├── parameters.json
├── plugins
└── test.py
</pre>
### docker-compose.yml
only changes I have made in common env section:
- added PYTONPATH env variable
- mounting config folder were my log customization code lives
<pre>
environment:
&airflow-common-env
PYTHONPATH: /opt/airflow/config
AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS: 'log_config.LOGGING_CONFIG'
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'false'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- ./config:/opt/airflow/config
user: "${AIRFLOW_UID:-50000}:0"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
</pre>
log_config.py (nothing special for now):
<pre>
from copy import deepcopy
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
</pre>
### What happened
When executing `docker-compose up` command, the program exists with the following errors:
<pre>
airflow_postgres_1 is up-to-date
airflow_redis_1 is up-to-date
Recreating airflow_airflow-init_1 ... done
ERROR: for airflow-scheduler Container "2b3fbb5a9e97" exited with code 1.
ERROR: for airflow-webserver Container "2b3fbb5a9e97" exited with code 1.
ERROR: for airflow-worker Container "2b3fbb5a9e97" exited with code 1.
ERROR: for flower Container "2b3fbb5a9e97" exited with code 1.
ERROR: for airflow-triggerer Container "2b3fbb5a9e97" exited with code 1.
ERROR: Encountered errors while bringing up the project.
</pre>
when looking at the error using docker logs command:
<pre>
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 41, in configure_logging
logging_config = import_string(logging_class_path)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'log_config'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 483, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 50, in configure_logging
raise ImportError(f'Unable to load custom logging from {logging_class_path} due to {err}')
ImportError: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
</pre>
### What you expected to happen
Being able to use my custom logger definition.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985566244
Then this is a bug if you can provide reproducible case. Please report it . The othr thing is not a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985011243
> same effect. Also in the initial docker file, dags, logs and plugins folders are all mounted into the container. https://airflow.apache.org/docs/apache-airflow/2.2.2/docker-compose.yaml
>
> IMHO we need to mount the config folder as well to be able to make the logging customization file available inside the container
If you want - yes. You should. Note that the docker-compose quick-start is just that, quick start to get up and running quickly. And if you want to customize it you are on your own there and you should modify the docker compose as you see fit:
https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html
> DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it.
So all good. Just modify your docker compose in the way that suits you. No issue for Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1023178110
> thanks for the replies. I tried it with Helm too, and it's working. Some strange thing is going on with compose and mounted file/folder permissions, but currently I don't have time to figure out the exact problem and solution.
Yeash this is **precisely** what I meant by adding this as a warning to our docuementation:
> DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985319737
Of course. You should use "standalone" command of airflow. rather than docker-compose.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#standalone
You are trying to shoot yoursefl in the foot by complicating your setup (and drag busy maintainer in it).
I think also it's good to read some advices:
First of all, you are on your own there (so any custom configuration there is definitely not a bug). Discussions are much better for that. You were advice to do that when you opened a "bug".
Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
```
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
```
I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985319737
Of course. You should use "standalone" command of airflow. rather than docker-compose.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#standalone
You are trying to shoot yoursefl in the foot by complicating your setup (and drag busy maintainer in it).
I think also it's good to read some advices:
First of all, you are on your own there (so any custom configuration there is definitely not a bug). Discussions are muh better for that. You were advice to do that when you opened a "bug".
Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
```
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
```
I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1021417097
> Summarizing: following the steps in the documentation leads to an error in the container startup.
@chriske - could you please open a new [discussion](https://github.com/apache/airflow/discussions) and explain all the details of errors, problems and how you tried to remedy it. Ideally a `gist` with logs and details of error message you had, and goal that you tried to achieve. There are many things you should check and describe:
* What is your configuration you applied?
* How did you check that the configuration is typo-free?
* How did you check that airflow properly picked it up?
* How did you expect it to be in your docker compose?
* How does it look when you exec in your docker container?
* What errors do you get?
* What did you expect and what did you get instead ?
Ideally showing the exact code and lists what you see in the container, including what you expected vs. what you see. What did you try to do to fix it?
There are many ways how custom logging might/might not work and many ways how you can **think** your docker compose setup is right, where it is not - you really need to understand very well all the nuances of how docker compose works and how docker containers works inside - this is a knowledge outside of airflow, it's pretty generic `docker-compose` and `container` knowledge and experience, and in many cases users don't really understand some of the consequences. This might not be your case of course but from the very "terse" description no-one is able to tell if this is your configuration, or airlfow instructions, or unexpected docker-compose behaviour and consequence, or misuderstanding or bad assumptions on either side.
Posting "same error" and "following the guide" is not nearly good enough to get any help with your problem. But if you describe your problem in detail in discussion, there is a chance that someone will help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1022668548
> Hi there,
>
> I'm facing the same issue as @xelita . I want to implement custom logging based on the official documentation https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
>
> Sure, for development it would be easier to test this locally without containers, but I will use the new logging configuration in a Kubernetes cluster and container based setup, which means the compose stack is just perfect to test it. I didn't modified the compose file, just added the config folder, and 2 files, based on the documentation, and setted a variable. This is not a change, it is an extension. So when we're using the same containers anywhere (docker, compose, Kubernetes), this issue will happen. I don't see why this is a compose related problem.
>
> Summarizing: following the steps in the documentation leads to an error in the container startup. @xelita did you found a solution?
No I have followed @potiuk recommendation and we are now using the official HELM chart to deploy Airflow which actually works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chriske commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
chriske commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1021179805
Hi there,
I'm facing the same issue as @xelita .
I want to implement custom logging based on the official documentation https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
Sure, for development it would be easier to test this locally without containers, but I will use the new logging configuration in a Kubernetes cluster and container based setup, which means the compose stack is just perfect to test it. I didn't modified the compose file, just added the config folder, and 2 files, based on the documentation, and setted a variable. This is not a change, it is an extension. So when we're using the same containers anywhere (docker, compose, Kubernetes), this issue will happen. I don't see why this is a compose related problem.
Summarizing: following the steps in the documentation leads to an error in the container startup.
@xelita did you found a solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984656869
Thanks @ephraimbuddy for your quick answer but it fails in the same way. The error message is slighly different:
<pre>
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 41, in configure_logging
logging_config = import_string(logging_class_path)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'config'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 483, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 50, in configure_logging
raise ImportError(f'Unable to load custom logging from {logging_class_path} due to {err}')
ImportError: Unable to load custom logging from config.log_config.LOGGING_CONFIG due to No module named 'config'
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
</pre>
common-env entry in my docker-compose:
<pre>
&airflow-common-env
#PYTHONPATH: /opt/airflow/config
#AIRFLOW__LOGGING__REMOTE_LOGGING: 'true'
AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS: 'config.log_config.LOGGING_CONFIG'
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'false'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- ./config:/opt/airflow/config
user: "${AIRFLOW_UID:-50000}:0"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984656869
Thanks @ephraimbuddy for your quick answer but it fails in the same way. The error message is a bit different:
<pre>
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 41, in configure_logging
logging_config = import_string(logging_class_path)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'config'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 483, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 50, in configure_logging
raise ImportError(f'Unable to load custom logging from {logging_class_path} due to {err}')
ImportError: Unable to load custom logging from config.log_config.LOGGING_CONFIG due to No module named 'config'
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985319737
Of cours. You should use "standalone" command of airflow. rather than docker-compose.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#standalone
You are trying to shoot yoursefl in the foot by complicating your setup (and drag busy maintainer in it).
I think also it's good to read some advices:
First of all, you are on your own there (so any custom configuration there is definitely not a bug). Discussions are muh better for that.
Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
```
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
```
I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984446682
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984759014
same effect. Also in the initial docker file, dags, logs and plugins folders are all mounted into the container.
https://airflow.apache.org/docs/apache-airflow/2.2.2/docker-compose.yaml
IMHO we need to mount the config folder as well to be able to load the logging customization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19962:
URL: https://github.com/apache/airflow/issues/19962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985569076
But please double check first if it's s not caused by some of the modifications of the compose file. The compose file is this - quick start. So if the error reportes cannot be reproduced using the standard docker compose and standard image - it means some of your modifications could have caused it. For me it is suspicious and i believe the error is not reported when you use it out of the box, so it is also possible that ypu have a bug in your.modified compose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chriske commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
chriske commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1023153889
Hi,
thanks for the replies. I tried it with Helm too, and it's working. Some strange thing is going on with compose and mounted file/folder permissions, but currently I don't have time to figure out the exact problem and solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985328420
> Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
> I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
Did you notice the Airflow version I am using by reading this ticket entirely?
<pre>
Apache Airflow
version | 2.2.2
executor | CeleryExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | postgresql+psycopg2://airflow:airflow@postgres/airflow
dags_folder | /opt/airflow/dags
plugins_folder | /opt/airflow/plugins
base_log_folder | /opt/airflow/logs
remote_base_log_folder |
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985569076
But please double check first if it's s not caused by so e.othe t modifications of the compose file. The compose file is this - quick start. So if the error reportes cannot be reproduced using the standard docker compose and standard image - it means some of your modifications could have caused it. For me it is suspicious and i believe the error is not reported when you use it out of the box, so it is also possible that ypu have a bug in your.modified compose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985319737
Of cours. You should use "standalone" command of airflow. rather than docker-compose.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#standalone
You are trying to shoot yoursefl in the foot by complicating your setup.
I think also it's good to read some advices:
First of all, you are on your own there (so any custom configuration there is definitely not a bug). Discussions are muh better for that.
Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
```
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
```
I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk removed a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk removed a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1023178248
> thanks for the replies. I tried it with Helm too, and it's working. Some strange thing is going on with compose and mounted file/folder permissions, but currently I don't have time to figure out the exact problem and solution.
Yeash this is **precisely** what I ment by adding this as a warning to our docuementation:
> DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984676704
Not sure why you are having errors. You can remove the config part and try again. config is already in the airflow path see https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#built-in-pythonpath-entries-in-airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984759014
same effect. Also in the initial docker file, dags, logs and plugins folders are all mounted into the container.
https://airflow.apache.org/docs/apache-airflow/2.2.2/docker-compose.yaml
IMHO we need to mount the config folder as well to be able to make the logging customization file available inside the container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985284705
@potiuk agree with you about not using the docker-compose for production deployment. However I just want to test a custom logger implementation locally with docker.compose.
In this case, even if I mount the config folder with the appropriate logging file, the stack fails saying it could not find the logger module.
Also what is your recommendation regarding Airflow local development (testing custom plugins, ...). I thought docker-compose was suitable for this kind of usecase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1021417097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1021417097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chriske commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
chriske commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1021179805
Hi there,
I'm facing the same issue as @xelita .
I want to implement custom logging based on the official documentation https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
Sure, for development it would be easier to test this locally without containers, but I will use the new logging configuration in a Kubernetes cluster and container based setup, which means the compose stack is just perfect to test it. I didn't modified the compose file, just added the config folder, and 2 files, based on the documentation, and setted a variable. This is not a change, it is an extension. So when we're using the same containers anywhere (docker, compose, Kubernetes), this issue will happen. I don't see why this is a compose related problem.
Summarizing: following the steps in the documentation leads to an error in the container startup.
@xelita did you found a solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1023178110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-984506588
Remove the PYTHONPATH and set logging class this way: `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=config.log_config.LOGGING_CONFIG` let me know if it fails after that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita commented on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985674124
Ok will check that. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985319737
Of cours. You should use "standalone" command of airflow. rather than docker-compose.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#standalone
You are trying to shoot yoursefl in the foot by complicating your setup (and drag busy maintainer in it).
I think also it's good to read some advices:
First of all, you are on your own there (so any custom configuration there is definitely not a bug). Discussions are muh better for that. You were advice to do that when you opened a "bug".
Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
```
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
```
I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xelita edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
xelita edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-985328420
> Secondly - you shoud also listen to some helpful messages printed. Your problem with docker compose might for example come from this very straightforward error that you have:
ERROR!!!: Too old Airflow version !
The minimum Airflow version supported: 2.2.0. Only use this or higher!
> I do not know if that's the problem, but if the compose tells you that you use wrong version, I think you should think twice before opening an issue classifed as "bug".
Did you notice the Airflow version I am using by reading this ticket entirely? The first thing you ask when creating a ticket is the Airflow version we are running. You should read twice before replying.
For your information
<pre>
Apache Airflow
version | 2.2.2
executor | CeleryExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | postgresql+psycopg2://airflow:airflow@postgres/airflow
dags_folder | /opt/airflow/dags
plugins_folder | /opt/airflow/plugins
base_log_folder | /opt/airflow/logs
remote_base_log_folder |
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19962:
URL: https://github.com/apache/airflow/issues/19962#issuecomment-1023178110
> thanks for the replies. I tried it with Helm too, and it's working. Some strange thing is going on with compose and mounted file/folder permissions, but currently I don't have time to figure out the exact problem and solution.
Yeash this is **precisely** what I meant by adding this as a warning to our documentation:
> DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org