You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/07 19:38:38 UTC
[GitHub] [airflow] pastushenkoy opened a new issue, #22842: Environment variables for elastic search logging have no effect
pastushenkoy opened a new issue, #22842:
URL: https://github.com/apache/airflow/issues/22842
### Apache Airflow version
2.2.5 (latest released)
### What happened
When I start airflow in official docker container with command:
```
docker run \
-e AIRFLOW__ELASTICSEARCH__JSON_FORMAT=True \
-e AIRFLOW__ELASTICSEARCH__JSON_FORMAT=True \
-e AIRFLOW__ELASTICSEARCH__LOG_ID_TEMPLATE="{dag_id}-{task_id}-{run_id}-{try_number}" \
-e AIRFLOW__ELASTICSEARCH__JSON_FIELDS="asctime, filename, lineno, levelname, message" \
apache/airflow:2.2.5 webserver
```
I've got the following output:
```
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2022-04-07 18:38:08,161] {dagbag.py:500} INFO - Filling up the DagBag from /dev/null
[2022-04-07 18:38:35,231] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[2022-04-07 18:40:11 +0000] [29] [INFO] Starting gunicorn 20.1.0
[2022-04-07 18:40:11 +0000] [29] [INFO] Listening at: http://0.0.0.0:8080 (29)
[2022-04-07 18:40:11 +0000] [29] [INFO] Using worker: sync
[2022-04-07 18:40:11 +0000] [38] [INFO] Booting worker with pid: 38
[2022-04-07 18:40:11 +0000] [39] [INFO] Booting worker with pid: 39
[2022-04-07 18:40:11 +0000] [40] [INFO] Booting worker with pid: 40
[2022-04-07 18:40:11 +0000] [41] [INFO] Booting worker with pid: 41
[2022-04-07 18:40:43,975] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-04-07 18:40:43,975] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-04-07 18:40:44,093] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-04-07 18:40:44,095] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
```
### What you think should happen instead
I expected to see json logs in Elasticsearch format in console.
### How to reproduce
```
mkdir temp
```
```
cd temp
```
```
docker run -v "$(pwd)":/opt/airflow apache/airflow:2.2.5 db init
```
```
docker run -v "$(pwd)":/opt/airflow \
-e AIRFLOW__ELASTICSEARCH__JSON_FORMAT=True \
-e AIRFLOW__ELASTICSEARCH__JSON_FORMAT=True \
-e AIRFLOW__ELASTICSEARCH__LOG_ID_TEMPLATE="{dag_id}-{task_id}-{run_id}-{try_number}" \
-e AIRFLOW__ELASTICSEARCH__JSON_FIELDS="asctime, filename, lineno, levelname, message" \
apache/airflow:2.2.5 webserver
```
### Operating System
macos 11.2.3
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pastushenkoy commented on issue #22842: Environment variables for elastic search logging have no effect
Posted by GitBox <gi...@apache.org>.
pastushenkoy commented on issue #22842:
URL: https://github.com/apache/airflow/issues/22842#issuecomment-1094702538
That's much clearer now. Thant you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22842: Environment variables for elastic search logging have no effect
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22842:
URL: https://github.com/apache/airflow/issues/22842#issuecomment-1092146627
This is complete misunderstanding of elasticsearch configuration https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/logging/index.html
You won't get logs in elasticsearch in console. if you configure elasticsearch, they will be sent to elasticsearch but logs in console come with the format defined in your loging configuration. You can still update it if you want but ELASTICSEARCH configuration only affects logs sent to elasticsearch,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #22842: Environment variables for elastic search logging have no effect
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22842:
URL: https://github.com/apache/airflow/issues/22842#issuecomment-1092129741
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #22842: Environment variables for elastic search logging have no effect
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #22842: Environment variables for elastic search logging have no effect
URL: https://github.com/apache/airflow/issues/22842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pastushenkoy commented on issue #22842: Environment variables for elastic search logging have no effect
Posted by GitBox <gi...@apache.org>.
pastushenkoy commented on issue #22842:
URL: https://github.com/apache/airflow/issues/22842#issuecomment-1092743213
@potiuk Thank you for your answer.
Can you then tell me what `AIRFLOW__ELASTICSEARCH__WRITE_STDOUT=True` should do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22842: Environment variables for elastic search logging have no effect
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22842:
URL: https://github.com/apache/airflow/issues/22842#issuecomment-1092895003
> Can you then explain to me what `AIRFLOW__ELASTICSEARCH__WRITE_STDOUT=True` should do?
It does 100% what is written in the documentation:
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#write-stdout
> Write the task logs to the stdout of the worker, rather than the default file.
WORKER not WEBSERVER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org