You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/18 23:14:26 UTC
[GitHub] [airflow] scrottty opened a new issue, #25145: Logs from import modules not showing in Airflow Log
scrottty opened a new issue, #25145:
URL: https://github.com/apache/airflow/issues/25145
### Apache Airflow version
2.2.3
### What happened
When running code from an import module, e.g. `boto3`, the logs are not printed to the Airflow log. When running on my local machine, the logs are printed to the console after settings `logging.basicConfig(level=logging.DEBUG)`. Setting the environmental variable `AIRFLOW__LOGGING__LOGGING_LEVEL` didnt help either.
Here is a copy of the DAG code used:
```python
import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
import logging
import boto3
def test_log():
logger = logging.getLogger('test_logger')
logger.debug('debug')
logger.info('info')
logger.warn('warn')
logger.error('error')
logger.critical('critical')
logging.basicConfig(level=logging.DEBUG)
client = boto3.client('s3')
with DAG(
'test_log',
description='test log',
schedule_interval=None,
start_date=datetime.datetime(2022,7,18),
catchup=False,
tags = ['logs']
) as dag:
task = PythonOperator(
task_id='test_log_task',
python_callable=test_log,
dag=dag
)
```
And here is the Airflow log file:
```
*** Reading remote log from s3://airflow-logs-dev-ollie/v1/test_log/test_log_task/2022-07-18T21:54:30.228267+00:00/6.log.
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: test_log.test_log_task manual__2022-07-18T21:54:30.228267+00:00 [queued]>
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: test_log.test_log_task manual__2022-07-18T21:54:30.228267+00:00 [queued]>
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1242} INFO - Starting attempt 6 of 6
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1243} INFO -
--------------------------------------------------------------------------------
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1262} INFO - Executing <Task(PythonOperator): test_log_task> on 2022-07-18 21:54:30.228267+00:00
[2022-07-18, 22:56:59 UTC] {standard_task_runner.py:52} INFO - Started process 13 to run task
[2022-07-18, 22:56:59 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'test_log', 'test_log_task', 'manual__2022-07-18T21:54:30.228267+00:00', '--job-id', '53291', '--raw', '--subdir', 'DAGS_FOLDER/log_test.py', '--cfg-path', '/tmp/tmpgninlo77', '--error-file', '/tmp/tmpfjgy5csk']
[2022-07-18, 22:56:59 UTC] {standard_task_runner.py:77} INFO - Job 53291: Subtask test_log_task
[2022-07-18, 22:56:59 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: test_log.test_log_task manual__2022-07-18T21:54:30.228267+00:00 [running]> on host testlogtestlogtask.482126fd3d2f40509477941f31efd31b
[2022-07-18, 22:56:59 UTC] {configuration.py:668} WARNING - Ignoring unknown env var 'AIRFLOW__KUBERNETES_SECRETS__aws_creds'
[2022-07-18, 22:56:59 UTC] {configuration.py:668} WARNING - Ignoring unknown env var 'AIRFLOW__KUBERNETES_SECRETS__aws_config'
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1427} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test_log
AIRFLOW_CTX_TASK_ID=test_log_task
AIRFLOW_CTX_EXECUTION_DATE=2022-07-18T21:54:30.228267+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-07-18T21:54:30.228267+00:00
[2022-07-18, 22:56:59 UTC] {log_test.py:14} INFO - info
[2022-07-18, 22:56:59 UTC] {log_test.py:15} WARNING - warn
[2022-07-18, 22:56:59 UTC] {log_test.py:16} ERROR - error
[2022-07-18, 22:56:59 UTC] {log_test.py:17} CRITICAL - critical
[2022-07-18, 22:56:59 UTC] {python.py:152} INFO - Done. Returned value was: None
[2022-07-18, 22:56:59 UTC] {taskinstance.py:1270} INFO - Marking task as SUCCESS. dag_id=test_log, task_id=test_log_task, execution_date=20220718T215430, start_date=20220718T225659, end_date=20220718T225659
[2022-07-18, 22:56:59 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
```
The log statements from within the DAG are shown down to `DEBUG` but the logs from boto3 aren't shown.
### What you think should happen instead
The log statements from the called module should be printed to the Airflow log to the level set in `logging.basicConfig(level=logging.DEBUG)` or in
```
logger = logging.getLogger('logger_name')
logger.setLevel(level=logging.DEBUG)
```
### How to reproduce
The code above should reproduce the issue. Let me know if you need more!
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==2.4.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==3.0.0
apache-airflow-providers-docker==2.3.0
apache-airflow-providers-elasticsearch==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.1.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-azure==3.3.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.2.0
apache-airflow-providers-slack==4.1.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Deployed on EKS v1.22
### Anything else
Occurs everytime
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25145: Logs from import modules not showing in Airflow Log
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25145:
URL: https://github.com/apache/airflow/issues/25145#issuecomment-1188423984
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25145: Logs from import modules not showing in Airflow Log
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25145: Logs from import modules not showing in Airflow Log
URL: https://github.com/apache/airflow/issues/25145
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25145: Logs from import modules not showing in Airflow Log
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25145:
URL: https://github.com/apache/airflow/issues/25145#issuecomment-1188695264
You should follow https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html?highlight=advanced#advanced-configuration and configur your loggers as you want (for example you can selectively disable or enable boto3 logging level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org