You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/10 05:45:30 UTC

[GitHub] [airflow] nirutgupta opened a new issue #20779: Logs of tasks when running is not available with kubernetesexecutor on webserver UI

nirutgupta opened a new issue #20779:
URL: https://github.com/apache/airflow/issues/20779


   ### Description
   
   Provide any configuration, or a sidecar container to achieve task logs streaming in kubernetes executor to webserver UI when task are in running state.
   
   ### Use case/motivation
   
   Moving out from celery executor where I can see the running logs , it is very difficult for me to have the patience when using kubernetes executor for seeing task logs of long running tasks since they will only come up after the task succeeds or fails. 
   
   ### Related issues
   
   Using kubernetes executor. When I run a DAG, I am only able to see the logs when task is in running state, only thing I can see is. 
   ```
   *** Falling back to local log
   *** Trying to get logs (last 100 lines) from worker pod sleeptestsleeptest.dda6a168c2ad40f4b8bc933779db1732 ***
   
   
   /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:357 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
   [2022-01-06, 12:00:07 UTC] {dagbag.py:500} INFO - Filling up the DagBag from /opt/***/dags/repo/qainfra/sample_dags/1.0/hang_debug.py
   ```
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #20779: Logs of tasks when running is not available with kubernetesexecutor on webserver UI

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20779:
URL: https://github.com/apache/airflow/issues/20779#issuecomment-1008559601


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nirutgupta edited a comment on issue #20779: Logs of tasks when running is not available with kubernetesexecutor on webserver UI

Posted by GitBox <gi...@apache.org>.
nirutgupta edited a comment on issue #20779:
URL: https://github.com/apache/airflow/issues/20779#issuecomment-1011786738


   k8s task handler.py which we can use https://gist.github.com/szeevs/938ad3cf96e732d4b1b55a74015aed5b
   log_config.py will look like 
   ```
   from copy import deepcopy
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   from airflow.configuration import conf
   
   import os
   import sys
   
   sys.path.append('/opt/airflow/custom_log')
   
   BASE_LOG_FOLDER: str = conf.get('logging', 'BASE_LOG_FOLDER')
   FILENAME_TEMPLATE: str = conf.get('logging', 'LOG_FILENAME_TEMPLATE')
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   LOGGING_CONFIG['handlers']['k8stask'] = {
       'class': 'k8s_task_handler.KubernetesTaskHandler',
       'formatter': 'airflow',
       'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
       'filename_template': FILENAME_TEMPLATE
   }
   
   LOGGING_CONFIG['loggers']['airflow.task']['handlers'].append('k8stask')
   ```
   
   Make sure while deploying, configure this env
     - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
       value: "log_config.LOGGING_CONFIG"
   
   This will do the job. :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nirutgupta commented on issue #20779: Logs of tasks when running is not available with kubernetesexecutor on webserver UI

Posted by GitBox <gi...@apache.org>.
nirutgupta commented on issue #20779:
URL: https://github.com/apache/airflow/issues/20779#issuecomment-1011785432


   I see a good working solution of this with https://szeevs.medium.com/handling-airflow-logs-with-kubernetes-executor-25c11ea831e4
   All we need to do is make this k8s_task_handler file available under utils/log/ and configure log_config.py file to link airflow.tasks to this new handler as well. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nirutgupta edited a comment on issue #20779: Logs of tasks when running is not available with kubernetesexecutor on webserver UI

Posted by GitBox <gi...@apache.org>.
nirutgupta edited a comment on issue #20779:
URL: https://github.com/apache/airflow/issues/20779#issuecomment-1011786738


   k8s task handler.py which we can use https://gist.github.com/szeevs/938ad3cf96e732d4b1b55a74015aed5b
   log_config.py will look like 
   ```
   from copy import deepcopy
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   from airflow.configuration import conf
   
   import os
   import sys
   
   sys.path.append('/opt/airflow/custom_log')
   
   BASE_LOG_FOLDER: str = conf.get('logging', 'BASE_LOG_FOLDER')
   FILENAME_TEMPLATE: str = conf.get('logging', 'LOG_FILENAME_TEMPLATE')
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   LOGGING_CONFIG['handlers']['k8stask'] = {
       'class': 'k8s_task_handler.KubernetesTaskHandler',
       'formatter': 'airflow',
       'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
       'filename_template': FILENAME_TEMPLATE
   }
   
   LOGGING_CONFIG['loggers']['airflow.task']['handlers'].append('k8stask')
   ```
   
   Make sure while deploying configure this env
     - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
       value: "log_config.LOGGING_CONFIG"
   
   This will do the job. :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] nirutgupta commented on issue #20779: Logs of tasks when running is not available with kubernetesexecutor on webserver UI

Posted by GitBox <gi...@apache.org>.
nirutgupta commented on issue #20779:
URL: https://github.com/apache/airflow/issues/20779#issuecomment-1011786738


   k8s task handler.py which we can use https://gist.github.com/szeevs/938ad3cf96e732d4b1b55a74015aed5b
   log_config.py will look like 
   `from copy import deepcopy
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   from airflow.configuration import conf
   
   import os
   import sys
   
   sys.path.append('/opt/airflow/custom_log')
   
   BASE_LOG_FOLDER: str = conf.get('logging', 'BASE_LOG_FOLDER')
   FILENAME_TEMPLATE: str = conf.get('logging', 'LOG_FILENAME_TEMPLATE')
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   LOGGING_CONFIG['handlers']['k8stask'] = {
       'class': 'k8s_task_handler.KubernetesTaskHandler',
       'formatter': 'airflow',
       'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
       'filename_template': FILENAME_TEMPLATE
   }
   
   LOGGING_CONFIG['loggers']['airflow.task']['handlers'].append('k8stask')
   `
   Make sure while deploying configure this env
     - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
       value: "log_config.LOGGING_CONFIG"
   
   This will do the job. :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org