You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/13 13:50:13 UTC
[GitHub] [airflow] KulykDmytro opened a new issue #21548: Cant get Kubernetes task's log when using CeleryKubernetesExecutor
KulykDmytro opened a new issue #21548:
URL: https://github.com/apache/airflow/issues/21548
### Apache Airflow version
2.2.3 (latest released)
### What happened
Recently we're switched from `KubernetesExecutor` to `CeleryKubernetesExecutor` and faced with strange behaviour
Tasks which are sent to `kubernetes` queue unable to provide logs in `webserver` while executing
```
*** Falling back to local log
*** Log file does not exist: /opt/airflow/logs/sfmc_sync/notification_center_send_log/2022-02-12T07:00:00+00:00/6.log
*** Fetching from: http://10.47.204.47:8793/log/sfmc_sync/notification_center_send_log/2022-02-12T07:00:00+00:00/6.log
*** Failed to fetch log file from worker. [Errno 113] No route to host
```
webserver not even tried to reach tasks log over k8s api (like it happened when using `KubernetesExecutor`)
### What you expected to happen
When task is executed on `kubernetes` queue (despite of executor chosen), logs from container's stdout should be provided to webserver upon request
### How to reproduce
set executor to `CeleryKubernetesExecutor`
set default queue to `celery`
set additional queue for `kubernetes`
send task for execution to `queue='kubernetes'`
### Operating System
Linux 5.4.156-83.273.amzn2.x86_64
### Versions of Apache Airflow Providers
standard ones for official image
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
EKS deployment
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on issue #21548: Cant get Kubernetes task's log when using CeleryKubernetesExecutor
Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on issue #21548:
URL: https://github.com/apache/airflow/issues/21548#issuecomment-1048191891
@arkadiusz-bach i believe you've mentioned how to fix an issue in #11629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arkadiusz-bach edited a comment on issue #21548: Cant get Kubernetes task's log when using CeleryKubernetesExecutor
Posted by GitBox <gi...@apache.org>.
arkadiusz-bach edited a comment on issue #21548:
URL: https://github.com/apache/airflow/issues/21548#issuecomment-1042417578
This is a bug, it is trying to access Celery logs because of the following line:
https://github.com/apache/airflow/blob/710d6997035a5e050367db013ae6847cc90dc51a/airflow/utils/log/file_task_handler.py#L143
It should also check whether executor is equal to 'CeleryKubernetesExecutor' and the task queue is 'kubernetes'(queue value should be taken from config)
Once you have got that fixed, log will be empty in the running state, because it is trying to read container logs and by default tasks logs are written to files.
You will need to either add stdout handler for `airflow.task` logger in logging config or do it like I did, I just replaced the `read_namespaced_pod_log` with kubernetes exec command, which just executes `tail -n 100 log_file_path' in the task container in order to get last 100 lines of log.
As for the failed or succeeded tasks, it depends whether you have got remote_logging enabled, if you do then log will be taken from remote location otherwise it will be empty, because it seems that you are not using the same volumes for webserver and tasks containers and once the task container is gone then logs are gone as well.
You can just mount the same volume for logs directory on both webserver and tasks containers, it will work, but if you are not using the same volumes(or volumes at all) for logs then that is the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21548: Cant get Kubernetes task's log when using CeleryKubernetesExecutor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21548:
URL: https://github.com/apache/airflow/issues/21548#issuecomment-1081027895
> @arkadiusz-bach i believe you've mentioned how to fix an issue in #11629
This is adifferent story. I think this one is a real issue that needs to be handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] arkadiusz-bach commented on issue #21548: Cant get Kubernetes task's log when using CeleryKubernetesExecutor
Posted by GitBox <gi...@apache.org>.
arkadiusz-bach commented on issue #21548:
URL: https://github.com/apache/airflow/issues/21548#issuecomment-1042417578
This is a bug, it is trying to access Celery logs because of the following line:
https://github.com/apache/airflow/blob/710d6997035a5e050367db013ae6847cc90dc51a/airflow/utils/log/file_task_handler.py#L143
It should also check whether executor is equal to 'CeleryKubernetesExecutor' and the task queue is 'kubernetes'(queue value should be taken from config)
Once you have got that fixed, log will be empty in the running state, because it is trying to read container logs and by default tasks logs are written to files.
You will need to either add stdout handler for `airflow.task` logger in logging config or do it like I did, I just replaced the read namespaced pod with kubernetes exec command, which just executes `tail -n 100 log_file_path' in the task container in order to get last 100 lines of log.
As for the failed or succeeded tasks, it depends whether you have got remote_logging enabled, if you do then log will be taken from remote location otherwise it will be empty, because it seems that you are not using the same volumes for webserver and tasks containers and once the task container is gone then logs are gone as well.
You can just mount the same volume for logs directory on both webserver and tasks containers, it will work, but if you are not using the same volumes(or volumes at all) for logs then that is the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org