You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/19 13:40:44 UTC
[GitHub] [airflow] emredjan opened a new issue, #26489: Cannot fetch log from Celery worker
emredjan opened a new issue, #26489:
URL: https://github.com/apache/airflow/issues/26489
### Apache Airflow version
2.4.0
### What happened
When running tasks on a remote celery worker, webserver fails to fetch logs from the machine, giving a '403 - Forbidden' error on version 2.4.0. This behavior does not happen on 2.3.3, where the remote logs are retrieved and displayed successfully.
The `webserver / secret_key` configuration is the same in all nodes (the config files are synced), and their time is synchronized using a central NTP server, making the solution in the warning message not applicable.
My limited analysis pointed to the `serve_logs.py` file, and the flask request object that's passed to it, but couldn't find the root cause.
### What you think should happen instead
It should fetch and show remote celery worker logs on the webserver UI correctly, as it did in previous versions.
### How to reproduce
Use airflow version 2.4.0
Use CeleryExecutor with RabbitMQ
Use a separate Celery worker machine
Run a dag/task on the remote worker
Try to display task log on the web UI
### Operating System
Red Hat Enterprise Linux 8.6 (Ootpa)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==3.0.0
apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.0.0
apache-airflow-providers-hashicorp==3.0.0
apache-airflow-providers-http==3.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-microsoft-mssql==3.0.0
apache-airflow-providers-mysql==3.0.0
apache-airflow-providers-odbc==3.0.0
apache-airflow-providers-sftp==3.0.0
apache-airflow-providers-sqlite==3.0.0
apache-airflow-providers-ssh==3.0.0
```
### Deployment
Virtualenv installation
### Deployment details
Using CeleryExecutor / rabbitmq with 2 servers
### Anything else
All remote task executions has the same problem.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26489: Cannot fetch log from Celery worker
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26489:
URL: https://github.com/apache/airflow/issues/26489#issuecomment-1251047398
Could be connected to IPV6 issue #24846 - which has been fixed in Airflow 2.4.0 (relased today) see https://github.com/apache/airflow/issues/26201 - please upgrade to 2.4.0 and see if the problem persists (and please comment here). If it does not fix it (or even before) - taje a look at the logs of webserver and they should contain error logs/explanation where it was from.
Converting to discussion as this is likely solved or need more detailed looking in the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26489: Cannot fetch log from Celery worker
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26489:
URL: https://github.com/apache/airflow/issues/26489#issuecomment-1251037220
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26489: Cannot fetch log from Celery worker
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26489: Cannot fetch log from Celery worker
URL: https://github.com/apache/airflow/issues/26489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org