You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/07 10:34:27 UTC
[GitHub] [airflow] devashish-parmar opened a new issue, #26201: Airflow webserver UI unable to access worker node logs
devashish-parmar opened a new issue, #26201:
URL: https://github.com/apache/airflow/issues/26201
### Apache Airflow version
2.3.4
### What happened
We have set up a multi node Airflow configuration. We have the below setup.
- 4 nodes
- node 1: webserver, rabbitmq, flower
- node 2: meta db
- node 3: worker 1
- node 4: worker 2
Although the setup is working fine and DAGs are getting distributed among the worker nodes, however the logs generated in the worker nodes are not reflecting in the webserver UI in the master node.
Below is the error:
***** Log file does not exist: /home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** Fetching from: http://<hostname>:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://<hostname>:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log'
For more information check: https://httpstatuses.com/403**
We have checked all the folder permission, secret_key, timezone but still the problem persist.
can anyone help here?
### What you think should happen instead
_No response_
### How to reproduce
_No response_
### Operating System
Linux Red Hat
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26201: Airflow webserver UI unable to access worker node logs
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26201: Airflow webserver UI unable to access worker node logs
URL: https://github.com/apache/airflow/issues/26201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26201: Airflow webserver UI unable to access worker node logs
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26201:
URL: https://github.com/apache/airflow/issues/26201#issuecomment-1239211226
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] devashish-parmar commented on issue #26201: Airflow webserver UI unable to access worker node logs
Posted by GitBox <gi...@apache.org>.
devashish-parmar commented on issue #26201:
URL: https://github.com/apache/airflow/issues/26201#issuecomment-1241458311
yes, the directory/file does exist. the below command runs fine from the worker node
less "/home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jledoux-sonergia commented on issue #26201: Airflow webserver UI unable to access worker node logs
Posted by GitBox <gi...@apache.org>.
jledoux-sonergia commented on issue #26201:
URL: https://github.com/apache/airflow/issues/26201#issuecomment-1249429747
We've ran into the same issue, worker's gunicorn webserver does not start properly on 2.3.4, downgrading to 2.3.3 fix this issue.
I think it's related to this PR https://github.com/apache/airflow/pull/24846
And saimon46's comment explains it: https://github.com/apache/airflow/pull/24846#issuecomment-1231578766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26201: Airflow webserver UI unable to access worker node logs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26201:
URL: https://github.com/apache/airflow/issues/26201#issuecomment-1250884792
@jledoux-sonergia - The 2.4.0 is out today and It should have the IPV6 problem fixed. @devashish-parmar - if the request is forbidden, you should have looked at the webeserver logs and it should tell you the exact reason for the problem you have
> We have checked all the folder permission, secret_key, timezone but still the problem persist.
Timezone is not an issue. You should check if TIME is synchronized on all your machines.
Converting that into a discussion as either some of the problems mentioned here (IPV6) have been fixed in 2.4.0 or they need more looking at the deployment side and checking the webserver logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nsAstro commented on issue #26201: Airflow webserver UI unable to access worker node logs
Posted by GitBox <gi...@apache.org>.
nsAstro commented on issue #26201:
URL: https://github.com/apache/airflow/issues/26201#issuecomment-1241117550
Can you see if there are any files in the directory '/home/dvafadm/airflow/logs/' and if so, can you find the ones corresponding the the above DAG and list the file/directory structure? I'm curious if this is a name mismatch or if the logs just don't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org