You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/23 07:50:11 UTC
[GitHub] [airflow] armourshield opened a new issue, #26620: Airflow 2.2.2 remote worker logging getting 403 Forbidden
armourshield opened a new issue, #26620:
URL: https://github.com/apache/airflow/issues/26620
### Apache Airflow version
Other Airflow 2 version
### What happened
I have a setup where airflow is running in **kubernetes (EKS)** and **remote worker** running in *docker-compose* in a VM behind a firewall in a different location.
**Problem**
Airflow Web server in EKS is getting 403 forbidden error when trying to get logs on remote worker.
**Build Version**
- Airflow - 2.2.2
- OS - Linux - Ubuntu 20.04 LTS
**Kubernetes**
- 1.22 (EKS)
- Redis (Celery Broker) - Service Port exposed on 6379
- PostgreSQL (Celery Backend) - Service Port exposed on 5432
**Airflow ENV config setup**
```
AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
AIRFLOW__CELERY__BROKER_URL: redis://<username>:<password>@redis-master.airflow-dev.svc.cluster.local:6379/0
AIRFLOW__CELERY__RESULT_BACKEND: >-
db+postgresql://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
AIRFLOW__CLI__ENDPOINT_URL: http://{hostname}:8080
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__FERNET_KEY: <fernet_key>
AIRFLOW__CORE__HOSTNAME_CALLABLE: socket.getfqdn
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__CORE__SQL_ALCHEMY_CONN: >-
postgresql+psycopg2://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT: '8793'
AIRFLOW__WEBSERVER__BASE_URL: http://{hostname}:8080
AIRFLOW__WEBSERVER__SECRET_KEY: <secret_key>
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_PASSWORD: <username-webserver>
_AIRFLOW_WWW_USER_USERNAME: <password-webserver>
```
**Setup Test**
1. Network reach ability by ping - OK
2. Celery Broker reach ability for both EKS and remote worker - OK
3. Celery Backend reach ability for both EKS and remote worker - OK
4. Firewall Port expose for remote worker Gunicorn API - OK
5. curl -v telnet://<remote-worker>:8793 test - OK (Connected)
6. Airflow flower recognizing both workers from Kubernetes and remote worker - OK
7. All the ENV on both webserver, worker (EKS, remote) and scheduler are identical
8. Queue is setup so the DAG runs exactly in that particular worker
9. Time on both docker, VM and EKS is on UTC. There is a slight 5 to 8 seconds difference in docker and the pod in EKS
10. Ran webserver on the remote VM as well which can pick up and show logs
**Description**
Airflow is able to execute the DAG in remote worker, the logs can be seen in the remote worker. I have tried all combinations of setting but still keep getting 403.
Another test which was done was just normal curl with webserver auth
This curl was done both from EKS and remote server which hosts docker-compose. Results are the same on all the server.
```
curl --user <username-webserver> -vvv http:<remote-worker>:8793/logs/?<rest-of-the-log-url>
Getting 403 Forbidden
```
I might have miss configured it, but I doubt that is the case.
Any tips on what I am missing here? Many thanks in advance.
### What you think should happen instead
Airflow web-server in EKS should be able to access the remote logs from remote worker as the port is accessible and all the secret key, fernet key matches and all the env matches.
### How to reproduce
Use K3S, minikube or kubernetes of your choice. to replicate remote-worker run docker-compose setup for the installation.
**Build Version**
- Airflow - 2.2.2
- OS - Linux - Ubuntu 20.04 LTS
**Kubernetes**
- 1.22 EKS, K3S, minikube or kubernetes
- Redis (Celery Broker) - Service Port exposed on 6379
- PostgreSQL (Celery Backend) - Service Port exposed on 5432
**Airflow ENV config setup**
```
AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
AIRFLOW__CELERY__BROKER_URL: redis://<username>:<password>@redis-master.airflow-dev.svc.cluster.local:6379/0
AIRFLOW__CELERY__RESULT_BACKEND: >-
db+postgresql://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
AIRFLOW__CLI__ENDPOINT_URL: http://{hostname}:8080
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__FERNET_KEY: <fernet_key>
AIRFLOW__CORE__HOSTNAME_CALLABLE: socket.getfqdn
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__CORE__SQL_ALCHEMY_CONN: >-
postgresql+psycopg2://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT: '8793'
AIRFLOW__WEBSERVER__BASE_URL: http://{hostname}:8080
AIRFLOW__WEBSERVER__SECRET_KEY: <secret_key>
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_PASSWORD: <username-webserver>
_AIRFLOW_WWW_USER_USERNAME: <password-webserver>
```
### Operating System
Ubuntu 20.04 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
**Airflow ENV config setup**
```
AIRFLOW__API__AUTH_BACKEND: airflow.api.auth.backend.basic_auth
AIRFLOW__CELERY__BROKER_URL: redis://<username>:<password>@redis-master.airflow-dev.svc.cluster.local:6379/0
AIRFLOW__CELERY__RESULT_BACKEND: >-
db+postgresql://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
AIRFLOW__CLI__ENDPOINT_URL: http://{hostname}:8080
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__FERNET_KEY: <fernet_key>
AIRFLOW__CORE__HOSTNAME_CALLABLE: socket.getfqdn
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__CORE__SQL_ALCHEMY_CONN: >-
postgresql+psycopg2://<username>:<password>@db-postgresql.airflow-dev.svc.cluster.local/<db>
AIRFLOW__LOGGING__BASE_LOG_FOLDER: /opt/airflow/logs
AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT: '8793'
AIRFLOW__WEBSERVER__BASE_URL: http://{hostname}:8080
AIRFLOW__WEBSERVER__SECRET_KEY: <secret_key>
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_PASSWORD: <username-webserver>
_AIRFLOW_WWW_USER_USERNAME: <password-webserver>
```
### Anything else
This has been failing to fetch logs even there is there complete access to the port and remote worker. 403 Forbidden keeps saying secret key when it is the same across the whole environment.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26620: Airflow 2.2.2 remote worker logging getting 403 Forbidden
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26620: Airflow 2.2.2 remote worker logging getting 403 Forbidden
URL: https://github.com/apache/airflow/issues/26620
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26620: Airflow 2.2.2 remote worker logging getting 403 Forbidden
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26620:
URL: https://github.com/apache/airflow/issues/26620#issuecomment-1255898277
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org