You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "D2CIT (via GitHub)" <gi...@apache.org> on 2024/11/04 20:37:25 UTC
[D] Airflow API creating roles [airflow]
GitHub user D2CIT created a discussion: Airflow API creating roles
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I'm trying to create a role via the airflow api but keep getting an error message.
- I'm running airflow in a container . Local test env in podman and in AWS ECS contiainer
- Reading out roles via the API goes fine.
- tested via curl ( see picture below)
- tested via python ( see script at "how to reproduce")
- error on version 2.5.1 but also on 2.10
- same error on version 2.10.2
```
AIRFLOW_API_URL: http://localhost:8080/api/v1
Failed to create role: 500
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
</head>
<body>
<div class="container">
<h1> Ooops! </h1>
<div>
<pre>
Something bad has happened. For security reasons detailed information about the error is not logged.
* You should check your webserver logs and retrieve details of this error.
* When you get the logs, it might explain the reasons, you should also Look for similar issues using:
* <b><a href="https://github.com/apache/airflow/discussions">GitHub Discussions</a></b>
* <b><a href="https://github.com/apache/airflow/issues">GitHub Issues</a></b>
* <b><a href="https://stackoverflow.com/questions/tagged/airflow">Stack Overflow</a></b>
* the usual search engine you use on a daily basis
All those resources might help you to find a solution to your problem.
* if you run Airflow on a Managed Service, consider opening an issue using the service support channels
* only after you tried it all, and have difficulty with diagnosing and fixing the problem yourself,
get the logs with errors, describe results of your investigation so far, and consider creating a
<b><a href="https://github.com/apache/airflow/issues/new/choose">bug report</a></b> including this information.
Python version: 3.9.20
Airflow version: 2.10.2
Node: redacted
-------------------------------------------------------------------------------
Error! Please contact server admin.</pre>
</div>
</div>
</body>
</html>
```
### What you think should happen instead?
API bug
### How to reproduce
Python script to create roles
```
import requests
import json
# Configuratie
AIRFLOW_API_URL = "http://localhost:8080/api/v1"
USERNAME = 'admin'
PASSWORD = 'admin'
auth = (USERNAME, PASSWORD)
print(f"AIRFLOW_API_URL: {AIRFLOW_API_URL}")
# Functie voor het ophalen van rollen in Airflow via API
def get_airflow_roles(uri, auth):
url = f"{uri}/roles"
response = requests.get(url, auth=auth)
if response.status_code == 200:
print("Get roles:")
print(response.json())
else:
print(f"Failed to get roles: {response.status_code}")
print(response.text)
# Functie voor het aanmaken van een rol in Airflow via API
def create_airflow_role(uri, auth, role_name):
url = f"{uri}/roles"
headers = {'Content-Type': 'application/json'}
data = json.dumps({
"name": role_name
})
response = requests.post(url, headers=headers, auth=auth, data=data)
if response.status_code == 201:
print(f"Role '{role_name}' created.")
else:
print(f"Failed to create role: {response.status_code}")
print(response.text)
# Hoofdfunctie
if __name__ == "__main__":
# Voorbeeld: Ophalen van bestaande rollen
#get_airflow_roles(AIRFLOW_API_URL, auth)
# Voorbeeld: Aanmaken van een nieuwe rol zonder permissies
nieuwe_rol_naam = "admin_config"
create_airflow_role(AIRFLOW_API_URL, auth, nieuwe_rol_naam)
```
### Operating System
MacOS 15.1
### Versions of Apache Airflow Providers
No extra providers. Just default airflow installation
### Deployment
Docker-Compose
### Deployment details
pod man-compose.yml local test environment;
```
version: '3.7'
services:
postgres:
image: postgres:14-alpine
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
POSTGRES_HOST_AUTH_METHOD: trust
ports:
- "5432:5432"
volumes:
- postgres_db:/var/lib/postgresql/data
airflow-init:
# image: apache/airflow:2.5.1-python3.9
image: apache/airflow:2.10.2-python3.9
environment:
AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
SQLALCHEMY_WARN_20: '1'
SQLALCHEMY_SILENCE_UBER_WARNING: '1'
volumes:
- ./dags:/opt/airflow/dags
entrypoint: ["airflow", "db", "init"]
depends_on:
- postgres
airflow-create-user:
image: apache/airflow:2.10.2-python3.9
environment:
AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session" # Geïdentificeerde typfout gecorrigeerd
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
volumes:
- ./dags:/opt/airflow/dags
entrypoint: ["airflow", "users", "create", "--username", "admin", "--password", "admin", "--firstname", "Admin", "--lastname", "User", "--role", "Admin", "--email", "admin@example.com"]
depends_on:
- postgres
- airflow-init
airflow-webserver:
image: apache/airflow:2.10.2-python3.9
environment:
AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
AIRFLOW__WEBSERVER__RBAC: 'True'
SQLALCHEMY_WARN_20: '1'
SQLALCHEMY_SILENCE_UBER_WARNING: '1'
AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'True'
volumes:
- ./dags:/opt/airflow/dags
# - ./config/webserver_config.py:/opt/airflow/webserver_config.py
command: webserver
ports:
- "8080:8080"
depends_on:
- postgres
- airflow-init
airflow-scheduler:
image: apache/airflow:2.10.2-python3.9
environment:
AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
SQLALCHEMY_WARN_20: '1'
SQLALCHEMY_SILENCE_UBER_WARNING: '1'
volumes:
- ./dags:/opt/airflow/dags
depends_on:
- postgres
- airflow-init
volumes:
postgres_db:
```
### Anything else?
Same error in
- different versions (2.5.1 and 2.10.2)
- tested in podman, docker and AWS ECS container
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
GitHub link: https://github.com/apache/airflow/discussions/43658
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "D2CIT (via GitHub)" <gi...@apache.org>.
GitHub user D2CIT added a comment to the discussion: Airflow API creating roles
thank you for your response. I have created a GitHub page with my setup and logfiles of each step.
See --> https://github.com/D2CIT/Airflow
- command.sh : contains all the commands step by step
- podman-compose.yaml : is the setup for the local airflow enviroment.
- Config folder : contains the airflow.cfg and the webserver_config.py
- log folder : contains the logfiles from the container for each step
Is this enough?
thanks
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11153302
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "D2CIT (via GitHub)" <gi...@apache.org>.
GitHub user D2CIT edited a comment on the discussion: Airflow API creating roles
The log shows that a KeyError occurs because of the missing key 'permissions ’ in the JSON message I've send via the API. This means probably a missing required key in the API request.
I tried this suggestion
```
curl -X POST "http://localhost:8080/api/v1/roles" \
-H "Content-Type: application/json" \
--data '{
"name": "example_role",
"permissions": [
{
"action": {
"name": "can_read"
},
"resource": {
"name": "DagRuns"
}
}
]
}
```
but then I get the following error
```
{
"detail": null,
"status": 403,
"title": "Forbidden",
"type": "https://airflow.apache.org/docs/apache-airflow/2.10.2/stable-rest-api-ref.html#section/Errors/PermissionDenied"
}
```
This error message indicates that the permissions field is treated as unknown by the Airflow API, despite being expected according to the code.
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11153428
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow API creating roles
Converting to a discussion - this is not an actionable issue without log details and analysis of those.
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11147229
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "D2CIT (via GitHub)" <gi...@apache.org>.
GitHub user D2CIT edited a comment on the discussion: Airflow API creating roles
The log shows that a KeyError occurs because of the missing key 'permissions ’ in the JSON message I've send via the API. This means probably a missing required key in the API request.
I tried this suggestion
```
curl -X POST "http://localhost:8080/api/v1/roles" \
-H "Content-Type: application/json" \
--data '{
"name": "example_role",
"permissions": [
{
"action": {
"name": "can_read"
},
"resource": {
"name": "DagRuns"
}
}
]
}
```
but then I get the following error
```
{
"detail": null,
"status": 403,
"title": "Forbidden",
"type": "https://airflow.apache.org/docs/apache-airflow/2.10.2/stable-rest-api-ref.html#section/Errors/PermissionDenied"
}
```
This error message indicates that the permissions field is treated as unknown by the Airflow API, despite being expected according to the code.it seems there may be a nesting or incompatibility problem.
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11153428
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow API creating roles
As explained in the error message - you need to look for a reason in logs. Without finding the error in logs, there is no way to help you - and likely when you see the logs, you will find what's wrong.
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11147226
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: Airflow API creating roles
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11147223
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow API creating roles [airflow]
Posted by "D2CIT (via GitHub)" <gi...@apache.org>.
GitHub user D2CIT added a comment to the discussion: Airflow API creating roles
The log shows that a KeyError occurs because of the missing key 'permissions ’ in the JSON message you send via the API. This means you are probably missing a required key in your API request.
I tried this suggestion
```
curl -X POST "http://localhost:8080/api/v1/roles" \
-H "Content-Type: application/json" \
--data '{
"name": "example_role",
"permissions": [
{
"action": {
"name": "can_read"
},
"resource": {
"name": "DagRuns"
}
}
]
}
```
but then I get the following error
```
{
"detail": null,
"status": 403,
"title": "Forbidden",
"type": "https://airflow.apache.org/docs/apache-airflow/2.10.2/stable-rest-api-ref.html#section/Errors/PermissionDenied"
}
```
This error message indicates that the permissions field is treated as unknown by the Airflow API, despite being expected according to the code.
GitHub link: https://github.com/apache/airflow/discussions/43658#discussioncomment-11153428
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org