You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/18 11:20:28 UTC
[GitHub] [airflow] harshit283 opened a new issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
harshit283 opened a new issue #14874:
URL: https://github.com/apache/airflow/issues/14874
Airflow Setup Used
Airflow Version: 1.10.12
PostgreSQL version: 10
Airflow Configuration:
AIRFLOW__WEBSERVER__WORKER_REFRESH_BATCH_SIZE: 0
AIRFLOW__WEBSERVER__WORKERS:10
AIRFLOW__CORE__STORE_SERIALIZED_DAGS: True
AIRFLOW__CORE__STORE_DAG_CODE: True
AIRFLOW__WEBSERVER__WORKER_CLASS: gevent
**Issue details:**
So we are doing some performance tests on airflow webserver to benchmark how many requests per second can a single container (kubernetes) of airflow webserver can handle, we are testing the airflow [experimental APIs](https://airflow.apache.org/docs/apache-airflow/stable/rest-api-ref.html#post--api-experimental-dags--DAG_ID--dag_runs). We are able to handle upto 8 RPS and on investigation it appears that the bottleneck is at the authentication layer. We are using the password based authentication where `check_password_hash` [method ](https://github.com/apache/airflow/blob/1.10.12/airflow/contrib/auth/backends/password_auth.py#L84) appears to be extremely slow in execution, due to which the authentication flow is contributing to 80-85% of total time taken for the Airflow API's.
After bypassing this method airflow webserver is easily able to serve 50 RPS
**We are looking for couple of things:**
- Is there any way to fix this issue or if this is expected should we opt for any other authentication protocol?
- Is this issue fixed in later versions of airflow?
- Are other authentication protocols more performant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14874:
URL: https://github.com/apache/airflow/issues/14874#issuecomment-803192762
@harshit283 The experimental API too has been deprecated from 2.0.0 -- since there were already experimental. They have been replaced by the Stable API: https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #14874:
URL: https://github.com/apache/airflow/issues/14874
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] harshit283 commented on issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
Posted by GitBox <gi...@apache.org>.
harshit283 commented on issue #14874:
URL: https://github.com/apache/airflow/issues/14874#issuecomment-802690100
Hi @kaxil We are testing out the **experimental APIs** [[Link](https://airflow.apache.org/docs/apache-airflow/stable/rest-api-ref.html#post--api-experimental-dags--DAG_ID--dag_runs)] in Airflow and not the Admin UI
Does this RBAC settings have any effect on our use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14874:
URL: https://github.com/apache/airflow/issues/14874#issuecomment-801873617
Hi @harshit283 , the old UI is deprecated and removed in Airflow 2 and we won't be porting any fixing to that UI.
Please consider using RBAC UI (https://airflow.apache.org/blog/airflow-1.10.12/#use-airflow-rbac-ui)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on issue #14874:
URL: https://github.com/apache/airflow/issues/14874#issuecomment-801873617
Hi @harshit283 , the old UI is deprecated and removed in Airflow 2 and we won't be porting any fixing to that UI.
Please consider using RBAC UI (https://airflow.apache.org/blog/airflow-1.10.12/#use-airflow-rbac-ui)
Do let us know if you face same issues with RBAC UI, in which case a new issue can be opened
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14874: Airflow Web Server - Performance Issues with Password Based Authentication
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14874:
URL: https://github.com/apache/airflow/issues/14874#issuecomment-801839966
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org