You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/11 20:22:24 UTC
[GitHub] [airflow] pat-s opened a new issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
pat-s opened a new issue #11436:
URL: https://github.com/apache/airflow/issues/11436
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 1.10.12
**Environment**:
- **Others**: Docker
**What happened**:
Trying to run LDAP login with v1.10.12 via the official `apache/airflow` Docker image.
I manually added the python `ldap3` module via `pip3 install --user ldap3`. Otherwise I get complaints that this module is missing.
Then, when starting the webserver, I get
```
airflow Webserver
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 26, in <module>
from airflow.bin.cli import CLIFactory
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/bin/cli.py", line 93, in <module>
api.load_auth()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/api/__init__.py", line 66, in load_auth
api_auth.client_auth = deprecated('use CLIENT_AUTH', api_auth.CLIENT_AUTH)
AttributeError: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
```
Seems related to #4343
**What you expected to happen**:
A normal startup of `airflow Webserver`.
Looks like the patch from #4343 needs also to be applied to the LDAP backend?
**How to reproduce it**:
Here's the Dockerfile:
```
FROM apache/airflow:1.10.12-python3.6
ARG ADDITIONAL_PYTHON_DEPS="ldap3"
RUN if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
pip3 install --user ${ADDITIONAL_PYTHON_DEPS}; \
fi
```
`entrypoint.sh`
```
#!/usr/bin/env bash
airflow initdb
airflow webserver
exec airflow "$@"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-708323040
@pat-s I described in the ticket the tasks that we have to do. I have a partial implementation, but it will be a while before I can publish it. If you want to do it quickly, you need to implement some code yourself based on the ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710919991
I would like to try using v2.0 (because we have not started using the service for now) however I cannot get the current master Docker container (build from source) to start. This should probably be discussed in a separate issue though.
All your CI runs for the production images are green but building them is just one thing 😄 Does the master branch currently run a stable webserver?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-708338886
Thanks. I don't think that the urgency is high enough to invest this much time into it for me right now.
I am happy to adapt once the implementation is ready. In the meantime I'll try to get the non-RBAC LDAP integration working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-707077118
Support for LDAP in FAB is not very good (see: https://github.com/dpgaspar/Flask-AppBuilder/issues/956) and I personally use Keycloak in similar projects. See: https://github.com/apache/airflow/issues/11305
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-706997168
I did not know that is until now but I guess the RBAC one since the other one is deprecated now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-707011806
When using RBAC, authentication with LDAP in the API is not supported. As of Airflow 1.10, only Kerberos authorization is supported.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-707076212
I never used non-RBAC UI because we dropped support for it. We have support for LDAP authorization in RBAC UI in the Airflow developer version, but this feature was never released. If you want to use it, you have to provide the necessary class on your own.
Here is docs:
https://airflow.readthedocs.io/en/latest/security/api.html#basic-authentication
https://airflow.readthedocs.io/en/latest/security/api.html#roll-your-own-api-authentication
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #11436:
URL: https://github.com/apache/airflow/issues/11436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-711081992
Success! 🎉
I built a new image using the following build args
```sh
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.6-slim-buster" \
--build-arg PYTHON_MAJOR_MINOR_VERSION="3.6" \
--build-arg AIRFLOW_EXTRAS="aws,azure,kubernetes,mysql,postgres,ssh,virtualenv" \
--build-arg ADDITIONAL_PYTHON_DEPS="python-ldap" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="unixodbc git vim" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc vim build-essential python3-dev python2.7-dev libldap2-dev libsasl2-dev slapd ldap-utils tox lcov valgrind" \
--build-arg ADDITIONAL_RUNTIME_ENV_VARS="ACCEPT_EULA=Y" \
```
In addition, I enabled LDAP auth in `webserver_config.py` following the instructions.
Last, here is my `docker-compose.yml`:
```sh
# adapted from https://github.com/apache/airflow/pull/8621
version: "3"
services:
postgres_airflow:
image: postgres
container_name: postgres_airflow
restart: always
environment:
POSTGRES_USER: "airflow"
POSTGRES_PASSWORD: "airflow"
POSTGRES_DB: "airflow"
POSTGRES_HOST: "postgres_airflow"
initdb_adduser:
image: my-airflow
container_name: airflow-adduser
depends_on:
- postgres_airflow
entrypoint: /bin/bash
command: -c 'airflow db init'
airflow-webserver:
image: my-airflow
container_name: airflow
restart: always
ports:
- "9000:8080"
command: webserver
volumes:
- ./:/logs
environment:
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres_airflow:5432/airflow
- AIRFLOW__CORE__EXECUTOR=LocalExecutor
- AIRFLOW__CORE__LOAD_EXAMPLES=False
- AIRFLOW__CORE__STORE_DAG_CODE=False
- AIRFLOW__CORE__STORE_SERIALIZED_DAGS=False
- AIRFLOW__CORE__DEFAULT_TIMEZONE=Europe/Berlin
- AIRFLOW__WEBSERVER__BASE_URL=<secret>
- AIRFLOW__WEBSERVER__AUTHENTICATE=True
- AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX=True
- AIRFLOW__SMTP__SMTP_HOST=<secret>
- AIRFLOW__SMTP__SMTP_USER=<secret>
- AIRFLOW__SMTP__SMTP_PASSWORD=<secret>
- AIRFLOW__SMTP__SMTP_PORT=587
- AIRFLOW__SMTP__SMTP_MAIL_FROM=<secret>
- AIRFLOW__LDAP__URI=ldap://<secret>
- AIRFLOW__LDAP__BIND_USER=<secret>
- AIRFLOW__LDAP__BIND_PASSWORD=<secret>
- AIRFLOW__LDAP__BASEDN=<secret>
```
I also tried Keycloak/openid-connect via `AUTH_TYPE = AUTH_OAUTH` but was not successful. Happy to switch once there is an official implementation one day.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-1005864066
Closing as seems to be 1.10 specific and resolved in Airflow 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710906464
Even with `AIRFLOW__WEBSERVER__RBAC=False` I get the same error in v1.10.x.
I did not try v1.9 since this version is too outdated.
Building and running a docker container from master failed (some pendulum issues).
Hence I think there is no way currently to use LDAP with airflow.
What confuses me somewhat is the fact that there is not real documentation on this in the docs.
Yes, there is this little sentence about
> This is for flask-admin based web UI only
but I think it needs more explanation. Users (like me) do not know that LDAP does not work at all with v1.10 and that even for trying, one needs to disable RBAC explicitly.
So I will not create users manually (😢 ) and hope for v2.0 to be released soon 🙂
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] singhalss commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
singhalss commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-724519933
I tried with AUTH_TYPE = AUTH_LDAP as per the documentation given but no luck. Also the logs are not giving any clue about the problem, is there anyway we can enable debug logs for FAB security manager to get more idea?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710925093
@pat-s I recommend to try Airflow 2.0 alpha 1. We have a Docker image ready for this version. See: https://lists.apache.org/thread.html/rf34558953ba367561574c194500a34d7f3c21fe2798b173b86fc309c%40%3Cdev.airflow.apache.org%3E
You can ask about a production Docker image on #prod-docker-image ([](https://s.apache.org/airflow-slack)) also
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-706993413
Do you want to use RBAC UI or non-RBAC UI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710918490
PS. Authorization with user accounts in the database also does not work properly in Airflow 1.10. Only Kerberos auth works in this version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710937172
Ah, promising! Will try it later.
Thanks for your quick replies!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-708285846
@mik-laj Keycloak via openid-connect would also be my favorite. Is there something you can share with respect to Keycloak?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pat-s commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
pat-s commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-707068292
Then I am happy to use the non-RBAC UI. This means when disabling RBAC, I should be able to login via LDAP in v1.10.12?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710915064
@pat-s You can use this module with Airflow 2.0 and it should work.
https://github.com/apache/airflow/blob/master/airflow/api/auth/backend/basic_auth.py
To load this module you have to copy it to `~/airflow/config` (See: https://airflow.readthedocs.io/en/latest/modules_management.html) and set up according to documentation: https://airflow.readthedocs.io/en/latest/security/api.html#basic-authentication
> but I think it needs more explanation. Users (like me) do not know that LDAP does not work at all with v1.10 and that even for trying, one needs to disable RBAC explicitly.
Documentation for Airflow 2.0 no longer describes the flask-admin UI, so everything is easier to understand. We are no longer developing Airflow 1.10, but only focusing on fixing critical bugs, including security, and working on improving the migration process to Airflow 2.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710915064
@pat-s You can use module from Airflow 2.0 and it should work.
https://github.com/apache/airflow/blob/master/airflow/api/auth/backend/basic_auth.py
To load this module you have to copy it to `~/airflow/config` (See: https://airflow.readthedocs.io/en/latest/modules_management.html) and set up according to documentation: https://airflow.readthedocs.io/en/latest/security/api.html#basic-authentication
> but I think it needs more explanation. Users (like me) do not know that LDAP does not work at all with v1.10 and that even for trying, one needs to disable RBAC explicitly.
Documentation for Airflow 2.0 no longer describes the flask-admin UI, so everything is easier to understand. We are no longer developing Airflow 1.10, but only focusing on fixing critical bugs, including security, and working on improving the migration process to Airflow 2.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #11436: LDAP: module 'airflow.contrib.auth.backends.ldap_auth' has no attribute 'CLIENT_AUTH'
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #11436:
URL: https://github.com/apache/airflow/issues/11436#issuecomment-710915064
@pat-s You can use module from Airflow 2.0 and it should work.
https://github.com/apache/airflow/blob/master/airflow/api/auth/backend/basic_auth.py
To use this backend you have to copy it to `~/airflow/config` (See: https://airflow.readthedocs.io/en/latest/modules_management.html) and set up according to documentation: https://airflow.readthedocs.io/en/latest/security/api.html#basic-authentication
> but I think it needs more explanation. Users (like me) do not know that LDAP does not work at all with v1.10 and that even for trying, one needs to disable RBAC explicitly.
Documentation for Airflow 2.0 no longer describes the flask-admin UI, so everything is easier to understand. We are no longer developing Airflow 1.10, but only focusing on fixing critical bugs, including security, and working on improving the migration process to Airflow 2.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org