You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/14 15:17:36 UTC
[GitHub] [airflow] Minyus opened a new issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Minyus opened a new issue #10330:
URL: https://github.com/apache/airflow/issues/10330
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
This questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 1.10.11
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**: All
- **OS** (e.g. from /etc/os-release): All
- **Kernel** (e.g. `uname -a`): All
- **Install tools**: All
- **Others**:
**What happened**:
<!-- (please include exact error messages if you can) -->
I'm trying to set up Google OAuth authentication for the web UI.
I found the document [google-authentication](https://airflow.apache.org/docs/stable/security.html#google-authentication) provides how to configure in `airflow.cfg` as follows, but I'm trying to configure using environment variables.
```
[webserver]
authenticate = True
auth_backend = airflow.contrib.auth.backends.google_auth
[google]
client_id = google_client_id
client_secret = google_client_secret
oauth_callback_route = /oauth2callback
domain = example1.com,example2.com
```
I referred to [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html), but I found the following 2 issues.
- Although "auth_backend" is listed under the section "webserver" in [google-authentication](https://airflow.apache.org/docs/stable/security.html#google-authentication), the environment variable name is "AIRFLOW__API__AUTH_BACKEND" in [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) . Is this section inconsistency ("WEBSERVER" or "API") between `airflow.cfg` and environment variable name expected?
- 4 keys ("client_id", "Client_secret", "oauth_callback_route", "domain") are listed under the section "google" in [google-authentication](https://airflow.apache.org/docs/stable/security.html#google-authentication), but I could not find "google" section in [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html).
**What you expected to happen**:
<!-- What do you think went wrong? -->
- All the available environment variable keys are correctly listed in [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html).
- The document clarifies if a key can be configured in `airflow.cfg` but cannot be configured as an environment variable.
**How to reproduce it**:
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
Minyus commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674588462
@mik-laj
Thanks for your suggestion.
Unfortunately, I couldn't find a Helm Chart tested with Airflow 2.0 right now, but I will consider in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674930130
BTW @Minyus -> if you see any places how the image documentation (especially the user-facing one) can be improved to be more clear and useful - feel free to make a PR or two. I created "good-first-issue" to update the documentation here: https://github.com/apache/airflow/issues/10363 and you are most welcome to help with that. I often find it difficult to write a good "user-facing" documentation as often many assumptions are obvious to me, but not at all to the users. So users that can help writing/improving user documentation are the best help ever!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
Minyus commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674391661
@alexbegg
Thanks for your clarification.
It seems the problem is [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) is incomplete thus confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674674326
@Minyus The helm chart in "charts" folder of master is tested with the production Dockerfile build image and it is tested with every PR :). You will find instructions in README how to use it . We are going to release it soon so this is the "target" helm chart for Airflow in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg edited a comment on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
alexbegg edited a comment on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674319661
@Minyus I just want to clarify some things. The same [Security](https://airflow.apache.org/docs/stable/security.html) documentation page explains both the `auth_backend` under [api] (which as of 1.10.11 defaults to `airflow.api.auth.backend.deny_all`) and the `auth_backend` under [webserver] (which does not seem to be included in the default config, I assume because by default `authenticate` is `False`).
The fact that both have the same key name can be confusing, but I am sure that both settings do different things. Setting your API `auth_backend` won't also set your webserver `auth_backend`, and vice-versa.
Also, not describing `auth_backend` under "webserver" on the [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) page or even the default configuration is confusing things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
Minyus commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-675530445
@potiuk
Thanks for your explanation.
I now know which image I should use and the roadmap of Airflow.
I'll continue to try Airflow and hope I can find some time to contribute in the future as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
alexbegg commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674319661
@Minyus I just want to clarify some things. The same [Security](https://airflow.apache.org/docs/stable/security.html) documentation page explains both the `auth_backend` under [api] (which as of 1.10.11 defaults to `airflow.api.auth.backend.deny_all`) and the `auth_backend` under [webserver] (which does not seem to be included in the default config, I assume because by default `authenticate` is `False`). That part is misleading, but I am sure that both settings do different things.
Setting your API `auth_backend` won't also set your webserver `auth_backend`, and vice-versa.
And not describing `auth_backend` under "webserver" on the [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) page is confusing things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674441498
I recommend enabling the RBAC UI. It has a completely different authorization setup. In Airflow 2.0, we plan to drop support for non-RBAC WEB UI.
Here is migration guide: https://github.com/apache/airflow/blob/master/UPDATING.md#drop-legacy-ui-in-favor-of-fab-rbac-ui
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674907479
All images starting with "master" are the 2.0 ones, and the one that you are looking for are the ones without the suffix (so master-python3.x). There are also images in v1-10-test- prefix (for work in progress on releasing stuff from 1.10 line) and last but not least 1.10.N- images (which are the ones that correspond to actual released versions - 1.10.10, 1.10.11 etc.). All those without a suffix are those that you should be interested in using.
You can see all the details about naming convention and description of the different images we use here: https://github.com/apache/airflow/blob/master/IMAGES.rst
It's rather comprehensive description of all the types of images we are using but it starts with naming convention and explanation of what the images are.
Also if you want to customize or extend the image I can heartily recommend my talk from the Airflow Summit where I talk about the Production Image, how and why it is done like it is, and how you can extend and customize it: https://youtu.be/wDr3Y7q2XoI and the IMAGES.rst contains plenty of examples and reference information on customizable parameters for the image.
Regarding the Helm chart - it was donated by Astronomer and while it still contains some references to Astronomer, they are different than the original Astronomer's chart (those references will be removed and replaced with the target repository once we release it).
For now this is really a development version yet, not yet released in any way, so there is no "repository" yet where it can be installed from. It really requires you to check out the repository and install it from the sources (as described in the README). I hope we can get there in a few months - once we release 1.10.12 we will likely put more effort into comprehensive and automated testing of the helm chart, to make sure that it is good before we release it officially. Hopefully, it will be released for the (hopefully) last 1.10.13 version from the 1.10 line, but for sure it will be there for 2.0 release from the very beginning.
I hope it is helpful information :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674907479
All images starting with "master" are the 2.0 ones, and the one that you are looking for are the ones without the suffix (so master-python3.x). There are also images in v1-10-test- prefix (for work in progress on releasing stuff from 1.10 line) and last but not least 1.10.N- images (which are the ones that correspond to actual released versions - 1.10.10, 1.10.11 etc.). All those without a suffix are those that you should be interested in using.
You can see all the details about naming convention and description of the different images we use here: https://github.com/apache/airflow/blob/master/IMAGES.rst
It's rather comprehensive description of all the types of images we are using but it starts with naming convention and explanation of what the images are.
Also if you want to customize or extend the image I can heartily recommend my talk from the Airflow Summit where I talk about the Production Image, how and why it is done like it is, and how you can extend and customize it: https://youtu.be/wDr3Y7q2XoI and the IMAGES.rst contains plenty of examples and reference information on customizable parameters for the image.
Regarding the Helm chart - it was donated by Astronomer and while it still contains some references to Astronomer, they are different than the original Astronomer's chart (those references will be removed and replaced with the target repository once we release it).
For now this is really a development version, not yet released in any way, so there is no "repository" where it can be installed from. It really requires you to check out the repository and install it from the sources (as described in the README). I hope we can get there in a few months - once we release 1.10.12 we will likely put more effort into comprehensive and automated testing of the helm chart, to make sure that it is good before we release it officially. Hopefully, it will be released for the (hopefully) last 1.10.13 version from the 1.10 line, but for sure it will be there for 2.0 release from the very beginning.
I hope it is helpful information :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
Minyus commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674896623
@potiuk
Thanks for sharing updates about the Helm Chart.
1. I checked the Docker Hub and I see severals images that start with "master" at [apache/airflow](https://hub.docker.com/r/apache/airflow/tags?page=1&name=master).
- master-python3.x
- master-python3.x-build
- master-python3.x-ci
- master-python3.x-ci-manifest
Are these all Airflow 2.0dev?
Which Docker images are used to test with the Helm Chart?
2. I checked the [chart README](https://github.com/apache/airflow/tree/master/chart)
and found that it mentions 2 helm repositories, `stable` (https://kubernetes-charts.storage.googleapis.com) and `astronomer` (https://helm.astronomer.io).
Which Airflow Chart is tested with Airflow 2.0dev?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-812854284
Documentation for Airflow 2.0 includes documentation for all environmental variables:
http://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html
http://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#environment-variables
I don't think we can improve anything. The documentation for Airflow 1.10 does not clearly describe the differences between RBAC and non-RBAC interfaces, but we are unlikely to fix that as this release only gets bug fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Minyus edited a comment on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
Minyus edited a comment on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674391661
@alexbegg
Thanks for your clarification.
It seems the problem is that [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) is incomplete thus confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-812587318
@potiuk is there a task to complete here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg edited a comment on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
alexbegg edited a comment on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674319661
@Minyus I just want to clarify some things. The same [Security](https://airflow.apache.org/docs/stable/security.html) documentation page explains both the `auth_backend` under [api] (which as of 1.10.11 defaults to `airflow.api.auth.backend.deny_all`) and the `auth_backend` under [webserver] (which does not seem to be included in the default config, I assume because by default `authenticate` is `False`).
The fact that both have the same key name can be confusing, but I am sure that both settings do different things. Setting your API `auth_backend` won't also set your webserver `auth_backend`, and vice-versa.
Also, not describing `auth_backend` under "webserver" on the [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) page is confusing things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674441757
There is very little chance that an active contributor will fix the Airflow 1.10 documentation errors as we are focused on Airflow 2.0 development. However, I will gladly review the proposed changes if someone submits a PR to Airflow 1.10 documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexbegg edited a comment on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
alexbegg edited a comment on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674319661
@Minyus I just want to clarify some things. The same [Security](https://airflow.apache.org/docs/stable/security.html) documentation page explains both the `auth_backend` under [api] (which as of 1.10.11 defaults to `airflow.api.auth.backend.deny_all`) and the `auth_backend` under [webserver] (which does not seem to be included in the default config, I assume because by default `authenticate` is `False`). That fact that both have the same key name can be confusing, but I am sure that both settings do different things.
Setting your API `auth_backend` won't also set your webserver `auth_backend`, and vice-versa.
And not describing `auth_backend` under "webserver" on the [Configuration reference](https://airflow.apache.org/docs/stable/configurations-ref.html) page is confusing things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #10330:
URL: https://github.com/apache/airflow/issues/10330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10330: [Documentation] Section inconsistency between airflow.cfg and environment variable names
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10330:
URL: https://github.com/apache/airflow/issues/10330#issuecomment-674124621
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org