You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/18 18:33:26 UTC
[GitHub] [airflow] luancaarvalho opened a new issue #19688: Airflow does not load default connections
luancaarvalho opened a new issue #19688:
URL: https://github.com/apache/airflow/issues/19688
### Apache Airflow version
2.2.1
### Operating System
linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
deploy with helm in EKS.
### What happened
When I deployed my airflow via helm, I wanted to use the aws_default connection in my dags, but passing this connection airflow logs that this connection does not exists. So I looked in airflow UI and in postgres connection table and this connection did not exist.
After that I checked the AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS env var and it was equals true.
### What you expected to happen
I expected that the default connections was created with the initial deploy.
### How to reproduce
You just need to deploy the airflow via helm chart and check that value of the AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS env var, goes to the connection table in the database and check that any connection was created.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974010264
> I remember your message, but I test to run db init and even with that my default connections are not created.
> And in another post I was advised to create this issue, because that is really a bug, please check it.
> https://apache-airflow.slack.com/archives/C027H098M1C/p1636638256018800
This is not helm's problem/job. the `db init` that you were advised to, should work just fine but it can (and should be) done outside of the helm chart and what you observed is not an issue with the helm chart, it's a problem on how you run your `airflow db init`. IMHO (see above) running `airlfow db init` and creating the default connections is outside of the scope of helm-chart completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974056084
To summarize - the scenario where you want to `play` or `develop` with Airflow and get "development" and "quick start" data it - chooing Helm Chart is a wrong choice. I do not think we should ever complicate Helm Chart to include `play` and `development` options to it where we already have different ways how our users can do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974200577
:heart: it @dstandish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-973358312
@kaxil @ephraimbuddy @jedcunningham if i remember correctly, `airflow db upgrade` does not load connections.
so i think that's what's going on here. the `migrate-database-job` runs upgrade and not init.
personally i think that it's reasonable for the helm chart to not load the default connections. those are intended as samples, but the heml chart is more production-oriented. otherwise i don't see very good solutions.
one would be to load connections on upgrade. but this would be a change in behavior and folks rely on the fact that upgrade does not load default connections.
the other would be to have the chart try to run `init` on first install, which i assume is possible with k8s. but it is complexity i'm not sure we really need.
lmkyt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974005030
@kaxil @dstandish - I think we should have a common conclusion. Do you think it is a bug? - I do not. I think "loading default connections" is the same class as "creating users". For Helm Chart this is not in scope and it should not work.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974047616
@ashb I agree with you that, by default, airflow helm chart should not create the default connections, but if I set the AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS env var to true he should create these connections, right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974053214
> @potiuk I know this is not a helm's problem, when I mentioned db init, I'm excluding all considerations about a problem with a helm chart, because I runned db init outside of helm was after finished the deploy, it's because of that I created a issue like airflow bug and not a helm chart bug.
The `airflow db init` works in the latest (and 2.2.1) release if run properly, with the right DB. I just re-checked 2.2.1 and it works as expected:
```
./breeze start-airflow --use-airflow-version 2.2.1 --load-default-connections
```
So you probably run `airflow db init` for a different database or did not set the variable when run it. Please double check it.
> Besides, I still thinking that in my first helm chart deploy, so it's not an upgrade it's my initial release, when I set the variable AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS to true I expected to see the default connections created in my connection table in postgres and this is not happening.
How this is not a bug ?
It's not. As mentioned already several times ```AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS``` is a development setting:
Via: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#load-default-connections
> Whether to load the default connections that ship with Airflow. It’s good to get started, but you probably want to set this to False in a production environment
Helm chart is our production deployment, where you should manage your data on your own. Helm is inteded to be used as production deployment so this setting should be (and is) not effective.
If you want to `play` with the default settings or get `quick start` with airflow with default connections - you can easiily use https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html - this is our "play" and "development" setting, not the Helm chart. And there we use `airflow db init` (there is even separate step to run which is called "initialize the database": https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initialize-the-database) that will create the default connections if you set this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974047616
@ashb I agree with you that, by default, airflow helm chart should not create the default connections, but if I set the AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS env var to true it should create these connections, right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974224583
Well - just to clarify - we still do not want to make it "built-in" to create connections @luancaarvalho. This will remain your work to create them in the database if you use chart. The "demo" is merely about renaming this section so that people will not think they can use it in the Helm Chart.
So for you nothing changes. You still should make sure that you add the right connections (those that you need) in the database (or secret storages) - independently from `helm install` command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974190683
@potiuk hey i have an idea.
We can create a `demo` section in config.
Then:
```
AIRFLOW__DEMO__LOAD_DEFAULT_CONNECTIONS
AIRFLOW__DEMO__LOAD_EXAMPLE_DAGS
```
And then maybe users would be more accepting of `demo` settings being disabled for certain scenarios.
As it is, from user perspective, maybe since it's a "core" setting, there is the expectation that it should always work.
But there is a negative to this idea because i'll bet sometimes people like example dags in the helm chart perhaps for testing or "demoing" reasons.
On the other hand maybe by collecting these under a demo section, we could more cleanly enable / disable even in helm chart.
It _is_ maybe a bit odd to "elevate" these kinds of settings to the `core` section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974005030
@kaxil @dstandish - I think we should have a common conclusion. do you think it is a bug - I do not. I think "loading default connections" is the same class as "creating users". For Helm Chart this is not in scope and it should not work.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974008306
Some more context: We had a long discussion with Matthew in the past on why it makes no sense to have it - basically if we do want to make it works, we need to get "terraform-like" desired state described. I.e. when we install a new version of chart, we should describe what is the "final" state of the DB configuration we want to achieve, rather than what actions to execute.
Otherwise we fall into all the kind of problems on how to merge changes that were manually done in the database - answering questions like "should we keeep existing values, or override them".
For me this is something that is a "terraform" (or other kind of deployment you do) and not "helm" job - and helm should only really create and update structure of the DB but never insert/modify data in the DB. at least that was my opinion and result of the discussion so far :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974053214
> @potiuk I know this is not a helm's problem, when I mentioned db init, I'm excluding all considerations about a problem with a helm chart, because I runned db init outside of helm was after finished the deploy, it's because of that I created a issue like airflow bug and not a helm chart bug.
The `airflow db init` works in the latest release if run properly, with the right DB. I just re-checked 2.2.1 and it works as expected:
```
./breeze start-airflow --use-airflow-version 2.2.1 --load-default-connections
```
So you probably run `airflow db init` for a different database or did not set the variable when run it. Please double check it.
> Besides, I still thinking that in my first helm chart deploy, so it's not an upgrade it's my initial release, when I set the variable AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS to true I expected to see the default connections created in my connection table in postgres and this is not happening.
How this is not a bug ?
It's not. As mentioned already several times ```AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS``` is a development setting:
Via: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#load-default-connections
| Whether to load the default connections that ship with Airflow. It’s good to get started, but you probably want to set this to False in a production environment
Where Helm chart is production deployment, where you should managed your data on your own. Helm is inteded to be used as production deployment so this setting should be (and is) not effective.
If you want to `play` with the default settings and get "quick start" with airflow with default connections - you can easiily use https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html - this is our "play" and "development" setting, not the Helm chart. And there we use `airflow db init` (there is even separate step to run which is called "initialize the database": https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initialize-the-database) that will create the default connections if you set this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974031485
> IMHO (see above) running airlfow db init and creating the default connections is outside of the scope of helm-chart completely.
I agree with that -- Helm chart, _should not_ create sample connections..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974206760
@potiuk @dstandish The ideas of the update in the docs and the additon of demo section are awesome ! Certainly both will help a lot, because even with helm existing for production enviroments, it's good to test the way of my production deployment in my development enviroment.
Besides that, with the demo section and the creation of default connections, for people like me that uses IAM instance profile to our connections have this connections created will help to speed up the development phase a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974002574
Hello @potiuk !
I remember your message, but I test to run db init and even with that my default connections are not created.
And in another post I was advised to create this issue, because that is really a bug, please check it.
https://apache-airflow.slack.com/archives/C027H098M1C/p1636638256018800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974002574
Hello @potiuk !
I remember your message, but I test to run db init and even with that my default connections are not created.
And in another post I was advised to create this issue, because that is really a bud, please check it.
https://apache-airflow.slack.com/archives/C027H098M1C/p1636638256018800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-973146163
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-973147184
Tagging you guys.
@kaxil @dstandish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-973309647
@luancaarvalho can you document how you set `AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS`? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974044926
@potiuk I know this is not a helm's problem, when I mentioned db init, I'm excluding all considerations about a problem with a helm chart, because I runned db init outside of helm was after finished the deploy, it's because of that I created a issue like airflow bug and not a helm chart bug.
Besides, I still thinking that in my first helm chart deploy, so it's not an upgrade it's my initial release, when I set the variable AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS to true I expected to see the default connections created in my connection table in postgres and this is not happening.
How this is not a bug ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] luancaarvalho commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
luancaarvalho commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974004865
@dstandish I set AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS to true in two ways.
First, helm install $RELEASE_NAME apache-airflow/airflow \
--namespace $NAMESPACE \
--set 'env[0].name=AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS,env[0].value=True'
Second, I created a values.yaml and in the extraENV propertie I defined the valeu to True
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974053214
> @potiuk I know this is not a helm's problem, when I mentioned db init, I'm excluding all considerations about a problem with a helm chart, because I runned db init outside of helm was after finished the deploy, it's because of that I created a issue like airflow bug and not a helm chart bug.
The `airflow db init` works in the latest (and 2.2.1) release if run properly, with the right DB. I just re-checked 2.2.1 and it works as expected:
```
./breeze start-airflow --use-airflow-version 2.2.1 --load-default-connections
```
So you probably run `airflow db init` for a different database or did not set the variable when run it. Please double check it.
> Besides, I still thinking that in my first helm chart deploy, so it's not an upgrade it's my initial release, when I set the variable AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS to true I expected to see the default connections created in my connection table in postgres and this is not happening.
How this is not a bug ?
It's not. As mentioned already several times ```AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS``` is a development setting:
Via: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#load-default-connections
| Whether to load the default connections that ship with Airflow. It’s good to get started, but you probably want to set this to False in a production environment
Where Helm chart is production deployment, where you should managed your data on your own. Helm is inteded to be used as production deployment so this setting should be (and is) not effective.
If you want to `play` with the default settings and get "quick start" with airflow with default connections - you can easiily use https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html - this is our "play" and "development" setting, not the Helm chart. And there we use `airflow db init` (there is even separate step to run which is called "initialize the database": https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initialize-the-database) that will create the default connections if you set this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974056084
To summarize - for the scenario where you want to `play` or `develop` with Airflow and get "development" and "quick start" with it, choosing Helm Chart is a wrong choice. I do not think we should ever complicate Helm Chart to include `play` and `development` options to it where we already have different ways how our users can do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974224583
Well - just to clarify - we still do not want to make it "built-in" to create connections via chart deployment @luancaarvalho. This will remain your work to create them in the database if you use chart. The "demo" is merely about renaming this section so that people will not think they can use it in the Helm Chart.
So for you nothing changes. You still should make sure that you add the right connections (those that you need) in the database (or secret storages) - independently from `helm install` command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974224583
Well - just to clarify - we still do not want to make it "built-in" to create connections @luancaarvalho. This will remain your work to create them in the database if you use chart. The "demo" is merely about renaming this section so that people will not think they can use it in the Helm Chart.
So for you nothing changes. You still should make sure that you have the right connections (those that you need) in the database (or secret storages) - independently from `helm install` command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974073515
@dstandish @kaxil @ashb I proposed update to the chart's configuration docs that will explicitly document this behaviour. I think the fact that @luancaarvalho was confused about it means that we should be pretty explicit about it (unless we come to conclusion of course the we want to support some form of it in our Helm - but if we agree on that, I think it would have to be a completely different thing based on importing exported connection MAYBE - we discussed it several times already without conclusions I think) https://github.com/apache/airflow/pull/19708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974190683
@potiuk hey i have an idea.
We can create a `demo` section in config.
Then:
```
AIRFLOW__DEMO__LOAD_DEFAULT_CONNECTIONS
AIRFLOW__DEMO__LOAD_EXAMPLE_DAGS
```
And then maybe users would be more accepting of `demo` settings being disabled for certain scenarios.
As it is, from user perspective, maybe since it's a "core" setting, there is the expectation that it should always work.
But there is a negative to this idea because i'll bet sometimes people like example dags in the helm chart perhaps for testing or "demoing" reasons.
On the other hand maybe by collecting these under a demo section, we could more cleanly enable / disable even in helm chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-973465341
@dstandish is completely right.
I already explained that to you ~ week ago @luancaarvalho https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1636638536256300
If you really want to add connections you need to run `db init` and it's not going to happen via helm chart. Helm chart is there to install application, not to load data to the database. When you install a new version of application you are not supposed to "reload alll the connections". You are supposed to add them manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19688:
URL: https://github.com/apache/airflow/issues/19688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974190683
@potiuk hey i have an idea.
We can create a `demo` section in config.
Then:
```
AIRFLOW__DEMO__LOAD_DEFAULT_CONNECTIONS
AIRFLOW__DEMO__LOAD_EXAMPLE_DAGS
```
And then maybe users would be more accepting of `demo` settings being disabled for certain scenarios.
As it is, from user perspective, maybe since it's a "core" setting, there is the expectation that it should always work.
But there is a negative to this idea because i'll bet sometimes people like example dags in the helm chart perhaps for testing or "demoing" reasons.
On the other hand maybe by collecting these under a demo section, we could more cleanly enable / disable even in helm chart.
It is maybe a bit odd to "elevate" these kinds of settings to the `core` section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974053214
> @potiuk I know this is not a helm's problem, when I mentioned db init, I'm excluding all considerations about a problem with a helm chart, because I runned db init outside of helm was after finished the deploy, it's because of that I created a issue like airflow bug and not a helm chart bug.
The `airflow db init` works in the latest (and 2.2.1) release if run properly, with the right DB. I just re-checked 2.2.1 and it works as expected:
```
./breeze start-airflow --use-airflow-version 2.2.1 --load-default-connections
```
So you probably run `airflow db init` for a different database or did not set the variable when run it. Please double check it.
> Besides, I still thinking that in my first helm chart deploy, so it's not an upgrade it's my initial release, when I set the variable AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS to true I expected to see the default connections created in my connection table in postgres and this is not happening.
How this is not a bug ?
It's not. As mentioned already several times ```AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS``` is a development setting:
Via: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#load-default-connections
> Whether to load the default connections that ship with Airflow. It’s good to get started, but you probably want to set this to False in a production environment
Where Helm chart is production deployment, where you should managed your data on your own. Helm is inteded to be used as production deployment so this setting should be (and is) not effective.
If you want to `play` with the default settings and get "quick start" with airflow with default connections - you can easiily use https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html - this is our "play" and "development" setting, not the Helm chart. And there we use `airflow db init` (there is even separate step to run which is called "initialize the database": https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#initialize-the-database) that will create the default connections if you set this variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19688: Airflow does not load default connections
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19688:
URL: https://github.com/apache/airflow/issues/19688#issuecomment-974053214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org