You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pschoenmakers-ah (via GitHub)" <gi...@apache.org> on 2023/02/21 08:00:45 UTC
[GitHub] [airflow] pschoenmakers-ah opened a new issue, #29650: Post installation upgrades fail for postgres
pschoenmakers-ah opened a new issue, #29650:
URL: https://github.com/apache/airflow/issues/29650
### Official Helm Chart version
1.8.0 (latest released)
### Apache Airflow version
2.5.1
### Kubernetes Version
1.23.16 (rancher-desktop on windows)
### Helm Chart configuration
After succesful installation of the chart I create a "values" file to add the webserver key as per official documentation. No further changes.
`helm show values apache-airflow/airflow > airflow.yaml`
### Docker Image customizations
No customisation
### What happened
After generating the "airflow.yaml" file and adding the webserver secret key the upgrade fails
`helm upgrade --install airflow apache-airflow/airflow -n airflow -f airflow.yaml --debug`
Gives output
```
history.go:56: [debug] getting history for release airflow
upgrade.go:142: [debug] preparing upgrade for airflow
Error: UPGRADE FAILED: execution error at (airflow/charts/postgresql/templates/secrets.yaml:20:15):
PASSWORDS ERROR: The secret "airflow-postgresql" does not contain the key "password"
helm.go:84: [debug] execution error at (airflow/charts/postgresql/templates/secrets.yaml:20:15):
PASSWORDS ERROR: The secret "airflow-postgresql" does not contain the key "password"
UPGRADE FAILED
main.newUpgradeCmd.func2
helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.5.0/command.go:918
main.main
helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
runtime/proc.go:250
runtime.goexit
runtime/asm_amd64.s:1571
```
When inspecting the secret the key "password" indeed is not there. Instead it has "postgres-password"
To attempt to let the upgrade continue I added an extra key "password" in the secret with the same value.
The upgrade now indeed proceeds but gets stuck on the migrations step
`client.go:540: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 5m0s
client.go:568: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
client.go:607: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
client.go:607: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0`
After some time the upgrade fails on a timeout.
```
upgrade.go:434: [debug] warning: Upgrade "airflow" failed: post-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: post-upgrade hooks failed: timed out waiting for the condition
helm.go:84: [debug] post-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
helm.sh/helm/v3/cmd/helm/upgrade.go:201
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.5.0/command.go:872
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.5.0/command.go:990
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.5.0/command.go:918
main.main
helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
runtime/proc.go:250
runtime.goexit
runtime/asm_amd64.s:1571
```
### What you think should happen instead
As with the 1.7.0 version of the chart I expect the webserver key to be applied without error messages
### How to reproduce
- Install rancher-desktop
- Make sure it has dockerd container engine and kubernetes 1.23.16 activated
- Install chart 1.8.0 as per https://airflow.apache.org/docs/helm-chart/stable/index
- create a values file and add a webserverSecretKey as per documentation
- Attempt an upgrade using the adjusted values file
### Anything else
I tried to install chart 1.7.0 by adding --version 1.7.0 but that also fails on the postgres secret, where the keys are now different. When adding the missing ones in this attempt an error gets thrown that postgres cannot be upgraded at all.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29650: Post installation upgrades fail for postgres
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29650:
URL: https://github.com/apache/airflow/issues/29650#issuecomment-1438019362
This ie expected - se release notes. Built-in postgres is "development" feature only and you should not rely on it as working without breaking changes. You need to fix things manually: https://airflow.apache.org/docs/helm-chart/stable/release_notes.html#airflow-helm-chart-1-8-0-2023-02-06
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29650: Post installation upgrades fail for postgres
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29650:
URL: https://github.com/apache/airflow/issues/29650#issuecomment-1438004572
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29650: Post installation upgrades fail for postgres
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #29650:
URL: https://github.com/apache/airflow/issues/29650#issuecomment-1438474529
> This was indeed the root cause but it was not really clear how to resolve. In the end I got it going like this:
Yep. That's a bit the pain with development settings, they might need sometimes manual tweaking, and sometimes even wiping things out and recreating from the scratch. Upgrading from previous "development" setting painlessly is not a goal for us and sometimes we might defer to manual process for users who attempted to use the development setting and want to keep the data (despite our recommendation is to use externally managed db).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #29650: Post installation upgrades fail for postgres
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #29650: Post installation upgrades fail for postgres
URL: https://github.com/apache/airflow/issues/29650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pschoenmakers-ah commented on issue #29650: Post installation upgrades fail for postgres
Posted by "pschoenmakers-ah (via GitHub)" <gi...@apache.org>.
pschoenmakers-ah commented on issue #29650:
URL: https://github.com/apache/airflow/issues/29650#issuecomment-1438429597
This was indeed the root cause but it was not really clear how to resolve. In the end I got it going like this:
- create a values file
- do a --dry-run install this indeed points to a password error.
`helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --dry-run`
- Edit the values.yaml replace the "~" for username: and password: with "postgres"
- add an extra key "password" to the airflow-postgresql secret with value "postgres"
- rerun the upgrade with the values.yaml file
This brought up all the pods correctly and allowed me to change the webserverSecretKey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org