You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/23 12:20:47 UTC
[GitHub] [airflow] ralleman-quasarsat opened a new issue #19771: Unable to get external postgresql working
ralleman-quasarsat opened a new issue #19771:
URL: https://github.com/apache/airflow/issues/19771
### Official Helm Chart version
1.3.0 (latest released)
### Apache Airflow version
2.2.2 (latest released)
### Kubernetes Version
Latest AKS
### Helm Chart configuration
Values file (airflow-values-deploy.yaml):
```
postgresql:
enabled: false
externalDatabase:
type: postgres
host: postgresql-postgresql-ha-pgpool
port: 5432
database: airflow
user: airflow
passwordSecret: airflow
passwordSecretKey: airflow-pwd
# use this for any extra connection-string settings, e.g. ?sslmode=disable
properties: ""
```
Azure DevOps pipeline step:
```
- task: HelmDeploy@0
displayName: 'Helm deploy Airflow'
inputs:
connectionType: 'Azure Resource Manager'
azureSubscription: xxx-gs-dev-rg-connection
azureResourceGroup: xxx-gs-dev-rg
kubernetesCluster: xxx-gs-dev-app-kube
namespace: airflow
command: upgrade
releaseName: airflow
chartName: apache-airflow/airflow
arguments: --debug
waitForExecution: false
valueFile: airflow-values-deploy.yaml
```
### Docker Image customisations
None
### What happened
The helm step fails with a timeout. Step log:
```
2021-11-23T05:53:50.9141556Z ##[section]Starting: Helm deploy Airflow
2021-11-23T05:53:50.9151048Z ==============================================================================
2021-11-23T05:53:50.9151430Z Task : Package and deploy Helm charts
2021-11-23T05:53:50.9151856Z Description : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running helm commands
2021-11-23T05:53:50.9152230Z Version : 0.183.0
2021-11-23T05:53:50.9152500Z Author : Microsoft Corporation
2021-11-23T05:53:50.9152816Z Help : https://aka.ms/azpipes-helm-tsg
2021-11-23T05:53:50.9153158Z ==============================================================================
2021-11-23T05:53:52.0554403Z [command]/usr/local/bin/helm upgrade --namespace airflow --install --values /home/vsts/work/1/s/airflow-values-deploy.yaml --debug airflow apache-airflow/airflow
2021-11-23T05:58:59.3606037Z history.go:56: [debug] getting history for release airflow
2021-11-23T05:58:59.3606666Z Release "airflow" does not exist. Installing it now.
2021-11-23T05:58:59.3613687Z install.go:178: [debug] Original chart version: ""
2021-11-23T05:58:59.3619896Z install.go:199: [debug] CHART PATH: /home/vsts/.cache/helm/repository/airflow-1.3.0.tgz
2021-11-23T05:58:59.3638133Z
2021-11-23T05:58:59.3640714Z client.go:299: [debug] Starting delete for "airflow-broker-url" Secret
2021-11-23T05:58:59.3653708Z client.go:328: [debug] secrets "airflow-broker-url" not found
2021-11-23T05:58:59.3654284Z client.go:128: [debug] creating 1 resource(s)
2021-11-23T05:58:59.3655016Z client.go:299: [debug] Starting delete for "airflow-fernet-key" Secret
2021-11-23T05:58:59.3655769Z client.go:328: [debug] secrets "airflow-fernet-key" not found
2021-11-23T05:58:59.3656305Z client.go:128: [debug] creating 1 resource(s)
2021-11-23T05:58:59.3657358Z client.go:299: [debug] Starting delete for "airflow-redis-password" Secret
2021-11-23T05:58:59.3658136Z client.go:328: [debug] secrets "airflow-redis-password" not found
2021-11-23T05:58:59.3658683Z client.go:128: [debug] creating 1 resource(s)
2021-11-23T05:58:59.3659220Z client.go:128: [debug] creating 29 resource(s)
2021-11-23T05:58:59.3659957Z client.go:299: [debug] Starting delete for "airflow-run-airflow-migrations" Job
2021-11-23T05:58:59.3660748Z client.go:328: [debug] jobs.batch "airflow-run-airflow-migrations" not found
2021-11-23T05:58:59.3661324Z client.go:128: [debug] creating 1 resource(s)
2021-11-23T05:58:59.3662104Z client.go:528: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 5m0s
2021-11-23T05:58:59.3663025Z client.go:556: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
2021-11-23T05:58:59.3663948Z client.go:595: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
2021-11-23T05:58:59.3664870Z client.go:556: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
2021-11-23T05:58:59.3665795Z client.go:595: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
2021-11-23T05:58:59.3666631Z Error: failed post-install: timed out waiting for the condition
2021-11-23T05:58:59.3667417Z helm.go:88: [debug] failed post-install: timed out waiting for the condition
2021-11-23T05:58:59.3712123Z ##[error]history.go:56: [debug] getting history for release airflow
install.go:178: [debug] Original chart version: ""
install.go:199: [debug] CHART PATH: /home/vsts/.cache/helm/repository/airflow-1.3.0.tgz
client.go:299: [debug] Starting delete for "airflow-broker-url" Secret
client.go:328: [debug] secrets "airflow-broker-url" not found
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "airflow-fernet-key" Secret
client.go:328: [debug] secrets "airflow-fernet-key" not found
client.go:128: [debug] creating 1 resource(s)
client.go:299: [debug] Starting delete for "airflow-redis-password" Secret
client.go:328: [debug] secrets "airflow-redis-password" not found
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 29 resource(s)
client.go:299: [debug] Starting delete for "airflow-run-airflow-migrations" Job
client.go:328: [debug] jobs.batch "airflow-run-airflow-migrations" not found
client.go:128: [debug] creating 1 resource(s)
client.go:528: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 5m0s
client.go:556: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
client.go:595: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:556: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
client.go:595: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: failed post-install: timed out waiting for the condition
helm.go:88: [debug] failed post-install: timed out waiting for the condition
2021-11-23T05:58:59.3729738Z ##[section]Finishing: Helm deploy Airflow
```
airflow-worker-0 log:
```
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 96, in check
db.check()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 940, in check
session.execute('select 1 as is_alive;')
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1295, in execute
return self._connection_for_bind(bind, close_with_result=True).execute(
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1151, in _connection_for_bind
engine, execution_options
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2340, in _wrap_pool_connect
e, dialect, self
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1584, in _handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "airflow-postgresql.airflow" to address: Name or service not known
```
Note the error at the end **sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "airflow-postgresql.airflow" to address: Name or service not known**
### What you expected to happen
It should connect to the external postgresql database. I have no idea why it's trying to connect to the host "airflow-postgresql.airflow" and I don't know where that comes from.
Notes:
1. We use postgresql-ha from bitnami using pgpool.
2. I can log into the postgresql server using the airflow user. No issues logging in.
3. The airflow database has not been initialised at the time the Helm deploy task runs in the Azure pipeline.
4. The airflow secret is working and is visible in the AKS secrets under the airflow namespace.
### How to reproduce
Using an external postgresql-ha database cluster, run the Helm deploy as described above.
### Anything else
I found this failed secret in the kubernetes secrets under the airflow namespace. All other secrets are fine.
```
{
"apiVersion": "v1",
"data": {
"release": "SDRzSUFBQ..." <-- this is huge
},
"kind": "Secret",
"metadata": {
"creationTimestamp": "2021-11-23T05:53:58Z",
"labels": {
"modifiedAt": "1637647139",
"name": "airflow",
"owner": "helm",
"status": "failed",
"version": "1"
},
"name": "sh.helm.release.v1.airflow.v1",
"namespace": "airflow",
"resourceVersion": "2207212",
"uid": "42e20a40-b2c8-40b4-b178-a53d0497ee09"
},
"type": "helm.sh/release.v1"
}
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-976487065
> Please don't close tickets before I've had a chance to respond.
We can always reopen the issue when you respond, but when it's clearly wrong, closing it good for community so that invalid requests are not looked at and do not take time of maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ralleman-quasarsat commented on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
ralleman-quasarsat commented on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-977492405
Thanks for the help on this. I was not aware that there were 2 repos/projects. It's working now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ralleman-quasarsat commented on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
ralleman-quasarsat commented on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-976475548
Please don't close tickets before I've had a chance to respond.
The configuration is defined here: https://github.com/airflow-helm/charts/tree/main/charts/airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-976483880
Plese open your issue where it belongs. The Helm Chart you are referring to is not maintained by the Airlfow Community so issues should be raised there (you mentioned 1.3.0 Airflow Chart above which is not the one you are using.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-976472448
Where did you get the `externalDatabase` configuration from ? I am scratching my head, because it's nowhere to be found in the code of netiher Airflow nor Helm Chart. The right way to configure the external database is here: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#database
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-976472448
Where did you get teh `externalPostgresDatabase` configuration from ? I am scratching my head, because it's nowhere to be found in the code of netiher Airflow nor Helm Chart. The right way to configure the external database is here: ghttps://airflow.apache.org/docs/helm-chart/stable/production-guide.html#database
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19771:
URL: https://github.com/apache/airflow/issues/19771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19771: Unable to get external postgresql working
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #19771:
URL: https://github.com/apache/airflow/issues/19771#issuecomment-976472448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org