You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "V0lantis (via GitHub)" <gi...@apache.org> on 2023/02/21 11:10:41 UTC
[GitHub] [airflow] V0lantis opened a new issue, #29654: Not able to specify redis as a result backend
V0lantis opened a new issue, #29654:
URL: https://github.com/apache/airflow/issues/29654
### Official Helm Chart version
1.8.0 (latest released)
### Apache Airflow version
2.4.3
### Kubernetes Version
v1.21.14-eks
### Helm Chart configuration
```
data:
resultBackendConnection:
protocol: redis
host: my.host.io
port: 6379
db: 1
```
### Docker Image customizations
_No response_
### What happened
```
Error: Failed to render chart: exit status 1: Error: values don't meet the specifications of the schema(s) in the following chart(s):
airflow:
- data.resultBackendConnection: pass is required
- data.resultBackendConnection: sslmode is required
- data.resultBackendConnection: Additional property password is not allowed
Error: plugin "diff" exited with error
```
### What you think should happen instead
As specified in Celery [documentation](https://docs.celeryq.dev/en/latest/userguide/configuration.html#redis-backend-settings), I shouldn't have any issue just passing a Url like
```
config:
celery:
result_backend: "redis://my.host.io:6379/1"
```
Although when I am doing so, I have error in my DagProcessor:
```
Liveness probe failed: You have configured a result_backend of redis://my.host.io:6379/1, it is highly recommended to use an alternative result_backend (i.e. a database).
No alive jobs found.
````
### How to reproduce
Just copy past my configuration
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #29654: Not able to specify redis as a result backend
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29654:
URL: https://github.com/apache/airflow/issues/29654#issuecomment-1484286052
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #29654: Not able to specify redis as a result backend
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29654:
URL: https://github.com/apache/airflow/issues/29654#issuecomment-1495157040
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #29654: Not able to specify redis as a result backend
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #29654: Not able to specify redis as a result backend
URL: https://github.com/apache/airflow/issues/29654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] V0lantis commented on issue #29654: Not able to specify redis as a result backend
Posted by "V0lantis (via GitHub)" <gi...@apache.org>.
V0lantis commented on issue #29654:
URL: https://github.com/apache/airflow/issues/29654#issuecomment-1443187351
Thank @hussein-awala, will try this !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] V0lantis commented on issue #29654: Not able to specify redis as a result backend
Posted by "V0lantis (via GitHub)" <gi...@apache.org>.
V0lantis commented on issue #29654:
URL: https://github.com/apache/airflow/issues/29654#issuecomment-1438332301
I would like to add that there is no `job_type` for the dag-processor and therefore, the liveness probe is pointless since it cannot find any job in the database
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #29654: Not able to specify redis as a result backend
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29654:
URL: https://github.com/apache/airflow/issues/29654#issuecomment-1442614738
Hello Arthur,
> Error: Failed to render chart: exit status 1: Error: values don't meet the specifications of the schema(s) in the following chart(s):
airflow:
data.resultBackendConnection: pass is required
data.resultBackendConnection: sslmode is required
data.resultBackendConnection: Additional property password is not allowed
Error: plugin "diff" exited with error
We build the Url from the [values](https://github.com/apache/airflow/blob/main/chart/templates/secrets/result-backend-connection-secret.yaml#L46-L46), so you should provide it as the following to create the secret:
```yaml
data:
resultBackendConnection:
protocol: redis
host: my.host.io
port: 6379
db: airflow
user: airflow_user
pass: airflow_pass
sslmode: disable
```
This creates a new secret:
```yaml
kind: Secret
apiVersion: v1
metadata:
name: release-name-airflow-result-backend
labels:
tier: airflow
release: release-name
chart: airflow
heritage: Helm
type: Opaque
data:
connection: "ZGIrcmVkaXM6Ly9haXJmbG93X3VzZXI6YWlyZmxvd19wYXNzQG15Lmhvc3QuaW86NjM3OS9haXJmbG93"
```
And here is the raw connection value
```bash
$ echo ZGIrcmVkaXM6Ly9haXJmbG93X3VzZXI6YWlyZmxvd19wYXNzQG15Lmhvc3QuaW86NjM3OS9haXJmbG93 | base64 -d
db+redis://airflow_user:airflow_pass@my.host.io:6379/airflow
```
And if you prefer to avoid providing the password in the values or creating the secret by helm, you can create the secret with the same structure and provide its name to `.Values.data.resultBackendSecretName` which is the recommended way.
> Although when I am doing so, I have error in my DagProcessor:
Liveness probe failed: You have configured a result_backend of redis://my.host.io:6379/1, it is highly recommended to use an alternative result_backend (i.e. a database).
No alive jobs found.
I think it's a problem in the provided url where the schema should be `db+redis` instead of `redis`.
PS: if you create the secret and provide its name to `.Values.data.resultBackendSecretName` or if you fill all the fields of `.Values.data.resultBackendConnection` as I did in my example, you don't need to provide the config `config.celery.config` where the connection will [be added](https://github.com/apache/airflow/blob/main/chart/templates/_helpers.yaml#L86-L92) to Airflow by the chart.
Can you please test the url format I suggested and check if you still have the same problem with the liveness probe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org