You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/18 14:37:42 UTC
[GitHub] [airflow] magnusfurugard opened a new issue #19044: Statsd host does not update
magnusfurugard opened a new issue #19044:
URL: https://github.com/apache/airflow/issues/19044
### Official Helm Chart version
1.2.0 (latest released)
### Apache Airflow version
2.1.4 (latest released)
### Kubernetes Version
v.1.21.2
### Helm Chart configuration
```
statsd:
enabled: false
env:
# Enable statsd here, disable chart default.
- name: AIRFLOW__METRICS__STATSD_ON
value: true
- name: AIRFLOW__METRICS__HOST
value: <service name for statsd>
- name: AIRFLOW__METRICS__STATSD_PORT
value: <port for statsd>
```
### Docker Image customisations
None.
### What happened
Setting `AIRFLOW__METRICS__HOST` seem to have no effect, even if we set `statsd.enabled=false` in the chart and enable it through the same config (`AIRFLOW__METRICS__STATSD_ON`). Airflow will continue to try to resolve the default host, `<name>-statsd`-service, even if set `AIRFLOW__METRICS__HOST`.
### What you expected to happen
I expected Airflow to honor the `AIRFLOW__METRICS__HOST` variable and resolve the service as specified there.
### How to reproduce
### Step 1 - vanilla install
Install the helm-chart with the following `values.yml`
```yml
# airflow-values.yml
statsd:
enabled: true # this is the default
```
```bash
helm install -f airflow-values.yml example apache-airflow/airflow --version 1.2.0 --timeout 1200s
```
### Step 2 - Replace statsd with custom statsd
We want to disable the chart-built-in statsd and use our own instead. To accommodate, set up _any_ statsd deployment in the same namespace. In my case, I use the prometheus exporter one but it can be absolutely anything.
```bash
# Install statsd.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install -f statsd-values.yml statsd prometheus-community/prometheus-statsd-exporter --timeout 1200s
# Chart-default is to accept udp/tcp on port 9125. Above config also creates a service for us called `statsd-prometheus-statsd-exporter`.
```
... And update Airflow to communicate with the new statsd
```yml
# airflow-values.yml
# Update Airflow to use the new statsd
statsd:
enabled: false
env:
- name: AIRFLOW__METRICS__STATSD_ON
value: true
- name: AIRFLOW__METRICS__HOST
value: statsd-prometheus-statsd-exporter
- name: AIRFLOW__METRICS__STATSD_PORT
value: 9125
```
```bash
helm upgrade -f airflow-values.yml example apache-airflow/airflow --version 1.2.0 --timeout 1200s
```
### Step 3 - Airflow fails to resolve the new host
Run any DAG from within Airflow, or restart the webserver to trigger an error, displaying that it's unable to resolve the service;
```
[2021-10-18 14:22:35,668] {stats.py:359} ERROR - Could not configure StatsClient: [Errno -2] Name or service not known, using DummyStatsLogger instead.
```
### Step 4 - Rename the new statsd-service to match what Airflow is looking for
We can fix this by overriding the name of the new service to match that which we _know_ Airflow is looking for (`<name>-statsd`).
```yml
# statsd-values.yml
# Remember we set the name of our helm install to `example` in step 1.
# This toggle alters the name of the service for our custom statsd to be whatever we put below.
fullnameOverride: example-statsd
```
```bash
helm upgrade -f statsd-values.yml statsd prometheus-community/prometheus-statsd-exporter --timeout 1200s
```
**Airflow is now able to log metrics to our custom statsd-client.**
And just to showcase that these variables are not picked up, let's set them to something that we know does not exist.
```yml
# airflow-values.yml
statsd:
enabled: false
env:
- name: AIRFLOW__METRICS__STATSD_ON
value: true
- name: AIRFLOW__METRICS__HOST
value: whatever-you-want-really
- name: AIRFLOW__METRICS__STATSD_PORT
value: 3123 # anything numeric
```
```bash
helm upgrade -f airflow-values.yml example apache-airflow/airflow --version 1.2.0 --timeout 1200s
```
**Airflow is _still_ able to log metrics to our custom statsd-client.**
### Anything else
I ran this with `executor=KubernetesExecutor`, but I doubt it has any effect. Made releases and upgrades faster.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-959450029
Hey @magnusfurugard, you need to use `AIRFLOW__METRICS__STATSD_HOST`, not `AIRFLOW__METRICS__HOST`:
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#statsd-host
I looked and didn't see any reference to the wrong env var name in our docs. Did you see it somewhere?
If you still have issues, let me know and I can reopen this. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-959450029
Hey @magnusfurugard, you need to use `AIRFLOW__METRICS__STATSD_HOST`, not `AIRFLOW__METRICS__HOST`:
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#statsd-host
I looked and didn't see any reference to the wrong env var name in our docs. Did you see it somewhere?
If you still have issues, let me know and I can reopen this. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19044:
URL: https://github.com/apache/airflow/issues/19044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-959450029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19044:
URL: https://github.com/apache/airflow/issues/19044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19044:
URL: https://github.com/apache/airflow/issues/19044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19044: Statsd host does not update
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-945843918
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org