You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/18 14:37:42 UTC

[GitHub] [airflow] magnusfurugard opened a new issue #19044: Statsd host does not update

magnusfurugard opened a new issue #19044:
URL: https://github.com/apache/airflow/issues/19044


   ### Official Helm Chart version
   
   1.2.0 (latest released)
   
   ### Apache Airflow version
   
   2.1.4 (latest released)
   
   ### Kubernetes Version
   
   v.1.21.2
   
   ### Helm Chart configuration
   
   ```
   statsd:
     enabled: false
   
   env:
     # Enable statsd here, disable chart default.
     - name: AIRFLOW__METRICS__STATSD_ON
       value: true
     - name: AIRFLOW__METRICS__HOST
       value: <service name for statsd>
     - name: AIRFLOW__METRICS__STATSD_PORT
       value: <port for statsd>
   ```
   
   ### Docker Image customisations
   
   None.
   
   ### What happened
   
   Setting `AIRFLOW__METRICS__HOST` seem to have no effect, even if we set `statsd.enabled=false` in the chart and enable it through the same config (`AIRFLOW__METRICS__STATSD_ON`). Airflow will continue to try to resolve the default host, `<name>-statsd`-service, even if set `AIRFLOW__METRICS__HOST`. 
   
   ### What you expected to happen
   
   I expected Airflow to honor the `AIRFLOW__METRICS__HOST` variable and resolve the service as specified there.
   
   ### How to reproduce
   
   ###  Step 1 - vanilla install
   Install the helm-chart with the following `values.yml`
   ```yml
   # airflow-values.yml
   
   statsd:
     enabled: true # this is the default
   ```
   ```bash
   helm install -f airflow-values.yml example apache-airflow/airflow --version 1.2.0 --timeout 1200s
   ```
   
   ### Step 2 - Replace statsd with custom statsd
   We want to disable the chart-built-in statsd and use our own instead. To accommodate, set up _any_ statsd deployment in the same namespace. In my case, I use the prometheus exporter one but it can be absolutely anything.
   
   ```bash
   # Install statsd. 
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install -f statsd-values.yml statsd prometheus-community/prometheus-statsd-exporter --timeout 1200s
   
   # Chart-default is to accept udp/tcp on port 9125. Above config also creates a service for us called `statsd-prometheus-statsd-exporter`.
   ```
   ... And update Airflow to communicate with the new statsd
   ```yml
   # airflow-values.yml
   
   # Update Airflow to use the new statsd
   statsd:
     enabled: false
   
   env:
     - name: AIRFLOW__METRICS__STATSD_ON
       value: true
     - name: AIRFLOW__METRICS__HOST
       value: statsd-prometheus-statsd-exporter
     - name: AIRFLOW__METRICS__STATSD_PORT
       value: 9125
   ```
   ```bash
   helm upgrade -f airflow-values.yml example apache-airflow/airflow --version 1.2.0 --timeout 1200s
   ```
   
   ### Step 3 - Airflow fails to resolve the new host
   Run any DAG from within Airflow, or restart the webserver to trigger an error, displaying that it's unable to resolve the service; 
   ```
   [2021-10-18 14:22:35,668] {stats.py:359} ERROR - Could not configure StatsClient: [Errno -2] Name or service not known, using DummyStatsLogger instead.
   ```
   
   ### Step 4 - Rename the new statsd-service to match what Airflow is looking for
   We can fix this by overriding the name of the new service to match that which we _know_ Airflow is looking for (`<name>-statsd`).
   
   ```yml
   # statsd-values.yml
   
   # Remember we set the name of our helm install to `example` in step 1.
   # This toggle alters the name of the service for our custom statsd to be whatever we put below.
   fullnameOverride: example-statsd
   ```
   ```bash
   helm upgrade -f statsd-values.yml statsd prometheus-community/prometheus-statsd-exporter --timeout 1200s
   ```
   
   **Airflow is now able to log metrics to our custom statsd-client.**
   
   And just to showcase that these variables are not picked up, let's set them to something that we know does not exist. 
   ```yml
   # airflow-values.yml
   
   statsd:
     enabled: false
   
   env:
     - name: AIRFLOW__METRICS__STATSD_ON
       value: true
     - name: AIRFLOW__METRICS__HOST
       value: whatever-you-want-really
     - name: AIRFLOW__METRICS__STATSD_PORT
       value: 3123 # anything numeric
   ```
   ```bash
   helm upgrade -f airflow-values.yml example apache-airflow/airflow --version 1.2.0 --timeout 1200s
   ```
   
   **Airflow is _still_ able to log metrics to our custom statsd-client.**
   
   
   ### Anything else
   
   I ran this with `executor=KubernetesExecutor`, but I doubt it has any effect. Made releases and upgrades faster.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham commented on issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-959450029


   Hey @magnusfurugard, you need to use `AIRFLOW__METRICS__STATSD_HOST`, not `AIRFLOW__METRICS__HOST`:
   https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#statsd-host
   
   I looked and didn't see any reference to the wrong env var name in our docs. Did you see it somewhere?
   
   If you still have issues, let me know and I can reopen this. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham commented on issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-959450029


   Hey @magnusfurugard, you need to use `AIRFLOW__METRICS__STATSD_HOST`, not `AIRFLOW__METRICS__HOST`:
   https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#statsd-host
   
   I looked and didn't see any reference to the wrong env var name in our docs. Did you see it somewhere?
   
   If you still have issues, let me know and I can reopen this. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham closed issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19044:
URL: https://github.com/apache/airflow/issues/19044






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham commented on issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-959450029






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham closed issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19044:
URL: https://github.com/apache/airflow/issues/19044


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] jedcunningham closed issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19044:
URL: https://github.com/apache/airflow/issues/19044


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #19044: Statsd host does not update

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19044:
URL: https://github.com/apache/airflow/issues/19044#issuecomment-945843918


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org