You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jordi-crespo (via GitHub)" <gi...@apache.org> on 2023/11/23 12:54:58 UTC
[I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
jordi-crespo opened a new issue, #35817:
URL: https://github.com/apache/airflow/issues/35817
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
airflow_version: 2.6.2
Deployed in: Azure Kubernetes Service
I am encountering a persistent and unusual issue with the StatsD configuration in Airflow. Despite following the guidelines provided in the [Datadog Integration Documentation](https://docs.datadoghq.com/integrations/airflow/?tab=containerized) and the [Airflow Helm Chart Production Guide](https://airflow.apache.org/docs/helm-chart/1.9.0/production-guide.html), I consistently receive an XML error message related to the statsd_host environment variable.
### Error Message:
The statsd_host variable is set as follows in my configuration:
```yaml
Copy code
- name: AIRFLOW__METRICS__STATSD_HOST
value: '10.0.32.180'
```
However, regardless of the configuration I use (including different host settings), the Airflow UI shows the following error under the "Configuration" tab in the "Metrics" section for statsd_host:
```xml
Copy code
<?xml version="1.0" encoding="utf-8"?>
<Error xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Code>ResourceNotFound</Code>
<Message>The specified resource does not exist.</Message>
<Details>'latest' isn't a valid resource name.</Details>
</Error>
```
<img width="1495" alt="image" src="https://github.com/apache/airflow/assets/43036171/ff4f3122-552d-4896-911c-28723d3ce790">
If I exec into one of the pods and I echo the `AIRFLOW__METRICS__STATSD_HOST` env variable I get: `10.0.32.180`
## Attempts to Resolve:
I've tried various configurations for the statsd_host variable, including using the service name, IP address, and even testing with localhost.
I've verified network connectivity and confirmed that there are no apparent network or firewall issues blocking the communication.
The error persists regardless of the method used to set the statsd_host.
## Seeking Help:
I am looking for guidance or suggestions on what might be causing this issue and how to resolve it. It appears that the configuration is correct according to the documentation, but the error suggests a deeper problem, possibly related to how Airflow is interpreting or fetching the environment variable.
Any help or insights from the community would be greatly appreciated.
### What you think should happen instead
When configuring the AIRFLOW__METRICS__STATSD_HOST environment variable in Airflow, I expect the system to correctly interpret and utilize the provided host value (in this case, 10.0.32.180) for sending metrics to StatsD/Datadog. The expected outcomes are:
Correct Resolution of Host Value: Airflow should resolve the statsd_host value without errors and use it for establishing a connection to the StatsD server or the Datadog agent.
No XML Error Messages: The configuration should not result in XML error messages like 'ResourceNotFound'. Instead, it should either successfully connect to the StatsD/Datadog service or provide a clear, relevant error message if the connection fails.
Visible Confirmation in Airflow UI: In the Airflow UI, under the "Configuration" tab in the "Metrics" section, the statsd_host setting should reflect the entered value without displaying unrelated XML error messages.
Successful Metrics Transmission: Metrics collected by Airflow should be successfully transmitted to the specified StatsD/Datadog host, allowing for monitoring and analysis through the Datadog platform.
In summary, I expect a seamless integration where Airflow can send metrics to the specified StatsD/Datadog service without encountering XML-related errors or misinterpretations of the configuration settings.
### How to reproduce
Deploy Airflow in Kubernetes: Use the Airflow Helm chart to deploy Airflow in a Kubernetes cluster. Make sure to include the following environment variable configuration in your Helm chart's values.yaml file or as part of your deployment configuration:
```yaml
Copy code
env:
- name: AIRFLOW__METRICS__STATSD_HOST
value: '10.0.32.180'
```
Access Airflow UI: Once Airflow is deployed, access the Airflow web UI.
Navigate to Configuration Tab: In the Airflow UI, navigate to the 'Admin' menu and select 'Configuration'.
Observe the Error: In the 'Metrics' section under the 'Configuration' tab, observe the XML error message displayed for the statsd_host setting.
### Operating System
Docker, creating the image from python:3.10-slim-buster
### Versions of Apache Airflow Providers
apache-airflow-providers-http==4.3.0
apache-airflow-providers-jdbc==3.3.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-snowflake==4.0.5
apache-airflow-providers-google==10.0.0
apache-airflow-providers-mysql==5.3.1
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-microsoft-azure==7.0.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #35817:
URL: https://github.com/apache/airflow/issues/35817#issuecomment-1824963562
Reading this I can confirm that statsd is working in our setup in K8s with the standard helm chart very stable sine ~1 year at least.
Am I correct to assume that your statsd container is _not_ deployed via the helm chart but you use helm just for the airflow deployment itself and point to an external statsd (hosted separately)?
Are you sure it is good to use a base IPv4 address and not a host name? Correct to assume your statsd is outside of K8s and the IP is stable and reachable (no firewall)?
From the screenshot I see that you missed to set the statsd reporting to "on"? Can you double check your config as described in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#statsd-on ? It is disabled per default if not explicitly set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #35817: Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable
URL: https://github.com/apache/airflow/issues/35817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #35817:
URL: https://github.com/apache/airflow/issues/35817#issuecomment-1826282503
Okay, then I assume the root cause originally is the error
```
ERROR - Unexpected error: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
```
Otherwise setting `statsd-on` will be required if you don't want to use the internal statsd (which I expect is fine to go with an external).
After reading the datadog docs, the setup uses the agents on all individual K8s hosts. Might be something has moved, the error from above is related to some host name parsing issues, seems the IP address in the Python code miserably fails. Can you please post not only the one file but the full log of a node starting such that the context can be understood?
As a second attempt can you try using the kubernetes hostname instead of the IP via `spec.nodeName` field instead of `status.hostIP`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
Posted by "jordi-crespo (via GitHub)" <gi...@apache.org>.
jordi-crespo commented on issue #35817:
URL: https://github.com/apache/airflow/issues/35817#issuecomment-1825630340
> Reading this I can confirm that statsd is working in our setup in K8s with the standard helm chart very stable sine ~1 year at least. Am I correct to assume that your statsd container is _not_ deployed via the helm chart but you use helm just for the airflow deployment itself and point to an external statsd (hosted separately)? Are you sure it is good to use a base IPv4 address and not a host name? Correct to assume your statsd is outside of K8s and the IP is stable and reachable (no firewall)? From the screenshot I see that you missed to set the statsd reporting to "on"? Can you double check your config as described in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#statsd-on ? It is disabled per default if not explicitly set.
According to the links to datadog and airflow, they say to use the environmental variables in this way in the containers:
https://docs.datadoghq.com/integrations/airflow/?tab=containerized#connect-airflow-to-dogstatsd and https://airflow.apache.org/docs/helm-chart/1.9.0/production-guide.html
The environmental variable statsd on was set to false, because if I set it to true I get the following error:
```
ERROR - Unexpected error: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
```
It is a host IP because according to the links I sent you, it says to use the IP of the node where airflow is running. I have checked that this works, so I guess the problem must be in airflow, but I don't understand where the problem lies.
I see that there is the possibility of creating a container for statsd, but I don't see how I could configure it to use it with datadog, since from what I understand in the guides, it has to be used using the env variables.
What do you think about this after I have given you more context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #35817:
URL: https://github.com/apache/airflow/issues/35817#issuecomment-1848790180
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow StatsD Configuration Error: XML 'ResourceNotFound' Message for statsd_host Environment Variable [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #35817:
URL: https://github.com/apache/airflow/issues/35817#issuecomment-1858993587
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org