[GitHub] [airflow] nicolamarangoni opened a new issue, #27180: Only partial metrics sent to statsd when the airflow services run on different hosts (AWS Fargate tasks)

[GitBox <gi...@apache.org>]

nicolamarangoni opened a new issue, #27180:
URL: https://github.com/apache/airflow/issues/27180

   ### Apache Airflow version
   
   2.4.1
   
   ### What happened
   
   We have an AirFlow-Deployment on **AWS Fargate** where each service (webserver, scheduler, dag-processor, celery-workers) are running in their own task, each task has a different hostname and ip.
   We also have a separated task with a **datadog-agent** running a statsd (**dogstatsd**) server to which we want to push airflow metrics.
   We have `statsd_host=DATADOG_AGENT_HOST` and `statsd_datadog_enabled="True"`.
   Statsd successfully collect metrics like **dagbag_size** and **scheduler_heartbeat** but **NOT** **healthy** and **can_connect**.
   We similar deployment on k8s in which all service containers run in the same pod, also with a single hostname and ip for everything. In some k8s setups we have the datadog-agent in the same pod in another pod. In the second case we set **statsd_host = DDAGENT_POD_HOST** and we successfully collect all metrics.
   
   My suspect is that metrics are not pushed to statsd from every service independently but **only by the scheduler service**.
   The scheduler service check **health** and **can_connect** and push them to statsd. However these 2 metrics originates in the webserver. If the webserver runs in a different host than the scheduler (like different tasks in Fargate), the metrics are not collected.
   
   Is it possible that the scheduler is the only service that push metrics to statsd and that it can collect metrics from other services always searching for them on localhost?
   
   ### What you think should happen instead
   
   All metrics should be sent to statsd.
   
   ### How to reproduce
   
   Setup AirFlow with every service running in a separated pod/host/task with its own ip and hostname.
   Activate statsd.
   
   ### Operating System
   
   Debian GNU/Linux 11 (bullseye)
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==6.0.0
   apache-airflow-providers-celery==3.0.0
   apache-airflow-providers-cncf-kubernetes==4.4.0
   apache-airflow-providers-common-sql==1.2.0
   apache-airflow-providers-databricks==3.3.0
   apache-airflow-providers-datadog==3.0.0
   apache-airflow-providers-dbt-cloud==2.2.0
   apache-airflow-providers-docker==3.2.0
   apache-airflow-providers-elasticsearch==4.2.1
   apache-airflow-providers-ftp==3.1.0
   apache-airflow-providers-google==8.3.0
   apache-airflow-providers-grpc==3.0.0
   apache-airflow-providers-hashicorp==3.1.0
   apache-airflow-providers-http==4.0.0
   apache-airflow-providers-imap==3.0.0
   apache-airflow-providers-microsoft-azure==4.3.0
   apache-airflow-providers-mysql==3.2.1
   apache-airflow-providers-odbc==3.1.2
   apache-airflow-providers-postgres==5.2.2
   apache-airflow-providers-redis==3.0.0
   apache-airflow-providers-sendgrid==3.0.0
   apache-airflow-providers-sftp==4.1.0
   apache-airflow-providers-slack==5.1.0
   apache-airflow-providers-snowflake==3.3.0
   apache-airflow-providers-sqlite==3.2.1
   apache-airflow-providers-ssh==3.2.0
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   AirFlow on AWS Fargate (ECS) with separated tasks for every service.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #27180: Only partial metrics sent to statsd when the airflow services run on different hosts (AWS Fargate tasks)

[GitBox <gi...@apache.org>]

potiuk closed issue #27180: Only partial metrics sent to statsd when the airflow services run on different hosts (AWS Fargate tasks)
URL: https://github.com/apache/airflow/issues/27180


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org