You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/18 12:27:07 UTC
[GitHub] [airflow] ankxyz opened a new issue #13741: Airflow 2.0 does not send metrics to statsD
ankxyz opened a new issue #13741:
URL: https://github.com/apache/airflow/issues/13741
**Apache Airflow version**:
2.0.0
**Environment**:
- **OS** (e.g. from /etc/os-release): Ubuntu 20.04 LTS
- **Python version**: 3.8
- **Kernel** (e.g. `uname -a`): x86_64 x86_64 x86_64 GNU/Linux 5.4.0-58-generic #64-Ubuntu
- **Install tools**: pip
**What happened**:
Airflow 2.0 does not send metrics to statsD.
I configure Airflow with official documentation (https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/metrics.html) and by this article https://dstan.medium.com/run-airflow-statsd-grafana-locally-16b372c86524 (but I used port 8125).
I turned on statsD:
```ini
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
```
But I do not see airflow metrics at http://localhost:9102/metrics (statsD metrics endpoint).
---
P.S. I noticed this error just using Airflow 2.0. In version 1.10.13 everything is ok in the same environment.
Thank you for advance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762218740
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #13741: Airflow 2.0 does not send metrics to statsD when Scheduler is run with Daemon mode
Posted by GitBox <gi...@apache.org>.
ashb closed issue #13741:
URL: https://github.com/apache/airflow/issues/13741
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762297940
I think you need to provide more logs showing what's going on.
Have you followed UPDATING.md / 2.0 migration process? Have you seen this change:
https://github.com/apache/airflow/blob/master/UPDATING.md#metrics-configuration-has-been-moved-to-new-section
I think you have some configuration problem but it's hard to believe stats are not working for 2.0. Maybe open a discussion in https://github.com/apache/airflow/discussions or in slack and provide some more logging information there, and maybe someone who has statsd experience and runs Airflow 2.0 with stats d will be able to help to diagnose it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz edited a comment on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz edited a comment on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-765069021
@potiuk Of course. Moreover, I generated `airflow.cfg` from scratch to test it.
Сlarification (recently noticed): the problem arises just if I run `airflow scheduler` in daemon mode:
```bash
airflow scheduler -D
```
If I start it normally (without param -D) or using nohup - everything ok:
```bash
nohup airflow scheduler >> ${AIRFLOW_HOMW}/logs/scheduler.log 2>&1 &
```
So I have the problem just with daemon mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762385325
But did you change the config as per https://github.com/apache/airflow/blob/master/UPDATING.md#metrics-configuration-has-been-moved-to-new-section ?
You will see that the configuration changed for metrics to [metrics] section from [scheduler]. Did you change it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762300016
There is an upgrade-check that you should run in 1.10 to tell you about configuration changes, that you should follow: http://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html#step-3-install-and-run-the-upgrade-check-scripts
I assume that this is the case - that you simply did not follow this migration guide and I close this ticket for now. Please let us know if this fixed your problem (and if not - please provide more information what you tried). We can still re-open this if you still have the problem and some more info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz edited a comment on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz edited a comment on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-765069021
@potiuk Of course. Moreover, I generated `airflow.cfg` from scratch to test it.
Сlarification (recently noticed): the problem arises just if I run `airflow scheduler` in daemon mode:
```bash
airflow scheduler -D
```
If I start it normally (without param -D) or using nohup - everything ok:
```bash
nohup airflow scheduler >> ${AIRFLOW_HOMW}/logs/scheduler.log 2>&1 &
```
So I have the problem just with daemon mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762253836
@potiuk Yes:
```bash
pip install apache-airflow[statsd]==2.0.0
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-765069021
@potiuk Of course. Moreover, I generated `airflow.cfg` from scratch to test it.
Сlarification (recently noticed): the problem arises just if I run `airflow scheduler` in daemon mode:
```bash
airflow scheduler -D
```
If I start it normally (without param -D) or using nohup - everything ok:
```bash
nohup airflow scheduler >> ${AIRFLOW_HOMW}/logs/scheduler.log 2>&1 &
```
So I have the problem just in daemon mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762318463
@potiuk
I installed Airflow 2.0 on pure Python venv, where older Airflow version was not installed.
As I said, if I create another venv but with Airflow 1.10.13 - it works.
---
What steps I do:
* create python venv (`python3 -m venv venv && source venv/bin/activate`)
* install Airflow (`pip install apache-airlfow[statsd]`)
* configure Airflow through `airflow.cfg`:
```ini
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
```
* run statsD, prometheus, grafana
Airflow 1.10.13 - works (metrics are available), 2.0.0 - does not work :-(
---
I have no errors or even warnings in logs, just airflow metrics are not available in statsD.
To be more clear I will create demo repository with Airflow 2.0 configuration for statsD.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Junnplus commented on issue #13741: Airflow 2.0 does not send metrics to statsD when Scheduler is run with Daemon mode
Posted by GitBox <gi...@apache.org>.
Junnplus commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-785703692
It seems that the `daemonContext` will close the socket of statsd.
```py
return self.statsd.incr(stat, count, rate)
File "/usr/local/lib/python3.8/site-packages/statsd/client/base.py", line 35, in incr
self._send_stat(stat, '%s|c' % count, rate)
File "/usr/local/lib/python3.8/site-packages/statsd/client/base.py", line 59, in _send_stat
self._after(self._prepare(stat, value, rate))
File "/usr/local/lib/python3.8/site-packages/statsd/client/base.py", line 74, in _after
self._send(data)
File "/opt/airflow/airflow/stats.py", line 40, in _send
self._sock.sendto(data.encode('ascii'), self._addr)
OSError: [Errno 9] Bad file descriptor
```
A simple fix is to let stastd client load lazily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-765069021
@potiuk Of course. Moreover, I generated `airflow.cfg` from scratch to test it.
Сlarification (recently noticed): the problem arises just if I run `airflow scheduler` in daemon mode:
```bash
airflow scheduler -D
```
If I start it normally (without param -D) or using nohup - everything ok:
```bash
nohup airflow scheduler >> ${AIRFLOW_HOMW}/logs/scheduler.log 2>&1 &
```
So I have the problem just in daemon mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13741: Airflow 2.0 does not send metrics to statsD when Scheduler is run with Daemon mode
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-786686170
Oh yes, daemon mode will close all open files and sockets.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762219348
Did you install statsd extra?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ankxyz edited a comment on issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
ankxyz edited a comment on issue #13741:
URL: https://github.com/apache/airflow/issues/13741#issuecomment-762318463
@potiuk
I installed Airflow 2.0 on pure Python venv, where older Airflow version was not installed.
As I said, if I create another venv but with Airflow 1.10.13 - it works.
---
What steps I do:
* create python venv (`python3 -m venv venv && source venv/bin/activate`)
* install Airflow (`pip install apache-airlfow[statsd]`)
* configure Airflow through `airflow.cfg`:
```ini
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
```
* run statsD, prometheus, grafana
Airflow 1.10.13 - works (metrics are available), 2.0.0 - does not work :-(
---
I have no errors or even warnings in logs, just airflow metrics are not available in statsD.
To be more clear I will create demo repository with Airflow 2.0 configuration for statsD.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13741: Airflow 2.0 does not send metrics to statsD
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13741:
URL: https://github.com/apache/airflow/issues/13741
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org