You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/22 15:55:25 UTC
[GitHub] [airflow] vDMG opened a new issue, #26601: Statsd aggregation issue on some metrics with multiple schedulers running
vDMG opened a new issue, #26601:
URL: https://github.com/apache/airflow/issues/26601
### Apache Airflow version
Other Airflow 2 version
### What happened
When I am running multiple schedulers (>1) the statsd exporter does not correctly sum the running tasks from the different scheduler but it does just take one metric from one scheduler and display it as the value.
the metric name : `airflow_executor_running_tasks`
### What you think should happen instead
airflow statsd exporter should sum up all running tasks from all the existing and running schedulers to calculate `airflow_executor_running_tasks`
### How to reproduce
Run an airflow cluster with 2 schedulers and with statsd enabled.
Create enough tasks to balance them between the 2 schedulers. Using the UI you can check that between the displayed value and the `airflow_executor_running_tasks` metric exposed by statsd you will not have the same number?
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.2
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Statsd aggregation issue on some metrics with multiple schedulers running [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-2067932490
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Statsd aggregation issue on some metrics with multiple schedulers running [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-3895321325
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1475372727
To my perspective each scheduler should have it's own metric as it's a separated process. Summing/aggregating might hide underlying issue. Most "metric tracker" tool hav the ability to aggregate from their side so I'd prefer to give data as raw as we can. However this is not my area of expertise thus lets hear what others think. cc @ferruzzi WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Statsd aggregation issue on some metrics with multiple schedulers running [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-2304890763
@ferruzzi Worth looking at this again as part of the workstream where you guys are looking to audit all metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Yaro1 commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "Yaro1 (via GitHub)" <gi...@apache.org>.
Yaro1 commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1474869855
Well, in my opinion we have two ways:
1. try to sum up that metric for every scheduler
2. send metric with some id of scheduler and then users can sum it by themself
I think the second way is better.
Also it's strange, but I found a duplicate of executor.running_tasks - scheduler.tasks.running.
But it seems like we have to delete it because it's always zero:
```
num_tasks_in_executor = 0
Stats.gauge("scheduler.tasks.running", num_tasks_in_executor)
```
It doesn't change and we have only two using of the variable num_tasks_in_executor.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Yaro1 commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "Yaro1 (via GitHub)" <gi...@apache.org>.
Yaro1 commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1474871897
But we can have some separate process for getting heartbeat from db and sending it, in that way it seems clear and better for me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Statsd aggregation issue on some metrics with multiple schedulers running [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-2011340957
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Yaro1 commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "Yaro1 (via GitHub)" <gi...@apache.org>.
Yaro1 commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1474871254
In 1. I mean, that we have to change logic of heartbeat for scheduler, cause for now it's sending for each scheduler independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Statsd aggregation issue on some metrics with multiple schedulers running [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
URL: https://github.com/apache/airflow/issues/26601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Yaro1 commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "Yaro1 (via GitHub)" <gi...@apache.org>.
Yaro1 commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1465201792
please assign to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Yaro1 commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "Yaro1 (via GitHub)" <gi...@apache.org>.
Yaro1 commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1475392909
okay, I will investigate the 2nd option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1475378622
1. Would be complex and brittle and is against the "distributed" system architecture, we would have to elect the "leader" (if we want to publish it by one of the schedulers or add a new entity to publish such metrics. Sounds complicating and smells zookeper (https://zookeeper.apache.org/) which is something we definitely want to avoid.
2. The problem is that currently we have no unique scheduler id that we can rely on. The schedulers are "identical" copies of the same process and we neither enumerate them, nor have another way of distinguishing them. So far at least.
Yes. Option 2. is better but the fact that we cannot distinuish them, makes it dificult, both to generate the metrics but even more difficult to aggregate them. The problem is that when we restart scheduler, the id should not change, otherwise the function aggregating the metrics will see that as "another" scheduler. This migh or might not be a problem - depending on the metrics.
We have few options there:
1) use uuid or simlar at the start of the scheduler to generate the name - we have a few UUID candidates there https://docs.python.org/3/library/uuid.html but none of them guarantees that the id will remain unchanged after the restart (especially in container environment). It could be random or hardware-based (or both).
2) similarly to workers we use hostname callable to get a unique id of teh scheduler when it starts. this is not a fool-proof either, because the hostname/IP might also (and often does) change after the restart.
The problem is tha Option 2) and non-random UUID generation does not guarantee uniqueness if a user chooses to run several schedulers at the same hardware without containerisation. This is not very likely, but possible scenario - because running multiple schedulers on the same hardware with multip-processing is generally a good idea (because such schedulers will be faster due to Python multi-prociessing/GIL limitaitons, Probably the UUID calculation and hostname should require also the port number on which scheduler is exposing healthcheck or something like that.
3) add a flag to specify scheduler ID when starting it (this might override 1) or 2) in cases when the ID is expected to remain the same for specific deployment.
I **think** the metrics in question should be fine to aggregate if the UUID is randomly allocated (I have not looked at those that are involved so it needs to be checked metrics-by-mytrics. However it will result in having changing uuids over time while scheduler is running and being restarted. Not sure how statsd monitoring software / aggregation will handle it - something to be investingated by someone doing such a change. Also the 3) might be a way to limit the impact of that, but at the expense of complicating deployment. Currently we add a new scheduler by **just** starting another scheduler process. If we also add the option to specify the id of the scheduler when starting, the Helm Chart of ours should likely be updated with that option. And documentation should be written to include that option.
Generally doable, but just wanted to make everyone aware in the discussion that is quite a bit more complex than "just send metrics with unique id". It would be simpler if we had the id, but we don't have it currently and there are implications of having it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1476687794
Thanks for the tag, @eladkal. Yeah, I agree with how the conversation has progressed for sure. Separate in name (and add the name to the metric tags) seems like it would be ideal. @potiuk's response about the difficulties in that definitely makes it sound like a challenge.
Since I mentioned taking on the OTel integration, I've had one or two people tell me that not all of the metrics listed on our [docs page](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html) work, so i planned to double check as many as I can once I have OTel working and update that page or he code as appropriate, but that's a bit of a longer-term goal at this point. Thanks for pointing this one out, @vDMG, and thanks for taking it on @Yaro1 :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26601: Statsd aggregation issue on some metrics with multiple schedulers running
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-1255229249
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Statsd aggregation issue on some metrics with multiple schedulers running [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #26601:
URL: https://github.com/apache/airflow/issues/26601#issuecomment-3986504854
@Yaro1 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org