You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Bowrna (via GitHub)" <gi...@apache.org> on 2024/03/22 08:26:32 UTC
[PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Bowrna opened a new pull request, #38400:
URL: https://github.com/apache/airflow/pull/38400
issue: #36188
related: #36188
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2017270573
when setting the `dag_processing.processes` as gauge instead of the counter, we may need to initialize the value.
Gauge works in the following way:
gaugor:333|g
the above command sets the param `gaugor` to 333
gaugor:433|g
the above command sets the param `gaugor` to 433
gaugor:-10|g
the above command sets the param `gaugor` to 423
gaugor:+4|g
the above command sets the param `gaugor` to 427
So to keep increasing the process count as it starts and decrease the process count as it stops(finishes or timeouts or error outs), we first have to initialize it to zero value at the beginning. I could set the gauge to zero at the start of the `airflow scheduler` command.
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#running-more-than-one-scheduler
But there is an option to run more than one scheduler. so setting the param as zero in the beginning of the `airflow scheduler` command wouldn't be the right way.
How do you think this case can be better handled? @potiuk @dirrao @ferruzzi @hussein-awala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2024398539
> when setting the `dag_processing.processes` as gauge instead of the counter, we may need to initialize the value.
>
> ```
> Gauge works in the following way:
> gaugor:333|g
> the above command sets the param `gaugor` to 333
> gaugor:433|g
> the above command sets the param `gaugor` to 433
> gaugor:-10|g
> the above command sets the param `gaugor` to 423
> gaugor:+4|g
> the above command sets the param `gaugor` to 427
> ```
>
> So to keep increasing the process count as it starts and decrease the process count as it stops(finishes or timeouts or error outs), we first have to initialize it to zero value at the beginning. I could set the gauge to zero at the start of the `airflow scheduler` command. https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#running-more-than-one-scheduler But there is an option to run more than one scheduler. so setting the param as zero in the beginning of the `airflow scheduler` command wouldn't be the right way.
>
> How do you think this case can be better handled? @potiuk @dirrao @ferruzzi @hussein-awala
@ferruzzi I think I will need help with the above part. If we are changing `dag_processing.processes` with a different name to gauge, I have to find out about initializing to zero value. Your views on this would be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2119179128
> > when setting the `dag_processing.processes` as gauge instead of the counter, we may need to initialize the value.
> > ```
> > Gauge works in the following way:
> > gaugor:333|g
> > the above command sets the param `gaugor` to 333
> > gaugor:433|g
> > the above command sets the param `gaugor` to 433
> > gaugor:-10|g
> > the above command sets the param `gaugor` to 423
> > gaugor:+4|g
> > the above command sets the param `gaugor` to 427
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > So to keep increasing the process count as it starts and decrease the process count as it stops(finishes or timeouts or error outs), we first have to initialize it to zero value at the beginning. I could set the gauge to zero at the start of the `airflow scheduler` command. https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#running-more-than-one-scheduler But there is an option to run more than one scheduler. so setting the param as zero in the beginning of the `airflow scheduler` command wouldn't be the right way.
> > How do you think this case can be better handled? @potiuk @dirrao @ferruzzi @hussein-awala
>
> @ferruzzi I think I will need help with the above part. If we are changing `dag_processing.processes` with a different name to gauge, I have to find out about initializing to zero value. Your views on this would be helpful.
>
> Edit: I will need help in this part to get unblocked
@ferruzzi do I have any update on this part?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2118514080
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna closed pull request #38400: changing dag_processing.processes from UpDownCounter to guage
URL: https://github.com/apache/airflow/pull/38400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2122995292
> @ferruzzi do I have any update on this part?
How would that work? A gauge is an arbitrary reading, you just tell it what the new/current value is. Unless you see some way for Airflow to read the last reported value from the metrics backend (StatsD, OTel, etc) you would need to store the last-reported value somewhere, increment/decrement it accordingly, then emit the new value. If you are doping that then it seems like the answer would be to add some logic which fetches thew last reported value and defaults to 0 if it isn't found.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1541522673
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
That said, where does that leave this PR? Is it still needed? There have been a few metrics-related ones making a bit of a tangled mess lately and I'm a bit lost on what needs to be looked at and where things are with each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1538280611
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
Not a bug as far as I am concerned. StatsD does not have the UpDownCounter concept and their Gauges can be incremented/decremented. In OTel, a Gauage is a "current reading" and can not be incremented or decremented. It's just an Otel vs StatsD semantics issue. In your other comments below, you point out the issues with making it a Gauage pretty well and more or less explain the reasons I did it the way I did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1539753482
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
`processor_timeouts` is a counter, and is listed as one in the docs on the website:

The only UpDownCounter in use is `dag_processing.processes`, which the docs also show in the docs as a Counter.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1541520266
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
It's not the most straight-forward thing, and the metric names are all so similar, I totally get it. If you see a way to make it more clear, please do! Documentation improvements and clarifications are often overlooked and at least as important as code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1538280611
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
Not a bug as far as I am concerned. StatsD does not have the UpDownCounter concept and their Gauges can be incremented/decremented. It's just an Otel vs StatsD semantics issue. In your other comments below, you point out the issues with making it a Gauage pretty well and more or less explain the reasons I did it the way I did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2016455324
> Add deprecation details around the old metric. Can you update the metrics documentation?
yes @dirrao Let me do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1540465952
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
ah, my bad @ferruzzi. My brain tricked me when i was seeing the documentation on website and sorry for bothering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1537109916
##########
docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst:
##########
@@ -219,6 +219,7 @@ Name Description
=================================================== ========================================================================
``dagbag_size`` Number of DAGs found when the scheduler ran a scan based on its
configuration
+``dag_processing.processes_count`` Number of currently running DAG parsing processes
``dag_processing.import_errors`` Number of errors from trying to parse DAG files
``dag_processing.total_parse_time`` Seconds taken to scan and import ``dag_processing.file_path_queue_size`` DAG files
Review Comment:
@potiuk @ferruzzi @hussein-awala Do you think param like ``dag_processing.total_parse_time` need to be under the metrics `gauges`? This is not a docs issue, but in the code this metric is set as gauge value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1538589431
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
@ferruzzi I could see that param "dag_processing.processor_timeouts" is only increased. I still didn't get the logic behind why it has to be incremented as counter but specified as gauge in the documentation. Can you help me understand it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on code in PR #38400:
URL: https://github.com/apache/airflow/pull/38400#discussion_r1535234940
##########
airflow/dag_processing/manager.py:
##########
@@ -1207,6 +1209,7 @@ def _kill_timed_out_processors(self):
processor.start_time.isoformat(),
)
Stats.decr("dag_processing.processes", tags={"file_path": file_path, "action": "timeout"})
+ Stats.gauge("dag_processing.processes_count", -1, delta=True, tags={"file_path": file_path, "action": "timeout"})
Stats.incr("dag_processing.processor_timeouts", tags={"file_path": file_path})
Review Comment:
@potiuk @hussein-awala @ferruzzi I could see from the docs that the metric "dag_processing.processor_timeouts" is specified as gauge. But in the code, it is getting incremented as counter type here. Is this a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #38400: changing dag_processing.processes from UpDownCounter to guage
URL: https://github.com/apache/airflow/pull/38400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38400:
URL: https://github.com/apache/airflow/pull/38400#issuecomment-2336881260
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org