You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rawwar (via GitHub)" <gi...@apache.org> on 2024/05/04 00:43:54 UTC
[I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
rawwar opened a new issue, #39400:
URL: https://github.com/apache/airflow/issues/39400
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
with `AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH` set to `true` and `AIRFLOW__METRICS__METRICS_ALLOW_LIST` set to `.*get_astronauts.*` , I notice several other metrics being sent to datadog that don't match the pattern:

### What you think should happen instead?
Following metrics shouldn't be showing up on datadog when the allow list is `.*get_astronauts.*`:
1. `airflow.dag.task.duration.*`
2. `airflow.ti.*`
3. `datadog.*` (Not sure if we have any control to block these)
### How to reproduce
env variables:
AIRFLOW__METRICS__STATSD_DATADOG_ENABLED | true
-- | --
AIRFLOW__METRICS__STATSD_DATADOG_TAGS | testNo:C2
DATADOG_API_KEY | ● ● ● ● ● ● ●
AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH | true
AIRFLOW__METRICS__METRICS_ALLOW_LIST | `.*print_astronaut_craft.*`
Once you update your datadog key here, you will start seeing metrics in datadog-->metrics-->summary . use the tag `testNo:C2` to filter
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-5660343428
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-5660676608
cc @hussein-awala maybe someone from datadog has the time to pick it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-5686639944
I had a look at this on main and I don't think it's still valid.
`metrics_use_pattern_match` was removed in Airflow 3 — pattern matching is the only
mode now, so the config in the repro doesn't exist anymore. And the allow list itself
is applied on every emit path in the Datadog logger (incr, decr, gauge, timing, timer),
including for legacy metric names, so nothing is slipping past the filter.
What was really happening is that `ti.start`, `ti.finish` and `task.duration` don't have
the dag_id in the metric *name* — it's in the tags. So a pattern like
`.*get_astronauts.*` was never going to match them. It only matches the legacy names
(`dag.{dag_id}.{task_id}.duration` and friends), which are behind `legacy_names_on`. The
`datadog.*` ones are the DogStatsd client's own telemetry and were always outside our
allow list.
So this isn't a filtering bug — it's that allow/block lists match on names and aren't
tag-aware. That's a reasonable thing to want, but it's a different change from what's
described here, so I'd rather see it as its own feature request. Closing this one.
---
Drafted-by: Claude Code (Opus 5); reviewed by @hussein-awala before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2694304298
@rawwar is this still an issue on main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2704361492
I'll try to replicate it later this week and reconfirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2708189757
@eladkal , Yes. i just tested this and I can still replicate it. I added a allow list and yet, I see airflow.dag.task.duration.*, airflow.ti.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2708190536
However, I've unassigned myself as I won't be able to work on it at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #39400: Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled
URL: https://github.com/apache/airflow/issues/39400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org