You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rawwar (via GitHub)" <gi...@apache.org> on 2024/05/04 00:43:54 UTC

[I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

rawwar opened a new issue, #39400:
URL: https://github.com/apache/airflow/issues/39400

   ### Apache Airflow version
   
   main (development)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   with `AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH` set to `true` and `AIRFLOW__METRICS__METRICS_ALLOW_LIST` set to `.*get_astronauts.*` , I notice several other metrics being sent to datadog that don't match the pattern:
   ![image](https://github.com/apache/airflow/assets/20266953/16510682-9e45-48c1-9a95-ce726996cca5)
   
   
   ### What you think should happen instead?
   
   Following metrics shouldn't be showing up on datadog when the allow list is `.*get_astronauts.*`:
   
   1. `airflow.dag.task.duration.*`
   2. `airflow.ti.*`
   3. `datadog.*` (Not sure if we have any control to block these)
   
   ### How to reproduce
   
   env variables:
   
   AIRFLOW__METRICS__STATSD_DATADOG_ENABLED | true
   -- | -- 
   AIRFLOW__METRICS__STATSD_DATADOG_TAGS | testNo:C2
   DATADOG_API_KEY | ● ● ● ● ● ● ●
   AIRFLOW__METRICS__METRICS_USE_PATTERN_MATCH | true
   AIRFLOW__METRICS__METRICS_ALLOW_LIST | `.*print_astronaut_craft.*`
   
   
   Once you update your datadog key here, you will start seeing metrics in datadog-->metrics-->summary . use the tag `testNo:C2` to filter
   
   ### Operating System
   
   Debian GNU/Linux 11 (bullseye)
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-5660343428

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-5660676608

   cc @hussein-awala maybe someone from datadog has the time to pick it up?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-5686639944

   I had a look at this on main and I don't think it's still valid.
   
   `metrics_use_pattern_match` was removed in Airflow 3 — pattern matching is the only
   mode now, so the config in the repro doesn't exist anymore. And the allow list itself
   is applied on every emit path in the Datadog logger (incr, decr, gauge, timing, timer),
   including for legacy metric names, so nothing is slipping past the filter.
   
   What was really happening is that `ti.start`, `ti.finish` and `task.duration` don't have
   the dag_id in the metric *name* — it's in the tags. So a pattern like
   `.*get_astronauts.*` was never going to match them. It only matches the legacy names
   (`dag.{dag_id}.{task_id}.duration` and friends), which are behind `legacy_names_on`. The
   `datadog.*` ones are the DogStatsd client's own telemetry and were always outside our
   allow list.
   
   So this isn't a filtering bug — it's that allow/block lists match on names and aren't
   tag-aware. That's a reasonable thing to want, but it's a different change from what's
   described here, so I'd rather see it as its own feature request. Closing this one.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @hussein-awala before posting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2694304298

   @rawwar is this still an issue on main?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2704361492

   I'll try to replicate it later this week and reconfirm. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2708189757

   @eladkal , Yes. i just tested this and I can still replicate it. I added a allow list and yet, I see airflow.dag.task.duration.*, airflow.ti.*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39400:
URL: https://github.com/apache/airflow/issues/39400#issuecomment-2708190536

   However, I've unassigned myself as I won't be able to work on it at this time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #39400: Extra metrics are being sent to DataDog when using Allow list with pattern matching enabled
URL: https://github.com/apache/airflow/issues/39400


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org