You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/02 13:24:24 UTC
[GitHub] [airflow] rjribeiro opened a new issue, #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
rjribeiro opened a new issue, #24111:
URL: https://github.com/apache/airflow/issues/24111
### Apache Airflow version
2.3.1 (latest released)
### What happened
I have the following mapping for statsd:
```
- match: "*.dag.*.*.duration"
match_metric_type: observer
name: "af_agg_dag_task_duration"
labels:
airflow_id: "$1"
dag_id: "$2"
task_id: "$3"
```
A metric that was in the form `<<AIRFLOW_ID>>_dag_<<DAG_ID>>_<TASK_ID>>_duration ` should be mapping to `af_agg_dag_task_duration{airflow_id=<<AIRFLOW_ID>>,dag_id=<<DAG_ID>>,task_id = <<TASK_ID>>, quantile="0.5"}` for example.
But this is not happening for tasks in task groups. The metric is not mapped in these cases
### What you think should happen instead
_No response_
### How to reproduce
_No response_
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==3.4.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.2
apache-airflow-providers-docker==2.7.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==7.0.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.9.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.4
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
URL: https://github.com/apache/airflow/issues/24111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hankehly commented on issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
hankehly commented on issue #24111:
URL: https://github.com/apache/airflow/issues/24111#issuecomment-1300723028
@rjribeiro Did you every resolve your issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hankehly commented on issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
hankehly commented on issue #24111:
URL: https://github.com/apache/airflow/issues/24111#issuecomment-1338627162
I think this is normal behavior, due to the hierarchy of tasks / task groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24111:
URL: https://github.com/apache/airflow/issues/24111#issuecomment-1338951088
yes. Closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24111:
URL: https://github.com/apache/airflow/issues/24111#issuecomment-1144861552
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hankehly commented on issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
hankehly commented on issue #24111:
URL: https://github.com/apache/airflow/issues/24111#issuecomment-1183832282
@rjribeiro You may be aware of this already, but tasks inside task groups have their IDs prefixed with the group id of the task group. This _may be_ related to your issue.
Can you try adding `prefix_group_id=False` to the `TaskGroup` definition?
Ex.
```py
with TaskGroup("my_group_id", prefix_group_id=False):
@task
def my_task():
...
```
https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html?highlight=taskgroup
> By default, child tasks/TaskGroups have their IDs prefixed with the group_id of their parent TaskGroup. This helps to ensure uniqueness of group_id and task_id throughout the DAG.
>
> To disable the prefixing, pass prefix_group_id=False when creating the TaskGroup, but note that you will now be responsible for ensuring every single task and group has a unique ID of its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rishav394 commented on issue #24111: Statsd dont mapping task duration metrics for tasks in TaskGroup
Posted by GitBox <gi...@apache.org>.
rishav394 commented on issue #24111:
URL: https://github.com/apache/airflow/issues/24111#issuecomment-1311512259
@hankehly @rjribeiro
Hey this might be useful to you.
```yaml
- match: "*.dag.*.*.*.duration"
match_metric_type: observer
name: "af_agg_dag_group_task_duration"
labels:
airflow_id: "$1"
dag_id: "$2"
group_id: "$3"
task_id: "$4"
```
Could not make `group_id` optional so I added a whole new mapping for adding `group_id`. This works for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org