You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "wu-sheng (via GitHub)" <gi...@apache.org> on 2023/02/04 09:35:00 UTC
[GitHub] [skywalking] wu-sheng opened a new issue, #10341: [Feature] Setup dashboard for Airflow monitoring
wu-sheng opened a new issue, #10341:
URL: https://github.com/apache/skywalking/issues/10341
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
This is an open issue for new contributors. Apache Airflow is a widely used workflow scheduler. We are encouraging someone new to the community to add a new level catalog(**workflow**) for Airflow.
## Metrics
Airflow exposes metrics through StatsD, https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html.
We could use StatsD + OpenTelemetry StatesD(https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/README.md) + OpenTelemetry OTEL exporter to ship the metrics to SkyWalking OTEL receiver.
Then use MAL to build metrics as well as a dashboard for those metrics. Notice, a new layer and new UI menu should be added.
## Logging
Airflow supports Fluents to ship metrics, https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-architecture.html. SkyWalking already has FluentD setup support, so we should be able to receive and catalog the logs.
Additionally, [Task Logs](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html) seems an interesting think. We could use LAL(Log Analysis) to group the logs by task name(or ID) by treating tasks as endpoints(SkyWalking concept).
### Use case
Add more observability for Airflow server.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426973836
What tag do you need? Tag is not required. For describing airflow server, that could be set through otel collector, like we did for mysql metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426983742
> > What tag do you need? Tag is not required. For describing airflow server, that could be set through otel collector, like we did for mysql metrics.
>
> I mean that all info is contained in the "metrics name" such as <job_name>, <job_id>, <dag_id>, <task_id>, <operator_name>, and so on. But I have no way to filter and process. Or I just don't consider these metrics? Because of the statsD data format, the otel data collected will not have "key value pair" tag attributes.
I doubt that you mixed the concept between "airflow job" and "opentelemetry job"? We use the OpenTelemetry Job name to distinguish data sources.
As for the metrics name like `"local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>"`, I think you should spit them in the OpenTelemetry processor and move the metadata into tags then send to OAP, anyway I'll take a look at Airflow's doc to see what's the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435522599
> Have you resolved the issue we discussed on slack?
I review the otel function, I find using the `replace_pattern` [function](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#replace_pattern) can clear the redundant string in the tag value. Also, I think the groovy can do it too, but I have not read this part completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435912206
Do you mean delta is increasement from last report period?
Could you explain a little more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440008428
It is one kind of data in opentelemetry protocol.
```
message Metric {
reserved 4, 6, 8;
string name = 1;
string description = 2;
string unit = 3;
oneof data {
Gauge gauge = 5;
Sum sum = 7;
Histogram histogram = 9;
ExponentialHistogram exponential_histogram = 10;
Summary summary = 11;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1459449597
> @mufiye Any update or block?
I think I should block it here temporarily. I am preparing to find an internship now and have no time to continue this issue in the last two weeks. You can unassign this issue to me.
I think the next step is to add one mal function to the meter analyzer. Then write the mal rule and build the dashboard. If anyone take over this task, I can also provide support such as the config file of otel collector.
I'm sorry for this situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426979285
> What tag do you need? Tag is not required. For describing airflow server, that could be set through otel collector, like we did for mysql metrics.
I mean that all info is contained in the "metrics name" such as <job_name>, <job_id>, <dag_id>, <task_id>, <operator_name>, and so on. But I have no way to filter and process. Or I just don't consider these metrics? Because of the statsD data format, the otel data collected will not have <key><value> pair tag attributes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427561102
> We could use this(replace ConvertCase to another function) to set the metric name without parameter.
>
> ```
> metrics:
> set(metric.name, ConvertCase(metric.name, "snake"))
> ```
>
> https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md#convert-metric-name-to-snake-case
>
> Meanwhile, we could `set(attributes["job_id"], replace(metric.name, xxx))`. Of course, the `set(attributes...)` must run first, otherwise, the metadata lost.
>
> Could you check what I am missing?
Yes, you are right. And I have tried this before by below config. The most important thing I think is how to process the attributes in `set(attributes["job_id"], replace(metric.name, xxx))`.
```yaml
processors:
transform:
metric_statements:
- context: resource
statements:
- context: datapoint
statements:
- set(attributes["job_id"], metric.name)
- context: metric
statements:
- replace_match(name, "system.*.cpu", "system.cpu")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435685198
> Good to see there is no block. Take your time.
> The planned release for 9.4.0 should be 2-3 weeks away. If your plan is adding this to 9.4.0, let's know. I will check with you before cutting release.
Is there any way to show the received original metrics in the log？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435530227
Good to see there is no block. Take your time.
The planned release for 9.4.0 should be 2-3 weeks away. If your plan is adding this to 9.4.0, let's know. I will check with you before cutting release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435475194
@mufiye Any update about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443572560
> I think you need to check what is delta counter. Counter is increasing or reset. How does delta apply to this case?
I think this dag_processing.process does not meet the Prometheus counter definition, it can decrease, I'm sure because I test it. It is the [pr](https://github.com/apache/airflow/pull/24316) I find.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440142954
If a time-series value is delta, let's say (-5, 4, 3, 1, -4), unless we know the initial value is 10(or any value), we could know the exact value of `process number`(use your example).
So, do we have that number or do we have the total of processes? If there isn't, we only could see the trend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443419647
> In my opinion, I think they always do the accumulation for the counter metrics whether they have been stored or not. But we can not do the accumulation for metrics that have been stored.
If you could push a counter to OAP, we could work on that. Your previous context is about there is a delta only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426981589
@mufiye Do you check OTEL side configurations? Is there a way to change their style? Otherwise, we maybe need to build a stated receiver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440223268
I think about how they visualize this type, so I think we could try this on Prometheus/Grafana.
AFAIK, we only could show this value as a trend, I don't know whether there is something we missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435969506
> I think about Delta, it should be converted as a gauge, no matter if it is `monotonic` or not. @kezhenxu94 What do you think? @mufiye Would you like to try this first as a separate PR?
What the separate PR mean? What should I do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440277029
I don't know as much as you are.
Pick a way you prefer, and we could discuss details when dashboards are out. Adjusting these is not hard.
Don't worry. Everytime, PR takes time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443381234
> If Prometheus could identify/use it as a counter, why can't we? We converted it to delta because it isn't cumulative. What is missed here?
In my opinion, I think they always do the accumulation for the counter metrics weather they have been stored or not. But we can not do the accumulation for metrics have been stored.
I have not verified the process of counter metrics in promethus because I have not learned the golang. It's my future plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443579021
> That is my point of asking. Only focus on this metric, whether they show, how they show.
Ok, I get it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1459378998
@mufiye Any update or block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1416921990
I have done one issue about the agent and I'm interested in oap. I think I can do this task, could you please assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427536236
We could use this(replace ConvertCase to another function) to set the metric name without parameter.
```
metrics:
set(metric.name, ConvertCase(metric.name, "snake"))
```
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md#convert-metric-name-to-snake-case
___
Meanwhile, we could `set(attributes["job_id"], replace(metric.name, xxx))`. Of course, the `set(attributes...)` must run first, otherwise, the metadata lost.
___
Could you check what I am missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427521436
> replace_match would change everything, it may not be a good one. `Split` should be good. You should have an expression to get the array and then use the index.
>
> And I can find the index relative docs, https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md#lists
I think we can not get the single string in the array. The doc says that "the grammar does not provide an accessor to individual list entries".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435969491
I mean you need a pull request to update `OpenTelemetryMetricRequestProcessor` by following above logic. That change should be done first before the airflow features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435948684
> What is the issue? If the source(from otel) is delta, then use SUM to build data per min/hour/day is correct. The recent AWS s3 monitoring is using this way.
My issue is that the skywalking oap can not receive counter data of airflow. I think the reason is the delta counter data will not be received. The relative code is shown below. Did I misunderstand this part of the code？
```java
if (metric.hasSum()) {
final Sum sum = metric.getSum();
if (sum
.getAggregationTemporality() != AGGREGATION_TEMPORALITY_CUMULATIVE) {
return Stream.empty();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435970106
> I mean you need a pull request to update `OpenTelemetryMetricRequestProcessor` by following above logic. That change should be done first before the airflow features.
I would like to try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427508468
Are you considering this too complex? In the transfer process, you should be able to hardcode most of them, right?
> it will replace the key
Tag key is static and hard codes, such as `task_id` a key.
replace_match would change everything, it may not be a good one. `Split` should be good. You should have an expression to get the array and then use the index.
And I can find the index relative docs, https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md#lists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435521598
Have you resolved the issue we discussed on slack?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435975057
> I think about Delta, it should be converted as a gauge, no matter if it is `monotonic` or not. @kezhenxu94 What do you think?
>
> @mufiye Would you like to try this first as a separate PR?
Sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435907540
> `OpenTelemetryMetricRequestProcessor#processMetricsRequest`. There are several debug logs in here.
By reading the skywalking log and code, I have found out why skywalking storage doesn't show some metrics. The "AggregationTemporality" is delta for counter metrics so the skywalking will discord them by `OpenTelemetryMetricRequestProcessor#adaptMetrics`. By viewing the otel collector log, I think "AggregationTemporality" of all counter metrics from airflow is delta. How should I handle this situation? Could you give me some suggestions?
By the way, the timer metrics from the airflow can only be transformed to "ExponentialHistogram" or "Summary" type. I think skywalking doesn't support ExponentialHistogram for otel receiver. So "Summary" type is the only option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435922878
> Do you mean delta is increasement from last report period? Could you explain a little more?
Yes, delta means increasement from last report period.
About skywalking hot to process counter(sum) data:
```java
if (metric.hasSum()) {
final Sum sum = metric.getSum();
if (sum
.getAggregationTemporality() != AGGREGATION_TEMPORALITY_CUMULATIVE) {
return Stream.empty();
}
```
Counter Data from airflow, the AggregationTemporality is Delta:
```
Descriptor:
-> Name: airflow_job_start
-> Description:
-> Unit:
-> DataType: Sum
-> IsMonotonic: false
-> AggregationTemporality: Delta
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435921687
> Is this different from histogram?
The summary shows count, sum and quantile of data. But histogram shows count, sum and bucket of data. I think we can use summary data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440207898
> There is no `all` concept. That is my point on `delta` issue, we never are able to find out the initial value.
>
> Could you check how this works on stated? Such as check and try [apache/airflow#29449](https://github.com/apache/airflow/pull/29449)?
You mean to check how airflow collect metrics and send out stated data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427519864
> Are you considering this too complex? In the transfer process, you should be able to hardcode most of them, right?
>
> > it will replace the key
I think I just say something wrong, I want to say that it will change the value of the relative key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435671717
> Good to see there is no block. Take your time. The planned release for 9.4.0 should be 2-3 weeks away. If your plan is adding this to 9.4.0, let's know. I will check with you before cutting the release.
I'm not sure whether I can finish this in 2-3 weeks. The next step for me is to comb the airflow metrics and write the rule. And I still encounter a problem that data collected by the otel-collector can not seen in the skywalking storage. This problem happens when I use otlp exporter option instead of opencensus option. I think I have lots of things to learn.
By the way, it's time for me to go back school, so I may allocate relatively less time to this. But no matter what, I will do my best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426980374
>
These metadata compose the name, such as "local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426984983
> > > What tag do you need? Tag is not required. For describing airflow server, that could be set through otel collector, like we did for mysql metrics.
> >
> >
> > I mean that all info is contained in the "metrics name" such as <job_name>, <job_id>, <dag_id>, <task_id>, <operator_name>, and so on. But I have no way to filter and process. Or I just don't consider these metrics? Because of the statsD data format, the otel data collected will not have "key value pair" tag attributes.
>
> I doubt that you mixed the concept between "airflow job" and "opentelemetry job"? We use the OpenTelemetry Job name to distinguish data sources.
>
> As for the metrics name like `"local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>"`, I think you should spit them in the OpenTelemetry processor and move the metadata into tags then send to OAP, anyway I'll take a look at Airflow's doc to see what's the case.
I think I say something wrong. It could add "key value pair" tag to the statsD message, but actually airflow only use the name to contain these metadata. I think using OpenTelemetry processor to process the data maybe a feasible method. About the "job", it is just the original [airflow metrics](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html) name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427431020
Could you use `set(target, value)` and `replace_match` to achieve this? Only at this time `replace_match` is using regex to split the value of the specific key, such as dag_id. Such as only matching the text after the 2nd dot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426959038
Hello, @wu-sheng . I find that all data opentelemetry collector received use tag to compose metrics name but have no tags to process. I have no idea how to write the mal rules to process these data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440000156
Could you share what is `exponentialHistogram`? What does `exponential` mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440118805
> Then, in this case, it seems we never get the absolute value, is it? Does it report absolute time somehow?
sorry, I can't get it, could you explain your perspective more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440111805
Then, in this case, it seems we never get the absolute value, is it? Does it report absolute time somehow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435515677
> @mufiye Any update about this?
Sorry, little update these days. I successfully send data to skywalking oap from airflow. I want to ask that for the otel-collector, which the exporter option should we choose, otlp or opencensus?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435521512
otlp is a better choice. oc is merged into otlp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1416923750
> Assigned. Good luck.
Ok, I will do my best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1417113921
> @mufiye You could take one step at a time. Make metrics available for airflow first. Then move forward on logs.
Ok, I will do the metrics part first. And I think I can refer to other similar issues about how to add the metrics dashboard, such as issue#9677.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426981376
> >
>
>
>
> These metadata compose the name, such as "local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>".
OK, if it has, we could write a small MAL script(groovy based) to split these matadata.
But meanwhile, I think how to match these metrics are a little challenging. @kezhenxu94 @hanahmily @wankai123 What do you suggest? Do we have to write a regex based analysis?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426980619
@potiuk Do you have time to help?
We want to monitor airflow with meteics having job/dag/task IDs to group metrics rather than just metrics for the whole airflow server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435912431
> "ExponentialHistogram" or "Summary" type. I think skywalking doesn't support ExponentialHistogram for otel receiver. So "Summary" type is the only option.
Is this different from histogram?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435969098
> I think about Delta, it should be converted as a gauge, no matter if it is `monotonic` or not. @kezhenxu94 What do you think? @mufiye Would you like to try this first as a separate PR?
Does the separate PR mean only metrics part for airflow monitor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435971745
I think it should be not hard. Try to follow `AGGREGATION_TEMPORALITY_CUMULATIVE`'s not `Monotonic` case.
If this metric is going to report at least once per minute, we should be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1417105711
@mufiye You could take one step at a time. Make metrics available for airflow first. Then move forward on logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] potiuk commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427239675
Not much time to help (catching up with some stuff) , but from what it is worth - statsd of Airflow is not the "best" to consume for Skywalking - unforatunately you'd indeed need to parse the metric name and while I am not sure how OTEL processor might work, regexp approach might be a good idea.
However just to give you perspective - Airflow's metrics are evolving.
Quite recently (coming in next version of Airflow) - 2.6 most likely @hussein-awala improved Statsd metrics with DataDog metadata tags - https://github.com/apache/airflow/pull/28961 and maybe, rather than focusing on pure statsd metrics you could integrate those.
Also - a bit more long term - In Airlfow we already approved Open Telemetry support for Airflow https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow and we even have a chance to progress with the implementation - @feruzzi is looking into the integration and even is adding better support for Airflow's statsd metrics testing in Breeze (Airflow development environment) with Grafana and Prometheus - https://github.com/apache/airflow/pull/29449
So maybe it could be a nice teamwork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427386874
I have done the research. And start a [new discussion](https://github.com/open-telemetry/opentelemetry-collector-contrib/discussions/18520) in the opentelemetry collector contrib.
I have tried to use the [metrics transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor), [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) and [attributes processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor).
The metrics transform processor's combine function can process the counter and gauge data, but can not process summary and histogram data. It will encounter some problems when processing histogram data it will report errors in the terminal, and for summary data, the datapoint part of the result data is null.
For the transform processor, I could use [functions](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#replace_pattern) like replace_pattern to make the metrics name concise, such as "ti.finish.<dag_id>.<task_id>." to "ti.finish", but I have no idea how to add the <dag_id> and <task_id> into the attributes, maybe the Split [function](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#Split) to split name by "." and set function, but it does not work.
Could anyone give me some suggestions about my next step? I think I can try to process the counter and gauge in the stasD -> otel collector -> skywalking case. Or anyone who is familiar with opentelemetry collector contrib could have solution to solve the problem that splitting metrics names into key-value pair attributes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426982146
> @mufiye Do you check OTEL side configurations? Is there a way to change their style? Otherwise, we maybe need to build a stated receiver.
I think maybe the processor of the otel collector can do this and I need to check this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435930813
>
>
>
>
>
>
> > Do you mean delta is increasement from last report period? Could you explain a little more?
>
>
>
> I have researched the stated receiver, processors, and otlp exporter of otel receiver to transform Delta metrics to CUMULATIVE metrics, but not found a component can do this.
>
>
Check my last comment. Delta is good, which mean increasement in the period. We sum them could get accurate count per dimension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435929628
> Do you mean delta is increasement from last report period? Could you explain a little more?
I have researched the stated receiver, processors, and otlp exporter of otel receiver to transform Delta metrics to CUMULATIVE metrics, but not found a component can do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1439991802
> > > Is this different from histogram?
> >
> >
> > The summary shows count, sum and quantile of data. But histogram shows count, sum and bucket of data. I think we can use summary data.
>
> AFAIK, MAL supports histogram, which could access counter, and bucket to get avg or percentile. But summary is not not supported, and it has less precision. If histogram works, we should never choose summary.
The stated receiver in otel collector will transform the `timer metric` in airflow to `exponentialHistogram` as histogram type . But our skywalking otel-receiver `OpenTelemetryMetricRequestProcessor#adaptMetrics` can not support this type. Should I make this `exponentialHistogram` type be supported or use summary type for timer metric?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440059988
About <1>, the easiest way is,
- `pool` seems a running env, we could catalog it as an instances, naming through `pool:xxx`. Is the pool shares among tasks?
- the `job`, `dag`, `operation_name` could be various endpoints as running processes. We could name them by following `/job/xxxx`, `/dag/yyy`. Does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443451981
> > In my opinion, I think they always do the accumulation for the counter metrics whether they have been stored or not. But we can not do the accumulation for metrics that have been stored.
>
> If you could push a counter to OAP, we could work on that. Your previous context is about there is a delta only.
I think I can only push a "delta type counter" to the oap by otel collector. I think maybe we can support to accumulate "delta type counter"? It may be complicated but I can try to do it.
Or I just show the data trend by "delta type counter" data and do the dashboard first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1426979959
Does the original statsd have these metadata?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440021773
I just read https://opentelemetry.io/docs/reference/specification/metrics/data-model/#exponentialhistogram, it seems it is just the typical Prometheus Histogram setup in practice.
Back to you question
> Should I make this exponentialHistogram type be supported or use summary type for timer metric?
We should transfer this to our histogram, I think. You need to get the bucket transfer correctly from `exponentialHistogram` to `histogram`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440062905
> But for some nonmonotonic delta counter metric, such as "dag_processing.processes" which means number of currently running DAG parsing processes, it can be negative in one period in my test. How should we handle this kind of metric?
How do the process could be negative? What does it mean originally? `number of currently running DAG parsing processes` should be 0 or positive logically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440235942
> I think about how they visualize this type, so I think we could try this on Prometheus/Grafana. AFAIK, we only could show this value as a trend, I don't know whether there is something we missed.
Ok, I get it. I will check how they visualize their metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435968567
I think about Delta, it should be converted as a gauge, no matter if it is `monotonic` or not. @kezhenxu94 What do you think?
@mufiye Would you like to try this first as a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435929696
> > Is this different from histogram?
>
>
>
> The summary shows count, sum and quantile of data. But histogram shows count, sum and bucket of data. I think we can use summary data.
AFAIK, MAL supports histogram, which could access counter, and bucket to get avg or percentile. But summary is not not supported, and it has less precision.
If histogram works, we should never choose summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440271005
> About <1>, the easiest way is,
>
> * `pool` seems a running env, we could catalog it as an instances, naming through `pool:xxx`. Is the pool shares among tasks?
> * the `job`, `dag`, `operation_name` could be various endpoints as running processes. We could name them by following `/job/xxxx`, `/dag/yyy`. Does this make sense?
I think tasks and pool are inclusion relation, but others are not. Furthermore, by the metric name, we can not get which task is in which pool. Maybe make these components' level same is the only way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443372879
> apache/airflow#29449
I think it is because the counter definition in [promethus metrics](https://prometheus.io/docs/concepts/metric_types/#counter). A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443375296
If Prometheus could identify/use it as a counter, why can't we? We converted it to delta because it isn't cumulative.
What is missed here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443544664
I think you need to check what is delta counter. Counter is increasing or reset. How does delta apply to this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1459455610
Got it. Thanks for the feedback.
Take your time for your own interest. That always matters primarily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1416923538
Assigned. Good luck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427567635
From OTEL perspective, I think you only have to do is splitting `local_task_job.task_exit.<job_id>.<dag_id>.<task_id>.<return_code>` as metric name with all other things as a parameter tag.
Because on the OAP side, the MAL engine is powered by Groovy, which is much more powerful and flexible than OTEL processor. You could finish the left processes there.
Is this possible? I am at Slack for DM, if you have any trouble, ping me there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1427498760
> Could you use `set(target, value)` and `replace_match` to achieve this? Only at this time `replace_match` is using regex to split the value of the specific key, such as dag_id. Such as only matching the text after the 2nd dot.
I try it but it doesn't work. Because the third argument of `replace_match` is the string, and it will replace the key, we can't get the dag_id and use it as the third argument in the `replace_match` function. I have also tried the `Split` function before, but it will return a string array and the transform processor does not provide the index operation of this array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435689889
`OpenTelemetryMetricRequestProcessor#processMetricsRequest`. There are several debug logs in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1435930019
> > Do you mean delta is increasement from last report period? Could you explain a little more?
>
>
>
> Yes, delta means increasement from last report period.
>
> About skywalking how to process counter(sum) data:
>
> ```java
>
> if (metric.hasSum()) {
>
> final Sum sum = metric.getSum();
>
> if (sum
>
> .getAggregationTemporality() != AGGREGATION_TEMPORALITY_CUMULATIVE) {
>
> return Stream.empty();
>
> }
>
> ```
>
> Counter Data from airflow, the AggregationTemporality is Delta:
>
> ```
>
> Descriptor:
>
> -> Name: airflow_job_start
>
> -> Description:
>
> -> Unit:
>
> -> DataType: Sum
>
> -> IsMonotonic: false
>
> -> AggregationTemporality: Delta
>
> ```
>
>
What is the issue? If the source(from otel) is delta, then use SUM to build data per min/hour/day is correct. The recent AWS s3 monitoring is using this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1443577422
That is my point of asking. Only focus on this metric, whether they show, how they show.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440151974
> If a time-series value is delta, let's say (-5, 4, 3, 1, -4), unless we know the initial value is 10(or any value), we could know the exact value of `process number`(use your example).
>
> So, do we have that number or do we have the total of processes? If there isn't, we only could see the trend.
I think we can't get the total number of processes unless we add every delta value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440165085
There is no `all` concept. That is my point on `delta` issue, we never are able to find out the initial value.
Could you check how this works on stated? Such as check and try https://github.com/apache/airflow/pull/29449?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440051786
> I just read https://opentelemetry.io/docs/reference/specification/metrics/data-model/#exponentialhistogram, it seems it is just the typical Prometheus Histogram setup in practice.
>
> Back to you question
>
> > Should I make this exponentialHistogram type be supported or use summary type for timer metric?
>
> We should transfer this to our histogram, I think. You need to get the bucket transfer correctly from `exponentialHistogram` to `histogram`.
I will try to do it later. And there are some other essential points that need to be discussed.
1. There are some labels in airflow metric names such as <job_name>, <task_id>, <dag_id>, <operator_name>, <pool_name> which represent the components in airflow. Which level should I classify these components to? In airflow concepts, `dag` contains lots of `tasks` to be run, `pool` is where tasks run in, `operator_name` is one kind of `task` and `job` I think is a larger concept than a task because it also includes the scheduler job. I think we classify all these components as endpoints?
2. As before, we transform the "delta counter metric" to "gauge metric", we can represent some monotonic delta metric as the current metric in one specific period of time. For example, `ti_successes` metric describes "Overall task instances successes", so we can think the gauge metric as "the successful task instances in this period". But for some nonmonotonic delta counter metric, such as "dag_processing.processes" which means number of currently running DAG parsing processes, it can be negative in one period in my test. How should we handle this kind of metric? Maybe just show them originally for showing the trend of the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mufiye commented on issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "mufiye (via GitHub)" <gi...@apache.org>.
mufiye commented on issue #10341:
URL: https://github.com/apache/skywalking/issues/10341#issuecomment-1440066587
> > But for some nonmonotonic delta counter metric, such as "dag_processing.processes" which means number of currently running DAG parsing processes, it can be negative in one period in my test. How should we handle this kind of metric?
>
> How do the process could be negative? What does it mean originally? `number of currently running DAG parsing processes` should be 0 or positive logically.
Because the total number which is the sum of the gauge value means currently running DAG parsing processes. So one delta value can be negative. The "originally" means we just show the gauge value whether they are negative or positive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #10341: [Feature] Setup dashboard for Airflow monitoring
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #10341: [Feature] Setup dashboard for Airflow monitoring
URL: https://github.com/apache/skywalking/issues/10341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org