You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "peachisai (via GitHub)" <gi...@apache.org> on 2024/02/04 13:00:35 UTC
[PR] Add `count` aggregation function for MAL [skywalking]
peachisai opened a new pull request, #11869:
URL: https://github.com/apache/skywalking/pull/11869
<!--
⚠️ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
<!-- ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👇 ====
### Fix <bug description or the bug issue number or bug issue link>
- [ ] Add a unit test to verify that the fix works.
- [ ] Explain briefly why the bug exists and how to fix it.
==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👆 ==== -->
<!-- ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👇 ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👆 ==== -->
<!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
This is my scenario
In rocketmq different consumer group has a different consumer offset
```
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group3",} 347.0
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group2",} 601.0
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group1",} 694.0
```
```
- name: consumer_group_count
exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).count(['group'])
```
Then I can get the consumer group number to caculate the blacklog messages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928645078
> I was talking with Kai, I am wondering whether number of the remaining messages should be good to measure by group? Rather all for all grouping? If something goes wrong, operation team should care more about the remaining per group.
Do you mean we should caculate the remaining msg in each group?
The topic dimenssion provides each group offset
```
- name: consumer_group_offset
exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).downsampling(MAX)
```
Or should I use this?
```
producer_offset-consumer_group_offset
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926033228
```
- name: consumer_group_count
exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).count(['group'])
```
I think the result of your expression is:
```
rocketmq_consumer_offset{group="group3"} 1
rocketmq_consumer_offset{group="group2"} 1
rocketmq_consumer_offset{group="group1"} 1
```
Is that expected?
please refer to the other UT test cases and add yours:
https://github.com/apache/skywalking/blob/master/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926112212
>Yes, After the sum aggregation. I will serve UT tests up later
`sum` is used to aggregate the value of the metrics, I think is unnecessary to use it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928658782
> Do you mean we should caculate the remaining msg in each group?
For MQ case, yes, that should be more useful.
> producer_offset-consumer_group_offset
Does this work? I think minus only works when labels are same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926122582
> I need to pre-aggregation on the ['cluster','topic','group'] dimension
you should aggregate the dimension in func `count` directly. Please refer other aggregate func `max/min/avg`, we don't need aggregate twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926119964
After the sum, why we still have group label to count?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926321428
From the rocketMQ official exporter doc I found these metrics samples:
```
# HELP rocketmq_producer_offset TopicOffset
# TYPE rocketmq_producer_offset counter
rocketmq_producer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_tfq",} 1878633.0
rocketmq_producer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_tfq",} 3843787.0
rocketmq_producer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_topic_tfq",} 2798195.0
rocketmq_producer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_topic_tfq",} 145
# HELP rocketmq_consumer_offset GroupOffset
# TYPE rocketmq_consumer_offset counter
rocketmq_consumer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_topic_tfq",group="DEV_CID_consumer_cfq",} 1462030.0
rocketmq_consumer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_tfq",group="DEV_CID_cfq",} 3843787.0
rocketmq_consumer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_topic_tfq",group="DEV_CID_consumer_cfq",} 2800569.0
rocketmq_consumer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_tfq",group="DEV_CID_cfq",} 1878633.0
```
Could you give more details on how to calculate the backlog messages?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926118902
> rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group3",} 347.0
```
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group3",} 347.0
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-b",topic="topic1",group="group3",} 347.0
```
I need to pre-aggregation on the ```['cluster','topic','group']``` dimension
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1925744177
Please provide UTs for the new functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #11869:
URL: https://github.com/apache/skywalking/pull/11869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928564055
> Could you explain (producer_offset*consumer_group_count-consumer_offset)? Producer is somehow related to group but consuming doesn't?
This is original metrics
```
```
> Could you explain (producer_offset*consumer_group_count-consumer_offset)? Producer is somehow related to group but consuming doesn't?
The concept of a consumer group is different from that of a producer group. Each consumer group has its own independent offset, and the group of producer is no longer in use in 5.x version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926412830
> From the rocketMQ official exporter doc I found these metrics samples:
>
> ```
> # HELP rocketmq_producer_offset TopicOffset
> # TYPE rocketmq_producer_offset counter
> rocketmq_producer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_tfq",} 1878633.0
> rocketmq_producer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_tfq",} 3843787.0
> rocketmq_producer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_topic_tfq",} 2798195.0
> rocketmq_producer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_topic_tfq",} 145
>
> # HELP rocketmq_consumer_offset GroupOffset
> # TYPE rocketmq_consumer_offset counter
> rocketmq_consumer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_topic_tfq",group="DEV_CID_consumer_cfq",} 1462030.0
> rocketmq_consumer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_tfq",group="DEV_CID_cfq",} 3843787.0
> rocketmq_consumer_offset{cluster="MQCluster",broker="broker-a",topic="DEV_TID_topic_tfq",group="DEV_CID_consumer_cfq",} 2800569.0
> rocketmq_consumer_offset{cluster="MQCluster",broker="broker-b",topic="DEV_TID_tfq",group="DEV_CID_cfq",} 1878633.0
> ```
>
> Could you give more details on how to calculate the backlog messages?
```
- name: producer_offset
exp: rocketmq_producer_offset.sum(['cluster','topic']).downsampling(MAX)
- name: consumer_group_count
exp: rocketmq_consumer_offset.count(['cluster','topic','group'])
- name: consumer_offset
exp: rocketmq_consumer_offset.sum(['cluster','topic']).downsampling(MAX)
```
producer_offset*consumer_group_count-consumer_offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928594577
The `count` function implementation looks good, I think we can reserve it. What we are concerned about is the calculation of `producer_offset*consumer_group_count-consumer_offse`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1926086153
> ```
> - name: consumer_group_count
> exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).count(['group'])
> ```
>
> I think the result of your expression is:
>
> ```
> rocketmq_consumer_offset{group="group3"} 1
> rocketmq_consumer_offset{group="group2"} 1
> rocketmq_consumer_offset{group="group1"} 1
> ```
>
> Is that expected?
>
> please refer to the other UT test cases and add yours: https://github.com/apache/skywalking/blob/master/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java
Yes, After the ```sum``` aggregation. I will serve UT tests up later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928680349
> I am not on my computer now, I will test it later, maybe MQE could works?
MQE also requires they have the same labels or `one side without any labels`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928585343
I was talking with Kai, I am wondering whether number of the remaining messages should be good to measure by group? Rather all for all grouping?
If something goes wrong, operation team should care more about the remaining per group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` aggregation function for MAL [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11869:
URL: https://github.com/apache/skywalking/pull/11869#issuecomment-1928677422
> > Do you mean we should caculate the remaining msg in each group?
>
> For MQ case, yes, that should be more useful.
>
> > producer_offset-consumer_group_offset
>
> Does this work? I think minus only works when labels are same.
I am not on my computer now, I will test it later, maybe MQE could works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org