You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "peachisai (via GitHub)" <gi...@apache.org> on 2024/02/01 14:43:30 UTC
[PR] Add count function for MAL down sampling. [skywalking]
peachisai opened a new pull request, #11859:
URL: https://github.com/apache/skywalking/pull/11859
<!--
⚠️ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
<!-- ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👇 ====
### Fix <bug description or the bug issue number or bug issue link>
- [ ] Add a unit test to verify that the fix works.
- [ ] Explain briefly why the bug exists and how to fix it.
==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👆 ==== -->
<!-- ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👇 ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👆 ==== -->
<!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
This is my scenario
In ```rocketmq``` different consumer group has a different consumer offset
```
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group3",} 347.0
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group2",} 601.0
rocketmq_consumer_offset{cluster="DefaultCluster",broker="broker-a",topic="topic1",group="group1",} 694.0
```
Downsampling the metrics count
```
- name: consumer_group_count
exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).downsampling(COUNT)
```
Then I can get the consumer group number to caculate the blacklog messages
```
producer_offset*consumer_group_count- consumer_offset
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922613784
> Sorry, I don't seem to understand the point. Do you expect the result of consumer_group_count is `3`, according to the following expression?
>
> ```
> - name: consumer_group_count
> exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).downsampling(COUNT)
> ```
Yes,I want to acqurie the number of consumer group count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922743531
> > Yes,I want to acqurie the number of consumer group count
>
> Then I think you need a `MAL count func rather than a downsampling func`, such as
>
> ```
> rocketmq_consumer_offset.count()
> ```
>
> Downsampling is used to aggregate the result of the metric in (m/h/d).
It seems cannot invoke
```
2024-02-02 11:28:54,508 org.apache.skywalking.oap.meter.analyzer.dsl.Expression 92 [main] ERROR [] - [10.0.0-SNAPSHOT-8fb408a] failed to run "(rocketmq_consumer_offset.sum(['cluster','topic','group']).count()).tag({tags -> tags.cluster = 'rocketmq::' + tags.cluster}).endpoint(['cluster'], ['topic'], Layer.ROCKETMQ)"
groovy.lang.MissingMethodException: No signature of method: org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily.count() is applicable for argument types: () values: []
Possible solutions: print(java.io.PrintWriter), print(java.lang.Object), minus(java.lang.Number), minus(org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily), wait(), any()
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:72) ~[groovy-4.0.15.jar:4.0.15]
at org.codehaus.groovy.vmplugin.v8.IndyGuardsFiltersAndSignatures.unwrap(IndyGuardsFiltersAndSignatures.java:163) ~[groovy-4.0.15.jar:4.0.15]
at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321) ~[groovy-4.0.15.jar:4.0.15]
at Script1.run(Script1.groovy:1) ~[?:?]
at org.apache.skywalking.oap.meter.analyzer.dsl.Expression.run(Expression.java:81) ~[classes/:?]
at org.apache.skywalking.oap.meter.analyzer.dsl.Expression.parse(Expression.java:59) ~[classes/:?]
at org.apache.skywalking.oap.meter.analyzer.Analyzer.build(Analyzer.java:92) ~[classes/:?]
at org.apache.skywalking.oap.meter.analyzer.MetricConvert.buildAnalyzer(MetricConvert.java:74) ~[classes/:?]
at org.apache.skywalking.oap.meter.analyzer.MetricConvert.lambda$new$2(MetricConvert.java:61) ~[classes/:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.apache.skywalking.oap.meter.analyzer.MetricConvert.<init>(MetricConvert.java:67) ~[classes/:?]
at org.apache.skywalking.oap.meter.analyzer.prometheus.PrometheusMetricConverter.<init>(PrometheusMetricConverter.java:73) ~[classes/:?]
at org.apache.skywalking.oap.server.receiver.otel.otlp.OpenTelemetryMetricRequestProcessor.lambda$start$8(OpenTelemetryMetricRequestProcessor.java:130) ~[classes/:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) [?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) [?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) [?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) [?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) [?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) [?:?]
at org.apache.skywalking.oap.server.receiver.otel.otlp.OpenTelemetryMetricRequestProcessor.start(OpenTelemetryMetricRequestProcessor.java:131) [classes/:?]
at org.apache.skywalking.oap.server.receiver.otel.OtelMetricReceiverProvider.start(OtelMetricReceiverProvider.java:93) [classes/:?]
at org.apache.skywalking.oap.server.library.module.BootstrapFlow.start(BootstrapFlow.java:46) [classes/:?]
at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:75) [classes/:?]
at org.apache.skywalking.oap.server.starter.OAPServerBootstrap.start(OAPServerBootstrap.java:52) [classes/:?]
at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:23) [classes/:?]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922792369
> > > > Yes,I want to acqurie the number of consumer group count
>
> > >
>
> > >
>
> > > Then I think you need a `MAL count func rather than a downsampling func`, such as
>
> > > ```
>
> > >
>
> > > rocketmq_consumer_offset.count()
>
> > > ```
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > Downsampling is used to aggregate the result of the metric in (m/h/d).
>
> >
>
> > And avg or latest for downsampling seems more reasonable?
>
>
>
> It seems not compatible
I mean it is suitable if you have counter for general function, rather than downsampling.
How to count for downsampling would be very tricky. One day could have dozens of merging, it doesn't indicate your original meaning anymore, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922168140
How to make this count accurate when downsampling mechanism affect in hour and day precision?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1925738888
> I will use `count` aggragation in place of the `count` downsampling
I am doubt about this logic. Because logically, count for downsampling only works for counting events, offset can't be aggregated or counted between latest data and existing data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922747223
> It seems not compatible
`count()` is not supported yet, you need to implement it if required.
https://github.com/apache/skywalking/blob/fc3cc4bd878e41b2eee84a0bbed3f3808b3ef9b9/docs/en/concepts-and-designs/mal.md#aggregation-operation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai closed pull request #11859: Add `count` function for MAL down sampling.
URL: https://github.com/apache/skywalking/pull/11859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922808031
> > > > > Yes,I want to acqurie the number of consumer group count
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > Then I think you need a `MAL count func rather than a downsampling func`, such as
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > rocketmq_consumer_offset.count()
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > Downsampling is used to aggregate the result of the metric in (m/h/d).
> >
> >
> > >
> >
> >
> > > And avg or latest for downsampling seems more reasonable?
> >
> >
> > It seems not compatible
>
> I mean it is suitable if you have counter for general function, rather than downsampling. How to count for downsampling would be very tricky. One day could have dozens of merging, it doesn't indicate your original meaning anymore, right?
Oh, I think I make some misunderstands, I mean the ```count``` aggregation function is not compatible with the code now, I just kown I should implement it. I also agree that ```count``` by aggregation function and downsampling the latest meter is better
I will try to implement the ```count``` aggregation function later.
> > > > > Yes,I want to acqurie the number of consumer group count
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > Then I think you need a `MAL count func rather than a downsampling func`, such as
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > rocketmq_consumer_offset.count()
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > >
> >
> >
> > > > Downsampling is used to aggregate the result of the metric in (m/h/d).
> >
> >
> > >
> >
> >
> > > And avg or latest for downsampling seems more reasonable?
> >
> >
> > It seems not compatible
>
> I mean it is suitable if you have counter for general function, rather than downsampling. How to count for downsampling would be very tricky. One day could have dozens of merging, it doesn't indicate your original meaning anymore, right?
Oh, I think I make some misunderstands, I mean the ```count``` aggregation function is not compatible with the code now, I just kown I should implement it. I also agree that ```count``` by aggregation function and downsampling the latest meter is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1925733792
I will use `count` aggragation in place of the `count` downsampling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922664493
> > Yes,I want to acqurie the number of consumer group count
>
>
>
> Then I think you need a `MAL count func rather than a downsampling func`, such as
>
> ```
>
> rocketmq_consumer_offset.count()
>
> ```
>
> Downsampling is used to aggregate the result of the metric in (m/h/d).
>
>
>
>
And avg or latest for downsampling seems more reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922744810
> groovy.lang.MissingMethodException: No signature of method: org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily.count() is applicable for argument types: () values: []
> Possible solutions: print(java.io.PrintWriter), print(java.lang.Object), minus(java.lang.Number), minus(org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily), wait(), any()
It seems not compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922634419
> Yes,I want to acqurie the number of consumer group count
Then I think you need a `MAL count func rather than a downsampling func`, such as
```
rocketmq_consumer_offset.count()
```
Downsampling is used to aggregate the result of the metric in (m/h/d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "wankai123 (via GitHub)" <gi...@apache.org>.
wankai123 commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922605774
Sorry, I don't seem to understand the point. Do you expect the result of consumer_group_count is `3`, according to the following expression?
```
- name: consumer_group_count
exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).downsampling(COUNT)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922756988
> > It seems not compatible
>
> `count()` is not supported yet, you need to implement it if required. https://github.com/apache/skywalking/blob/fc3cc4bd878e41b2eee84a0bbed3f3808b3ef9b9/docs/en/concepts-and-designs/mal.md#aggregation-operation
Okay,Thank you, I will have a try laterly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `count` function for MAL down sampling. [skywalking]
Posted by "peachisai (via GitHub)" <gi...@apache.org>.
peachisai commented on PR #11859:
URL: https://github.com/apache/skywalking/pull/11859#issuecomment-1922745954
> > > Yes,I want to acqurie the number of consumer group count
> >
> >
> > Then I think you need a `MAL count func rather than a downsampling func`, such as
> > ```
> >
> > rocketmq_consumer_offset.count()
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Downsampling is used to aggregate the result of the metric in (m/h/d).
>
> And avg or latest for downsampling seems more reasonable?
It seems not compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org