You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "weixiang1862 (via GitHub)" <gi...@apache.org> on 2024/01/17 01:40:23 UTC
[PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
weixiang1862 opened a new pull request, #11788:
URL: https://github.com/apache/skywalking/pull/11788
<!--
⚠️ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
### Fix topN record calculation
- [x] Add a unit test to verify that the fix works.
- [x] Explain briefly why the bug exists and how to fix it.
<!-- ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👇 ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👆 ==== -->
<!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
In `LimitedSizeBufferedData`, it uses `HashMap<StorageId, LinkedList<STORAGE_DATA>>` to stash records.
With data storageId as topN group key, topn records have no chance to compare with each other, so every record will be stashed into the map and finally persisent into db. In addition, continuous record stash in a topn period will bring pressure to oap memory.
So I change topn group key to `entityId + timeBucket`, then stash map can only reserver topn of every entity(db, cache) in minute level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11788:
URL: https://github.com/apache/skywalking/pull/11788#issuecomment-1894798635
I will check this later today. Thanks for addressing the kernel level issue.
fyi @apache/skywalking-committers This should exist from the beginning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11788:
URL: https://github.com/apache/skywalking/pull/11788#discussion_r1454813710
##########
docs/en/changes/changes.md:
##########
@@ -23,6 +23,7 @@
* Fix SumPerMinFunctionTest error function.
* Remove unnecessary annotations and functions from Meter Functions.
* Add `max` and `min` functions for MAL down sampling.
+* Change topN group key from `StorageId` to `entityId + timeBucket`.
Review Comment:
```suggestion
* Fix critical bug of uncontrolled memory cost of TopN statistics. Change topN group key from `StorageId` to `entityId + timeBucket`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
Posted by "ZhengBing520 (via GitHub)" <gi...@apache.org>.
ZhengBing520 commented on PR #11788:
URL: https://github.com/apache/skywalking/pull/11788#issuecomment-1991046398
> > > So I change topN group key to entityId + timeBucket, then stash map can only reserve topN of every entity(db, cache) in the minute level.
> >
> >
> > The timeBucket attribute in topN is in seconds, not minutes
>
> If this is the case, we face a bug. Could you fix this?
I saw this PR（ https://github.com/apache/skywalking/pull/11792 ）Fixed it, so I deleted the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11788:
URL: https://github.com/apache/skywalking/pull/11788#issuecomment-1991039971
> > So I change topN group key to entityId + timeBucket, then stash map can only reserve topN of every entity(db, cache) in the minute level.
>
>
>
> The timeBucket attribute in topN is in seconds, not minutes
If this is the case, we face a bug. Could you fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #11788:
URL: https://github.com/apache/skywalking/pull/11788
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Change topN group key from `StorageId` to `entityId + timeBucket`. [skywalking]
Posted by "ZhengBing520 (via GitHub)" <gi...@apache.org>.
ZhengBing520 commented on PR #11788:
URL: https://github.com/apache/skywalking/pull/11788#issuecomment-1991036936
> So I change topN group key to entityId + timeBucket, then stash map can only reserve topN of every entity(db, cache) in the minute level.
The timeBucket attribute in topN is in seconds, not minutes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org