You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "weixiang1862 (via GitHub)" <gi...@apache.org> on 2024/12/20 06:07:20 UTC
[PR] Fix `ServerStatusService.statusWatchers` concurrent modification. [skywalking]
weixiang1862 opened a new pull request, #12884:
URL: https://github.com/apache/skywalking/pull/12884
<!--
⚠️ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
### Fix `ServerStatusService.statusWatchers` concurrent modification.
- [ ] Add a unit test to verify that the fix works.
- [x] Explain briefly why the bug exists and how to fix it.
<!-- ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👇 ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking/blob/master/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👆 ==== -->
<!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
`ServerStatusService#registerWatcher` called in main thread and `ServerStatusService#rebalancedCluster` called in
cluster coordinator thread, this may cause concurrent modification of statusWatchers ArrayList when oap server bootstrap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix `ServerStatusService.statusWatchers` concurrent modification. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12884:
URL: https://github.com/apache/skywalking/pull/12884#discussion_r1893535536
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java:
##########
@@ -64,6 +64,10 @@ public void bootedNow(List<ApplicationConfiguration.ModuleConfiguration> configu
}
public void rebalancedCluster(long rebalancedTime) {
+ if (!bootingStatus.isBooted()) {
+ return;
+ }
Review Comment:
OK, thanks for the explanation. Then you need to move `bootingStatus.setBooted(true);` to the last line of the `bootedNow` method. Otherwise, the race condition could still happen, right?
> I think notice once is enough in bootstrap stage. If you concern about cluster_rebalanced_time telemetry metric, how about move this return judge after cluster_rebalanced_time creating?
That is not correct. Because actually, we don't do rebalancing notifications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix `ServerStatusService.statusWatchers` concurrent modification. [skywalking]
Posted by "weixiang1862 (via GitHub)" <gi...@apache.org>.
weixiang1862 commented on code in PR #12884:
URL: https://github.com/apache/skywalking/pull/12884#discussion_r1893532228
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java:
##########
@@ -64,6 +64,10 @@ public void bootedNow(List<ApplicationConfiguration.ModuleConfiguration> configu
}
public void rebalancedCluster(long rebalancedTime) {
+ if (!bootingStatus.isBooted()) {
+ return;
+ }
Review Comment:
ServerStatusWatcher will be notified by the method `ServerStatusService#bootNow` right after bootstrap complete.
I think notice once is enough in bootstrap stage. If you concern about `cluster_rebalanced_time` telemetry metric, how about move this return judge after `cluster_rebalanced_time` creating?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix `ServerStatusService.statusWatchers` concurrent modification. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #12884:
URL: https://github.com/apache/skywalking/pull/12884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix `ServerStatusService.statusWatchers` concurrent modification. [skywalking]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #12884:
URL: https://github.com/apache/skywalking/pull/12884#discussion_r1893519042
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java:
##########
@@ -64,6 +64,10 @@ public void bootedNow(List<ApplicationConfiguration.ModuleConfiguration> configu
}
public void rebalancedCluster(long rebalancedTime) {
+ if (!bootingStatus.isBooted()) {
+ return;
+ }
Review Comment:
I think we should wait here rather than `return`? If we skip this step, then rebalancedTime and the telemetry metric of `cluster_rebalanced_time` are not being updated. Worse, ServerStatusWatcher implementations would not get any notification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org