You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2022/06/29 02:52:35 UTC
[GitHub] [dubbo] tietang opened a new pull request, #10234: 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
tietang opened a new pull request, #10234:
URL: https://github.com/apache/dubbo/pull/10234
## What is the purpose of the change
原来为使用redis scan命令来扫描 /dubbo/com.xxx.yyy.XxxService/*的key
按照https://dubbo.apache.org/zh/docsv2.7/user/references/registry/redis/对redis注册中心的介绍
只有/dubbo/com.xxx.yyy.XxxService/providers 和/dubbo/com.xxx.yyy.XxxService/consumers 2种可能性。
在在使用ClusterRedisClient时，scan命令（参考：org.apache.dubbo.remoting.redis.jedis.ClusterRedisClient.scan）是迭代循环所有cluster node
来扫描每一个node，并合并扫描结果完成的，scan命令本来就很慢，使用ClusterRedisClient时会随着节点越多越慢
我的场景中有6个节点，每次需要5~20秒的扫描时间，每次启动时很慢且卡在初始化reference阶段；或者泛化每接口第一次调用或者硬编码每接口第一次调用时，都会构建Reference，构建reference时花费了很多时间
如果项目中service 接口很多时，有可能在启动或者重启后会导致worker线程阻塞而请求无法正常调用，目前在测试环境已经遇到了该现象，高并发请求时有可能引起雪奔，特别是存在dubbo http网关时的场景。
经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在300~1000ms即可构建好reference。
测试环境多线程（50线程）预创建48个reference需要，同机房1985ms（优化前9839ms）提升5倍，跨机房（同地域）3276ms（优化前35303ms）提升10倍。
本地环境（远程连接测试环境redis）多线程（50线程）预创建48个reference需要18288ms（优化前436235ms，约7分钟多，提升23倍）。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] tietang commented on pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by GitBox <gi...@apache.org>.
tietang commented on PR #10234:
URL: https://github.com/apache/dubbo/pull/10234#issuecomment-1169478184
#10227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] codecov-commenter commented on pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10234:
URL: https://github.com/apache/dubbo/pull/10234#issuecomment-1180073332
# [Codecov](https://codecov.io/gh/apache/dubbo/pull/10234?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10234](https://codecov.io/gh/apache/dubbo/pull/10234?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d3d1075) into [master](https://codecov.io/gh/apache/dubbo/commit/c1a76a06a8e60b4dd97041611ca3521ca6fc322a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c1a76a0) will **decrease** coverage by `0.41%`.
> The diff coverage is `66.66%`.
```diff
@@ Coverage Diff @@
## master #10234 +/- ##
============================================
- Coverage 60.95% 60.54% -0.42%
+ Complexity 448 444 -4
============================================
Files 1100 1100
Lines 44579 44484 -95
Branches 6493 6456 -37
============================================
- Hits 27175 26933 -242
- Misses 14437 14589 +152
+ Partials 2967 2962 -5
```
| [Impacted Files](https://codecov.io/gh/apache/dubbo/pull/10234?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...dubbo/remoting/redis/jedis/ClusterRedisClient.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVtb3RpbmcvZHViYm8tcmVtb3RpbmctcmVkaXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JlbW90aW5nL3JlZGlzL2plZGlzL0NsdXN0ZXJSZWRpc0NsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ubbo/remoting/redis/jedis/SentinelRedisClient.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVtb3RpbmcvZHViYm8tcmVtb3RpbmctcmVkaXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JlbW90aW5nL3JlZGlzL2plZGlzL1NlbnRpbmVsUmVkaXNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...org/apache/dubbo/registry/redis/RedisRegistry.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVnaXN0cnkvZHViYm8tcmVnaXN0cnktcmVkaXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JlZ2lzdHJ5L3JlZGlzL1JlZGlzUmVnaXN0cnkuamF2YQ==) | `59.20% <83.33%> (+0.77%)` | :arrow_up: |
| [...he/dubbo/remoting/redis/jedis/MonoRedisClient.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVtb3RpbmcvZHViYm8tcmVtb3RpbmctcmVkaXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JlbW90aW5nL3JlZGlzL2plZGlzL01vbm9SZWRpc0NsaWVudC5qYXZh) | `74.41% <100.00%> (-7.64%)` | :arrow_down: |
| [...a/org/apache/dubbo/rpc/filter/AccessLogFilter.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcnBjL2R1YmJvLXJwYy1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JwYy9maWx0ZXIvQWNjZXNzTG9nRmlsdGVyLmphdmE=) | `27.69% <0.00%> (-50.77%)` | :arrow_down: |
| [...va/org/apache/dubbo/rpc/support/AccessLogData.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcnBjL2R1YmJvLXJwYy1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JwYy9zdXBwb3J0L0FjY2Vzc0xvZ0RhdGEuamF2YQ==) | `53.16% <0.00%> (-37.98%)` | :arrow_down: |
| [...ng/transport/dispatcher/all/AllChannelHandler.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVtb3RpbmcvZHViYm8tcmVtb3RpbmctYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kdWJiby9yZW1vdGluZy90cmFuc3BvcnQvZGlzcGF0Y2hlci9hbGwvQWxsQ2hhbm5lbEhhbmRsZXIuamF2YQ==) | `68.96% <0.00%> (-27.59%)` | :arrow_down: |
| [...bo/remoting/redis/support/AbstractRedisClient.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVtb3RpbmcvZHViYm8tcmVtb3RpbmctcmVkaXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2R1YmJvL3JlbW90aW5nL3JlZGlzL3N1cHBvcnQvQWJzdHJhY3RSZWRpc0NsaWVudC5qYXZh) | `21.05% <0.00%> (-26.32%)` | :arrow_down: |
| [...ng/transport/dispatcher/WrappedChannelHandler.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVtb3RpbmcvZHViYm8tcmVtb3RpbmctYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kdWJiby9yZW1vdGluZy90cmFuc3BvcnQvZGlzcGF0Y2hlci9XcmFwcGVkQ2hhbm5lbEhhbmRsZXIuamF2YQ==) | `47.82% <0.00%> (-15.22%)` | :arrow_down: |
| [...org/apache/dubbo/registry/dubbo/DubboRegistry.java](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZHViYm8tcmVnaXN0cnkvZHViYm8tcmVnaXN0cnktZGVmYXVsdC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZHViYm8vcmVnaXN0cnkvZHViYm8vRHViYm9SZWdpc3RyeS5qYXZh) | `42.00% <0.00%> (-14.00%)` | :arrow_down: |
| ... and [51 more](https://codecov.io/gh/apache/dubbo/pull/10234/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dubbo/pull/10234?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dubbo/pull/10234?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c1a76a0...d3d1075](https://codecov.io/gh/apache/dubbo/pull/10234?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on a diff in pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on code in PR #10234:
URL: https://github.com/apache/dubbo/pull/10234#discussion_r909284560
##########
dubbo-registry/dubbo-registry-redis/src/main/java/org/apache/dubbo/registry/redis/RedisRegistry.java:
##########
@@ -266,7 +268,30 @@ public void doSubscribe(final URL url, final NotifyListener listener) {
}
}
} else {
- doNotify(redisClient.scan(service + PATH_SEPARATOR + ANY_VALUE), url, Collections.singletonList(listener));
+// doNotify(redisClient.scan(service + PATH_SEPARATOR + ANY_VALUE), url, Collections.singletonList(listener));
+ //@tietang.wang 2022.06.29
+ // 原来为使用redis scan命令来扫描 /dubbo/com.xxx.yyy.XxxService/*的key
+ // 按照https://dubbo.apache.org/zh/docsv2.7/user/references/registry/redis/对redis注册中心的介绍
+ //只有/dubbo/com.xxx.yyy.XxxService/providers 和/dubbo/com.xxx.yyy.XxxService/consumers 2种可能性。
+ //在在使用ClusterRedisClient时，scan命令（参考：org.apache.dubbo.remoting.redis.jedis.ClusterRedisClient.scan）是迭代循环所有cluster node
+ // 来扫描每一个node，并合并扫描结果完成的，scan命令本来就很慢，使用ClusterRedisClient时会随着节点越多越慢
+ //我的场景中有6个节点，每次需要5~20秒的扫描时间，每次启动时很慢且卡在初始化reference阶段；或者泛化每接口第一次调用或者硬编码每接口第一次调用时，都会构建Reference，构建reference时花费了很多时间
+ //如果项目中service 接口很多时，有可能在启动或者重启后会导致worker线程阻塞而请求无法正常调用，目前在测试环境已经遇到了该现象，高并发请求时有可能引起雪奔，特别是存在dubbo http网关时的场景。
+ // 经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在300~1000ms即可构建好reference。
+ // 测试环境多线程（50线程）预创建48个reference需要，同机房1985ms（优化前9839ms）提升5倍，跨机房（同地域）3276ms（优化前35303ms）提升10倍。
+ // 本地环境（远程连接测试环境redis）多线程（50线程）预创建48个reference需要18288ms（优化前436235ms，约7分钟多，提升23倍）。
Review Comment:
这些描述在 PR 里面就行了，不需要写在代码中。另外代码必须全英文
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] liufeiyu1002 commented on a diff in pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by GitBox <gi...@apache.org>.
liufeiyu1002 commented on code in PR #10234:
URL: https://github.com/apache/dubbo/pull/10234#discussion_r945168299
##########
dubbo-remoting/dubbo-remoting-redis/src/main/java/org/apache/dubbo/remoting/redis/jedis/SentinelRedisClient.java:
##########
@@ -49,6 +49,14 @@ public SentinelRedisClient(URL url) {
sentinelPool = new JedisSentinelPool(masterName, sentinels, getConfig(), url.getPassword());
}
+ @Override
+ public boolean exists(String key) {
+ Jedis jedis = sentinelPool.getResource();
+ boolean result = jedis.exists(key);
+ jedis.close();
Review Comment:
这些不使用 try finally 包起来么？ 在 finally 里关闭(归还) jedis 么
```
try(Jedis jedis = sentinelPool.getResource()){
boolean result = jedis.exists(key);
return result;
}catch (Exception e){
throw e;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] chickenlj commented on pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by GitBox <gi...@apache.org>.
chickenlj commented on PR #10234:
URL: https://github.com/apache/dubbo/pull/10234#issuecomment-1235085178
2.6.x has been announced as EOF. Please file another pr to 2.7.x or 3.1.x and fix the issues mentioned in the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] chickenlj closed pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by GitBox <gi...@apache.org>.
chickenlj closed pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
URL: https://github.com/apache/dubbo/pull/10234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] yx-Yaoxaing commented on a diff in pull request #10234: #10227 优化subscribe时初始化性能，经过优化后，该场景在开发环境下从5~20秒性能提升至300~1000ms，提升5~20倍，基本上在30…
Posted by "yx-Yaoxaing (via GitHub)" <gi...@apache.org>.
yx-Yaoxaing commented on code in PR #10234:
URL: https://github.com/apache/dubbo/pull/10234#discussion_r1278267482
##########
dubbo-remoting/dubbo-remoting-redis/src/main/java/org/apache/dubbo/remoting/redis/jedis/SentinelRedisClient.java:
##########
@@ -49,6 +49,14 @@ public SentinelRedisClient(URL url) {
sentinelPool = new JedisSentinelPool(masterName, sentinels, getConfig(), url.getPassword());
}
+ @Override
+ public boolean exists(String key) {
+ Jedis jedis = sentinelPool.getResource();
+ boolean result = jedis.exists(key);
+ jedis.close();
Review Comment:
不是已经try with了...
##########
dubbo-remoting/dubbo-remoting-redis/src/main/java/org/apache/dubbo/remoting/redis/jedis/SentinelRedisClient.java:
##########
@@ -49,6 +49,14 @@ public SentinelRedisClient(URL url) {
sentinelPool = new JedisSentinelPool(masterName, sentinels, getConfig(), url.getPassword());
}
+ @Override
+ public boolean exists(String key) {
+ Jedis jedis = sentinelPool.getResource();
+ boolean result = jedis.exists(key);
+ jedis.close();
Review Comment:
不是已经try with了...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org