You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2022/08/02 16:28:30 UTC
[GitHub] [incubator-brpc] 52coder opened a new issue, #1870: 使用example中的例子验证熔断
52coder opened a new issue, #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870
**Describe the bug (描述bug)**
基于example中的例子验证可选熔断(根据失败率）
原始代码：https://github.com/52coder/incubator-brpc/tree/master/example/asynchronous_echo_c%2B%2B
client.cpp代码修改点：
https://github.com/52coder/incubator-brpc/blob/master/example/asynchronous_echo_c%2B%2B/client.cpp#L65后添加：
options.enable_circuit_breaker = true;
L30行修改为rr :
DEFINE_string(load_balancer, "rr", "The algorithm for load balancing");
server.cpp中未改动，先编译出来echo_server,然后在https://github.com/52coder/incubator-brpc/blob/master/example/asynchronous_echo_c%2B%2B/server.cpp#L63前面增加如下代码：
cntl->SetFailed(brpc::EREQUEST, "Fail to parse request");然后编译出二进制echo_server_fail
这里的想法是先构造一个100%失败的server触发熔断。
**To Reproduce (复现方法)**
验证方法，三个终端下分别执行
./echo_server --port=8001
./echo_server --port=8002
./echo_server_fail --port=8003
client端运行：
./echo_client --server="list://192.168.49.1:8001,192.168.49.1:8002,192.168.49.1:8003"
**Expected behavior (期望行为)**
期望行为是由于client端开启了熔断，其中一个server一直返回失败，所以应该会触发熔断才对，实际结果是跑了一晚上，一直出错的那个server并未摘除
`
I0803 00:26:21.137101 1872894 client.cpp:45] Received response from 192.168.49.1:8002: hello world (attached=bar) latency=603us
I0803 00:26:22.137842 1872886 client.cpp:45] Received response from 192.168.49.1:8003: hello world (attached=bar) latency=734us
W0803 00:26:23.138861 1872894 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8001][E1003]Fail to parse request
I0803 00:26:24.139042 1872886 client.cpp:45] Received response from 192.168.49.1:8002: hello world (attached=bar) latency=590us
I0803 00:26:25.139412 1872894 client.cpp:45] Received response from 192.168.49.1:8003: hello world (attached=bar) latency=639us
W0803 00:26:26.140028 1872886 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8001][E1003]Fail to parse request
I0803 00:26:27.140468 1872894 client.cpp:45] Received response from 192.168.49.1:8002: hello world (attached=bar) latency=733us
I0803 00:26:28.141491 1872886 client.cpp:45] Received response from 192.168.49.1:8003: hello world (attached=bar) latency=739us
W0803 00:26:29.142124 1872894 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8001][E1003]Fail to parse request
I0803 00:26:30.142181 1872886 client.cpp:45] Received response from 192.168.49.1:8002: hello world (attached=bar) latency=484us
I0803 00:26:31.142817 1872894 client.cpp:45] Received response from 192.168.49.1:8003: hello world (attached=bar) latency=737us
W0803 00:26:32.143875 1872886 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8001][E1003]Fail to parse request
I0803 00:26:33.144834 1872894 client.cpp:45] Received response from 192.168.49.1:8002: hello world (attached=bar) latency=748us
I0803 00:26:34.145134 1872886 client.cpp:45] Received response from 192.168.49.1:8003: hello world (attached=bar) latency=676us
W0803 00:26:35.145437 1872894 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8001][E1003]Fail to parse request
I0803 00:26:36.145931 1872886 client.cpp:45] Received response from 192.168.49.1:8002: hello world (attached=bar) latency=739us
`
**Versions (各种版本)**
OS:
Compiler:
brpc:
protobuf:
**Additional context/screenshots (更多上下文/截图)**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] TousakaRin commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
TousakaRin commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1228234396
我用你描述的代码是能正常熔断的，默认配置下如果server 错误率100%，第151个请求会触发熔断。
这是我用的代码：https://github.com/TousakaRin/brpc/tree/circuit_breaker_test
有一种可能是你的代码比较老，没有这个commit: https://github.com/apache/incubator-brpc/commit/c59aecdded0fbc01cda5b5462ef99c4d084b31ae
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] TousakaRin commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
TousakaRin commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1240492557
可以在client的connection页面看一下错误数量有没有正确记录
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] cdjingit commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
cdjingit commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1206011245
测试的qps多少？加大qps看看？理论上应该能触发，后续我看看能不能在我环境复现一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] TousakaRin commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
TousakaRin commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1207770415
检查一下是否有Socket[xxx] isolated by circuit breaker 的log，brpc的默认重连方式只需要tcp能够建立连接，有可能熔断之后又恢复了。
另外connections监控页面也可以检查各个连接的健康状况。
连接超时的熔断需要用iptables drop掉对应端口的包，直接sleep是不行的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] 52coder commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
52coder commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1220296413
@TousakaRin @cdjingit 有事情耽误，今天有空看了下，其中一个server回包的时候设置了cntl->SetFailed(brpc::EREQUEST, "Fail to parse request");，没有触发熔断的日志，server还能源源不断收到请求。
超时熔断验证没问题，现在失败率(可选)熔断使用example中的例子没有验证通过，业务中暂时没有开启。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] TousakaRin commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
TousakaRin commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1221741065
可以贴一下完整的代码么，我在server端添加`cntl->SetFailed(brpc::EREQUEST, "Fail to parse request");`之后，在client看到的日志是`W0822 11:08:00.715427 748755 client.cpp:90] [E1003][10.227.87.49:8000][E1003]Failed to parse request`，而你贴的日志是Fail to **send** EchoRequest, 比较奇怪。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] 52coder commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
52coder commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1207373473
@cdjingit 去掉了例子中的sleep，增大了qps也没有触发熔断，学习了这个https://github.com/apache/incubator-brpc/issues/387有点不太理解熔断跟qps的关系，有没有触发熔断的机制详细介绍?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] 52coder commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
52coder commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1229211645
@TousakaRin 不知道我得测试方法是否有问题，和你的代码一样（缺少了dummy_server.port),server端编译出来两个版本，一个包含cntl->SetFailed(brpc::EREQUEST, "Failed to parse request");，一个不包含这行代码:
./echo_server_ok --port=8001
./echo_server_fail --port=8002
client.cpp中DEFINE_string(server, "list://0.0.0.0:8001,0.0.0.0:8002", "IP Address of server");
./echo_client 直接运行
辛苦帮review下，实在没看出哪里有问题导致没有触发熔断？
client终端输出一直是（20min+)：
W0827 23:17:06.067692 13282 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8002][E1003]Failed to parse request
I0827 23:17:07.068715 13280 client.cpp:45] Received response from 0.0.0.0:8001: hello world (attached=bar) latency=1281us
W0827 23:17:08.069341 13282 client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8002][E1003]Failed to parse request
I0827 23:17:09.070102 13280 client.cpp:45] Received response from 0.0.0.0:8001: hello world (attached=bar) latency=1048us
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] 52coder commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
52coder commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1229352138
编译信息及文件列表：
make之后文件列表：
root@52coder:~/incubator-brpc/example/enable_circuit_breaker_c++# ls -lt
total 373920
-rwxr-xr-x 1 root root 95129344 8月 27 23:13 echo_server
-rwxr-xr-x 1 root root 95059624 8月 27 23:13 echo_client
-rw-r--r-- 1 root root 496992 8月 27 23:13 client.o
-rw-r--r-- 1 root root 4803 8月 27 23:13 client.cpp
-rwxr-xr-x 1 root root 95129344 8月 27 23:06 echo_server_ok
-rw-r--r-- 1 root root 948400 8月 27 23:06 server.o
-rw-r--r-- 1 root root 4203 8月 27 23:06 server.cpp
-rwxr-xr-x 1 root root 95129432 8月 27 23:05 echo_server_fail
-rw-r--r-- 1 root root 5 8月 27 23:05 dummy_server.port
-rw-r--r-- 1 root root 926128 8月 27 22:54 echo.pb.o
-rw-r--r-- 1 root root 22009 8月 27 22:54 echo.pb.h
-rw-r--r-- 1 root root 4878 8月 27 22:52 CMakeLists.txt
-rw-r--r-- 1 root root 1073 8月 27 22:52 echo.proto
-rw-r--r-- 1 root root 815 8月 27 22:52 Makefile
root@52coder:~/incubator-brpc/example/enable_circuit_breaker_c++# make
> Generating echo.pb.cc
/usr/local/bin/protoc --cpp_out=. --proto_path=. echo.proto
> Compiling echo.pb.o
g++ -c -I/usr/include/ -I../../output/include -DBRPC_WITH_GLOG=0 -DGFLAGS_NS=google -g -std=c++0x -DNDEBUG -O2 -D__const__= -pipe -W -Wall -Wno-unused-parameter -fPIC -fno-omit-frame-pointer echo.pb.cc -o echo.pb.o
> Compiling client.o
g++ -c -I/usr/include/ -I../../output/include -DBRPC_WITH_GLOG=0 -DGFLAGS_NS=google -g -std=c++0x -DNDEBUG -O2 -D__const__= -pipe -W -Wall -Wno-unused-parameter -fPIC -fno-omit-frame-pointer client.cpp -o client.o
> Linking echo_client
g++ -L/usr/lib/x86_64-linux-gnu -L../../output/lib -Xlinker "-(" echo.pb.o client.o -Wl,-Bstatic -lgflags -lprotobuf -lleveldb -lsnappy -lbrpc -Wl,-Bdynamic -Xlinker "-)" -lpthread -lssl -lcrypto -ldl -lz -lrt -o echo_client
> Compiling server.o
g++ -c -I/usr/include/ -I../../output/include -DBRPC_WITH_GLOG=0 -DGFLAGS_NS=google -g -std=c++0x -DNDEBUG -O2 -D__const__= -pipe -W -Wall -Wno-unused-parameter -fPIC -fno-omit-frame-pointer server.cpp -o server.o
In file included from ../../output/include/butil/resource_pool.h:89,
from ../../output/include/brpc/socket.h:30,
from ../../output/include/brpc/redis.h:33,
from ../../output/include/brpc/server.h:43,
from server.cpp:22:
../../output/include/butil/resource_pool_inl.h: In instantiation of ‘static butil::ResourcePool<T>* butil::ResourcePool<T>::singleton() [with T = brpc::Socket]’:
../../output/include/butil/resource_pool.h:118:38: required from ‘int butil::return_resource(butil::ResourceId<T>) [with T = brpc::Socket]’
../../output/include/brpc/socket_inl.h:112:32: required from here
../../output/include/butil/resource_pool_inl.h:368:17: warning: ‘new’ of type ‘butil::ResourcePool<brpc::Socket>’ with extended alignment 64 [-Waligned-new=]
368 | p = new ResourcePool();
| ^~~~~~~~~~~~~~~~~~
../../output/include/butil/resource_pool_inl.h:368:17: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
../../output/include/butil/resource_pool_inl.h:368:17: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
> Linking echo_server
g++ -L/usr/lib/x86_64-linux-gnu -L../../output/lib -Xlinker "-(" echo.pb.o server.o -Wl,-Bstatic -lgflags -lprotobuf -lleveldb -lsnappy -lbrpc -Wl,-Bdynamic -Xlinker "-)" -lpthread -lssl -lcrypto -ldl -lz -lrt -o echo_server
rm echo.pb.cc
root@52coder:~/incubator-brpc/example/enable_circuit_breaker_c++#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] 52coder commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
52coder commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1229210436
> 可以贴一下完整的代码么，我在server端添加`cntl->SetFailed(brpc::EREQUEST, "Fail to parse request");`之后，在client看到的日志是`W0822 11:08:00.715427 748755 client.cpp:90] [E1003][10.227.87.49:8000][E1003]Failed to parse request`，而你贴的日志是Fail to **send** EchoRequest, 比较奇怪。
代码是拷贝的asynchronous_echo_c++目录，修改代码是和你的https://github.com/TousakaRin/brpc/tree/circuit_breaker_test代码一样，除了没有dummy_server.port这个文件，client端打印client.cpp:42] Fail to send EchoRequest, [E1003][127.0.1.1:8003][E1003]Failed to parse request也是正常的吧，因为server端的回报设置了failed，打印这个应该没问题。
重新检查了一遍，其中一个server 100%失败，跑半小时还是没有触发熔断。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] 52coder commented on issue #1870: 使用example中的例子验证熔断功能(可选熔断和连接超时熔断)
Posted by GitBox <gi...@apache.org>.
52coder commented on issue #1870:
URL: https://github.com/apache/incubator-brpc/issues/1870#issuecomment-1228174728
@TousakaRin 代码就是使用的example中的例子
原始代码：https://github.com/52coder/incubator-brpc/tree/master/example/asynchronous_echo_c%2B%2B
client.cpp修改点：
https://github.com/52coder/incubator-brpc/blob/master/example/asynchronous_echo_c%2B%2B/client.cpp#L65
添加：
options.enable_circuit_breaker = true;
L30行修改为rr :
DEFINE_string(load_balancer, "rr", "The algorithm for load balancing");
server.cpp在改动前，先编译出来echo_server,然后在[https://github.com/52coder/incubator-brpc/blob/master/example/asynchronous_echo_c%2B%2B/server.cpp#L63前面增加如下代码：
cntl->SetFailed(brpc::EREQUEST, "Fail to parse request");
然后编译出二进制echo_server_fail，这里的想法是生成一个100%失败的server触发熔断。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org