You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by GitBox <gi...@apache.org> on 2022/12/02 06:11:46 UTC
[GitHub] [incubator-brpc] changchengx opened a new issue, #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
changchengx opened a new issue, #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031
**Describe the bug (描述bug)**
When running rdma_performance built from rdma-old, it reports below errro:
```
example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
```
**To Reproduce (复现方法)**
1. Build rdma-old branch
```
$ git clone https://github.com/apache/incubator-brpc.git verbs_old
$ cd verbs_old
$ git checkout rdma-old
$ bash config_brpc.sh --headers=/usr/include --libs=/usr/lib64
$ mkdir build
$ cd build
$ cmake -DWITH_RDMA=on ..
$ make
```
2. Build rdma-performance
```
$ cd verbs_old/example/rdma_performance
$ mkdir build
$ cd build
$ cmake ..
$ make
```
3. Start running two server process on one host, suppose the bond1 ip address is 22.22.30.6
```
verbs_old/example/rdma_performance/build/]$ numactl -i netdev:ens3f0np0 ./perf_server --use_rdma=true --rdma_device=mlx5_bond_1 --port=8003
```
&
```
verbs_old/example/rdma_performance/build/]$ numactl -i netdev:ens3f0np0 ./perf_server --use_rdma=true --rdma_device=mlx5_bond_1 --port=8004
```
4. Start running one client process on one host, suppose the bond1 ip address is 22.22.30.7
```
verbs_old/example/rdma_performance/build/]$ numactl -i netdev:ens3f0np0 ./perf_client --use_rdma=true --rdma_device=mlx5_bond_1 --servers=22.22.30.6:8003+22.22.30.6:8004 --echo_attachment=false --attachment_size=16 --thread_num=16384
```
Sometimes, the client process will reports below error:
```
verbs_old/example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
verbs_old/example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8003
```
**Expected behavior (期望行为)**
There should be no Reached timeout error.
The client and server host are connected through one switch. the RTT is very small.
**Versions (各种版本)**
OS:
```
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
```
Compiler:
```
$ gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-4)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```
brpc:
```
rdma-old branch
```
protobuf:
```
$ rpm -qa | grep protobuf
protobuf-c-1.3.0-6.el8.x86_64
protobuf-devel-3.5.0-13.el8.x86_64
protobuf-compiler-3.5.0-13.el8.x86_64
protobuf-3.5.0-13.el8.x86_64
```
**Additional context/screenshots (更多上下文/截图)**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] Tuvie commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
Tuvie commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1336313733
> How to improve it? What's the backend mechanism?
As I said, we create QPs at the beginning to reduce the overhead of ibv_create_qp at runtime. You don't need to care about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] wwbmmm commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
wwbmmm commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1334944220
the rdma-old branch is no longer maintained
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] changchengx commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
changchengx commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1338860829
@Tuvie Feedback later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] Tuvie commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
Tuvie commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1336312248
> 1. Is it normal to run two server processes and use one client process to connect with the two server processes when running rdma_performance with using RDMA to get the performance data?
Yes
> 2\. It rdma-old branch, when connecting one client process & one server process, it only create one QP when running rdma_performance/verbs. However, in master branch, the client will create lots of QPs when it connects with one server process. Is this the expected behavior? You can check the QPs info with below command:
In new branch, there is also one QP between one client and one server. But in order to improve the performance of connection establishment, we create a lot of QPs at the beginning in advance. Only one will be chosen from this pool when the connection is established.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] changchengx commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
changchengx commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1334955599
@wwbmmm
1. Is it normal to run two server processes and use one client process to connect with the two server processes when running rdma_performance with using RDMA to get the performance data?
3. It rdma-old branch, when connecting one client process & one server process, it only create one QP when running rdma_performance/verbs. However, in master branch, the client will create lots of QPs when it connects with one server process. Is this the expected behavior? You can check the QPs info with below command:
```
# ls -d /sys/kernel/debug/mlx5/*/QPs
/sys/kernel/debug/mlx5/0000:17:00.0/QPs /sys/kernel/debug/mlx5/0000:17:00.1/QPs /sys/kernel/debug/mlx5/0000:ca:00.0/QPs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] changchengx commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
changchengx commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1336331743
@Tuvie @wwbmmm
>
>we create QPs at the beginning to reduce the overhead of ibv_create_qp at runtime.
>
Could you give the test cases to show the usage scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] Tuvie commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
Tuvie commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1338859738
@changchengx You can set rdma_prepared_qp_cnt=0 to disable this feature to see the difference in your setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
[GitHub] [incubator-brpc] changchengx commented on issue #2031: example/rdma_performance/client.cpp:155] RPC call failed: [E1008]Reached timeout=2000ms @22.22.30.6:8004
Posted by GitBox <gi...@apache.org>.
changchengx commented on issue #2031:
URL: https://github.com/apache/incubator-brpc/issues/2031#issuecomment-1336312495
@Tuvie
>
> But in order to improve the performance of connection establishment
>
How to improve it? What's the backend mechanism?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org