You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by GitBox <gi...@apache.org> on 2021/04/28 09:09:41 UTC
[GitHub] [ratis] runzhiwang opened a new pull request #474: RATIS-1371. Support multi-client when transfer data between servers
runzhiwang opened a new pull request #474:
URL: https://github.com/apache/ratis/pull/474
## What changes were proposed in this pull request?
If server1 use only one client to transfer data to server2, there is bottleneck, so we should use multi-client.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-1371
## How was this patch tested?
No need new ut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r621978992
##########
File path: ratis-server-api/src/main/java/org/apache/ratis/server/RaftServerConfigKeys.java
##########
@@ -445,6 +445,19 @@ static int asyncWriteThreadPoolSize(RaftProperties properties) {
static void setAsyncWriteThreadPoolSize(RaftProperties properties, int port) {
setInt(properties::setInt, ASYNC_WRITE_THREAD_POOL_SIZE_KEY, port);
}
+
+ String CLIENT_NUM_KEY = PREFIX + ".client.number";
Review comment:
Let's call it "client.pool.size".
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -109,17 +111,22 @@ void close() {
private final EventLoopGroup bossGroup = new NioEventLoopGroup();
private final EventLoopGroup workerGroup = new NioEventLoopGroup();
private final ChannelFuture channelFuture;
+ private final int clientNum;
Review comment:
I think we can declare clientNum (or rename it to clientPoolSize) as a local variable in the constructor and use proxies.size() elsewhere. Then, we don't need this field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r621979428
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
use streamId to mod maybe not the best choice, because when there are 1000 clients, each only send one stream to server, all the streamId maybe zero, then we can not balance the proxies. Maybe we can use a static variable in server to record the stream number, and use stream number to mod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r621979428
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
use streamId to mod maybe not the best choice, because when there are 1000 clients, each only send one stream to server, all the streamId maybe zero, then we can not balance the proxies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] szetszwo commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r622001055
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
We may use clientId as well:
`
int index = Math.toIntExact(((0xFFFFFFFFL & request.getClientId().hashCode()) + request.getStreamId()) % clientNum);
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r622002012
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
Sure, good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r621979428
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
use streamId to mod maybe not the best choice, because when there are 1000 clients, each only send one stream to server, all the streamId maybe zero, then we can not balance the proxies. Maybe we can use a static variable in server to record how many stream, and use this variable to mod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang merged pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang merged pull request #474:
URL: https://github.com/apache/ratis/pull/474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r621979428
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
use streamId to mod maybe not the best choice, because when there are 1000 clients, each only send one stream to server, all the streamId maybe zero, then we can not balance the proxies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ratis] runzhiwang commented on a change in pull request #474: RATIS-1371. Support multi-client when transfer data between servers
Posted by GitBox <gi...@apache.org>.
runzhiwang commented on a change in pull request #474:
URL: https://github.com/apache/ratis/pull/474#discussion_r621979428
##########
File path: ratis-netty/src/main/java/org/apache/ratis/netty/server/NettyServerStreamRpc.java
##########
@@ -174,7 +183,9 @@ public void channelRead(ChannelHandlerContext ctx, Object msg) {
}
final DataStreamRequestByteBuf request = requestRef.set((DataStreamRequestByteBuf)msg);
- requests.read(request, ctx, proxies::getDataStreamOutput);
+
+ int index = (int)request.getStreamId() % clientNum;
Review comment:
use streamId to mod maybe not the best choice, because when there are 1000 clients, each only send one stream to server, all the streamId maybe zero, then we can not balance the proxies. Maybe we can use a static variable in server to record how many stream, and use stream number to mod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org