You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "YutaLin (via GitHub)" <gi...@apache.org> on 2026/05/05 03:35:38 UTC
[PR] HDDS-15149. Limit Connections Created by DataNode GRPC Server [ozone]
YutaLin opened a new pull request, #10184:
URL: https://github.com/apache/ozone/pull/10184
## What changes were proposed in this pull request?
Currently there is no limited control of network connections created by Datanode GRPC server, which caused "too many open files" issue in Datanode, 126920 TCP connections out of 128103 total open file handlers.
Introduce `GrpcConnectionLimitFilter` to control connections and add set default max connections to 5,000, leaves most of the fds for files/DBs.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15149
## How was this patch tested?
Add test for connection limit filter and node configuration, also run ci tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15149. Limit Connections Created by DataNode GRPC Server [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10184:
URL: https://github.com/apache/ozone/pull/10184#discussion_r3198878358
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java:
##########
@@ -136,7 +141,8 @@ public XceiverServerGrpc(DatanodeDetails datanodeDetails,
.executor(readExecutors)
.addService(ServerInterceptors.intercept(
xceiverService.bindServiceWithZeroCopy(),
- new GrpcServerInterceptor()));
+ new GrpcServerInterceptor()))
+ .addTransportFilter(connectionLimitFilter);
Review Comment:
@YutaLin , could you figure out what will be the client side behavior when connection is closed by GrpcConnectionLimitFilter? When SO_BACKLOG is hit, client will resend the SYNC message. If connection termination will cause client to fail completely, then we could only keep SO_BACKLOG, though GrpcConnectionLimitFilter is welly implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15149. Limit Connections Created by DataNode GRPC Server [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #10184:
URL: https://github.com/apache/ozone/pull/10184#discussion_r3186951387
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java:
##########
@@ -136,7 +141,8 @@ public XceiverServerGrpc(DatanodeDetails datanodeDetails,
.executor(readExecutors)
.addService(ServerInterceptors.intercept(
xceiverService.bindServiceWithZeroCopy(),
- new GrpcServerInterceptor()));
+ new GrpcServerInterceptor()))
+ .addTransportFilter(connectionLimitFilter);
Review Comment:
@YutaLin , thanks for working on this.
Could you try use NettyServerBuilder's withChildOption, to set ChannelOption.SO_BACKLOG?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15149. Limit Connections Created by DataNode GRPC Server [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #10184:
URL: https://github.com/apache/ozone/pull/10184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15149. Limit Connections Created by DataNode GRPC Server [ozone]
Posted by "YutaLin (via GitHub)" <gi...@apache.org>.
YutaLin commented on code in PR #10184:
URL: https://github.com/apache/ozone/pull/10184#discussion_r3190526452
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java:
##########
@@ -136,7 +141,8 @@ public XceiverServerGrpc(DatanodeDetails datanodeDetails,
.executor(readExecutors)
.addService(ServerInterceptors.intercept(
xceiverService.bindServiceWithZeroCopy(),
- new GrpcServerInterceptor()));
+ new GrpcServerInterceptor()))
+ .addTransportFilter(connectionLimitFilter);
Review Comment:
Hi @ChenSammi, thanks for the review,
I've added so_backlog to prevent burst connections, also keep connection limit filter to control sustained high connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15149. Limit Connections Created by DataNode GRPC Server [ozone]
Posted by "YutaLin (via GitHub)" <gi...@apache.org>.
YutaLin commented on PR #10184:
URL: https://github.com/apache/ozone/pull/10184#issuecomment-4402685521
HI @ChenSammi, thanks for the feedback! Your concern is correct, we don't make client retry again when we have `RESOURCE_EXHAUSTED` exception. I've removed GrpcConnectionLimitFilter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org